Queue Allocation in Machine Learning Accelerators

Abstract

The present disclosure generally provides solutions for improving the performance of a customized packet-switched TPU accelerator-side communication network. Specifically, a set of solutions for improving flow control behavior by tuning packet buffer queues in on-chip routers in a distributed training supercomputer network is described.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63 / 091,708, filed on October 14, 2020. Technical Field

[0003] The present disclosure relates to allocation of shared memory for traffic queuing in a machine learning accelerator communication network. Background Art

[0004] Electronic devices may consist of multiple different computing units that need to communicate data between them in order for the electronic device to operate. Data communication between computing units can be non-deterministic. For example, data communication experiences variable latency between the transmission time of one unit and the reception time of another unit. That is, the time it takes for data to propagate from one unit to another is not constant, but rather is subject to many different sources of variation in the transmission time. Summary of the invention

[0005] In general, the present disclosure relates to a method for memory allocation in a machine learning accelerator communication network. In general, the method includes accessing metadata associated with multiple communication ports of an application-specific integrated circuit (ASIC). The metadata identifies, for each port, whether the particular port is used in the current configuration and the communication medium associated with the particular port. Based on the metadata, an expected latency for each port is determined, and a portion of a shared memory is allocated to each port. The shared memory is allocated by assigning zero memory to unused ports, determining the memory allocation for each port based on the expected latency, and assigning a start address and a stop address of the shared memory to each port. Implementations may optionally include one or more of the following features.

[0006] In some implementations, assigning the start address and the stop address of the shared memory to each port includes calling an application programming interface (API) by a device other than the ASIC.

[0007] In some implementations, a process is performed on the ASIC using the machine learning accelerator communication network and the allocated shared memory. In some implementations, the process is training a neural network.

[0008] In some implementations, the ASIC is a tensor processing unit (TPU).

[0009] In some implementations, the communication medium identified in the metadata is a copper cable medium, an optical medium, a printed circuit board (PCB) medium, or any combination thereof.

[0010] In some implementations, memory is allocated according to the following equation:

[0011] In an alternative implementation, a method for memory allocation in a machine learning accelerator communication network includes determining a network topology for a network of machine learning accelerator ASICs. Access metadata associated with multiple communication ports of each ASIC within the network. The metadata identifies, for each of the multiple ports, whether a particular port is used in the current configuration and the communication medium associated with the particular port. For each port used in the network topology, a round trip time (RTT) delay is determined. A portion of a shared memory is allocated to each of the multiple ports by: determining a memory allocation for each port proportional to the RTT delay, and assigning a start address and a stop address of the shared memory to each port for the determined memory allocation. A process is executed on the machine learning accelerator that sends profiled traffic into the network for a predetermined duration. The number of received traffic packets is determined for each port, and a portion of the shared memory is reallocated to each of the multiple ports by: determining a memory allocation for each port proportional to the received traffic packet count, and reallocating a start address and a stop address for each port of the determined memory allocation.

[0012] In some implementations, assigning and reassigning the start address and the stop address of the shared memory to each port includes calling an API by a device other than the ASIC.

[0013] In some implementations, the memory allocation proportional to the determined RTT delay for each port is determined according to the following formula:

[0014] In some implementations, the memory allocation proportional to the received packet count for each port is determined according to the following formula:

[0015] In some implementations, the RTT delay is calculated prior to execution by transmitting and receiving one or more timing messages to determine the latency.

[0016] In some implementations, profiling the traffic includes all-to-all traffic, nearest neighbor traffic, composite traffic profiles, or any suitable combination thereof.

[0017] In an alternative implementation, a method for allocating memory for a machine learning accelerator communication network includes determining a network topology for a network of machine learning accelerator application-specific integrated circuits (ASICs). Access metadata associated with a plurality of communication ports of each ASIC within the network. The metadata identifies, for each of the plurality of ports, whether a particular port is used in a current architecture and a communication medium associated with the particular port. The metadata identifies, for each of the plurality of ports, whether a particular port is used in a current configuration and a communication medium associated with the particular port. For each port used in the network topology, a round trip time (RTT) delay is determined. A portion of a shared memory is allocated to each of the plurality of ports by: determining a memory allocation for each port proportional to the RTT delay, and assigning a start address and a stop address of the shared memory to each port for the determined memory allocation. A process is executed on the ASIC, the process using the machine learning accelerator communication network with the allocated shared memory. During execution of the process, a number of message packets received at each of the plurality of ports in a first time period is determined. An expected portion size of the shared memory for each of the plurality of ports is determined based on the number of message packets received at each port. The process is paused for a second time period and it is determined that the shared memory is clear of pending message packets. When the shared memory is clear, the start and stop addresses of the shared memory for each port are reassigned according to the expected portion size and execution of the process is resumed.

[0018] Various implementations provide one or more of the following advantages. For example, in some implementations, by more efficiently allocating message buffer queues, a machine learning hardware accelerator can execute faster and avoid bottlenecks in the accelerator's onboard communication network.

[0019] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic diagram illustrating an example machine learning accelerator according to an implementation of the present disclosure.

[0021] Figure 2 is a diagram illustrating an example shared memory with buffer queues allocated for port-to-port traffic.

[0022] Figure 3 is a flow chart of an example process for allocating shared memory based on expected latency according to an implementation of the present disclosure.

[0023] Figure 4 is a flow diagram of an example process for allocating shared memory based on profiling traffic operations according to implementations of the present disclosure.

[0024] Figure 5 is a flow chart of an example process for allocating shared memory based on traffic during execution according to an implementation of the present disclosure.

[0025] Figure 6 The icon can be found in Figure 1 Schematic diagram of an example of a dedicated logic chip used in a multi-chip system.

[0026] Figure 7 An example neural network (NN) compute tile 700 that may be used in an ASIC is illustrated. DETAILED DESCRIPTION

[0027] In general, the present disclosure relates to allocating shared memory between different communication channels or ports in a high-speed communication network (e.g., a communication network including a machine learning accelerator of multiple application-specific integrated circuits (ASICs)). Deep learning training requires distributed parallel processing. The distribution can divide a large amount of training data into different copies (replicas) (e.g., data parallelism), or divide a very large model into smaller modules (e.g., model parallelism). The divided training data and model parameters are placed on different processing units for simultaneous calculation.

[0028] Distributed training occurs in a synchronous, iterative, and incremental cycle. Based on data parallelism, each processing unit ingests a mini-batch of data at each step, computes local gradients, and then exchanges all local gradients across the entire network of computing units in an all-reduce manner to compute the final global consistent gradient, which is used to update the model weights at the end of the step.

[0029] According to model parallelism, each processing unit takes model activation input from its local training data or from the output of another processing unit operating on the hidden layer before itself. The processing unit then calculates the activation output, which can be the final model output, or used as the activation input of another processing unit. Gradients are calculated on the processing units including the final layer and sent back to the previous layers to update the segmented sub-model. This process can be pipelined to operate on consecutive mini-batches. According to this approach, intermediate activation outputs are sent around the network, as well as gradients at the model segmentation boundaries.

[0030] In practice, data and model parallelism can be combined to achieve the highest performance. For example, for a model with tens of billions of weight parameters, a large amount of computational resources and communication are required to converge the model to the desired level of accuracy.

[0031] To speed up the training process, ASICs such as customized accelerator chips, tensor processing units (TPUs) are designed to be used as processing units to accelerate deep learning calculations. In some implementations, chip types (e.g., FPGAs, GPGPUs, or CPUs) are used. At the same time, a co-designed high-speed communication network between accelerators is also established to accelerate communication between processing units. In short, the training system can provide exaFLOP-level computing performance, equivalent to a supercomputer in the prior art.

[0032] For simplicity, the custom network of the TPU accelerator is designed to reduce network processing overhead. Due to the basic features of the interconnect network, such as addressing, error detection and correction, routing and flow control, a large amount of network processing is performed on the accelerator chip hardware to accelerate processing.

[0033] The present disclosure focuses on improving the performance of a customized, packet-switched TPU accelerator-side communication network. Specifically, a set of solutions are proposed to improve flow control behavior by tuning packet buffer queues in on-chip routers in a distributed training supercomputer network.

[0034] The solutions disclosed in this article are applicable to the synchronous data parallel and model parallel training modes discussed above, and can also be generally applicable to asynchronous distributed training in accelerator networks.

[0035] Figure 1 is a schematic diagram illustrating an example machine learning accelerator 100 according to an implementation of the present disclosure. In this example, a TPU network is illustrated that is interconnected by an on-chip network stack processing hardware module. In this example, each TPU 106 has six network ports, which are linked to each other through buffer queues to form an intra-chip crossbar. In some implementations, each TPU 106 can have more or less than six ports (e.g., ten, three, or other numbers). In some implementations, the links between the ports can be further divided into multiple virtual communication channels. In addition, before executing a job within the TPU network, it can be determined at the software level which ports are connected. For example, Figure 1 The plurality of TPUs 106 are illustrated in a two-dimensional configuration, however, may have a three-dimensional (eg, spherical) configuration or other geometric shapes based on the connections between the TPU ports.

[0036] The accelerator-side network stack of the TPU 106 is implemented in an on-chip hardware processing module or router 102 on each TPU 106. Communications during distributed training are formatted as multiple network packets. In operation, packets between a pair of source and destination TPU chips are jumped on one or more TPU chips as intermediate steps before reaching the final destination. At each TPU chip hop, the on-chip network stack interfaces with the various network ports of the chip and forwards packets received from one port to another as instructed by a user-specified routing algorithm. Port-to-port transmission within the chip is performed in buffer queues. The cross-connected buffer queues within the chip form a crossbar for fast chip-level network processing.

[0037] Flow control can be enforced between each pair of directly connected TPUs 106 in a supercomputer network or machine learning accelerator 100, for example, to prevent a faster sender TPU 106 accelerator from overwhelming a slower receiver TPU 106 accelerator. When the TPU network begins forwarding activation output and weight gradient traffic across different accelerators, the buffer queues in the on-chip network stack of each chip continuously input packets from the input port to the output port, and the utilization of the buffer space changes dynamically according to the traffic in and out rate at the TPU chip 106. When the buffer queue corresponding to the port is fully utilized, packets cannot be received at the port because there is no remaining buffer space, and the upstream TPU accelerator should stop sending messages. The sender is regulated by opportunistically sending flow control messages that tell the sender the amount of buffer queue space available at the receiver port, denoted as "credit" below. The sender should not transmit any message that is larger than the available credit it receives. In some implementations, the link between any two connected TPU accelerators 106 is bidirectional, and credit messages are also sent bidirectionally.

[0038] The TPU accelerator side communication network may be a heterogeneous network environment. In other words, the communication time between ports may vary from port to port and from TPU 106 to TPU 106. In addition, the amount of traffic distributed on the ports may not be uniform, and therefore intelligent and dynamic message queuing can significantly enhance communication in the TPU communication network.

[0039] This non-uniformity can be caused by several different factors. For example, for a large-scale deep learning training supercomputer that scales to thousands of TPU accelerators 106, a variety of link media can be used to wire the entire network. Example media used in the link include, but are not limited to, PCB traces used to exchange data between TPU accelerators 106 soldered on the same tray, copper cables for low-cost short-distance communications, and optical cables for long-distance communications. Different media materials present heterogeneous link performance in terms of throughput and capacity, which affects queue utilization in routers.

[0040] In addition, large-scale supercomputer networks inevitably involve links with different lengths. Thousands of TPU accelerators 106 span at least multiple racks, resulting in intra-rack cables with shorter distances and lower latency, and inter-rack cables with longer distances, longer latency, and greater variation between different links. Delay affects the utilization of buffer queues. These cable links have various quality levels after being used for a period of time, measured by bit error rate (BER). The varying BER makes the packet transmission retry behavior different among all links, resulting in heterogeneous link pressure and queue saturation rate.

[0041] The network ports of a TPU are usually not used uniformly, depending on the topology of the distributed training job. For example, if a user training job requests TPU resources in a 2D grid topology among all TPU resources available in a supercomputer, then the TPUs on the edge of the grid do not use all ports—ports in the grid that are not internally facing are connected to neighboring accelerators, but do not receive traffic compared to other active send ports.

[0042] Large-scale supercomputer networks can inherently provide fault tolerance by routing around failed chips, failed hosts, or failed inter-TPU link media. Fault-tolerant routing can cause link traffic pressure to be unbalanced, even if the fault-free topology is initially a homogeneous topology. A TPU 106 whose ports are all actively used and connected to the media with the same type, distance, and quality may experience highly heterogeneous traffic because some ports are on more critical routes that bypass the failure site. In addition, accelerators adjacent to the failure site exacerbate the unbalanced use of ports, where ports directly connected to the failed chip or failed link do not receive traffic.

[0043] Distributed training methods inherently introduce heterogeneity. Although in data parallel models, all TPUs 106 exchange the same amount of local gradients with each other, more advanced techniques such as model partitioning and model parallelism make the gradients and activation outputs exchanged through the network inherently heterogeneous across partition boundaries. Communication latency, rather than throughput, also becomes more important in model parallelism because inference on a partitioned model often depends on output from another partition. In addition, reinforcement learning training also makes TPU network traffic heterogeneous, with one group of TPU workers 106 acting as agents that perform inference, while another group of TPU workers 106 performs a classic iterative training loop. Traffic within and between these groups is inherently different. Therefore, dynamic queue allocation is desired to address this heterogeneity.

[0044] In addition to multiple communication ports, channels, and on-chip hardware processing modules (routers 102), each TPU 106 also includes a shared programmable buffer queue storage structure 104. The buffer queue of each port can be controlled by software to different depths and is stored in a shared random access memory (RAM) (e.g., static RAM (SRAM) or dynamic RAM (DRAM)). In some implementations, each port of the TPU 106 is assigned a start address and an end address in the shared RAM 104 to specify the queue area of ​​the specific port within the shared memory space.

[0045] The size of the buffer queues associated with the various network ports of the TPU accelerators 106 is critical in tuning the overall training communication throughput. For a given link that experiences a lot of traffic, allocating too little buffer queue space for the port at the receiver TPU 106 will cause the sender TPU 106 to frequently stop its packet emission, slowing down traffic and potentially becoming a bottleneck for synchronous training communications. Each queue can be further divided into a number of virtual channels for queue allocations per channel. All queues correspond to unidirectional links between two accelerators, although the interconnect link is bidirectional.

[0046] Figure 2 An example shared memory 200 is illustrated with buffer queues allocated for port-to-port traffic. Traffic in the buffer queues can be outbound-to-port traffic (e.g., traffic generated from a TPU and outbound to a different TPU), port-to-inbound traffic (e.g., traffic generated outside a particular TPU and to be ingested by a particular TPU), or port-to-port traffic (e.g., traffic passing through a TPU in a TPU network). In addition, each port can be subdivided into multiple virtual channels, each of which provides a unique communication path. Although Figure 2Only port-to-port queues allocated in shared memory are depicted, but all three types of traffic or any suitable combination thereof may exist and be within the scope of the present disclosure. In the example shown, the FIFO (first-in, first-out) pointers of each queue use their start and end addresses as references for pushing and popping data from the queue. Port-to-inbound queues and outbound-to-port queues follow the same addressing scheme, with a smaller number of queues sharing RAM capacity. The same mechanism applies to accelerators with more or fewer ports than those illustrated.

[0047] These start and stop addresses (e.g., address 0 and address 1 as illustrated) can be controlled or assigned via software. A set of application programming interfaces (APIs) are exposed through the device driver of the TPU accelerator to write to the PCIe configuration space registers that specify the size of each buffer queue. The API converts the queue size into the start and end addresses of each queue for hardware operations. The disclosed solution can call these APIs to dynamically configure and allocate queue sizes for all ports in the TPU communication network.

[0048] Figure 3 is a flow chart of an example process for allocating shared memory based on expected latency according to an implementation of the present disclosure. The buffer queue allocation process checks the actively used ports in the (sub)networks participating in the training process and allocates queue sizes (credits) by weight of the uplink round-trip latency of each port. The scheme takes into account the heterogeneity of link media, inactive ports in non-homogeneous networks, and inactive ports directly connected to a failed link or a failed TPU accelerator chip. The scheme is divided into two steps: offline link calibration and online credit allocation.

[0049] During the offline link calibration, the average round trip time (RTT) for the different link media used in the TPU communication network is determined (302). This can be determined based on historical values, ping operations, or other methods, and provides a general expected latency for a given link based on the medium of the link. For example, a PCT trace may have a lower expected latency than a fiber optic cable.

[0050] During the online credit allocation phase, topology discovery may be performed to determine the network topology of the TPU communication network required or requested for a particular job to be executed (304). Topology discovery may identify which TPUs will participate in the upcoming job and which ports will be available in each TPU.

[0051] After determining the network topology, metadata can be accessed that provides information about the communication ports to be used for the job in the TPU communication network 306. For example, each TPU can save metadata in on-chip memory that identifies the port ID of each port, whether the port is used in the current topology, the media associated with each port (e.g., copper, optical, PCB, etc.), and the cable length associated with the port, etc.

[0052] The accessed metadata, combined with the expected latency determined during the offline calibration, can be used to determine the expected or desired RTT delay for each port, which can then be used to allocate a queue buffer in a shared memory for each port based on the expected RTT delay or latency (308). The queue size is calculated for each port in each TPU by the TPU itself. The TPU initially assigns a zero queue size to any unused port and then assigns the remaining queue size for each port. In some implementations, the queue size calculation process occurs independently and simultaneously for all TPUs with pending jobs in the TPU communication network. It indexes into the offline calibrated link latency table and calculates the queue size for each port in proportion to the round-trip time of the input port. In some implementations, the queue size is calculated by the following equation, where p is the port being calculated and N is the total number of ports actively used at the chip:

[0053]

[0054] Once the queue size for each port and channel is determined, the start and end addresses of the shared memory can be assigned for each port according to the size allocation (310). In some implementations, this is done using one or more APIs provided by the TPU architecture. For example, the API can write to a PCIe configuration space register that specifies the size of each buffer queue. The API converts the queue size into the start and end addresses of each queue for hardware operations. The disclosed solution can call these APIs to dynamically configure and allocate queue sizes for all ports in the TPU communication network.

[0055] When the assignment of addresses to the shared memory is complete, the TPU communication network can be enabled and an inference process or training run can be started (312). An inference process can be a long running process aimed at training a neural network or weighting it for future machine learning output.

[0056] Figure 4is a flow chart of an example process for allocating shared memory based on a profiled traffic run according to an implementation of the present disclosure. By performing a profiled traffic run, a more realistic representation of traffic demand during job execution can be determined. In some implementations, the process for allocating shared memory based on a profiled traffic run begins by completing the steps described in reference to Figure 3 The described process (eg, 302-310) is followed, and traffic queues are given initial allocations.

[0057] Once the initial buffer queue is assigned, one or more message traffic profiles may be selected for a profiling run on the network (402). Example profiles may include, but are not limited to, all-to-all, nearest neighbor traffic, or another synthetic pattern generated by a machine learning model (e.g., a ResNet-50 for model segmentation or a bidirectional encoder representation from a transformer (BERT) model). In some implementations, all previous traffic profiles or a combination thereof are combined for a profiling run. Before starting a profiling run, the software clears the hardware performance count registers in each TPU, effectively resetting them to zero. The hardware TPU performance count registers record the number of packets successfully received at each channel of each port of the TPU. Profiling traffic is injected into the TPU communication network, and the number of packets successfully received at each port of the TPU is counted by the hardware registers (404). A separate software system (e.g., via an API) reads the count registers and determines the total value of received packets for each port (and channel) during the profiling run. In some implementations, the profiling run is significantly shorter than the actual training run to be performed. For example, profiling runs can be on the order of minutes or hours, while training jobs can take days or weeks.

[0058] Once the profiling run is completed, the number of packets received at each lane of each port can be used to determine the queue size for that port (406). In some implementations, the queue size is determined using the following equation, where p is the port, N is the total number of ports on the ASIC, and packet_num is the number of packets received during the profiling run. It should be noted that the following equation produces a zero value for unused ports, so unused ports will be assigned a zero queue size.

[0059]

[0060] Once the queue size for each port and channel is determined, the start and end addresses of the shared memory can be assigned for each port according to the size allocation (310). In some implementations, this is done using one or more APIs provided by the TPU architecture. For example, the API can write to a PCIe configuration space register that specifies the size of each buffer queue. The API converts the queue size into the start and end addresses of each queue for hardware operations. The disclosed solution can call these APIs to dynamically configure and allocate queue sizes for all ports in the TPU communication network.

[0061] When the assignment of addresses to the shared memory is complete, the TPU communication network can be enabled and an inference process or training run can be started (312). An inference process can be a long running process aimed at training a neural network or weighting it for future machine learning output.

[0062] Profile-guided queue adjustment can reflect packet retransmissions caused by poor link quality, unbalanced link traffic caused by routing algorithms or faulty chips / links. Therefore, it can be a better solution to allocate more credits to ports / links that experience more traffic and need more buffer space for forwarding messages from one port to another.

[0063] Figure 5 is a flow chart of an example process for allocating shared memory based on traffic during execution according to an implementation of the present disclosure. Figure 5 The process described in can be similar to Figure 4 and Figure 3 The process (e.g., 302 to 312) begins. In some implementations, some or all of the processes occur sequentially. For example, the system may be as described with respect to Figure 3 Initially allocate memory based on predicted latency as described, then perform a profiling run and as described with respect to Figure 4 The memory is reallocated based on the profiling run as described, and finally, during the execution of the training run, according to the following reference Figure 5 The described processes are used to further reallocate memory. The present disclosure contemplates any suitable combination of these processes.

[0064] Once (via Figure 3 or Figure 4, or both) to achieve initial queue allocation, the training job can begin execution. During execution, the controller can monitor the packet counts for each port as recorded by the hardware TPU performance counter registers as described above. Initially, the packet statistics and hardware counters for each port can be cleared at the beginning of the run (502). The training job can execute for a predetermined amount of time, and then the queues can be adjusted based on the packet counts for each port during job execution. In some implementations, message queues are periodically reallocated during execution, or a set number of times (e.g., once an hour for the first five hours of execution). In some implementations, message queues are reallocated more frequently during the start of the training job, and the frequency becomes less frequent as execution proceeds (e.g., once every 10 minutes for the first hour, then once an hour for the next five hours, and then once a day thereafter). In some implementations, the period is configured by the user on a case-by-case basis.

[0065] When the predetermined time has expired and the queue is to be reallocated, the packet statistics for each port are read from the hardware TPU performance counter registers (504). The desired queue size is calculated (506). In some implementations, the queue size is calculated using the method described in reference to Figure 4 The desired queue size is calculated using the equation described above.

[0066] When the queue size is determined, the training job execution can be paused, allowing the TPU to fully ingest and process any pending message traffic and clear their queues (508). In some implementations, the controller verifies that the queue is clear (e.g., the shared RAM has no pending messages for each TPU) before continuing (510). In some instances, the pause lasts for a predetermined amount of time, and if the shared RAM is not clear (512), the training job execution resumes (516). In these instances, the controller can retry the reallocation at a later time. If it is determined that the shared RAM is clear, the queues in the shared RAM are each assigned new start and stop addresses (e.g., via an API as described above) (514). The controller can then clear the per-port packet statistics stored in the hardware TPU performance count registers, and the training job then resumes execution (502). During the execution of the training job, this process can be repeated as needed.

[0067] Figure 6 The diagram can be used for Figure 1 Figure 2 shows an example of a specialized logic chip in a machine learning accelerator. Figure 6 A block diagram of an example computing system 600 is shown that shows an ASIC used in a machine learning hardware accelerator as a method for accelerating tensor computations associated with a deep neural network (DNN). The system 600 may be, for example, as described with reference to Figure 1ASIC 102. System 600 typically includes a controller 602, a host interface 608, an input / output (I / O) link 610, a plurality of tiles including a first tile set 612 and a second tile set 614, a classifier portion 616, and a data bus identified in a bus diagram 618 (which is shown for clarity but is not included in system 600). Controller 602 typically includes a data memory 604, an instruction memory 606, and at least one processor configured to execute one or more instructions encoded in a computer-readable storage medium. Instruction memory 606 may store one or more machine-readable instructions executable by one or more processors of controller 602. Data memory 604 may be any of a variety of data storage media for storing and subsequently accessing various data related to calculations occurring within system 600.

[0068] The controller 602 is configured to execute one or more instructions related to tensor computations within the system 600, including instructions stored in the instruction memory 606. In some implementations, the data memory 604 and the instruction memory 606 are (one or more) volatile memory units. In some other implementations, the data memory 604 and the instruction memory 606 are (one or more) non-volatile memory units. The data memory 604 and the instruction memory 606 can also be in another form of a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device or device array, including a device in a storage area network or other configuration. In various implementations, the controller 602 can also be referenced or referred to as the core manager 602.

[0069] As depicted, the host interface 608 is coupled to the I / O link 610, the controller 602, and the classifier portion 616. The host interface 608 receives instructions and data parameters from the I / O link 610 and provides the instructions and parameters to the controller 602. In general, instructions can be provided to one or more devices in the system 600 via an instruction bus 624 (described below) and parameters can be provided to one or more devices in the system 600 via a ring bus 628 (described below). In some implementations, instructions are received by the controller 602 from the host interface 618 at an initial time and are stored in the instruction memory 606 at a later time for execution by the controller 602.

[0070] The classifier portion 616 is similarly coupled to the controller 602 and tile 7 of the second tile set 614. In some implementations, the classifier portion 616 is implemented as a separate tile within the system 600. In an alternative implementation, the classifier portion 616 is arranged or positioned within the controller 602 as a subcircuit or sub-device of the controller 602. The classifier portion 616 is typically configured to perform one or more functions on the accumulated pre-activation value, which is received as the output of the fully connected layer. The fully connected layer can be divided across the tiles in the tile sets 612 and 614. Therefore, each tile is configured to generate a subset of pre-activation values ​​(i.e., linear outputs) that can be stored in (one or more) memory cells of the tile. The classification result bus 620 provides a data path from the classifier portion 616 to the controller 602. Data including the post-function value (i.e., result) is provided to the controller 602 from the classifier portion 616 via the classification result bus 620.

[0071] The bus diagram 618 illustrates a data bus that provides one or more interconnected data communication paths between tiles of the first tile set 612 and the second tile set 614. The bus diagram 618 provides a legend for identifying the classification result bus 620, the CSR / host bus 622, the instruction bus 624, the mesh bus 626, and the ring bus 628, as shown in FIG. Figure 6 . In general, tiles are core components within the accelerator architecture of system 600 and are the focal point for tensor computations that occur in the system. Each tile is an individual computational unit that collaborates with other tiles in the system to accelerate computations across one or more layers of a multi-layer neural network. While tiles in tile sets 612, 614 are able to share execution of tensor computations associated with a given instruction, individual computational units are self-contained computational components that are configured to independently perform a subset of tensor computations relative to other corresponding tiles within tile sets 612, 614.

[0072] The CSR bus 622 is a single-master-multiple-slave bus that enables the controller 602 to transmit one or more instructions to set program configurations and read status registers associated with one or more tiles. The CSR bus 622 can be connected in a single daisy-chain configuration with one master bus segment and multiple slave bus segments. Figure 6 As shown in , the CSR bus 622 provides a communication coupling by connecting the tiles in the tile sets 612, 614 and the controller 602 in the ring to the bus data path of the host interface 610. In some implementations, the host interface 610 is a single master of the CSR bus ring and the entire CSR bus address space is memory mapped to the memory space in the host interface 610.

[0073] The CSR bus 622 can be used by the host interface 610 to perform one or more operations, including, for example, programming a memory buffer pointer in the controller 602 so that the controller 602 can begin to fetch instructions from the instruction memory 606, updating / programming various tile settings that remain static during one or more calculations (e.g., coefficient tables for polynomial approximation calculations), and / or loading / reloading firmware to the classification portion 616. In one example, a firmware reload may include a new function to be applied to a linear output (i.e., a pre-activation value). Therefore, each slave with access to the CSR bus 622 will have a different node identifier (node ​​ID) that is bound to the slave and identifies it. The node ID will be part of the instruction address and will be used, checked, or otherwise verified by the CSR slave (i.e., the controller 602, tiles 612, 614, and classifier 616) to determine whether the CSR packet is addressed to the slave.

[0074] In some implementations, one or more instructions can be transmitted by the host interface 602 through the controller 602. The instruction can be, for example, 32 bits wide, and the first 7 bits include header information indicating the instruction address / destination where the instruction will be received and executed. The first 7 bits of the header can contain data parameters representing a specific node ID. The slave (e.g., each tile) on the CSR bus ring can therefore check the header of the instruction to determine whether the request through the host (host interface 610) is addressed to the tile that checks the header. If the node ID of the header does not indicate that the destination is the check tile, the check tile copies the input CSR instruction packet to the CSR bus input connected to the next tile for inspection by the next tile.

[0075] The instruction bus 624 originates from the controller 602 and, similar to the CSR bus 622, also provides a communicative coupling via a bus data path connecting the tiles of the tile sets 612, 614 in the ring back to the controller 602. In one implementation, the controller 602 broadcasts one or more instructions via the instruction bus 624. The instructions broadcast by the controller 602 may be different than the instructions provided via the CSR bus 622. However, the manner in which the tiles receive and / or consume or execute instructions received via the bus 624 may be similar to the process for executing instructions received via the CSR bus 622.

[0076] In one example, the header (i.e., bitmap) of the instruction indicates to the receiving tile that the receiving tile needs to consume a specific instruction based on the bitmap associated with the instruction. The bitmap may have a specific width defined according to the bit. Instructions are typically forwarded from one tile to the next tile based on the parameters of the instruction. In one implementation, the width of the instruction bus 624 may be configured to be less than the size / width of the instruction. Therefore, in such a configuration, the transmission of instructions will be on several cycles and the bus stop of the instruction bus 624 will cause the decoder to place the instruction received at the tile in the appropriate target instruction buffer associated with the tile.

[0077] As further described below, the tiles in tile sets 612, 614 are typically configured to support two major categories of instructions. The two major categories can also be referred to as instruction types. Instruction types include tensor operations (TensorOp) instructions and direct memory access (DMAOp) instructions. In some implementations, DMAOp instructions have one or more specializations that allow concurrency. One or more specializations can be referred to as DMAOp instruction subtypes or opcodes. In some cases, each unique and / or valid DMAOp instruction type / subtype tuple will have a separate instruction buffer within a specific tile.

[0078] At a particular tile of tiles 612, 614, a bus stop associated with the instruction bus 624 will check the header bitmap to determine the instruction type / subtype. The instruction can be received by the tile and subsequently written to the instruction buffer of the tile before the instruction is executed by the tile. The instruction buffer of the tile to which the instruction is written can be determined by the type / subtype indicator / field of the instruction. The instruction buffer can include a first-in-first-out (FIFO) control scheme that gives consumption priority to one or more related instructions. Therefore, under the FIFO control scheme, instructions of the same type / subtype will always be executed in the order in which the instructions arrive at the instruction bus.

[0079] The different instruction buffers within a tile are the TensorOp instruction buffer and the DMAOp instruction buffer. As indicated above, instruction types include TensorOp instructions and DMAOp instructions. With respect to DMAOp instructions, instruction subtypes (indicating a "write" buffer location) include the following: 1) mesh inbound instruction buffer; 2) mesh outbound instruction buffer; 3) narrow-wide DMA instruction buffer; 4) wide-narrow DMA instruction buffer; and 5) ring bus DMA instruction buffer. Reference will be made to Figure 7These buffer locations are described in more detail. Wide and narrow designations are used throughout the specification and generally refer to the approximate size in terms of the width (bits / bytes) of one or more memory cells. As used herein, "narrow" may refer to one or more memory cells each having a size or width of less than 16 bits and "wide" may refer to one or more memory cells each having a size or width of less than 64 bits.

[0080] The mesh bus 626 provides a different data communication path from the CSR bus 622, the command bus 624, and the ring bus 628 (described below). Figure 6 , the mesh bus 626 provides a communication path that couples or connects each tile to its corresponding adjacent tile in both the X and Y dimensions. In various implementations, the mesh bus 626 can be used to transport input activations between one or more narrow memory cells in adjacent tiles. As shown, the mesh bus 626 does not allow input activation data to be forwarded directly to non-adjacent tiles.

[0081] In various implementations, the mesh bus 626 and the various tiles connected via the mesh bus 626 can have the following configuration. The four corner tiles of the mesh have two outbound ports and two inbound ports. The four edge tiles of the mesh have three inbound ports and three outbound ports. All non-edge non-corner tiles have four inbound ports and four outbound ports. In general, given an example N x N tile layout, an edge tile is a tile with only three adjacent tiles, while a corner tile is a tile with two adjacent tiles. With respect to the data flow methodology via the mesh bus 626, in general, each input activation arriving via the mesh bus 626 for a particular tile must be submitted to one or more narrow memory cells of the tile. Moreover, for tile configurations with less than four inbound ports, a DMAOp instruction can write a zero value to a location in the narrow memory of the tile instead of waiting for data on the absent input port. Likewise, for tile configurations with less than four egress ports, the DMAOp instruction will not perform narrow memory reads and port writes associated with transfers to any absent ports.

[0082] In some implementations, the location or address of the narrow memory cell(s) to which a particular input activation will be written or read from will be generated by the tensor traversal unit (hereinafter "TTU") based on the inbound / outbound DMAOp provided via the mesh bus 626. The inbound DMAOp and outbound DMAOp may be executed concurrently and any required synchronization will be managed by a synchronization flag control scheme implemented by the controller 602. Figure 7 Describe TTU in more detail.

[0083] The ring bus 628 originates from the controller 602 and, similar to the CSR bus 622 and instruction bus 624, also provides communication coupling via bus data paths connecting tiles 612, 614 in the ring back to the controller 602. In various implementations, the ring bus 628 typically connects or couples all wide memory cells in all tiles 612, 614 (see below). Figure 7 614). Thus, the payload width of the ring bus 628 corresponds to the width of the wide memory cells arranged within each tile of the tile sets 612, 614. As discussed above, the ring bus 628 also includes a bitmap header indicating the tiles that need to consume payload data including instructions or parameters passed via the ring bus 628.

[0084] With respect to data (i.e., payload) received at a particular tile via the ring bus 628, in response to receiving the information, each tile will zero (i.e., clear) the location data indicated in the bitmap header unique to the receiving tile before forwarding the data to another tile. Thus, when the header bitmap does not have the remaining bit set data indicating the particular tile that will receive the payload, forwarding of the payload to another tile will cease. Payload data generally refers to activations and weights used by one or more tiles during tensor computations performed based on deeply nested loop execution.

[0085] In some implementations, the controller 602 may be described as part of the ring bus 628. In one example, for a DMAOp instruction executed within a particular tile, the controller 602 may be used to pop data / payload from a ring bus stop and forward the payload to a ring bus stop in the next tile in the ring. The controller 602 may also cause the payload data to be submitted to one or more wide memory cells of the tile if such action is required by an instruction in the bitmap header. The address of the one or more wide memory cells to which the data needs to be written may be generated by a DMAOp instruction within a particular tile.

[0086] In various implementations, each tile of the tile set 612, 614 can be either a producer of payload data or a consumer of payload data. When a tile is a producer of payload data, the tile reads data from one or more of its wide memory cells and multicasts the data through the ring bus 628 for consumption by one or more other tiles. When a tile is a consumer of payload data, the tile receives and writes the data to one or more wide memory cells within the tile and forwards the payload data for consumption by one or more other tiles. With regard to the movement of payload data via the ring bus 628, there is usually only one producer / host of data on the ring bus 628 at any given time. The DMAOp instruction execution order (e.g., FIFO control scheme) in all tiles will ensure that there is only one producer / host of data on the ring bus 628 at a given time.

[0087] In some implementations, the controller 602 uses a sync flag control architecture to ensure that only one producer / master of payload data is present on the ring bus 628 at a given time. In one example, each write by a tile to the ring output will trigger an increment of the corresponding sync flag count. The controller 602 can examine the payload data to determine the number of data blocks or segments that comprise the payload. The controller 602 then monitors execution by the tile to ensure that the expected number of data segments are forwarded and / or consumed by the tile before another tile executes in master mode.

[0088] When there are local multicast groups connected via the ring bus 628 that have no overlapping areas on the ring bus, an exception occurs to ensure that there is only one producer / host of data at a given time on the ring bus 628. For example, tile 0 (host) may multicast (i.e., produce data) to tiles in the tile 0 - tile 3 group, while tile 4 (host) may do so to tiles in the tile 4 - tile 7 group. An important requirement of this dual-host multicast approach is that different multicast groups must not be allowed to see each other's data packets, as packet overlap may occur and result in one or more data calculation errors.

[0089] As in Figure 6As shown in , the controller 602 provides a communication data path that couples or connects the tiles in the tile sets 612, 614 to the I / O 610, and includes several core functions. The core functions of the controller 602 generally include: feeding one or more I / O input activations to the tiles in the tile sets 612, 614, feeding one or more input activations and parameters received from the I / O 610 to the tiles, feeding one or more instructions received from the I / O 610 to the tiles, sending I / O output activations to the host interface 608, and serving as a ring stop for the CSR bus 622 and the ring bus 628. As described in more detail below, the first tile set 612 and the second tile set 614 each include a plurality of tiles that are used to perform one or more tensor calculations based on a deep loop nest including an inner loop and an outer loop.

[0090] The system 600 generally operates as follows. The host interface 608 provides one or more instructions to the controller 602, which defines the direct memory access operation (DMAOp) that occurs for a given calculation. The descriptor associated with the instruction fed to the controller 602 will include information required by the controller to facilitate large-scale dot product calculations associated with multidimensional data arrays (tensors). In general, the controller 602 receives input activations, tile instructions, and model parameters (i.e., weights) from the host interface 608 for performing tensor calculations for a given layer of a neural network. The controller 602 can then cause the instruction to be multi-waved to tiles 612, 614 in a data stream manner defined by (one or more) instructions. As discussed above, the tile consuming the instruction can then broadcast new / subsequent instructions to another tile based on the bitmap data in the instruction header.

[0091] With respect to data flow, input activations and parameters are transmitted to tiles of tile sets 612, 614 via ring bus 628. Each of tiles 612, 614 will store a subset of input activations required to compute the subset of output activations assigned to that particular tile. The DMAOp instruction of the tile will cause the input activations to be moved from wide memory to narrow memory. When the required input activations, parameters / weights, and computation instructions (TTU operations, memory addresses, etc.) are available in the tile, computation within the tile begins. The computation occurring within the tile ends when the MAC operator (described below) within the tile completes all dot product operations defined by the instruction set and pre-activation function is applied to the result of the multiplication operation (i.e., output activation).

[0092] The result of one or more tensor calculations includes writing the output activation of the calculation layer to the (one or more) narrow memory cells of the tile performing the calculation. For some tensor calculations, there will be a transmission of the output edge activation to the adjacent tile via the mesh bus 626. When the calculation spans multiple layers, it is required to transmit the output edge activation to the adjacent tile to calculate the output activation for the subsequent layer. When the calculation of all layers is completed, DMAOp will move the last activation to the classifier tile 616 through the ring bus 628. The controller 602 will then read the last activation from the classifier tile 616 and execute DMAOp to move the last activation to the host interface 608. In some implementations, the classifier portion 616 performs the calculation of the output layer (i.e., the last layer) of the NN. In other implementations, the output layer of the NN is one of the classifier layer, regression layer, or another layer type typically associated with a neural network.

[0093] Figure 7 The diagram shows that Figure 1 An exemplary neural network (NN) computation block 700 used in the ASIC (e.g., TPU 106) is shown. In general, the exemplary tile 700 may correspond to the reference Figure 6 Any of the tiles within the first tile set 612 and the second tile set 614 discussed above. In various implementations, the computing tile 700 may also be referenced or referred to as a computing unit 700. Each computing tile 700 is a self-contained computing unit that is configured to execute instructions independently relative to other corresponding tiles within the tile sets 612, 614. As briefly discussed above, each computing tile 700 executes two types of instructions, TensorOp instructions and DMAOp instructions. In general, each instruction type will include a computing operation associated with a deep loop nest and therefore each instruction type will typically be executed at multiple times to ensure completion of all loop iterations.

[0094] As discussed in more detail below, different instruction types are executed by independent control units within the compute tile 700 that synchronize on data through synchronization flag control managed within the compute tile 700. The synchronization flag control manages concurrency between the execution of different instruction types within the compute tile 700. Each compute operation associated with each instruction type will be executed in a strict order (i.e., first in, first out) as issued. With respect to two instruction types (TensorOP and DMAOp), there is no ordering guarantee between these different instruction types and each type is handled by the compute tile 700 as a separate thread of control.

[0095] With respect to data flow construction, compute tile 700 generally includes data path 702 and data path 705, each of which provides a communication path for data flow entering and leaving compute tile 700. As described above, system 600 includes three different data bus structures arranged in a ring configuration - CSR bus 622, instruction bus 624, and ring bus 628. Figure 7 , data path 705 corresponds to instruction bus 624, while data path 702 generally corresponds to one of CSR bus 622 and ring bus 628. As shown, data path 702 includes ring output 703 providing an output path for data to leave compute tile 700 and ring input 704 providing an input path for data to enter compute tile 700.

[0096] The compute tile 700 also includes: a TensorOp control 706, which includes a TensorOp tensor traversal unit (TTU) 726; and a DMAOp control 708, which includes a DMAOp TTU 728. The TensorOp control 706 generally manages writing and reading TensorOpTTU registers 732 and performs traversal operations for execution by the TensorOp TTU 726. Likewise, the DMAOp control 708 generally manages writing and reading DMAOp TTU registers 734 and performs traversal operations for execution by the DMAOp TTU 728. The TTU registers 732 include an instruction buffer for storing one or more instructions, the instructions including operations to be performed by the TensorOP TTU 726 when the instructions are executed by the TensorOp control 706. Likewise, the TTU registers 734 include an instruction buffer for storing one or more instructions, the instructions including operations to be performed by the TensorOP TTU 708 when the instructions are executed by the DMAOp control 708. As described further below, the TTU is used by the compute tile 700 to traverse array elements of one or more tensors that typically reside in narrow memory 710 and wide memory 712 .

[0097] In some implementations, certain instructions for execution by the compute tile 700 arrive at the tile via the data path 705 (i.e., part of the instruction bus 624). The compute tile 700 will examine the header map to determine the instruction type (TensorOp or DMAOp) and the instruction subtype (read operation or write operation). The instruction(s) received by the compute tile 700 are then written to a specific instruction depending on the instruction type. In general, instructions are received and stored (i.e., written to a buffer) before being executed by components of the compute tile 700. As shown in Figure 7As shown in , the instruction buffers (ie, TensorOp TTU registers 732 and DMAOp TTU registers 734) may each include a first-in-first-out (FIFO) control scheme that prioritizes consumption (execution) of one or more related instructions.

[0098] As briefly discussed above, a tensor is a multidimensional geometric object and example multidimensional geometric objects include metrics and data arrays. An algorithm including a deep nested loop can be executed by the computation tile 700 to perform tensor calculations by iterating one or more nested loops to traverse an N-dimensional tensor. In an example computation process, each loop of the loop nest can be responsible for traversing a specific dimension of the N-dimensional tensor. As described herein, the TensorOp control 706 typically performs one or more tensor operations that drive the dimensional elements of a particular tensor construction to be traversed and accessed to complete a sequence of calculations defined by the deep nested loops.

[0099] The compute tile 700 also includes a narrow memory 710 and a wide memory 712. The narrow and wide designations generally refer to the size of the width (bits / bytes) of the memory cells of the narrow memory 710 and the wide memory 712. In some implementations, the narrow memory 710 includes memory cells each having a size or width of less than 16 bits, and the wide memory 712 includes memory cells each having a size or width of less than 32 bits. Typically, the compute tile 700 receives input activations via the data path 705, and the DMA control 708 performs operations to write the input activations into the narrow memory 710. Similarly, the compute tile 700 receives parameters (weights) via the data path 702, and the DMA control 708 performs operations to write the parameters into the wide memory 712. In some implementations, the narrow memory 710 can include a memory arbiter, which is typically used in a shared memory system to decide which control device (e.g., TensorOp control 706 or DMAOp control 708) will be allowed to access the shared memory cells of the narrow memory 710 for each memory cycle.

[0100] The computation tile 700 also includes an input activation bus 716 and a MAC array 714, including a plurality of cells, each of which includes a MAC operator 715 and a summing register 720. In general, the MAC array 714 performs tensor calculations including arithmetic operations related to dot product calculations using the MAC operators 715 and the summing register 720 across the plurality of cells. The input activation bus 716 provides a data path in which input activations are provided one by one from the narrow memory 710 for corresponding accesses of each MAC operator 715 through the MAC array 714. Thus, based on the one-by-one broadcasting of the input activations, a single MAC operator 715 of a particular cell activates each received input. The arithmetic operations performed by the MAC operators of the MAC array 714 typically include multiplying the input activations provided by the narrow memory 710 with parameters accessed from the wide memory 712 to produce a single output activation value.

[0101] During arithmetic operations, partial sums may be accumulated and stored in corresponding, e.g., sum registers 720 or written to wide memory 712 and re-accessed by a particular cell of MAC array 714 to complete a subsequent multiplication operation. The tensor computation can be described as a first part and a second part. The first part is completed when an output activation is generated, e.g., by completing a multiplication of an input activation and a parameter to generate an output activation multiplication operation. The second part includes applying a nonlinear function to the output activation, and the second part is completed when the output activation is written to narrow memory 710 after application of the function.

[0102] The computation tile 700 also includes: an output activation bus 718; a nonlinear unit (NLU) 722, including an output activation pipeline 724; an NLU control 738; and a reference diagram 730, which indicates the core attributes of the components in the computation tile 700. The reference diagram 730 is shown for clarity, but is not included in the computation tile 700. The core attributes include whether a particular component is a unit, a storage device, an operator, a control device, or a data path. In general, when completing the first part of the tensor calculation, the output activation is provided from the MAC array 714 to the NLU 722 via the output activation bus 718. After reaching the NLU 722, the data of the specified activation function received via the activation pipeline 724 is applied to the output activation and the output activation is then written to the narrow memory 710. In some implementations, the output activation bus 718 includes at least one pipeline shift register 736 and completing the second part of the tensor calculation includes shifting the output activation to the narrow memory 710 using the shift register 736 of the activation bus 718.

[0103] With respect to, for example, dot product calculations of two two-dimensional data arrays for a single computation tile 700, the MAC array 714 provides robust single instruction multiple data (SIMD) functionality. SIMD generally means that all parallel units (multiple MAC operators 715) share the same instruction (based on deep loop nesting), but each MAC operator 715 executes the instruction on a different data element. In a basic example, adding arrays [1, 2, 3, 4] and [5, 6, 7, 8] element by element in one cycle to obtain array [6, 8, 10, 12] would typically require four arithmetic units to perform operations on each element. By using SIMD, four units are able to share the same instruction (e.g., "add") and perform calculations in parallel. Therefore, the system 600 and computation tile 700 provide enhanced acceleration and parallelism in tensor calculations over previous approaches.

[0104] In one example, and as described in more detail below, a single instruction can be provided by the controller 602 to multiple compute tiles 700 (see Figure 6 The computing tile 700 can thus provide at least two forms of parallelism: 1) one form includes partitioning the output activations (corresponding to subsets of output neurons) among multiple tiles of the tile sets 612, 614; and 2) another form includes simultaneous computation (using a single instruction) of multiple subsets of the partitioned output neurons in the tiles of the tile sets 612, 614.

[0105] While the disclosed inventive concepts include those defined in the appended claims, it should be understood that the inventive concepts may also be defined in terms of the following embodiments.

[0106] Embodiment 1 is a method for memory allocation in a machine learning accelerator communication network. Typically, the method includes accessing metadata associated with multiple communication ports of an application-specific integrated circuit (ASIC). The metadata identifies, for each port, whether a particular port is used in the current configuration and the communication medium associated with the particular port. An expected latency is determined for each port based on the metadata, and a portion of a shared memory is assigned to each port. The shared memory is allocated by assigning zero memory to unused ports, determining the memory allocation for each port based on the expected latency, and assigning the start address and stop address of the shared memory to each port. Implementations may optionally include one or more of the following features.

[0107] Embodiment 2 is the method of embodiment 1, wherein assigning the start address and the stop address of the shared memory to each port comprises calling an application programming interface (API) by a device different from the ASIC.

[0108] Embodiment 3 is the method of embodiment 1 or 2, wherein a process using the machine learning accelerator communication network and the allocated shared memory is performed on the ASIC. In some embodiments, the process is training a neural network.

[0109] Embodiment 4 is the method of any one of embodiments 1 to 3, wherein the ASIC is a tensor processing unit (TPU).

[0110] Embodiment 5 is the method of any one of embodiments 1 to 4, wherein the communication medium identified in the metadata is a copper cable medium, an optical medium, a printed circuit board (PCB) medium, or any combination thereof.

[0111] Embodiment 6 is the method of any one of embodiments 1 to 5, wherein the memory is allocated according to the following equation:

[0112] Embodiment 7 is a system comprising: one or more computers and one or more storage devices storing instructions, the instructions being operable, when executed by the one or more computers, to cause the one or more computers to perform a method according to any one of claims 1 to 6.

[0113] Embodiment 8 is a computer storage medium encoded with a computer program, the program comprising instructions which, when executed by a data processing apparatus, are operable to cause the data processing apparatus to perform a method according to any one of claims 1 to 6.

[0114] Embodiment 9 is a method for memory allocation in a machine learning accelerator communication network, comprising determining a network topology for a network of machine learning accelerator ASICs. Access metadata associated with multiple communication ports of each ASIC within the network. The metadata identifies, for each of the multiple ports, whether a particular port is used in the current configuration and the communication medium associated with the particular port. For each port used in the network topology, determine the round trip time (RTT) delay. Allocate a portion of a shared memory to each of the multiple ports by: determining a memory allocation for each port proportional to the RTT delay, and assigning a start address and a stop address of the shared memory to each port for the determined memory allocation. Execute a process on the machine learning accelerator that sends profiled traffic into the network for a predetermined duration. Determine the number of received traffic packets for each port, and reallocate a portion of the shared memory to each of the multiple ports by: determining a memory allocation for each port proportional to the received packet count, and reallocating a start address and a stop address for each port of the determined memory allocation.

[0115] Embodiment 10 is the method of embodiment 9, wherein assigning and reassigning the start address and the stop address of the shared memory to each port comprises calling an API by a device different from the ASIC.

[0116] Embodiment 11 is the method of embodiment 9 or 10, wherein the memory allocation proportional to the determined RTT delay of each port is determined according to the following formula:

[0117]

[0118] Embodiment 12 is the method of any one of embodiments 9 to 11, wherein the memory allocation for each port proportional to the received packet count is determined according to the following formula:

[0119]

[0120] Embodiment 13 is the method of any one of embodiments 9 to 12, wherein the RTT delay is calculated prior to execution by transmitting and receiving one or more timing messages to determine the delay.

[0121] Embodiment 14 is the method of any one of embodiments 9 to 13, wherein profiling traffic includes all-to-all traffic, nearest neighbor traffic, synthetic traffic profiles, or any suitable combination thereof.

[0122] Embodiment 15 is the method of any one of embodiments 9 to 14, wherein assigning a start address and a stop address of the shared memory to each port comprises calling an application programming interface (API) by a device different from the ASIC.

[0123] Embodiment 16 is the method of any one of embodiments 9 to 15, wherein the process is performed on the ASIC using a machine learning accelerator communication network and the allocated shared memory. In some embodiments, the process is training a neural network.

[0124] Embodiment 17 is the method of any one of embodiments 9 to 16, wherein the ASIC is a tensor processing unit (TPU).

[0125] Embodiment 18 is a system comprising: one or more computers and one or more storage devices storing instructions, which instructions, when executed by the one or more computers, are operable to cause the one or more computers to perform a method according to any one of claims 9 to 17.

[0126] Embodiment 19 is a computer storage medium encoded with a computer program, the program comprising instructions which, when executed by a data processing apparatus, are operable to cause the data processing apparatus to perform a method according to any one of claims 9 to 17.

[0127] Embodiment 20 is a method for allocating memory for a machine learning accelerator communication network, including determining a network topology for a network of machine learning accelerator application-specific integrated circuits (ASICs). Access metadata associated with multiple communication ports of each ASIC within the network. The metadata identifies, for each of the multiple ports, whether a particular port is used in the current architecture and the communication medium associated with the particular port. The metadata identifies, for each of the multiple ports, whether a particular port is used in the current configuration and the communication medium associated with the particular port. For each port used in the network topology, a round trip time (RTT) delay is determined. A portion of a shared memory is allocated to each of the multiple ports by: determining a memory allocation for each port that is proportional to the RTT delay, and assigning a start address and a stop address of the shared memory to each port for the determined memory allocation. A process is executed on the ASIC, the process using the machine learning accelerator communication network with the allocated shared memory. During execution of the process, the number of message packets received at each of the multiple ports in a first time period is determined. The desired portion size of the shared memory for each of the multiple ports is determined based on the number of message packets received at each port. The process is paused during a second time period and it is determined that the shared memory has been emptied of pending message packets. When the shared memory has been emptied, the start address and stop address of the shared memory for each port are reassigned according to the expected portion size and execution of the process is resumed.

[0128] Embodiment 21 is the method of embodiment 20, wherein assigning a start address and a stop address of the shared memory to each port comprises calling an application programming interface (API) by a device different from the ASIC.

[0129] Embodiment 22 is the method of embodiment 20 or 21, wherein the process is performed on the ASIC using the machine learning accelerator communication network and the allocated shared memory. In some embodiments, the process is training a neural network.

[0130] Embodiment 23 is the method of any one of embodiments 20 to 22, wherein the ASIC is a tensor processing unit (TPU).

[0131] Embodiment 24 is a system comprising: one or more computers and one or more storage devices storing instructions, which instructions, when executed by the one or more computers, are operable to cause the one or more computers to perform a method according to any one of claims 20 to 23.

[0132] Embodiment 25 is a computer storage medium encoded with a computer program, the program comprising instructions which, when executed by a data processing apparatus, are operable to cause the data processing apparatus to perform a method according to any one of claims 20 to 23.

[0133] The above description is provided in the context of one or more specific implementations. Various modifications, changes and arrangements may be made to the disclosed implementations without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the described or illustrated implementations, but will be given the widest scope consistent with the principles and features disclosed herein.

[0134] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, while bus lines are described as "controllable," not all bus lines need to have the same level of control. For example, there may be varying degrees of controllability, with some bus lines being controllable only where some bus lines are limited in the number of tiles from which they can source data or from which they can send data. In another example, some bus lines may be dedicated to providing data in a single direction, such as north, east, west, or south as described herein. In some cases, the actions described in the claims may be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

[0135] Embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in computer hardware, including the structures disclosed in this specification and their structural equivalents or a combination of one or more thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by a data processing device or for controlling the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, for example, a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0136] The term "data processing apparatus" includes various apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0137] The processes and logic flows described in this specification can be performed by one or more programmable computers that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by dedicated logic circuits such as FPGAs, ASICs, or GPGPUs (general purpose graphics processing units), and the apparatus can also be implemented as dedicated logic circuits such as FPGAs, ASICs, or GPGPUs (general purpose graphics processing units).

[0138] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or what can be claimed, but rather descriptions of features that can be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate implementations can also be implemented in a single implementation in combination. Conversely, the various features described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features of the claimed combination can be cut out from the combination, and the claimed combination can involve a sub-combination or a variation of the sub-combination.

[0139] Similarly, although the operations are depicted in the accompanying drawings in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order, or all of the illustrated operations are performed to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. Moreover, the separation of the various system components in the implementation described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can be generally integrated together in a single software product or encapsulated in multiple software products.

[0140] In addition to the above description, controls can be provided to the user to allow the user to select whether and when the systems, programs, or features described herein can enable the collection of user information. In addition, certain data can be processed in one or more ways before being stored or used to remove personally identifiable information. For example, the identity of the user can be processed so that the user's personally identifiable information cannot be determined, or the user's skin data and / or diagnosis cannot be identified as associated with the user. Therefore, the user can control what information is collected about the user and how the information is used.

[0141] Thus, specific implementations of the subject matter have been described. Other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequence order shown to achieve the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented memory allocation method for a machine learning accelerator communication network, the method comprising: For a current machine learning training process, access metadata associated with a plurality of communication ports of an application specific integrated circuit ASIC including a plurality of processing units, wherein the metadata identifies, for each of the plurality of communication ports, whether the corresponding communication port is used in a current network topology of the plurality of processing units for the current machine learning training process and a communication medium associated with the corresponding communication port; determining an expected latency for each of the plurality of communication ports based on the accessed metadata; Allocating a portion of the shared memory of the ASIC to each of the plurality of communication ports by: Assigning zero memory to unused communication ports; determining a memory allocation for each used communication port based on the expected latency; and A start address and a stop address of the shared memory are assigned to each communication port.

2. The method according to claim 1, wherein: Assigning the start address and the stop address of the shared memory to each communication port includes calling, by a device different from the ASIC, an application programming interface API provided by the ASIC to assign the start address and the stop address of the shared memory to each communication port.

3. The method of claim 1 , comprising executing a process on the ASIC that uses the machine learning accelerator communication network and the allocated shared memory.

4. The method according to claim 3, wherein: The process includes training a neural network.

5. The method according to claim 1, wherein: The ASIC is a Tensor Processing Unit TPU.

6. The method according to claim 1, wherein: The communication medium identified in the metadata includes at least one of the following: Copper cable medium; Optical media; or Printed Circuit Board PCB medium.

7. The method according to claim 1, wherein: Memory allocation is based on to be sure.

8. A computer-implemented memory allocation method for a machine learning accelerator communication network, the method comprising: Determine a current network topology of a network for a machine learning accelerator application specific integrated circuit ASIC for a current machine learning training process; accessing metadata associated with a plurality of communication ports of each ASIC within the network for the current machine learning training process, wherein the metadata identifies, for each of the plurality of communication ports, whether the corresponding communication port is used in the current network topology for the current machine learning training process and a communication medium associated with the corresponding communication port; Determine a round trip time (RTT) delay for each used communication port in the network topology; Allocating a portion of a shared memory of a corresponding ASIC to each of the plurality of communication ports by: determining a memory allocation for each communication port that is proportional to the RTT delay; and assigning a start address and a stop address of the shared memory to each communication port for the determined memory allocation; executing a first process on the machine learning accelerator to send profiled traffic into the network for a predetermined duration; determining, for each communication port, a number of received traffic packets; and reallocating a portion of the shared memory to each of the plurality of communication ports by: determining a memory allocation for each communication port that is proportional to the number of traffic packets received; and For the determined memory allocation, a start address and a stop address of the shared memory are reassigned for each communication port.

9. The method according to claim 8, wherein: Assigning and reassigning the start address and the stop address of the shared memory to each communication port includes calling, by a device different from the ASIC, an application programming interface API provided by the ASIC to assign and reassign the start address and the stop address of the shared memory to each communication port.

10. The method of claim 8, comprising executing a second process on the ASIC, the second process using the machine learning accelerator communication network and the allocated shared memory.

11. The method according to claim 10, wherein: The second process includes training a neural network.

12. The method according to claim 8, wherein: The ASIC is a Tensor Processing Unit TPU.

13. The method according to claim 8, wherein: The memory allocation proportional to the determined RTT delay is based on to be sure.

14. The method according to claim 8, wherein: The memory allocation for each communication port is proportional to the number of packets received based on to be sure.

15. The method according to claim 8, wherein: The RTT delay is calculated by transmitting and receiving one or more timing messages to determine the delay prior to execution of the current machine learning training process.

16. The method according to claim 8, wherein: The analyzed traffic is at least one of the following: All to all traffic; Nearest neighbor traffic; or Synthetic traffic profile.

17. A computer-implemented memory allocation method for a machine learning accelerator communication network, the method comprising: Determine a current network topology of a network for a machine learning accelerator application specific integrated circuit ASIC for a current machine learning training process; For the current machine learning training process, accessing metadata associated with a plurality of communication ports of each ASIC, wherein the metadata identifies, for each communication port of the plurality of communication ports, whether the corresponding communication port is used in the current network topology for the current machine learning training process and a communication medium associated with the corresponding communication port; Determine a round trip time (RTT) delay for each used communication port in the network topology; Allocating a portion of a shared memory of a corresponding ASIC to each of the plurality of communication ports by: determining a memory allocation for each communication port that is proportional to the RTT delay; and assigning a start address and a stop address of the shared memory to each communication port for the determined memory allocation; executing a process on the ASIC, the process using a machine learning accelerator communication network having an allocated shared memory; and During the execution of the process: determining a number of message packets received at each of the plurality of communication ports during a first time period; determining a desired portion size of the shared memory for each of the plurality of communication ports based on a number of message packets received at each communication port; pausing the process for a second period of time; Determining that the shared memory has been cleared of pending message packets; reassigning a start address and a stop address of the shared memory for each communication port for the expected portion size; and Resuming execution of the process.

18. The method according to claim 17, wherein: Assigning and reassigning the start address and the stop address of the shared memory to each communication port includes calling, by a device different from the ASIC, an application programming interface API provided by the ASIC to assign the start address and the stop address of the shared memory to each communication port.

19. The method according to claim 17, wherein: The ASIC is a Tensor Processing Unit TPU.

20. The method according to claim 17, wherein: The process includes training a neural network.

Images