Neural processing unit synchronization system and method

Abstract

Systems and methods for exchanging synchronization information between various processing units using a synchronization network are disclosed. Systems and methods of the present disclosure include a device that includes a host and associated various neural processing units. Each neural processing unit can include a command communication module and a synchronization communication module. The command communication module can include circuitry for communicating with the host device over a host network. The synchronization communication module can include circuitry that enables communication between the various neural processing units over a synchronization network. The various neural processing units can be configured to obtain a synchronized update of a machine learning model by each neural processing unit. The synchronized update can be obtained at least in part by exchanging synchronization information using the synchronization network. Each neural processing unit can maintain a version of the machine learning model and can synchronize the version using the synchronized update.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a neural processing unit synchronization system and method. BACKGROUND

[0002] Distributed execution of machine learning applications among multiple computing nodes (e.g., devices, central processing units, neural processing units, hardware accelerators, single cores, etc.) can improve performance of machine learning applications (e.g., support more complex machine learning tasks, execute machine learning tasks faster, etc.). During such distributed execution, each of the multiple computing nodes can maintain a state. Correct execution of the machine learning application can require repeated synchronization of these states. Such synchronization can include obtaining synchronization information generated by the individual computing nodes, generating updated states, and configuring the individual computing nodes with the updated states. Depending on the application, synchronization can take up a significant portion of the execution time of the machine learning application. For example, in some conventional distributed implementations, training time of a neural network can be dominated by the time required to synchronize the neural network parameters among the individual computing nodes during training. SUMMARY

[0003] Systems and methods of the present disclosure relate to a processing unit configurable to exchange synchronization information with other processing units using a synchronization network.

[0004] Embodiments of the present disclosure include a processing unit. The processing unit can include a first communication module including circuitry to communicate between a host unit and the processing unit over a first communication network. The processing unit can be configurable to receive training information for a machine learning model from the host unit using the first communication module.

[0005] The processing unit can include a kernel. The processing unit can be configurable to generate update information using the kernel using the training information.

[0006] The processing unit can include a second communication module including circuitry to communicate between the processing unit and at least one second processing unit over a second communication network. The second communication network can be separate from the first communication network. The processing unit can be configurable to transmit first synchronization information for updating the machine learning model to the at least one second processing unit using the training information. The first synchronization information can include or be based on the update information.

[0007] Embodiments of the present disclosure include an apparatus. The apparatus can include a first host device and a plurality of first processing units. Each first processing unit can include a first communication module and a second communication module. The first communication module can include circuitry to communicate with the first host device over a first network. The second communication module can include circuitry to enable communication between each of the plurality of first processing units and at least one other first processing unit over a second network. The first processing units can be configured to obtain a synchronized update of a machine learning model by each of the plurality of first processing units at least in part by exchanging synchronization information using the second network. Each first processing unit can maintain a version of the machine learning model and each first processing unit can synchronize each version by updating each version using the synchronized update.

[0008] Various embodiments of the present disclosure include a method of training a machine learning model. The method can include obtaining a selected training subset of a training dataset. The selection can be obtained by a first processing unit of a first group of processing units associated with a first host device from a first host device of a group of host devices using a first communication network. The method can also include generating a first update information of the machine learning model by applying a batch of the training subset to the machine learning model. The first update information can be generated using a command processor of the first processing unit. The method can also include obtaining a synchronized update at least in part by providing a first synchronization information to at least one other processing unit. The synchronized update can be obtained using a communication processor of the first processing unit and a second communication network. The first synchronization information can be the first update information or can be based at least in part on the first update information. The method can include updating the machine learning model using the synchronized update.

[0009] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure embodiments claimed. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings, which constitute a part of this specification, illustrate several embodiments and together with the description, explain the principles and features of the embodiments of the present disclosure. In the drawings:

[0011] FIG. 1A An exemplary system including a synchronization network interconnecting various neural processing units is depicted in accordance with some embodiments of the present disclosure.

[0012] FIG. 1B An exemplary cloud computing system including a plurality of computing servers is depicted in accordance with some embodiments of the present disclosure.

[0013] FIGS. 1C-1E An exemplary synchronization information exchange is depicted in accordance with some embodiments of the present disclosure.

[0014] FIG. 2A Logical components of an exemplary neural processing unit are depicted in accordance with some embodiments of the present disclosure.

[0015] FIG. 2B An exemplary kernel architecture is depicted in accordance with some embodiments of the present disclosure.

[0016] FIG. 3A and FIG. 3B An exemplary process for synchronizing updates between exemplary neural processing units is depicted in accordance with some embodiments of the present disclosure.

[0017] FIG. 4 Distribution of training tasks to various exemplary central processing units and distribution of training tasks to various neural processing units associated with each exemplary central processing unit in accordance with various embodiments of the present disclosure are depicted.

[0018] FIG. 5 An exemplary method for training a machine learning model using a synchronous network in accordance with some embodiments of the present disclosure is depicted. DETAILED DESCRIPTION

[0019] Reference will now be made in detail to the exemplary embodiments, which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings and the following description to refer to the same or like parts. Unless otherwise defined, technical or scientific terms have the meaning commonly understood by one of ordinary skill in the art. The embodiments of the present disclosure are described in sufficient detail to enable those skilled in the art to practice the embodiments of the present disclosure. It is to be understood that other embodiments can be utilized and that changes can be made without departing from the scope of the embodiments of the present disclosure. Accordingly, the materials, methods, and examples are illustrative only and not necessarily limiting.

[0020] Performance of machine learning (ML) tasks can be improved using processors or hardware accelerators (e.g., neural processing units) that are suitable for such tasks. A neural processing unit (NPU) according to various embodiments of the present disclosure can be configured to separate execution of an ML task and synchronization of the NPU. Such an NPU can configure two communication modules. A host communication module can support communication between the NPU and a host system using a first network. A synchronization module can support communication between the NPU and other NPUs (not necessarily controlled by the same host system or even not necessarily controlled by the same part of a computing device) using a second network. Further, the NPU can be configured to support propagation of updates state along computations (e.g., updates state can be generated and distributed as part of the exchange of synchronization information between NPUs). In some embodiments, the NPU can include a communication processor configured to support propagation of computations along updates state. Such a configuration can not require a host processor or device to perform synchronization of the NPU. With these technical improvements, the time for synchronization used by an ML task can be reduced, thereby reducing the total time required to execute the ML task.

[0021] Various embodiments of the present disclosure can combine out-of-order execution (e.g., software or hardware pipelining, etc.) to accelerate execution of ML tasks. An NPU according to various embodiments of the present disclosure can be configured to reorder instructions or distribute instructions among NPU components to improve utilization of computing resources. For example, the NPU can be configured to generate updates for a neural network using a command processor and a kernel, and distribute the updates using a communication processor. The NPU can be configured to use the command processor and the kernel to perform other tasks distributed to the NPU during distribution of the updates. Based on the structure of the ML task (e.g., whether data dependencies exist or the type of data dependencies, etc.), the other tasks can be related to the ML task (e.g., the task can be related to a next iteration of the ML task, etc.) or can not involve the ML task (e.g., the task can involve a separate instance of the ML task, another ML task, or another application).

[0022] Various embodiments of the present disclosure can support reduced complexity and increased performance of computing clusters. Some traditional computing cluster layouts include computing devices connected to a hierarchy of switches. For example, a cluster of computing devices can be connected to a aggregation switch, or to a top-of-rack switch, which in turn is connected to an aggregation switch. Multiple clusters of computing devices can be connected to an aggregation switch in this manner. The aggregation switch in turn can be connected to a router to provide connectivity to remote computing devices. The inclusion of switches and routers increases the complexity of the computing cluster and decreases the performance of the computing cluster. According to various embodiments of the present disclosure, interconnections between NPUs in computing devices can replace or supplement switches and routers. Performance and functionality requirements of traditional switches and routers can be reduced in consequence. In some embodiments, interconnections between NPUs can be implemented within a cluster of NPUs (not necessarily within the same computing device) instead of implementing a distributed system within a cluster of computing devices.

[0023] Distributed training of neural networks can use synchronous stochastic gradient descent across many distributed computing nodes to accelerate training. NPUs according to various embodiments of the present disclosure can exploit the data parallelism of this approach, and thus are particularly well-suited for distributed training of neural networks. However, while described herein with respect to such machine learning applications, embodiments of the present disclosure are not limited thereto. Rather, the described technical improvements (e.g., in application execution speed, NPU utilization, and computing cluster performance and complexity) can be realized by various embodiments of the present disclosure in the context of distributed applications (e.g., big data processing and querying, etc.) and computing clusters configured to perform such distributed applications.

[0024] FIG. 1A An exemplary system 100 is depicted that includes a synchronization network interconnecting various NPUs. The NPUs can be controlled (and communicate with) a host device using a host network. The host devices can communicate using an inter-host network. By offloading distribution of update information to the synchronization network, the amount of time required for the NPUs to synchronize in performing ML tasks can be reduced. Further, requirements on the inter-host network or the host network can be relaxed (e.g., to support simpler, cheaper, or more flexible implementations), or these networks can be configured to perform additional tasks (e.g., to support additional functionality of existing components or architectures, more efficient utilization, etc.). In some embodiments, the system 100 can be or implement a computing cluster or a group of communicatively connected computing clusters. For example, the system 100 can be implemented using processor clusters (e.g., CPUs and NPUs) disposed on a single device, separate devices (e.g., distributed computing devices), or some combination of single-processor or multi-processor devices.

[0025] Host devices (e.g., host 101a and host 101b, etc.) can be configured to control associated NPUs (e.g., NPU 107a and NPU 107b, etc.). The host devices and associated NPUs can be part of the same computing device or not. In some embodiments, a host device can push data or instructions to an NPU associated with the host device. Such data and instructions can be processed by the associated NPU, as discussed herein.

[0026] A host device can include a processor, such as a central processing unit, an application-specific integrated circuit, a field-programmable gate array, a graphics processing unit, etc. In some cases, a host device can be configured to perform an ML task. Such performance processing can include distributing a task to other host devices or NPUs associated with the host device (or receiving results from other host devices or NPUs associated with the host device). In various cases, a host device can be configured to receive a task from another host device and distribute the task (based on the received task) to an NPU associated with the host device. In such cases, the host device can receive results from the associated NPU and provide these results (or output based on these results) to other host devices.

[0027] A host device can include a host memory. The host memory can be an integrated memory or an external memory associated with the host device. The host memory can be implemented using techniques or devices known in the art (e.g., cache, RAM, solid state drive, magnetic drive, optical storage, etc.). For example, the host memory can be a double data rate synchronous dynamic random access memory (e.g., DDR SDRAM), etc. In comparison to memory integrated into an associated NPU, the host memory can be configured to store a large amount of data at a slower access speed. In some embodiments, the host memory can be configured to act as an advanced cache for an associated NPU. For example, data or instructions stored in the host memory can be transferred to one or more NPUs associated with the host device for performing an ML task.

[0028] In some embodiments, a host device can implement or support a compiler. A compiler can be a program or computer software that converts computer code written in a programming language into instructions executed by an NPU. In an ML task, a compiler can perform a variety of operations, such as preprocessing, lexical analysis, syntax analysis, semantic analysis, conversion of input program to an intermediate representation, initialization of a neural network, code optimization, and code generation, or various combinations of the above. For example, a compiler can generate a neural network with a predetermined architecture (e.g., connections between individual neurons and weights of individual neurons).

[0029] The inter-host network 103 may include a wired or wireless network capable of communication between various host devices. Such a network can be implemented using known network technologies, such as Ethernet, WiFi, cellular data links, and protocols like TCP / IP and UDP. In some embodiments, the inter-host network 103 may be implemented using network technologies such as Infiniband and RapidIO. The inter-host network 103 may include switches, routers, firewalls, or other components positioned between the various host devices (not shown).

[0030] The host network (e.g., host network 105a, host network 105b, etc.) enables communication between the host device and one or more NPUs. In some embodiments, the host network can be implemented using a bus interface (such as a PCIe bus) or network technologies such as Infiniband, RapidIO, NVLink, BlueLink, etc.

[0031] An NPU (e.g., NPU107a, NPU107b, etc.) can be configured to execute instructions received from an associated host device using a host network. The NPU can execute instructions received from the associated host device. The NPU can use data received from the associated host device or from other sources (e.g., data received from another system, database, streaming endpoint, etc.) using a host network to execute instructions. The NPU can also use synchronization data received from other NPUs using synchronization network 109 to execute instructions received from the associated host device. This document will combine... FIG. 2A Describe the various components of an exemplary NPU.

[0032] Synchronous network 109 can be configured to support communication between NPUs. This communication can be point-to-point. For example, an NPU can use synchronous network 109 to communicate directly with other NPUs without needing to insert any switches, hosts, routers, or other network devices between the NPUs. Synchronous network 109 can be implemented using high-bandwidth serial communication protocols such as NVLink and InfiniBand. FIG. 1A As shown, the synchronization network 109 can connect to different computing devices or to the NPU in different host devices.

[0033] The synchronization network 109 is not limited to any particular network topology. In various embodiments, the NPUs in a cluster can be connected according to a ring, hub-and-spoke, mesh, or other known topology. In some embodiments, each NPU in a cluster can be connected to all other NPUs in the cluster. In various embodiments, the synchronization network 109 can connect the NPUs into a hypercube topology or a ring-based topology as disclosed in U.S. Patent Application No. 16 / 777,731, filed January 30, 2020, and incorporated by reference herein in its entirety. Systems including NPUs connected into a hypercube topology or a ring-based topology can implement advanced ring-based and tree-based synchronization algorithms as described in U.S. Patent Application No. 16 / 777,771, filed January 30, 2020, and U.S. Patent Application No. 16 / 777,683, filed January 30, 2020, both of which are incorporated by reference herein in their entirety.

[0034] FIG. 1B An exemplary cloud computing system 130 including a plurality of computing servers (e.g., computing server 132 and computing server 134) is depicted. The cloud computing system 130 can be configured and interconnected as described for the FIG. 1A cloud computing system 100. For example, as shown in FIG. 1B computing server 132 can be configured to communicate internally using the host network 105a and externally using the inter-host network 103 and the synchronization network 109. The host device 101a can be configured to control the NPUs (e.g., NPU 111a and NPU 111b) using the host network 105a and communicate with other host devices using the inter-host network 103. The NPUs in the computing server 132 can be configured to communicate with each other and with the NPUs in other computing servers (e.g., computing server 134) in the cluster 130 using the synchronization network 109.

[0035] The cloud system 130 can provide cloud services with artificial intelligence (AI) capabilities, including image recognition, facial recognition, translation, 3D modeling, etc. It should be appreciated that the NPUs of the present disclosure can be deployed to computing devices in other forms. For example, such NPUs can also be integrated into portable computing devices such as a laptop, a smartphone, a tablet, or a wearable device.

[0036] FIGS. 1C-1DAn exemplary synchronization information exchange according to embodiments of the present disclosure is depicted. The synchronization information exchange can be an AllReduce exchange including a reduction phase and a broadcast phase. According to various embodiments of the present disclosure, distributed nodes (e.g., individual hardware accelerators, individual processors, or individual computing devices) can generate synchronization information. In the reduction phase, the synchronization information can be obtained from the individual distributed nodes. Then, a value can be generated from the obtained synchronization information. In a pass-through computation, the obtaining of the synchronization information and the generation of the value can be combined. For example, the synchronization information can be modified as it is passed between nodes, each node transmitting an updated value generated using a local value and a received value. In the distribution phase, the value is broadcast to the distributed nodes. According to various embodiments of the present disclosure, multiple values can be updated in this manner simultaneously.

[0037] FIG. 1C An exemplary synchronization information exchange between three nodes (e.g., node 151, node 153, and node 155) arranged in a linear topology is depicted. In this non-limiting example, each node has generated three items of synchronization information (e.g., node 151 has generated the values "3," "9," and "1"). The reduction phase in this example includes step 141 and step 143, and the distribution phase includes step 145 and step 147. The state of the nodes after the AllReduce exchange is depicted in step 149.

[0038] In step 141, the synchronization information generated by node 151 can be provided by node 151 to node 153. In this example of a pass-through computation, the synchronization information received from node 151 can be added by node 153 to the synchronization information generated by node 153 to create updated synchronization information. While this simple example describes the accumulation of values, embodiments of the present disclosure are not so limited. According to various embodiments of the present disclosure, other operations can also be performed. In step 143, the updated synchronization information generated by node 153 can be provided by node 153 to node 155. The updated synchronization information received from node 153 can be added by node 155 to the synchronization information generated by node 155 to create final synchronization information. While this simple example describes the final synchronization information as the sum of the synchronization information generated by the individual nodes, embodiments of the present disclosure are not so limited. According to various embodiments of the present disclosure, the final synchronization information can also be a function of the synchronization information (e.g., a weighted average or an unweighted average, etc.).

[0039] In step 145, the final synchronization information can be distributed from node 155 to node 153. The final synchronization information received by node 153 can overwrite or otherwise replace the synchronization information stored by node 153. In step 147, the final synchronization information can be distributed from node 153 to node 151. The final synchronization information received by node 151 can overwrite or otherwise replace the synchronization information stored by node 151. As shown in step 149, the state of all nodes can be synchronized in this manner.

[0040] FIG. 1D An exchange of synchronization information between three nodes (e.g., node 151, node 153, and node 155) arranged in a tree topology is depicted. The reduction phase includes a step 161 in which node 155 and node 153 communicate synchronization information to node 151. The synchronization information generated by node 151 is then added to the received synchronization information by node 151 to generate final synchronization information. While this example includes a two-level node tree including a single root node (e.g., node 151) and two leaf nodes (e.g., node 155 and node 153), embodiments of the present disclosure are not limited to such an arrangement. According to various embodiments of the present disclosure, an AllReduce exchange of a tree topology can include each root node corresponding to more or fewer leaf nodes, or other levels in the tree topology (e.g., node 151 can be a child of another node, which in turn can be a child of another node, etc.). The distribution phase includes a step 163 in which node 151 broadcasts the final synchronization information to node 155 and node 153. The final synchronization information received can overwrite or otherwise replace the synchronization information stored by node 153 and node 155. As shown in step 165, the state of all nodes can be synchronized in this manner.

[0041] FIG. 1E An exchange of synchronization information between three nodes (e.g., node 151, node 153, and node 155) arranged in a ring topology is depicted. As shown in this example, a given N nodes in a ring can each communicate 2*(N-l) times with two adjacent nodes in the ring. During each communication, a node transmits and receives synchronization information. In a first round of N-l communications (e.g., step 171 and step 173), each node accumulates the received synchronization information. In a second round of N-l communications (e.g., step 175 and step 177), each node overwrites or replaces the stored synchronization information with the respective received synchronization information. As shown in step 179, this can synchronize the state of all nodes.

[0042] FIG. 2ALogical components of an exemplary neural processing unit (NPU 201) are depicted in accordance with some embodiments of the present disclosure. The NPU 201 can include generation components that support distributed ML task execution, such as compute kernels 221, kernel memory 217, command processor 205, command DMA unit 207, and host communication module 203. The NPU 201 can also include synchronization components that support synchronization of the NPU 201 with other NPUs, such as inter-chip communication module 209 (ICC module 209), communication processor 211, communication memory 215, and communication DMA unit 213. The NPU 201 can include an on-chip network 223 that enables communication between the various components of the NPU 201 and the NPU memory 219. In some embodiments, the NPU 201 can include modules that support other debugging and communication functions, such as JTAG module 225, I2C module 227, and on-chip encoder / decoder for processing video and images (e.g., CoDEC 229). In accordance with various embodiments of the present disclosure, the NPU 201 can be configured to support at least partially separate execution of ML tasks and synchronization of the NPU 201. Such separate execution can support improved ML task performance, reduced complexity, and performance improvements in computing clusters that include such NPUs.

[0043] The host communication module 203 can be configured to support communication between the NPU 201 and a host device (e.g., a CPU that controls execution of one or more NPUs including the NPU 201) using a host network 105a. In some embodiments, the host network 105a can be an inter-chip bus connection component of a computing device. The host network 105a can be a PCIe bus, and the host communication module 203 can be a PCIe controller. Although described with respect to a PCIe bus and controller, other interfaces can be used (e.g., Infiniband, RapidIO, NVLink, bluelink, etc.), and the above description is not intended to be limiting.

[0044] In some embodiments, the NPU 201 can be configured to receive information (e.g., data or instructions) regarding an ML task from a host device (e.g., host 101a) using the host communication module 203. When the ML task is a training task, the information can include training information such as training instructions, ML model parameters (e.g., weights and biases of a neural network, or similar model parameters of other ML models), and training data (or information that enables retrieval of training data such as training data location, memory address, range of memory addresses, etc.). The information can also include hyperparameters (e.g., learning rate, loss function, random seed, number of nearest neighbors, distance metric, kernel type, or other hyperparameters applicable to the ML model being trained).

[0045] The command processor 205 can be configured to control tasks performed by the NPUs 201. In some embodiments, the command processor 205 can be configured to transfer information between a host device and the NPUs 201. For example, the command processor 205 can obtain data or instructions from the host device. In some embodiments, the command DMA unit 207 can obtain data or instructions in response to instructions (e.g., DMA load / store instructions, etc.) received from the command processor 205. In some cases, the command processor 205 can process the obtained data or instructions. For example, the command processor 205 can be configured to use data or instructions obtained from the host device to manage computations performed by the cores 221. Such data or instructions can include training information used to manage the cores to perform a training task. The training task can include generating update information for a machine learning model. In various cases, the command processor 205 can store the obtained data or instructions to the NPU memory 219, the core memory 217, a cache, buffer, or memory contained in the cores 221, or other memory of the NPUs 201 (e.g., by providing instructions to the command DMA 207). In various embodiments, the command processor 205 can instruct one or more cores 221 to perform computations according to previously stored instructions. In some embodiments, the command processor 205 can cause the NPUs 201 (or groups of one or more cores 221) to perform tasks in an ML task manner by providing a series of commands to other components of the NPUs 201. The command processor 205 can be configured to provide multiple sequences of such commands simultaneously to perform multiple ML tasks simultaneously (e.g., by multiple groups of one or more cores 221).

[0046] In some embodiments, the command processor 205 can interact with the host device under the supervision of a kernel mode driver (KMD). In some embodiments, the command processor 205 can modify commands for each core 221 to increase the probability of the core 221 processing in parallel. The modified commands can be stored in an instruction buffer. In some embodiments, the command processor 205 can be configured to coordinate one or more cores 221 to perform in parallel.

[0047] Command DMA 207 can be a direct memory access controller configured to assist in transferring data between memory associated with the host device and NPU 201. For example, command DMA 207 can assist in loading data or instructions from memory associated with the host device to command processor 205, NPU memory 219, kernel memory 217, a cache, a buffer or memory contained within a kernel, or other memory of NPU 201. In some embodiments, command DMA 207 can also assist in transferring data between multiple accelerators using a host network. Command DMA 207 can allow off-chip devices to access both on-chip and off-chip memory without causing an interrupt in a processor (e.g., CPU, etc.) of the host device. In addition, command DMA 207 can assist in transferring data between various components of NPU 201. For example, command DMA 207 can assist in transferring data between multiple kernels 221 or within one of kernels 221. Accordingly, command DMA 207 can also generate memory addresses and initiate memory read and write cycles. Command DMA 207 can also contain a number of hardware registers that can be read and written by command processor 205 or kernels 221, including memory address registers, byte count registers, one or more control registers, and other types of registers. These registers can specify the source of the transfer, the destination of the transfer, the direction of the transfer (read from or write to an input / output (I / O) device), the size of the transfer unit, or the number of bytes to transfer in one burst of data.

[0048] Inter-chip communication module 209 (ICC module 209) can enable NPU 201 to directly exchange information (e.g., data or instructions) with other devices (e.g., other NPUs). Such information can be exchanged using a synchronous network. As noted above with respect to FIG. 1A such a synchronous network can be separate from a host network that connects NPU 201 with a host device and separate from an inter-host network that connects various host devices. As described herein, in some embodiments, ICC module 209 can be physically or logically divided into multiple ports.

[0049] In some embodiments, the ICC module 209 can be configured to interact with other components of the NPU 201 through the communication memory 215. The ICC module 209 can be configured to store data received from other devices in the communication memory 215, or read data from the communication memory 215 for transmission to other devices. As described herein, in some embodiments, the communication memory 215 can include multiple physical or logical partitions, each corresponding to a port of the ICC module 209. The ICC module 209 can be configured to store information received by a port into a corresponding partition in the communication memory 215 (or read information for transmission from a corresponding partition in the communication memory 215 via a port) through the port from which the information was received. In some embodiments, each ICC module 209 can correspond to one of the individual ports.

[0050] The communication processor 211 can be configured to generate synchronization information, distribute synchronization information within the NPU 201, or manage synchronization information exchange with other NPUs. The communication processor 211 can perform the generation, distribution, and management according to instructions received from at least one of a host device or the command processor 205. According to various embodiments of the present disclosure, the communication processor 211 can provide command sequences to other components of the NPU (e.g., the communication DMA unit 213, the ICC module 209, the cores 221, etc.). Such commands, when executed by these other components of the NPU 201, can cause them to provide data to the communication processor 211 or exchange synchronization information with other NPUs.

[0051] The communication processor 211 can generate synchronization information using data generated by the NPU 201 (e.g., by one or more of the cores 221) or received from other NPUs. The communication processor 211 can obtain (e.g., using the communication DMA unit 213) data from the memory of the NPU 201 (e.g., the NPU memory 219, the core memory 217, or the communication memory 215). As described herein, the communication processor 211 can generate synchronization information as part of propagating a computation. In embodiments in which the ICC module 209 includes multiple ports, the communication processor 211 can be configured with hardware or software resources dedicated to each port in the ICC module 209.

[0052] The communication processor 211 can distribute (e.g., using the communication DMA unit 213) synchronization information between the memory of the NPU 201 (e.g., the NPU memory 219, the core memory 217, or the communication memory 215). For example, the communication processor 211 can obtain synchronization information from the communication memory 215 and the NPU memory 219, use the obtained information to generate updated synchronization information, and store the updated synchronization information into the NPU memory 219 or the core memory 217.

[0053] The communication processor 211 can manage the exchange of synchronization information with other NPUs by providing instructions to the ICC module 209 or managing data stored in the communication memory 215. For example, the communication processor 211 can store synchronization information to the communication memory 215 and cause the ICC module 209 to transmit the stored synchronization information to other NPUs. As another example, the communication processor 211 can load (e.g., into the communication processor 211) or distribute (e.g., into other memory of the NPU 210) synchronization information received by the ICC module 209 and stored in the communication memory 215. Continuing with this example, the communication processor 211 can load the synchronization information from the communication memory 215, generate updated synchronization information using the loaded synchronization information, write the updated synchronization information to the communication memory 215, and cause the ICC module 209 to transmit the updated synchronization information to other NPUs.

[0054] The communication DMA 213 can be a direct memory access controller configured to assist in transferring data between the ICC module 209 and other components of the NPU 201. For example, the communication DMA 207 can assist in transferring data or instructions from the communication memory 215 to the communication processor 211 or other memory in the NPU 201 (e.g., NPU memory 219, core memory 217, cache, buffer, or memory included in the core, or other memory of the NPU 201), or from the communication processor 211 or other memory in the NPU 201 to the communication memory 215. Similar to the command DMA 207, the communication DMA 213 can generate memory addresses and initiate individual read or write cycles of memory. The communication DMA 213 also contains a number of hardware registers that can be read and written by the communication processor 211 or the core 221, including memory address registers, byte count registers, one or more control registers, and other types of registers. These registers can specify the source of the transfer, the destination of the transfer, the direction of the transfer (read from or write to an input / output (I / O) device), the size of the transfer unit, or the number of bytes to transfer in one data burst.

[0055] Communication memory 215 can be a memory configured to support synchronization of NPU 201 with other NPUs. Communication memory 215 can be accessible by at least one of communication processor 211 and inter-chip communication module 209 (e.g., using communication DMA unit 213). In some embodiments, communication memory 215 can be implemented using SRAM. In various embodiments, communication memory 215 can be logically or physically divided into individual blocks corresponding to individual ports of ICC module 209. In some embodiments, each block can be configured to store data or instructions received from or provide stored data or instructions to the corresponding port of ICC module 209. In some such embodiments, communication processor 211 can have access to all blocks of communication memory 215.

[0056] Kernel memory 217 can be a memory configured to provide data storage to kernel 221 with higher bandwidth or lower latency than NPU memory 219. In some embodiments, kernel memory 217 can be implemented using SRAM. In some embodiments, kernel memory 217 can be organized into caches or buffers for shared use by kernel 221.

[0057] NPU memory 219 can be configured as a global memory for NPU 201. NPU memory 219 can be accessible by command processor 205 and kernel 221 (e.g., using command DMA unit 207). NPU memory 219 can also be accessible by communication processor 211 and communication memory 215 (e.g., using communication DMA unit 213). NPU 201 can use NPU memory 219 to transfer data or instructions between command processor 205 or kernel 221 and communication processor 211 or communication memory 215. For example, NPU 201 can store instructions or data received from a host device or command processor 205 in NPU memory 219 (e.g., using command DMA unit 207). NPU 201 can then distribute the stored data or instructions to communication processor 211 or communication memory 215 (e.g., using communication DMA unit 213).

[0058] In some embodiments, NPU memory 219 can include multiple memory blocks. Individual memory blocks can be DRAM memory and can form an integrated package with kernel 221 or other components of NPU 201. As a non-limiting example, NPU memory 219 can include eight on-chip high-bandwidth memory blocks (e.g., HBM2, etc.).

[0059] In some embodiments, the NPU 201 can also include a memory controller (not shown) configured to manage data transfers between other components of the NPU 201 (e.g., the command DMA unit 207, the communication DMA unit 213, or the cores 221) and specific memory blocks within the NPU memory 219. The memory controller can generate memory addresses and initiate memory read and write cycles. The memory controller can contain several hardware registers that can be read and written by one or more processors. The registers can include memory address registers, byte count registers, one or more control registers, and other types of registers. These registers can specify the source of the transfer, the destination of the transfer, the direction of the transfer (read from or write to an input / output (I / O) device), the size of the transfer unit, the number of bytes to transfer in one burst, or other typical features of a memory controller. In some embodiments, the NPU 201 can include a memory controller for each memory block within the NPU memory 219.

[0060] According to various embodiments of the present disclosure, the cores 221 can include one or more processing elements for performing tasks. The processing elements can be or include a single instruction multiple data (SIMD) architecture. Such an architecture can include one or more processing units configured to perform one or more operations (e.g., multiplication, complex multiplication, addition, multiply-accumulate, etc.) based on commands received from the command processor 704. The cores 221 can be communicatively coupled directly or through the on-chip network 223. As a non-limiting example of a direct connection, the cores 221 can be connected directly using a unidirectional ring bus (or other configuration suitable for efficient pipelining of neural network training or inference tasks). The cores 221 can be configured to perform various operations according to instructions. The instructions can be obtained by the cores 221 from the command processor 205, a host device, or an instruction buffer in a memory of the NPU 201 (e.g., the NPU memory 219, the core memory 217, or other memory of the NPU 201).

[0061] The on-chip network 223 can be a network configured to communicatively connect at least some components of the NPU 201. The on-chip network 223 can transfer data or instructions between these communicatively connected components of the NPU 201. In some embodiments, the on-chip network 223 can be a packet-switched network. Embodiments of the present disclosure are not limited to a particular implementation of the on-chip network 223.

[0062] The JTAG 225 can be or include a JTAG / TAP controller that can specify a dedicated debug port for implementing a serial communication interface (e.g., a JTAG interface) that enables low-overhead access to the NPU 201 without direct external access to the system address and data buses. The JTAG 225 can also have an on-chip test access port interface (e.g., a TAP interface) that implements a protocol for accessing a set of test registers that provide chip logic level and individual component functionality.

[0063] The I2C 227 can be a serial communication bus controller. When connected to a properly configured serial communication bus (e.g., an I2C bus) of a computing device, the I2C 227 can enable communication between the NPU 201 and other components of the computing device (e.g., off-chip memory or peripherals) that are connected to the serial communication bus.

[0064] The CoDEC 229 can be an encoder / decoder. The CoDEC 229 can be configured to perform image or video compression and decompression. The NPU 201 can be configured to use the CoDEC 229 to transfer images or video between the NPU 201 and a host device using a host network. For example, the NPU 201 can be configured to use the CoDEC 229 to decompress image or video data received from a host device or to compress image or video data prior to transmission to a host device. In this way, the CoDEC 229 using the NPU 201 reduces the host network bandwidth requirements for processing image or video data.

[0065] FIG. 2A The classification of the components depicted is not intended to be limiting. Multiple depicted components can be combined into a fewer number of components (or a larger number of components divided into a greater number of components) with equivalent functionality. For example, in some embodiments, the communication memory 215 can be combined into the NPU memory 219 or the core memory 217. Similarly, a processor can combine the functionality of the communication processor 211 and the command processor 204, or a DMA unit can combine the functionality of the communication DMA unit 213 and the command DMA unit 207. For example, in various embodiments, the inter-chip module 209 can be divided into multiple modules, each associated with a separate processor and communication memory. In some embodiments, certain functionality can not be present. For example, the NPU memory 219, the JTAG module 225, the I2C module 227, or the CoDEC 229 can not be present.

[0066] Further, while described above as including multiple on-chip memory blocks, the NPU memory 219 is not limited to such an architecture. The NPU memory 219 can instead be formed as a single memory block or include off-chip memory (e.g., DDR / GDDR memory, etc.).

[0067] FIG. 2B FIG. 13 illustrates an exemplary architecture of a kernel 240 (e.g., one of the one or more kernels 221) in accordance with some embodiments of the present disclosure. As shown, the kernel 240 can include an operation unit 241, a memory engine 243, a sequencer 244, an instruction buffer 245, a constant buffer 247, a local memory 246, etc. The depicted architecture is not intended to be limiting, as the systems and methods of the present disclosure can be used with other kernel architectures. FIG. 2B

[0068] The operation unit 241 can include one or more processing units configured to perform operations on received data (e.g., feature maps of a neural network). These operations can include multiplication, complex multiplication, addition, multiply-add, element-wise operations, etc. In some embodiments, the operation unit 241 can be configured to accelerate the performance of convolution operations or matrix multiplication operations. These operations can also include adjustment operations (e.g., pooling, de-pooling, up-sampling, or down-sampling operations), region of interest (ROI) operations, etc. In some embodiments, the operation unit 241 can include an adjustment unit, a pooling data path, etc.

[0069] The memory engine 243 can be configured to perform data copying within the kernel 240 or between the kernel 240 and other kernels. In some embodiments, the memory engine 243 can be assisted by a DMA unit (e.g., the command DMA unit 207 or the communication DMA unit 213) of the NPU 201. For example, such a DMA unit can perform (e.g., in response to an instruction from the memory engine 243) data copying from a local memory (e.g., the local memory 246 of the kernel 240) to the operation unit 241 or to other kernels. In some embodiments, the memory engine 243 can be configured to perform matrix transposition for fitting matrices for use in the operation unit. FIG. 2B

[0070] The sequencer 244 can be coupled with the instruction buffer 245 and configured to retrieve commands and distribute the commands to various components of the kernel 240. For example, the sequencer 244 can distribute operation commands (e.g., convolution commands, multiplication commands, pooling commands, etc.) to the operation unit 241 or distribute data copying commands to the memory engine 243. The sequencer 244 can also be configured to monitor the execution of a neural network task and parallelize sub-tasks of the neural network task to improve execution efficiency. In some embodiments, the operation unit 241 and the memory engine 243 can run in parallel under the control of the sequencer 244 according to instructions stored in the instruction buffer 245.

[0071] ​​Instruction buffer 245 can be configured to store instructions for core 240. In some embodiments, instruction buffer 245 is coupled with and provides instructions to sequencer 244. In some embodiments, instructions stored in instruction buffer 245 can be transferred or modified by command processor 205.

[0072] Constant buffer 247 can be configured to store constant values. In some embodiments, constant values stored in constant buffer 247 can be used by operation units 241 for batch normalization, quantization, dequantization, etc.

[0073] Local memory 246 can provide storage space with fast read / write speed. Local memory 246 can be part of core memory 217 or can be separate from core memory 217. The size of local memory 246 can be determined based on on-chip space requirements and data access requirements of core 240 during expected use. The size of memory 246 can be determined to reduce the frequency of data access requests to NPU memory 219. In this way, data access latency can be reduced. In some embodiments, local memory 246 can have a capacity of at least 100 MB. In some embodiments, local memory 246 can be implemented using SRAM to minimize data loading latency and energy consumption or evenly distributed on the chip to alleviate dense wiring and heating issues.

[0074] FIG. 3A and FIG. 3B Exemplary processes for synchronizing updates among various exemplary neural processing units are depicted in accordance with some embodiments of the present disclosure. These processes can involve the exchange of synchronization information between exemplary NPU 201 and other NPUs (e.g., NPU 301a, NPU 301b, NPU 301c, etc.). NPU 201 can be configured to generate or exchange synchronization information using ICC module 209, communication memory 215, communication processor 211, NPU memory 219, and core 221. NPU 201 can be so configured in accordance with various commands provided by command processor 205, a host device, etc. The various commands can be received directly from command processor 205, a host device, etc., or from one or more instruction buffers in NPU 201 that are configured to store such commands.

[0075] In FIG. 3A and FIG. 3BIn the illustrated non-limiting embodiment, the ICC module 209, the communication memory 215, and the communication processor 211 each include logical or physical subcomponents. The ICC module 209 can include ports (e.g., ports 302a, 302b, and 302c as illustrated). In some embodiments, each port can correspond to other NPUs (e.g., NPUs 301a, 301b, 301c, etc.). The communication memory 215 can include partitions (e.g., partitions 303a, 303b, 303c as illustrated). Each partition can correspond to a port included in the ICC module 209. In some embodiments (as illustrated in FIGS. 3A-3C), the communication processor 211 can include multiple processors (e.g., processors 304a, 304b, and 304c as illustrated). Each processor of the multiple processors can perform processing to generate synchronization information or manage interactions with ports included in the ICC module 209. In this way, FIG. 3A and FIG. 3B each port, each partition, and each processor can interact in accordance with various embodiments of the present disclosure and generate and exchange synchronization information between the NPU 201 and other NPUs. This generation and exchange of synchronization information can occur largely in parallel with ML tasks performed by the NPU 201, enabling faster execution of ML tasks and improved overall performance of the NPU 201. FIG. 3A and FIG. 3B each port, each partition, and each processor can interact in accordance with various embodiments of the present disclosure and generate and exchange synchronization information between the NPU 201 and other NPUs. This generation and exchange of synchronization information can occur largely in parallel with ML tasks performed by the NPU 201, enabling faster execution of ML tasks and improved overall performance of the NPU 201.

[0076] Each port in the ICC module 209 can be implemented using hardware, software, or a combination of hardware and software. Each port can support a communication link with another device (e.g., other NPUs) using a synchronization network (e.g., the synchronization network 109). Depending on the architecture of the NPU or data transfer requirements, the ICC module 209 can include port 1, port 2, port 4, port 8, or more ports. For example, an NPU with six ports can support a more complex synchronization network (and possibly faster or more efficient) than an NPU with four ports. In some embodiments, the communication link can be a point-to-point connection (e.g., a connection between each communication endpoint without intermediate routing or switching). In various embodiments, the communication link can be a serial connection. Each port can include a number of lanes (e.g., between 1 and 32 or more depending on the architecture of the NPU or data transfer requirements). For example, each port can include 8 lanes. Each lane can be implemented as a pair of unidirectional serial data links (e.g., two unidirectional differential conductor pairs). The ICC module 209 can include serialization units that support serialization / deserialization of transferred data (e.g., data can be converted from parallel to serial for transmission, or from serial to parallel upon reception).

[0077] The communication processor 211 can implement execution of the various processes depicted in FIGS. 1 and 2. See FIG. 3A and FIG. 3B (e.g., process 310, process 320, and process 330). In some embodiments, the communication processor 211 can implement execution of these processes using hardware or software resources dedicated to each port in the ICC module 209. For example, the communication processor 211 can include multiple processors, each running a process for managing a corresponding port in the ICC module 209 (e.g., checking received individual values, providing instructions to transmit individual values stored in the communication memory 215, configuring a port to communicate with a particular NPU, etc.). As another example, the communication processor 211 can include a single processor running multiple concurrent processes (e.g., processes), each for managing a corresponding port in the ICC module 209. In some embodiments, the communication processor 211 can include one or more arithmetic logic units (ALUs) for processing synchronization information. In some embodiments, each port in the ICC module 209 can have a corresponding dedicated ALU in the communication processor 211.

[0078] The individual partitions in the communication memory 215 can be implemented using hardware, software, or a combination of hardware and software. For example, the NPU 201 can implement the communication memory 215 using multiple separate on-chip memories. Each separate memory can correspond to one partition. As another example, the NPU 201 can implement the communication memory 215 using fewer separate on-chip memories than individual partitions (e.g., a single on-chip memory and two partitions). In such implementations, at least two partitions can be implemented as logical partitions in a single physical memory. In some embodiments, a port in the ICC module 209 can be configured to read and write synchronization information from a corresponding partition in the communication memory 215. In some implementations, each port in the ICC module 209 can only be able to access a corresponding partition in the communication memory 215. In some embodiments, all processes executed by the communication processor 211 can read and write to all partitions in the communication memory 215.

[0079] FIG. 3A and FIG. 1ESome of the processing described in the middle can generate synchronization information (e.g., updates 311, etc.) during execution of the ML task. For example, a kernel 221 can perform a training task on a deep learning neural network using a batch of training data, generating loss function gradients for updating the neural network parameters. In this non-limiting example, the generated synchronization information can be the loss function gradients, be a portion of the loss function gradients, or include the loss function gradients. In some embodiments, NPU memory 219 can be used to transfer the generated synchronization information from the generating component (e.g., kernel 221, command processor 205, kernel memory 217, etc.) to the synchronization component (e.g., communication processor 211, communication memory 215, ICC module 209, etc.) of NPU 201. In such various embodiments, the generating component can not be directly accessible by the synchronization component, and NPU 219 can be used to exchange synchronization information between the generating component and the synchronization component. Mediating the exchange of data between the generating component and the synchronization component in this way can reduce the interference of the synchronization component with the execution of the ML task, improving the performance of the system.

[0080] FIG. 1C An exemplary along-propagation computation synchronization process 310 according to various embodiments of the present disclosure is depicted. In some embodiments, process 310 can be performed according to or in response to instructions provided by communication processor 211 (e.g., by processing running on processor 304a). Process 310 can be performed as part of an AllReduce exchange (e.g., process 310 can implement an AllReduce exchange as described in U.S. Patent Application No. 16 / 209, 1 13, filed December 3, 2018, which is incorporated by reference in its entirety). FIG. 3A(Step 175 of the AllReduce exchange described in the diagram). As part of processing 310, kernel 221 may execute an ML task for generating update 311 that can be stored in NPU memory 219 (e.g., using command DMA unit 207). In some embodiments, update 311 may be transferred to a memory partition (e.g., partition 303a) of communication memory 215. Synchronization information 313 may be received from NPU 301a using a corresponding port (e.g., port 302a) of ICC module 209. Synchronization information 313 may be written to partition 303a (e.g., directly via port 302a using communication DMA unit 213, etc.). (e.g., by communication DMA unit 213 in response to instructions received from communication processor 211, etc.) Update 311 and synchronization information 313 will be loaded into processor 304b. In some embodiments, update 311 may be loaded directly from NPU memory 219. Processor 304b can generate synchronization information 319 (e.g., by performing logical or arithmetic operations using update 311 and synchronization information 313). Then, synchronization information 319 can be written to partition 303b of communication memory 215 (e.g., using communication DMA unit 213). Then, port 302b can obtain synchronization information 319 and transmit it to NPU 301b (directly via port 302b using communication DMA unit 213, etc.).

[0081] In some cases, synchronization information 319 can be synchronization information used by the NPU to update their state (e.g., as...). FIG. 1C (Step 175). In this case, synchronization information 319 can also be written to NPU memory 219 (e.g., using communication DMA unit 213, etc.). Synchronization information 319 can then be accessed by core 221 for ML tasks (e.g., using command DMA 207, etc.). In addition, NPU 301b can use the synchronization information 319 received from NPU 201 to update its state. NPU 301b can also transfer synchronization information 319 to other NPUs.

[0082] In various circumstances ( FIG. 3B (Not depicted in the text), after providing synchronization information 319 to NPU301b, NPU201 can receive synchronization information from NPU301b for updating its state (e.g., as shown in the reference). FIG. 3BPort 302b can store the received synchronization information in partition 303b according to instructions provided by the processing executing on processor 304b (as described in step 147). The received synchronization information can be copied to NPU 219, which is subsequently accessed by core 221 for ML tasks. The received synchronization information can also be copied to partition 303a (e.g., according to instructions provided by the processing executing on processor 304b). Port 302a can then transmit this synchronization information to NPU 301a, respectively.

[0083] FIG. 1DAn exemplary along- propagation compute synchronization process 320 according to various embodiments of the present disclosure is depicted. In some embodiments, process 320 can be performed according to or in response to instructions provided by communication processor 211 (e.g., by a process running on processor 304b). Process 320 can be performed as part of an AllReduce exchange. As a non-limiting example, NPU 201 can be an intermediate node in a tree of nodes. NPU 201 can be configured to receive synchronization information from child nodes in the tree (e.g., NPUs 301a and 301b), generate updated synchronization information based on the received synchronization information, and provide the updated synchronization information to a parent node in the tree of nodes. As part of process 320, kernel 221 can execute an ML task (e.g., using command DMA unit 207) that can be stored in NPU memory 219 for generating update 321. In some embodiments, update 321 can be transferred to a memory partition of communication memory 215 (e.g., partition 303a). Synchronization information 323 can be received from NPU 301a using a corresponding port of ICC module 209 (e.g., port 302a). Synchronization information 323 can be written to partition 303a (e.g., directly through port 302a using communication DMA unit 213, etc.). Synchronization information 325 can be received from NPU 301b using a corresponding port of ICC module 209 (e.g., port 302b). Synchronization information 325 can be written to partition 303b (e.g., directly through port 302b using communication DMA unit 213, etc.). Update 321, synchronization information 323, and synchronization information 325 will be loaded into processor 304b (e.g., by communication DMA unit 213 in response to instructions received from communication processor 211, etc.). In some embodiments, update 321 can be loaded directly from NPU memory 219. Communication processor 211 can generate synchronization information 329 (e.g., by performing logical or arithmetic operations using update 311 and synchronization information 313). Synchronization information 329 can then be written to partition 303c of communication memory 215 (e.g., using communication DMA unit 213, etc.). Port 302c can then obtain synchronization information 329 (directly through port 302c using communication DMA unit 213, etc.) and transfer synchronization information 329 to NPU 301c (e.g., a parent node in the tree of nodes).

[0084] In some cases (e.g., as shown), synchronization information 329 can be the synchronization information that the NPUs use to update their state (e.g., as shown in FIG. 3B). FIG. 3B FIG. 3B ​step 163). In this case, the synchronization information 329 can be written to the NPU memory 219 (e.g., using the communication DMA unit 213). The synchronization information 329 can then be accessed by one or more of the cores 221 for ML tasks (e.g., using the command DMA 207, etc.). In some such instances (not shown in FIG. 3A instead of writing the synchronization information 329 to the partition 303c, the synchronization information 329 can be written to the partition 303a and the partition 303b. The port 302a and the port 302b can then obtain the synchronization information 329 from the partition 303a and the partition 303b and transmit the synchronization information to the NPU 301a and the NPU 301b, respectively (e.g., child nodes in a tree of nodes).

[0085] In various instances (not shown in FIG. 3B After providing the synchronization information 329 to the NPU 301c, the NPU 201 can receive synchronization information from the NPU 301c for updating its state, in accordance with instructions provided by the processing performed on the processor 304c. The port 302c can store the received synchronization information in the partition 303c, in accordance with instructions provided by the processing performed on the processor 304c. The received synchronization information can be copied to the NPU 219 and subsequently accessed by the cores 221 for ML tasks. The received synchronization information can also be copied to the partition 303a and the partition 303b (e.g., in accordance with instructions provided by the processing performed on the processor 304c). The port 302a and the port 302b can then transmit the synchronization information to the NPU 301a and the NPU 301b, respectively.

[0086] From the foregoing description, it can be appreciated that the NPU 201 is not limited to a particular synchronization topology or synchronization processing. Rather, the connections between the NPUs in a cluster can depend on how the NPUs are connected using a synchronization network (e.g., the synchronization network 109). The topology can in turn depend on the number of connections supported by each NPU (e.g., the number of ports supported by each ICC module). In some embodiments, a program executing on a computing cluster formed by a plurality of NPUs can specify how to pass synchronization information between the NPUs so connected.

[0087] As noted above with respect to FIG. 4 and FIG. 4The synchronization information exchanged between the NPUs can be used by the kernels in the ML task. In some embodiments, the synchronization information can be used to update a state of the NPUs. The state can be saved in the NPU memory 219, the kernel memory 217, or a memory in each kernel 221. The synchronization information can be used to overwrite or update the state. For example, the state can include a version of a machine learning model, and the synchronization information can be used to overwrite or update the version of the machine learning model. Continuing this example, the state can be or include neural network weights and biases, and the synchronization information can be loss function gradients used to update these neural network weights and biases.

[0088] FIG. 1C Distributions of training tasks to various exemplary central processing units, and distributions of training tasks to various neural processing units associated with each exemplary central processing unit, are depicted in accordance with various embodiments of the present disclosure. In accordance with various embodiments of the present disclosure, a training task can be executed or interpreted (e.g., “performed”) by a system comprising a plurality of central processing units. At least one neural processing unit can be associated with each central processing unit. The neural processing units of the system can be configured to perform a synchronization task by exchanging synchronization information using a synchronization network. In this way, the system can perform machine learning tasks more quickly and efficiently.

[0089] In accordance with various embodiments of the present disclosure, the program 401 specifies the performance of a training task. In some embodiments, the system can be configured to modify the program 401 to support the performance of a synchronization task. In this example, the training task comprises training a neural network using mini-batch stochastic gradient descent. However, embodiments of the present disclosure are not limited thereto.

[0090] In accordance with some embodiments of the present disclosure, the program 401 can be executed by a host CPU of a computing system. In some embodiments, the computing system can comprise other central processing units (e.g., the computing system can be a computing cluster, a cloud computing system, etc.). In various embodiments, the computing system can be communicatively connected to other central processing units (e.g., the computing system can be a user device such as a tablet, laptop, desktop, etc. that is communicatively connected to a computing cluster, a cloud computing system, etc.). In various embodiments, the host CPU can distribute the performance of the training task to other CPUs. However, embodiments of the present disclosure are not limited thereto. In some embodiments, the host CPU can be associated with an NPU and can participate in the performance of the training task (e.g., the host CPU can be a CPU0 depicted in FIG. 1). FIG. 1C The host CPU and the CPU0 depicted in FIG. 1 can be the same CPU.

[0091] As one non-limiting example, the program 401 can specify the following training task:

[0092] For (e in epochs){

[0093] re_shuffle(&samples)

[0094] parallel_for(c in cpu_nodes){

[0095] parallel_for(w in workers[c]){

[0096] NPU_run(samples[w]);

[0097] }

[0098] }

[0099] }

[0100] Program 401 specifies that the training task is to be performed for a set of epochs. At each epoch, the training data will be reshuffled and a training run process (e.g., NPU_run) is performed using a different set of samples. Program 401 specifies that the training run process is to be performed in parallel by each of a set of central processing units (e.g., the central processors identified in “cpu_nodes”). Further, program 401 specifies that the training run process is to be performed in parallel by each of a set of processing devices (e.g., each NPU) associated with each central processing unit.

[0101] According to various embodiments of the present disclosure, a computing system can generate a set of tasks (e.g., task 411, task 413) for execution by a central processing unit. Each task can be tailored to the execution of the CPU. In the present non-limiting example, task 411 can be adapted for execution by CPU0 (which can be the host CPU in some embodiments):

[0102] parallel_for(w in workers[c0]){

[0103] NPU_run(samples[w]);

[0104] }

[0105] In this example, the task specifies that the training run processes are executed in parallel on processing devices (e.g., NPUs) associated with CPUO (e.g., w in processing device [c0]). Each training run process can be executed by a processing device using training data (e.g., samples[w]) associated with the processing device. According to various embodiments of the present disclosure, the training data sets associated with each processing device can be disjoint or overlapping. Various embodiments of the present disclosure are not limited to any particular method of associating training data with processing devices.

[0106] According to various embodiments of the present disclosure, each CPU can generate a set of sub-tasks for execution by processing devices (e.g., NPUs) associated with that CPU. For example, CPUO can generate sub-task 421 and sub-task 423, and CPU N can generate sub-task 425. In this non-limiting example, the sub-tasks can divide the execution of a training run process for a sample set w into batches as follows:

[0107] For (b in batches){

[0108] load(samples[b][w0]);

[0109] compute(samples[b][w0]);

[0110] all_reduce(workers);

[0111] b++;

[0112] }

[0113] In this example, the sub-tasks specify that the processing devices load training data corresponding to the batch and processing device combination (e.g., load(samples[b][w0] for worker w0). The processing devices then use the loaded training data and stored neural network model to compute a loss (e.g., compute(samples[b][w0]). An exchange is then performed that causes the all-reduce task (e.g., all_reduce(workers)) that synchronizes each processing device with the other processing devices to synchronize information across the entire processing device. In this non-limiting example, the loss information can be combined to generate an average loss. The stored neural network model used by each processing device can then be updated based on the average loss. In this way, the stored neural network model used by all processing devices remains synchronized.

[0114] According to various embodiments of the present disclosure, the processing device can be an NPU. The command processor of the NPU can execute instructions contained in sub-tasks. In some embodiments, the command processor can be configured to support execution of multiple independent sub-tasks. Continuing the previous example, in response to the “load (samples[b][w0])” instruction, the command processor can issue a DMA-LOAD command to the command DMA unit. The DMA-LOAD command can cause the command DMA unit to fetch the specified training data (e.g., from a memory external to the NPU or a memory of the NPU) and load the specified samples into a memory associated with at least one kernel of the NPU (e.g., a shared kernel memory, a memory dedicated to the kernel, etc.). Continuing the example, in response to the “compute (samples([b][w0])” instruction, the command processor can start a COMPUTE task on at least one kernel of the NPU. In some embodiments, starting the COMPUTE task can include configuring the at least one kernel to perform a computation task using the training data. The command processor can also issue an all-reduce command to a communication processor of the NPU in response to the “all_reduce(workers)” instruction. In some embodiments, issuing the all-reduce command can include configuring the communication processor to perform an appropriate all-reduce task.

[0115] According to various embodiments of the present disclosure, in response to the start of the COMPUTE task, the at least one kernel of the NPU can determine synchronization information (e.g., loss information) for synchronizing various NPUs. For example, in response to the first start of the computation task, the at least one kernel can perform the following operations:

[0116] error = forward(samples[b0][w0]);

[0117] grad[] = backprop(error);

[0118] HBM[*vars_ptr] = grad[];

[0119] In this non-limiting example, the at least one kernel computes a bias using training data corresponding to a combination of a batch and a group of NPUs; computes a gradient from the bias information; and stores the gradient to a location in a main memory (e.g., NPU memory 219, etc.) of the NPU.

[0120] In some embodiments, multiple kernels can be configured to determine the gradient together. For example, multiple kernels using training data corresponding to a combination of a kernel, a batch, and an NPU can generate multiple biases. In some embodiments, other kernels can obtain the multiple biases (e.g., from a multiple kernel pipeline, from memory, etc.) and compute the gradient.

[0121] In some embodiments, after completing the COMPUTE task (e.g., immediately after completion or at some time after completion), the communication processor of the NPU can perform an AllReduce task. As one non-limiting example:

[0122] REG[x] = get_reduce_dest(LOCAL ID);

[0123] REG[y] = get_brdcast_dest(LOCAL ID);

[0124] var_local = HBM(var);

[0125] if (!is_first_node[])

[0126] while (!receive(&var, flag_id);

[0127] var += var_local;

[0128] }

[0129] if (!is_last_node[]

[0130] send(var, port[Reg[x]], flag_id);

[0131] if (is_last_node[]

[0132] reduced = var / N;

[0133] if (!is_last_node[])

[0134] while (!receive(&reduced, flag_id_2);

[0135] HBM[var] = reduced;

[0136] }

[0137] send(reduced, port[Reg[y]], flag_id_2);

[0138] In some embodiments, the communication processor can process these exemplary instructions to implement the AllReduce exchange depicted in FIG. 4 In some embodiments, the communication processor can use DMA commands to cause the communication DMA unit 213 to move individual values between the NPU memory 219 and the communication memory 215. Depending on the individual instruction, the NPU can be configured with two other NPUs as a reduction destination and a broadcast destination. For example, when the NPU is node 153 in FIG. 5 In some embodiments, the communication processor can process these exemplary instructions to implement the AllReduce exchange depicted in

[0139] As FIG. 1AAs shown, the all reduce operation can be performed after each batch compute is complete. Even if these NPUs are associated with different CPUs (e.g., NPUs 0-M associated with CPU N can share synchronization information with NPUs 0-N associated with CPU 0), the all reduce operation can share synchronization information among the various NPUs. In some embodiments, the various kernels can be configured to perform other tasks while the communication processor performs the all reduce operation, thereby improving the efficiency of the NPUs. In some embodiments, at least some portions of the first batch of all reduce operations and the subsequent batch of compute operations can run in parallel, thereby improving the speed of the NPUs.

[0140] FIG. 2A An exemplary method 500 for training a machine learning model using a synchronization network is depicted in accordance with various embodiments of the present disclosure. The method 500 can be performed using a system including at least one host device (e.g., for FIGS. 1C-1E the system 100 described above). Each host device can include a CPU and an associated NPU. The NPUs in the host devices can be communicatively connected using a synchronization network. The NPUs can include a generation component that supports training of a machine learning model and a synchronization component that supports exchange of synchronization information between the NPUs (e.g., as described for FIG. 3A the system 100 described above). The synchronization network can connect the various NPUs according to a topology (e.g., a linear topology, a tree topology, or a ring topology as depicted in FIG. 3B , or other suitable topology such as those described in U.S. Patent Application Nos. 16 / 777,731, 16 / 777,771, or 16 / 777,683). The topology of the synchronization network can be determined during creation of the system (e.g., based on physical connections between the host devices) or during execution of the method 500 (e.g., using software or firmware control of the synchronization network). Using the synchronization network, the system can perform an AllReduce exchange of synchronization information as described in FIG. 4 , FIG. 4 and FIG. 4 . By offloading synchronization of the NPUs performing the training task onto a dedicated synchronization network, the method 500 can improve the speed of training of the machine learning model and improve the utilization efficiency of the NPUs.

[0141] The method 500 can begin at step 501. In step 501, the system can be configured to train a machine learning model. In some embodiments, a program can configure the system to perform the training as described above for FIGS. 1C-1DThe program can be obtained by the system through interaction with a user or other computing devices. The program can specify that the training is divided into a plurality of epochs, in each of which the model is trained on a portion of the training data. In some cases, the portion of training data can be the entire training dataset. In some embodiments, the program can specify a topology of the synchronization network that can be implemented using software or firmware controls in the host devices.

[0142] According to various embodiments of the present disclosure, the system can be configured to distribute training tasks among a set of host devices. Each host device can include a CPU and a set of associated NPUs. Each CPU can be configured to distribute training subtasks to the NPUs associated with the CPU. A training task or subtask can indicate a computation to be performed by the CPU or NPU. A training task or subtask can indicate or include training data for performing the computation. For example, training data to be used by the CPU in performing a training task can be indicated in the task or distributed with the task. Similarly, training data to be used by the NPU in performing a training subtask can be indicated in the subtask or distributed with the subtask. In embodiments in which an indication of training data is distributed, the CPU (or NPU) can use the indication to obtain the appropriate training data (e.g., by retrieving the appropriate training data from a memory associated with the system).

[0143] A subtask distributed by the CPU can configure the NPU to perform steps 503-509 of method 500. For clarity of discussion, these steps are described with respect to a single NPU of a particular host device. But various embodiments of the present disclosure are not so limited, and some or all of the following steps can be performed by some or all of the NPUs in the system.

[0144] In step 503, the NPU can obtain a selection of a training subset of the training dataset. The NPU can obtain the selection from the CPU of the host device. The training subset can be a portion of a larger subset of training data distributed to the CPU. The NPU can obtain the selection of the training subset by obtaining the training subset (e.g., the actual training data) or by obtaining an identifier or location of the training subset (e.g., a URL of a storage location of the subset, one or more memory addresses or address ranges of the training subset, or other suitable retrieval information).

[0145] In step 505, the NPU can generate update information by applying a batch of the training subset to the machine learning model. The batch can be a portion of the training subset. Various embodiments of the present disclosure are not limited to any particular method of selecting the batch. The generating component of the NPU can use the machine learning model and a batch of training information to compute the update information. For example, as FIG. 3AAs shown, the command processor can configure the cores of the NPUs with biases and gradients of a computer machine learning model.

[0146] In step 507, the NPU can obtain synchronized update information. The NPU can obtain the synchronized update information at least in part by exchanging synchronization information with at least one other NPU. Exchanging synchronization information can include providing or receiving synchronization update information. For example, the NPU can receive synchronization update information from at least one other NPU or provide synchronization update information to at least one other NPU. Additionally or alternatively, exchanging synchronization information can include providing or receiving synchronization information used to generate the synchronization update information. In some cases, the provided synchronization information can be the update information generated in step 505 or include the update information generated in step 505. In various cases, the provided synchronization information can be based at least in part on the update information generated in step 505. For example, the communication processor NPU can generate the provided synchronization information using the update information generated in step 505 and synchronization information received from one or more other NPUs. For example, as described in FIG. 3B 、 FIG. 4 、 FIG. 1A and ​ , the communication processor can be configured to generate the provided synchronization information by accumulating locally generated update information and received synchronization information. In some embodiments, the NPU can participate in an AllReduce exchange of synchronization information.

[0147] The NPU can exchange synchronization information with at least one other NPU associated with the same CPU (e.g., part of the same host device) or at least one NPU associated with at least one different CPU. For example, the system can include a set of host devices, the NPU can be associated with a first host device, and can exchange synchronization information with one of a second set of NPUs associated with a second host device.

[0148] As described herein, the NPU can exchange synchronization information with at least one other NPU using a serializer / deserializer module corresponding to the at least one other NPU. The synchronization information can be stored in a partition of the communication memory as it is received from (or transmitted to) the at least one other NPU. The partition can correspond to the at least one other NPU.

[0149] At step 509, the NPU can update the machine learning model using the synchronized update information. In some embodiments, the communication processor of the NPU can write the synchronized update information to the memory of the NPU. Then, the command processor of the NPU can use the synchronized update information to update the machine learning model. For example, when the machine learning model is a neural network and the synchronized update information is gradient information, the instruction processor can cause the NPU to generate new neural network parameters based on the existing parameters and the gradient information.

[0150] At step 511, the NPU can determine whether the epoch is complete. In some embodiments, the training subset can be divided into individual batches, and the epoch can end when each batch has been used in training. In various embodiments, the individual batches can be generated by sampling from the training subset, and the epoch can end when a predetermined number of batches have been used for training. In some embodiments, the NPU can continue to generate first update information, obtain synchronized updates, and update the machine learning model using different batches of the training subset until the epoch is complete.

[0151] At step 513, the NPU can determine whether the training is complete. The system can be configured to perform the training task until a condition is met. Suitable conditions can include or depend on, but are not limited to, a number of epochs of training, a duration of training, a performance of the machine learning model (e.g., classification accuracy, confusion matrix, area under curve, mean squared error, or other suitable metrics), etc.

[0152] At step 515, the system can be configured to provide the trained version of the machine learning model. The trained version of the machine learning model can be stored in memory by the system, provided to another computing device, displayed to a user, etc. Various embodiments of the present disclosure are not limited to a particular method of providing the trained version of the machine learning model. In some embodiments, the host device of the system can provide the trained version. The CPU of the host device can be configured to receive the trained version of the machine learning model from the NPU associated with the CPU (e.g., using a host network as described in ​ various embodiments, the NPU can provide the current version of the machine learning model or the synchronized update information to the associated CPU after each synchronization. When the CPU is configured to receive the synchronized update information, the CPU can be configured to maintain an updated version of the machine learning model using the received synchronized update information.

[0153] At step 517, the method 500 can end. In some embodiments, after the method 500 is completed (e.g., immediately upon completion or at some time after completion), the system can release resources associated with training the machine learning model.

[0154] The foregoing description has been presented for purposes of illustration. It is not exhaustive and is not limited to the precise form or embodiments disclosed. Modifications and adaptations will be apparent to those skilled in the art in view of the specification and practice of the embodiments. For example, the described implementations include hardware, but systems and methods consistent with the present disclosure can be implemented in hardware and software. In some embodiments, components described herein (e.g., modules, etc.) can be implemented at least partially using circuitry (e.g., discrete components, integrated circuits, a combination of discrete and integrated components, or other suitable electronic or optical circuitry), Moreover, while certain components have been described as being coupled, such components can be integrated or distributed in any suitable manner.

[0155] Moreover, while illustrative embodiments have been described herein, the scope includes any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations or alterations based on the disclosure. The elements of the claims are to be construed in the broadest way consistent with the language and the context. The embodiments described are illustrative only and are not intended to be limiting. Further, the steps of the disclosed methods can be modified in any suitable manner, including by reordering steps or inserting or removing steps.

[0156] Features and advantages of the present disclosure are apparent from the detailed description, and thus, it is intended that the appended claims cover all systems and methods falling within the true spirit and scope of the present disclosure. As used herein, the indefinite articles “a” and “an” means “one or more.” Similarly, the use of the plural is not necessarily indicative of a plurality, unless it is explicitly stated that it is. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the present disclosure to the exact construction and operation described herein, accordingly, all suitable modifications and equivalents can be resorted to, falling within the scope of the disclosure.

[0157] As used herein, unless specifically stated otherwise, the term “or” includes all possible combinations, unless it is infeasible. For example, if a component is stated as including A or B, then unless specifically stated otherwise or infeasible, the component can include A, or B, or A and B. As a second example, if a component is stated as including A, B, or C, then unless specifically stated otherwise or infeasible, the component can include A, or B, or C, or A and B, or A and C, or B and C, or A and B and C.

[0158] Embodiments can be further described using the following clauses:

[0159] 1. A processing unit comprising: a first communication module comprising circuitry for communicating between a host unit and the processing unit over a first communication network, the processing unit being configurable to receive training information for a machine learning model from the host unit using the first communication module; a kernel, the processing unit being configurable to manage generation of update information by the kernel using the training information; and a second communication module comprising circuitry for communicating between the processing unit and at least one second processing unit over a second communication network, the second communication network being separate from the first communication network, the processing unit being configurable to transmit first synchronization information for updating the machine learning model to the at least one second processing unit using the training information, the first synchronization information comprising or being based on the update information.

[0160] 2. The processing unit of clause 1, further comprising: a command processor configurable to manage generation of the update information by the kernel using the training information; and a communication processor configurable to manage providing the first synchronization information to the at least one second processing unit using the training information.

[0161] 3. The processing unit of clause 2, further comprising: a communication memory accessible by a communication processor, wherein: the communication processor is configurable to obtain update information from the communication memory; and generate the first synchronization information using the update information and second synchronization information received from the at least one second processing unit; and the kernel is configurable to update the machine learning model based at least in part on the first synchronization information.

[0162] 4. The processing unit of clause 3, further comprising: a shared memory accessible by the communication processor and the kernel, wherein: the communication processor is further configurable to store the first synchronization information in the shared memory; and the kernel is further configurable to obtain the first synchronization information from the shared memory prior to updating the machine learning model.

[0163] 5. The processing unit of clause 3, wherein: the second communication module is further accessible by the communication memory; and the second communication module comprises circuitry configurable to obtain the first synchronization information from the communication memory and provide the first synchronization information to one of the at least one second processing unit.

[0164] 6. The processing unit of clause 2, further comprising a communication memory accessible by the communication processor and the second communication module, wherein: the communication processor is configurable to store the update information in the communication memory; and the second communication module comprises circuitry configurable to obtain the update information from the communication memory and provide the update information as first synchronization information to one of the at least one second processing unit.

[0165] 7. The processing unit of clause 2, wherein: the processing unit further comprises a communication memory configurable to comprise a plurality of partitions; the second communication module comprises a plurality of serialization units, each serialization unit comprising circuitry configurable to read the first synchronization information from a corresponding one of the plurality of partitions and provide the first synchronization information to a corresponding one of the at least one second processing unit; and receive the second synchronization information from a corresponding one of the at least one second processing unit and write the second synchronization information to a corresponding one of the plurality of partitions.

[0166] 8. The processing unit of clause 2, wherein: the processing unit further comprises a communication memory; and the communication processor comprises individual processors corresponding to individual ones of the at least one second processing unit; or the communication processor is configurable to perform individual processes corresponding to individual ones of the at least one second processing unit; and each individual processor or process is configurable to independently read received synchronization information from the communication memory, generate updated synchronization information, and write the updated synchronization information to the communication memory.

[0167] 9. The processing unit of any one of clauses 1 to 8, wherein: the first communication module is configured to communicate using a PCI express protocol; and the second communication module comprises circuitry to support point-to-point communication between the factoring processing unit and the at least one second processing unit.

[0168] 10. The processing unit of any one of clauses 1 to 9, wherein: the machine learning model is a neural network, and the update information comprises gradient information to update neural network weights.

[0169] 11. An apparatus comprising: a first host apparatus; and a plurality of first processing units, each first processing unit comprising: the first communication module comprising circuitry for communicating with the first host apparatus over a first network; and the second communication module comprising circuitry capable of communicating between each first processing unit and at least one other first processing unit over a second network; and wherein the first processing units are configured to obtain a synchronized update of a machine learning model by each of the plurality of first processing units at least in part by exchanging synchronization information using the second network, each first processing unit maintaining a version of the machine learning model; and by each first processing unit, synchronizing each version by updating each version using the synchronized update.

[0170] 12. The apparatus of clause 11, wherein: a first processing unit of the plurality of first processing units further comprises: a communication processor; and a communication memory; and the first processing unit is configured to exchange synchronization information at least in part by: receiving first synchronization information from the at least one second processing unit using the second communication module and storing the first synchronization information in the communication memory; generating second synchronization information using the first synchronization information by the communication processor and storing the second synchronization information in the communication memory; and providing the second synchronization information to the at least one third processing unit.

[0171] 13. The apparatus of clause 12, wherein: the at least one second processing unit or the at least one third processing unit comprises: a processing unit configured to communicate with a second host apparatus over a third network, the second host apparatus being separate from the first host apparatus.

[0172] 14. The apparatus of any one of clauses 12 to 13, wherein: the communication memory comprises a plurality of partitions; the second communication module comprises a plurality of serialization units, each serialization unit comprising circuitry configured to read and write from one of the plurality of partitions; and at least one serialization unit comprises circuitry configured to communicate with the at least one second processing unit, respectively.

[0173] 15. The apparatus of any one of clauses 11 to 14, wherein: the second communication module comprises circuitry capable of point-to-point communication between each first processing unit and at least one other first processing unit over the second network.

[0174] 16. The apparatus of any one of clauses 11 to 15, wherein: the plurality of first processing units are configured to each obtain the synchronized update by implementing an AllReduce exchange of the synchronization information.

[0175] 17. A method of training a machine learning model, comprising: obtaining, by a first processing unit of a first group of processing units associated with a first host device via a first communication network, a training subset of a training dataset selection from a first host device of a group of host devices; generating, using a command processor of the first processing unit, a first update to the machine learning model by applying a batch of the training subset to the machine learning model; obtaining a synchronization update at least in part by using a communication processor of the first processing unit and a second communication network to provide first synchronization information to the at least one other processing unit, the first synchronization information being the first update or based at least in part on the first update; and updating the machine learning model using the synchronization update.

[0176] 18. The method of clause 17, wherein: the at least one other processing unit comprises: a processing unit of the first group of processing units; or a processing unit of a second group of processing units associated with a second host device of the group of host devices.

[0177] 19. The method of any one of clauses 17-18, further comprising: repeating the generating the first update, obtaining the synchronization update, and updating the machine learning model using different batches of the training subset by the first processing unit; and providing the updated machine learning model to the first host device via the first communication network after the repeating is complete.

[0178] 20. The method of any one of clauses 17-19, wherein: obtaining the synchronization update further comprises: receiving the synchronization update from a first one of the at least one other processing unit, or second synchronization information, the first synchronization information being based at least in part on the first update and the second synchronization information.

[0179] 21. The method of clause 20, wherein: obtaining the synchronization update further comprises: receiving the second synchronization information; receiving the second synchronization information comprises: receiving the second synchronization information by a first serializer configured to communicate with the first one; and storing the second synchronization information in the communication memory; and providing the first synchronization information comprises: generating the first synchronization information using the second synchronization information and the first update by the communication processor; storing the first synchronization information in a partition of the communication memory accessible by a second serializer configured to communicate with a second one of the at least one other processing unit; and providing the first synchronization information to the second one.

[0180] 22. The method of clause 21, wherein: the first synchronization information is generated using at least one third synchronization information received from at least one third unit of the at least one other processing unit, or the first synchronization information comprises the synchronization update.

[0181] 23. The method of any of clauses 17-22, wherein: the first processing unit and the at least one other processing unit perform an AllReduce exchange to jointly generate a synchronization update.

[0182] 24. The method of any of clauses 17-23, wherein: the second communication network is a point-to-point network.

[0183] Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. The specification and examples are intended to be exemplary only and the true scope and spirit of the disclosure is indicated by the claims which follow.

Claims

1. A processing unit comprising: a first communication module comprising circuitry for communicating between a host unit and the processing unit over a first communication network, the processing unit being configurable to receive training information for a machine learning model from the host unit using the first communication module; a kernel, the processing unit being configurable to manage generation of update information for the kernel using the training information; and a second communication module comprising circuitry for communicating between the processing unit and at least one second processing unit over a second communication network, the second communication network being separate from the first communication network, the processing unit being configurable to transmit first synchronization information for updating the machine learning model to the at least one second processing unit using the training information, the first synchronization information comprising or being based on the update information; a communication handler configured to obtain the update information from a communication memory, generate the first synchronization information using the update information and second synchronization information received from the at least one second processing unit, and provide the first synchronization information to the at least one second processing unit; a shared memory, the communication handler being configured to store the first synchronization information in the shared memory for access by the kernel. a command handler configurable to manage generation of the update information for the kernel using the training information.

2. The processing unit of claim 1, further comprising:

3. The processing unit of claim 2, the kernel being configurable to: update the machine learning model based at least in part on the first synchronization information.

4. The processing unit of claim 3, the kernel being further configurable to obtain the first synchronization information from the shared memory prior to updating the machine learning model.

5. The processing unit of claim 3, wherein: the second communication module is further accessible by the communication memory; and the second communication module comprises circuitry configurable to: obtain the first synchronization information from the communication memory and provide the first synchronization information to one of the at least one second processing unit.

6. The processing unit of claim 2, further comprising: a communication memory accessible by the communication handler and the second communication module, wherein: the communication handler is configurable to: store the update information in the communication memory; and the second communication module comprises circuitry configurable to: obtain the update information from the communication memory and provide the update information as the first synchronization information to one of the at least one second processing unit.

7. The processing unit of claim 2, wherein: the processing unit further comprises a communication memory configurable to comprise a plurality of partitions; the second communication module comprises a plurality of serialization units, each serialization unit comprising circuitry configurable to: read the first synchronization information from a corresponding one of the plurality of partitions and provide the first synchronization information to a corresponding one of the at least one second processing unit; and the communication memory is further configurable to comprise a plurality of partitions. ​ receiving the second synchronization information from a corresponding one of the at least one second processing unit and writing the second synchronization information to a corresponding one of the plurality of partitions.

8. The processing unit of claim 2, wherein: the processing unit further comprises a communication memory; and the communication processor comprises individual processors corresponding to individual ones of the at least one second processing unit; or the communication processor is configurable to perform individual processes corresponding to individual ones of the at least one second processing unit; and the individual processors or processes are each configurable to independently read received synchronization information from the communication memory, generate updated synchronization information, and write the updated synchronization information to the communication memory.

9. The processing unit of claim 1, wherein: the first communication module is configured to communicate using a PCI express protocol; and the second communication module comprises circuitry to support point-to-point communication between the processing unit and the at least one second processing unit.

10. The processing unit of claim 1, wherein: the machine learning model is a neural network, and the update information comprises gradient information to update neural network weights.

11. A processor cluster, comprising: a first host device; and a plurality of first processing units, each first processing unit comprising: a first communication module comprising circuitry to communicate with the first host device over a first network; and a second communication module comprising circuitry to enable communication between each first processing unit and at least one other first processing unit over a second network; and a communication processor configured to obtain update information from a communication memory, generate first synchronization information using the update information and second synchronization information received from the at least one second processing unit, and provide the first synchronization information to the at least one second processing unit; a shared memory, the communication processor configured to store the first synchronization information in the shared memory for access by a kernel, wherein the first processing units are configured to: obtain, by each of the plurality of first processing units, a synchronized update of a machine learning model at least in part by exchanging synchronization information using the second network, each first processing unit maintaining a version of the machine learning model; and synchronize, by each first processing unit, each version by updating each version using the synchronized update.

12. The processor cluster of claim 11, wherein: a first processing unit of the plurality of first processing units further comprises: a communication processor; and a communication memory; and the first processing unit is configured to exchange synchronization information at least in part by: receiving first synchronization information from the at least one second processing unit using the second communication module and storing the first synchronization information in the communication memory; generating, by the communication processor, second synchronization information using the first synchronization information and storing the virtually second synchronization information in the communication memory; providing the second synchronization information to the at least one third processing unit.

13. The processor cluster of claim 12, wherein: the at least one second processing unit or the at least one third processing unit comprises: a processing unit configured to communicate with a second host device over a third network, the second host device being separate from the first host device.

14. The processor cluster of claim 12, wherein: the communication memory comprises a plurality of partitions; the second communication module comprises a plurality of serialization units, each serialization unit comprising circuitry configured to read and write from one of the plurality of partitions; and at least one serialization unit comprises circuitry configured to communicate with the at least one second processing unit, respectively.

15. The processor cluster of claim 11, wherein: the second communication module comprises circuitry capable of point-to-point communication between each first processing unit and at least one other first processing unit over the second network.

16. The processor cluster of claim 11, wherein: the plurality of first processing units are configured to obtain the synchronization update by each first processing unit implementing an AllReduce exchange of the synchronization information.

17. A method of training a machine learning model, comprising: obtaining, by a first processing unit in a first group of processing units associated with a first host device over a first communication network, a training subset of a training dataset selection from a first host device in a set of host devices; generating, using a command processor of the first processing unit, a first update information of the machine learning model by applying a batch of the training subset to the machine learning model; obtaining a synchronization update at least in part by using a communication processor of the first processing unit and a second communication network to provide a first synchronization information to the at least one other processing unit, the first synchronization information being the first update information or based at least in part on the first update information, the communication processor being configured to: obtain the update information from a communication memory, generate the first synchronization information using the update information and a second synchronization information received from the at least one second processing unit, and provide the first synchronization information to the at least one second processing unit; and the communication processor being configured to store the first synchronization information in a shared memory, and the kernel having access to the shared memory to update the machine learning model using the synchronization update.

18. The method of claim 17, wherein: the at least one other processing unit comprises: one of the first group of processing units; or one of a second group of processing units associated with a second host device in the set of host devices.

19. The method of claim 17, further comprising: repeating the generating the first update information, obtaining the synchronization update, and updating the machine learning model using different batches of the training subset by the first processing unit; and ​ ​ ​ Upon completion of the repeated generation, providing the updated machine learning model to the first host device over the first communication network.

20. The method of claim 17, wherein: obtaining the synchronization update further comprises receiving the synchronization update from a first one of the at least one other processing units, or second synchronization information, the first synchronization information based at least in part on the first update information and the second synchronization information.

21. The method of claim 20, wherein: obtaining the synchronization update further comprises receiving second synchronization information; receiving the second synchronization information comprises: receiving the second synchronization information through a first serializer configured to communicate with the first unit; and storing the second synchronization information in the communication memory; and providing the first synchronization information comprises: generating the first synchronization information with the second synchronization information and the first update information by the communication processor; storing the first synchronization information in a partition of the communication memory accessible by a second serializer, the second serializer configured to communicate with a second one of the at least one other processing units; and providing the first synchronization information to the second unit.

22. The method of claim 21, wherein: the first synchronization information is generated using at least one third synchronization information received from at least one third one of the at least one other processing units, or the first synchronization information comprises the synchronization update.

23. The method of claim 17, wherein: the first processing unit and the at least one other processing unit perform an AllReduce exchange to jointly generate a synchronization update.

24. The method of claim 17, wherein: the second communication network is a point-to-point network.

Images