USE OF RESOURCES IN A MIXTURE OF EXPERTS PROCESSING

Abstract

The use of resources employing improved mixture-of-experts processing is described. An example device comprises one or more network ports; one or more direct-access memory (DMA) engines; and a circuit for processing a mixture of experts (MoE) on the network, wherein the circuit includes at least a circuit for tracking the routing of tokens in the MoE processing, a prediction circuit for generating predictions regarding the MoE processing, including predictions of future token loads for the MoE processing, and a routing management circuit for managing the routing of tokens in the MoE processing, at least partially based on the predictions regarding the MoE processing.

Description

BACKGROUND OF THE REVELATION

[0001] Mixture of Experts (MoE) is an emerging paradigm for reducing the computational demands of large language models (LLMs) while maintaining comparable accuracy. Applying MoE processing allows models to be pretrained with less computational effort, thus dramatically increasing the size of a model or dataset for the same computational budget as a dense model. In particular, an MoE model can achieve the same or similar quality as its dense counterpart, with the quality being achieved much faster during pretraining.

[0002] In MoE operation, a sparse mixture-of-experts layer is used instead of dense feedforward network layers. A layer can comprise a specific number of experts (e.g., 16, 32, or 128), where an expert itself is either a feedforward network or a mixture of experts, resulting in a hierarchical mix of experts.

[0003] However, MoE technology brings with it certain complications in processing, including in the selection of the experts to be used in a specific case. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The embodiments described herein are illustrated by way of example and without limitation in the figures of the accompanying drawings, in which the same references denote similar elements and in which: Fig. Figure 1 is a block diagram showing a computer system configured to implement one or more aspects of the embodiments described here; Fig. Figure 2 is a block diagram of a system containing selected components of a data center; Fig. Figure 3 is a block diagram of a part of a data center, according to one or more examples in the present patent specification; Fig. 4A-4C illustrates programmable forwarding elements and adaptive routing; Fig. Figures 5A-5B show an example of network interface devices; Fig. Figure 6 is a block diagram illustrating a programmable network interface and a data processing unit; Fig. Figure 7 is a block diagram showing an IP core development system; Fig. Figure 8 is an illustration of a computer system or computing device with routing functions for mixing expert inputs into processing resources, according to some embodiments; Fig. Figure 9 illustrates the application of a mix of experts in AI processing; Fig. Figure 10 is an illustration of routing in a mixture-of-experts operation; Fig. Figures 11A-11F are illustrations of routing algorithms in a mixture-of-experts operation; Fig. Figure 12 is an illustration of a system for providing resources using an improved mixture-of-experts processing according to some embodiments; Fig. Figure 13A is an illustration of a resource deployment using an improved mixture-of-experts processing according to some embodiments; Fig. Figure 13B illustrates expert data that is retained for mixture-of-experts processing according to some embodiments; and Fig. Figure 14 is a flowchart illustrating a process for a mixture-of-experts operation in network processing according to some embodiments. DETAILED DESCRIPTION

[0005] In some embodiments, a device, system, or method enables the provision of resources using an improved Mixture of Experts (MoE) operation.

[0006] The following description includes numerous specific details to facilitate better understanding. However, it is obvious to a person skilled in the art that the embodiments described here can also be implemented without one or more of these specific details. In other cases, known features have not been described so as not to obscure the details of the present embodiments.

[0007] Fig. Figure 1 is a block diagram showing a computer system 100 configured to implement one or more aspects of the embodiments described herein. The computer system 100 comprises a processing subsystem 101 with one or more processors 102, such as central processing units (CPUs) or other host processors, and a system memory 104, which can communicate via a link path that may include a memory hub 105. The memory hub 105 may be a separate component within a chipset or be integrated into the processor(s) 102. The memory hub 105 is connected via a communication link 106 to an I / O subsystem 111. The I / O subsystem 111 includes an I / O hub 107, which may enable the computer system 100 to receive input from one or more input devices 108.Furthermore, the I / O hub 107 can enable a display controller, which may be contained in the processor(s) 102, to provide outputs to one or more display devices 110A. In one embodiment, the one or more display devices 110A connected to the I / O hub 107 may comprise a local, internal, or embedded display device.

[0008] The processing subsystem 101, for example, comprises one or more parallel processors 112 connected to the memory hub 105 via a communication link 113, such as a bus or fabric. The communication link 113 can be any number of standards-based communication link technologies or protocols, such as PCI Express, or a vendor-specific communication interface or structure. The one or more parallel processor(s) 112 can form a computationally focused parallel or vector processing system that may include a large number of processing cores and / or processing clusters, such as a many-integrated core (MIC) processor. For example, the parallel processor(s) 112 form a graphics processing subsystem capable of outputting pixels to one of the display device(s) 110A connected via the I / O hub 107.The one or more parallel processor(s) 112 may also include a display controller and a display interface (not shown) to enable a direct connection to one or more display device(s) 110B.

[0009] Within the I / O subsystem 111, a system storage unit 114 can be connected to the I / O hub 107 to provide a storage mechanism for the computer system 100. An I / O switch 116 can be used to provide an interface mechanism that enables connections between the I / O hub 107 and other components, such as a network adapter 118 and / or a wireless network adapter 119, which can be integrated into the platform, and various other devices that can be added via one or more add-in devices 120. The add-in device(s) 120 can, for example, also include one or more external graphics processing units, graphics cards, and / or computational accelerators. The network adapter 118 can be an Ethernet adapter or another wired network adapter.The wireless network adapter 119 can include one or more Wi-Fi, Bluetooth, Near Field Communication (NFC) or other network devices with one or more wireless radios.

[0010] The computer system 100 can contain other components not explicitly shown, including USB or other port connections, optical storage drives, video recording devices, and the like, which can also be connected to the I / O hub 107. The communication paths that connect the various components in Fig. 1. Connecting them together can be implemented using any suitable protocols, for example PCI (Peripheral Component Interconnect)-based protocols (e.g., PCI Express) or any other bus or point-to-point communication interfaces and / or protocols, for example, the NVLink high-speed connection, Compute Express Link™ (CXL™) (e.g., CXL.mem), Infinity Fabric (IF), Ethernet (IEEE 802.3), Remote Direct Memory Access (RDMA), InfiniBand, Internet Wide Area RDMA Protocol (iWARP), Transmission Control Protocol (TCP), Internet Protocol (IP), User Datagram Protocol (UDP), Quick UDP Internet Connections (QUIC), RDMA over Converged Ethernet (RoCE), Ultra Ethernet Transport (UET), Intel QuickPath Interconnect (QPI), Intel Ultra Path Interconnect (UPI), Intel On-Chip System Fabric (IOSF), Omnipath, HyperTransport, Advanced Microcontroller Bus Architecture (AMBA)-Zwischenverbindung, Open Coherent Accelerator Processor Interface (CAPI), Gen-Z, Cache Coherent Interconnect for Accelerators (CCIX), 3rd Generation Partnership Projects (3GPP) Long Term Evolution (LTE) (z. B. 4. Generation (4G)), 3GPP 5. Generation (5G) und Variationen davon oder in der Technik bekannte drahtgebundene oder drahtlose Zwischenverbindungsprotokolle.In some examples, data can be copied or stored on virtualized storage nodes using a protocol such as Non-Volatile Memory Express (NVMe) over Fabrics (NVMe-oF) or NVMe. In one embodiment, time-dependent communication protocols are supported, including time-dependent RDMA, time-dependent NVMe, and time-dependent NVMe-oF, where precise timing and rate of data consumption are used to control data transfer.

[0011] The one or more parallel processor(s) 112 may contain circuitry optimized for graphics and video processing, such as video output circuitry, and constitute a graphics processing unit (GPU). Alternatively or additionally, the parallel processor(s) 112 may contain circuitry optimized for general-purpose processing, while retaining the underlying computing architecture described in more detail herein. The components of the computer system 100 may be integrated with one or more other system elements on a single integrated circuit. For example, the parallel processor(s) 112, the memory hub 105, the processor(s) 102, and the I / O hub 107 may be integrated into an integrated system-on-a-chip (SoC) circuit.Alternatively, the components of the computer system 100 can be integrated into a single package to form a system-in-package (SiP) configuration. In one embodiment, at least some of the components of the computer system 100 can be integrated into a multi-chip module (MCM) that can be interconnected with other multi-chip modules to form a modular computer system.

[0012] In some configurations, the computing system 100 includes, in addition to the processor(s) 102 and the one or more parallel processor(s) 112, one or more accelerator device(s) 130 connected to the memory hub 105. The accelerator device(s) 130 are configured to perform domain-specific acceleration of workloads to handle computationally intensive tasks or to achieve high throughput. The accelerator device(s) 130 can reduce the load on the processor(s) 102 and / or the parallel processor(s) 112 of the computing system 100.The accelerator device(s) 130 may include, but are not limited to, intelligent network interface cards, data processing units, cryptographic accelerators, memory accelerators, artificial intelligence (AI) accelerators, neural processing units (NPUs), memory accelerators and / or video transcoding accelerators.

[0013] It is clear that the computer system 100 shown here is only an example and that variations and modifications are possible. The connection topology, including the number and arrangement of the bridges, the number of processors 102, and the number of parallel processors 112, can be changed as desired. For example, the system memory 104 can be connected directly to the processor(s) 102 without a bridge, while other devices communicate with the system memory 104 via the memory hub 105 and the processor(s) 102. In other alternative topologies, the parallel processors 112 are connected to the I / O hub 107 or directly to one of the processors 102, rather than to the memory hub 105. In other embodiments, the I / O hub 107 and the memory hub 105 can be integrated on a single chip.It is also possible for two or more sets of processors 102 to be connected via multi-sockets, which can be coupled with two or more instances of the parallel processor(s) 112.

[0014] Some of the components shown here are optional and need not be included in all implementations of Informatiksystem 100. For example, any number of expansion cards or peripherals can be supported, or some components can be omitted. Furthermore, some architectures may use different terminology for components than that shown in [reference to relevant documentation]. Fig. 1 illustrated similarities.

[0015] Fig. Figure 2 is a block diagram of a System 200 containing selected data center components. The components of the depicted data center could be located at a cloud service provider (CSP) or another data center, such as a traditional enterprise data center, a private cloud, or a public cloud providing services like Infrastructure as a Service (IaaS), Platform as a Service (PaaS), or Software as a Service (SaaS). The System 200 includes a number of workload clusters, including but not limited to Workload Cluster 218A and Workload Cluster 218B. The workload clusters can consist of single servers, blade servers, rackmount servers, or any other suitable server topology.

[0016] The System 200 can contain workload clusters 218A-218B. Workload clusters 218A-218B can occupy a Rack 248, housing multiple servers (e.g., Server 246). The Rack 248 and the servers of the Workload Clusters 218A-218B can conform to the rack unit standard (“U” standard), where a rack unit corresponds to a 19-inch wide rack frame and a full-size industry-standard rack can accommodate 42 units (42U) of equipment. A device unit (1U) (e.g., a 1U server) can be 1.75 inches high and approximately 36 inches deep. In various configurations, compute resources such as processors, memory, storage, accelerators, and switches can fit into multiples of rack units within a Rack 248.

[0017] A Server 246 can contain a standalone operating system configured to provide server functions, or the servers can be virtualized. A virtualized server can be controlled by a Virtual Machine Manager (VMM), hypervisor, and / or orchestrator and host one or more virtual machines, virtual servers, or virtual devices. Workload clusters 218A-218B can be located in a single data center or in different geographic data centers. Depending on contractual agreements, some servers may be dedicated to specific enterprise customers or tenants, while other servers may be shared.

[0018] The various devices in a data center can be interconnected via a Switching Fabric 270, which may include one or more high-speed routing and / or switching devices. The Switching Fabric 270 can provide north-south traffic 202 (e.g., traffic to and from the Wide Area Network (WAN), such as the Internet) and east-west traffic 204 (e.g., traffic throughout the data center). Historically, north-south traffic 202 constituted the majority of network traffic, but as web services have become more complex and distributed, the volume of east-west traffic 204 has increased. In many data centers, east-west traffic 204 now represents the largest share of traffic. Furthermore, traffic volume can increase further with the increasing performance of a server 246.For example, a Server 246 can offer multiple processor slots, with each slot accommodating a processor with four to eight cores and sufficient memory for those cores. This allows a server to host a number of virtual machines (VMs), which can generate traffic.

[0019] To handle the high traffic volume in a data center, a high-performance implementation of Switching Fabric 270 can be deployed. The Switching Fabric 270 implementation shown is an example of a flat network where a Server 246 can have a direct connection to a Top-of-Rack switch (ToR switch 220A-220B) (e.g., a "star" configuration). The ToR switch 220A can connect to a Workload Cluster 218A, while the ToR switch 220B can connect to the Workload Cluster 218B. A ToR switch 220A-220B can be coupled to a Core Switch 260. This two-tier flat network architecture is shown only as an illustrative example, and other architectures can also be used, such as...Three-tiered star or leaf rib topologies (also called "fat tree" topologies) based on the "Clos" architecture, hub-and-spoke topologies, mesh topologies, ring topologies or 3D mesh topologies, to name just one example.

[0020] The Switching Fabric 270 can be deployed through any suitable connection using any suitable connection protocol. For example, a Server 246 can contain a Fabric interface (FI), a Network Interface Card (NIC), or another host interface. The host interface itself can be connected to one or more processors via a link or bus such as PCI, PCIe, or similar, and in some cases, this link bus can be considered part of the Switching Fabric 270. The Switching Fabric can also use physical PCIe connections to implement more advanced protocols such as Compute Express Link (CXL).

[0021] The connectivity technology can be provided through a single connection or a hybrid connection. For example, PCIe might handle on-chip communication, 1Gb or 10Gb copper Ethernet provides relatively short connections to a 220A-220B toroidal switch, and optical cabling enables relatively long connections to the 260 core switch. Connectivity technologies include, for example, Ultra Path Interconnect (UPI), Fibre Channel, Ethernet, Fibre Channel over Ethernet (FCoE), InfiniBand, PCIe, NVLink, and fiber optic cables, to name just a few. Some are better suited for certain applications or functions than others, and selecting the appropriate fabric for a given application is a task for a specialist.

[0022] In one embodiment, the switching elements of the Switching Fabric 270 are configured to implement switching techniques to improve network performance in high-utilization scenarios. Examples of advanced switching techniques include adaptive routing, adaptive fault resolution, and adaptive and / or telemetry-based congestion control.

[0023] Adaptive Routing allows a ToR switch 220A-220B and / or a Core switch 260 to select the outbound port to which traffic is directed based on the load of the selected port, provided unrestricted port selection is enabled. An adaptive routing table can configure the forwarding tables of the switches in the Switching Fabric 270 to select between multiple ports between switches when multiple connections exist between a given set of switches in an adaptive routing group. Adaptive fault correction (e.g., self-healing) allows the automatic selection of an alternative port if the port selected by the forwarding table is in a failed or inactive state, enabling rapid recovery in the event of a switch-to-switch port failure.A notification can be sent to neighboring switches when adaptive routing or adaptive troubleshooting becomes active on a particular switch. With adaptive congestion control, a switch is configured to send a notification to neighboring switches when the congestion of a port on that switch exceeds a configured threshold. This can then prompt those neighboring switches to adaptively switch to uncongested ports on that switch or to switches connected via an alternative route to the destination.

[0024] Telemetry-based congestion control utilizes real-time monitoring of telemetry from network devices, such as switches within the Switching Fabric 270, to detect when congestion begins to impact the Switching Fabric 270's performance and to proactively adjust the switching tables within the network devices to prevent or mitigate the impending congestion. A ToR Switch 220A-220B and / or Core Switch 260 can implement an integrated telemetry-based congestion control algorithm or provide an application programming interface (API) through which a programmable telemetry-based congestion control algorithm can be implemented. A continuous feedback loop can be implemented in which the telemetry-based congestion control system continuously monitors the network and adjusts traffic flow in real time based on the ongoing telemetry data.The telemetry-based congestion control system can learn and adapt by adjusting to changing network conditions and improving its congestion control strategies based on historical data and trends.

[0025] Note, however, that while high-end fabrics are provided here for illustrative purposes, Switching Fabric 270 can generally encompass any suitable link or bus for the specific application, including legacy links used to implement local area networks (LANs), synchronous optical networks (SONETs), asynchronous transmission mode (ATM) networks, wireless networks such as Wi-Fi and Bluetooth, 4G wireless, 5G wireless, digital subscriber line connections (DSL), multimedia-over-coax alliance (MoCA) links, or similar wired or wireless networks. It is also expressly assumed that new networking technologies will emerge in the future that complement or replace some of those listed here, and that such future network topologies and technologies may be, or become, part of Switching Fabric 270.

[0026] Fig. Figure 3 is a block diagram of a part of a data center 300, according to one or more examples in the present patent specification. The depicted part of the data center 300 is not intended to include all components of a data center. The depicted part can be duplicated multiple times within the data center 300 and / or the data center 300 can include parts beyond those depicted, depending on the capacity and functionality to be provided by the data center 300. The data center 300 can, in various embodiments, include components of the data center of the system 200. Fig. 2 or another data center.

[0027] The Data Center 300 comprises a set of logic elements that form a multitude of nodes, where a node can be a physical server, a group of servers, or other hardware. A server can also host one or more virtual machines, depending on the application. A Fabric 370 is provided to connect various aspects of the Data Center 300. The Fabric 370 can be provided by any suitable interconnect technology, including but not limited to InfiniBand, Ethernet, PCIe, or CXL. The Data Center 300's Fabric 370 can be a version of the System 200's Switching Fabric 270. Fig. 2 and / or include elements thereof. The Fabric 370 of the Data Center 300 can connect elements of the Data Center, including server nodes (e.g., Storage Server Node 304, Heterogeneous Compute Server Node 306, CPU Server Node 308, Storage Server Node 310), Accelerators 330, Gateways 340A-340B to other Fabrics, Fabric Architectures or Connectivity Technologies, and an Orchestrator 360.

[0028] The server nodes of data center 300 can include, but are not limited to, a storage server node 304, a heterogeneous compute server node 306, a CPU server node 308, and a storage server node 310. The heterogeneous compute server node 306 and a CPU server node 308 can perform independent operations for different tenants or cooperative operations for a single tenant. The heterogeneous compute server node 306 and a CPU server node 308 can also host virtual machines that provide virtual server functionality to the data center's tenants.

[0029] The server nodes can be connected to the Fabric 370 via a Fabric 372 interface. The specific type of Fabric 372 interface used depends, at least in part, on the technology or protocol used to implement the Fabric 370. For example, if the Fabric 370 is an Ethernet fabric, the Fabric 372 interface can be an Ethernet network interface controller. If the Fabric 370 is a PCIe-based fabric, the Fabric interfaces can be PCIe-based connections. If Fabric 370 is an InfiniBand fabric, the Fabric interface 372 of the heterogeneous compute server node 306 and a CPU server node 308 can be a Host Channel Adapter (HCA), while the Fabric interface 372 of the storage server node 304 and the storage server node 310 can be a Target Channel Adapter (TCA).The TCA functionality can be an implementation-specific subset of the HCA functionality. The various fabric interfaces can be implemented as IP (Intellectual Property) blocks that can be inserted as a modular unit into an integrated circuit, just like other circuits within the Data Center 300.

[0030] The heterogeneous compute server node 306 includes multiple CPU sockets that can accommodate a CPU 319, which can be, but is not limited to, an Intel® Xeon™ multi-core processor. For example, the CPU 319 could also be a data center-class ARM® multi-core CPU, such as an NVIDIA® Grace™ CPU. The heterogeneous compute server node 306 includes storage devices 318 for storing runtime data and storage devices 316 for persistent data storage on non-volatile storage devices. The heterogeneous compute server node 306 can perform heterogeneous processing through the presence of GPUs (e.g., GPU 317), which can be used for high-performance computing (HPC), media servers, cloud gaming servers, and / or machine learning computations.In one configuration, the GPUs and CPUs of the heterogeneous compute server node 306 can be interconnected via connection technologies such as PCIe, CXL or NVLink.

[0031] The CPU server node 308 comprises a variety of CPUs (e.g., CPU 319), memory (e.g., memory devices 318), and storage (memory devices 316) to run applications and other program code that provide server functions, such as web servers or other types of functions that clients of the CPU server node 308 can access remotely. The CPU server node 308 can also execute program code that provides services or microservices enabling complex enterprise functions.Fabric 370 is provided with sufficient throughput to allow concurrent access to CPU server node 308 by a large number of clients, while simultaneously maintaining sufficient throughput for use by heterogeneous compute server node 306 and enabling the use of storage server node 304 and storage server node 310 by both the heterogeneous compute server node 306 and CPU server node 308. Furthermore, in a configuration where CPU server node 308 primarily relies on distributed services provided by storage server node 304 and storage server node 310, the memory and storage capacity of CPU server node 308 may not be sufficient for all operations it may require.Instead, a large pool of high-speed or specialized storage can be dynamically provisioned among a number of nodes, giving nodes access to a large resource pool without those resources remaining unused when not needed. This type of distributed architecture is made possible by the high speeds and low latency offered by the Fabric 370 architecture of modern data centers and can be advantageous because it avoids the need for excessive resource provisioning to server nodes.

[0032] Storage server node 304 can include storage node 305 with memory technologies suitable for storing data used during the execution of program code by the heterogeneous compute server node 306 and the CPU server node 308. Storage node 305 can include volatile memory modules, such as DRAM modules, and / or non-volatile memory technologies capable of operating at speeds similar to DRAM, providing sufficient throughput and latency performance metrics to be used as a layer of system memory at runtime. Storage server node 304 can be connected to the heterogeneous compute server node 306 and / or the CPU server node 308 via technologies such as CXL.mem, which enable memory access from a host to a device.In such a configuration, CPU 319 of heterogeneous compute server node 306, CPU server node 308, can connect to storage server node 304 and access storage node 305 of storage server node 304 in a similar way to how CPU 319 of heterogeneous compute server node 306 can access the device memory of a GPU within heterogeneous compute server node 306. For example, storage server node 304 can grant remote memory access (RDMA) to storage node 305, whereby CPU server node 308, for example, can access the memory resources of storage server node 304 via structure 370 using DMA (Direct Memory Access) operations, similar to how the CPU would access its own onboard memory.

[0033] Storage server node 304 can be used by heterogeneous compute server node 306 and CPU server node 308 to extend the runtime memory available for memory-intensive activities such as training machine learning models. A multi-tiered storage system can be enabled, allowing model data to be swapped in and out of storage devices 318 of heterogeneous compute server node 306 to the memory of storage server node 304, which offers higher performance and / or lower latency than local storage (e.g., storage devices 316). During workload setup, the entire workload dataset can be loaded into one or more of the storage nodes 305 of storage server node 304 and the storage devices 318 of heterogeneous compute server node 306, as needed during the execution of a heterogeneous workload.

[0034] Storage server node 310 provides storage functionality for heterogeneous compute server node 306, CPU server node 308, and potentially storage server node 304. Storage server node 310 can provide a networked bundle of disks or "Just a Bunch of Disks" (JBOD), program flash memory (PFM), a redundant array of independent disks (RAID), a redundant array of independent nodes (RAIN), network attached storage (NAS), or other non-volatile storage solutions. In one configuration, storage server node 310 can be coupled with heterogeneous compute server node 306, CPU server node 308, and / or storage server node 304, such as for NVMe-oF, enabling the implementation of the NVMe protocol over Fabric 370. In such configurations, the Fabric interfaces 372 of these servers can be intelligent interfaces containing hardware to accelerate NVMe-oF operations.

[0035] The Accelerators 330 within the Data Center 300 can provide various accelerated functions, including hardware or coprocessor acceleration for functions such as packet processing, encryption, decryption, compression, decompression, network security, or other accelerated functions within the data center. In some examples, the Accelerators 330 may include deep learning accelerators, such as neural processing units (NPUs), which can offload matrix multiplication operations or other neural network operations from the heterogeneous compute server node 306 or the CPU server node 308. In some configurations, the Accelerators 330 may reside in a dedicated accelerator server or be distributed across the various server nodes of the Data Center 300.For example, an NPU can be directly connected to one or more CPU cores within the heterogeneous Compute Server Node 306 or CPU Server Node 308. In some configurations, the Accelerators 330 can include or integrate intelligent network controllers, infrastructure processing units (IPUs), or data processing units that combine network controller functions with accelerator, processor, or coprocessor functions. The Accelerators 330 can also include edge processing units (EPUs) to perform real-time inference operations at the network edge. In some embodiments, the Accelerators 330 include hardware circuitry to enable mixture-of-experts operations, including tracking, predicting, and forwarding experts, as well as migrating and replicating specific experts, as described in [example missing in original text]. Fig. 8 and Fig. 12-13B further illustrated.

[0036] In a Data Center 300 configuration, gateways 340A-340B can connect Fabric 370 to other fabrics, fabric architectures, or interconnect technologies. For example, if Fabric 370 is an InfiniBand fabric, gateways 340A-340B can be gateways to an Ethernet fabric. If Fabric 370 is an Ethernet fabric, gateways 340A-340B can include routers to forward data to other parts of Data Center 300 or to a larger network, such as the internet. For instance, a first gateway 340A can connect to another network or subnet within Data Center 300, while a second gateway 340B can be a router to the internet.

[0037] Orchestrator 360 manages the provisioning, configuration, and operation of network resources within Data Center 300. Orchestrator 360 can consist of hardware or software running on a dedicated orchestration server. It can also be included in software running, for example, on CPU Server Node 308, which configures the Software Defined Networking (SDN) functionality of components within Data Center 300. In various configurations, Orchestrator 360 can enable the automated provisioning and configuration of Data Center 300 components by managing network resource allocation and template-based provisioning.Template-based provisioning is a method for deploying and managing IT resources using predefined templates. These templates can be based on standards required by government agencies, service providers, financial institutions, standards bodies, or customers. Service Level Agreements (SLAs) or Service Level Obligations (SLOs) can also be defined within the template. Orchestrator 360 can perform, but is not limited to, functions such as load balancing and traffic engineering, network segmentation, security automation, real-time telemetry monitoring, and adaptive switching management, including telemetry-based adaptive switching.In some configurations, Orchestrator 360 can also provide multi-tenancy and virtualization support by enabling the management of virtual networks, including the creation and deletion of virtual LANs (VLANs) and virtual private networks (VPNs), and tenant isolation for multi-tenant data centers.

[0038] Fig. 4A-4C illustrates programmable forwarding elements and adaptive routing. Fig. 4A illustrates a forwarding element that includes a control plane and a programmable data plane. Fig. Figure 4B illustrates a network with switching facilities configured for adaptive routing and telemetry-based congestion control. Fig. Figure 4C illustrates an InfiniBand switch with multiple ports and IB interfaces.

[0039] Fig. Figure 4A shows a forwarding element 400 that can be configured to forward data messages within a network based on a user-provided program. In some embodiments, the program includes instructions for forwarding data messages as well as for performing other processes such as firewalling, denial-of-service protection, and load balancing operations. The forwarding element 400 can be any type of forwarding device, including but not limited to a switch, router, or bridge. The forwarding element 400 can forward data messages associated with various technologies, such as but not limited to Ethernet, Ultra Ethernet, InfiniBand, or NVLink.

[0040] In various network configurations, the forwarding element is used as a non-edge forwarding element within the network to forward data messages from a source device to a destination device. In network configurations, the forwarding element 400 is used as an edge forwarding element at the network edge to connect to computing devices (e.g., standalone or host computers) that serve as sources and destinations of data messages. As a non-edge forwarding element, the forwarding element 400 forwards data messages between forwarding elements within the network, for example, via an intermediate network fabric. As an edge forwarding element, the forwarding element 400 forwards data messages to and from edge computing devices, to other edge forwarding elements, and / or to non-edge forwarding elements.

[0041] The forwarding element 400 contains circuitry for implementing a data plane 402, which performs the forwarding operations of the forwarding element 400 to relay data messages received by the forwarding element to other devices. The forwarding element 400 also includes circuitry for implementing a control plane 404, which configures the data plane circuitry. Furthermore, the forwarding element 400 includes physical ports 406, which receive data messages from and send data messages to devices outside the forwarding element 400. The data plane 402 includes ports 408, which receive data messages from the physical ports 406 for processing. The data messages are processed and forwarded to another port on the data plane 402, which is connected to another physical port of the forwarding element 400.In addition to the physical ports of the forwarding element 400, some of the ports 408 on data plane 402 may also be connected to other modules of data plane 402.

[0042] The data plane comprises programmable packet processor circuits that provide several programmable message processing stages. These stages can be configured to perform the forwarding operations of the forwarding element 400 data plane to process and forward data messages to their destinations. These message processing stages perform these forwarding operations by processing data tuples (e.g., message headers) associated with the data messages received by the data plane 402 to determine how the messages should be forwarded. The message processing stages include matching action units (MAUs) that attempt to match data tuples (e.g., header vectors) from messages with tables that specify which actions should be performed on the data tuples.In some embodiments, the table sets are populated by the control plane 404 and are not known when the data plane is configured to execute a program provided by a network user. The programmable message-handling circuits are grouped into multiple message-handling pipelines. The message-handling pipelines can be input or output pipelines before or after the traffic management stage of the routing element, which directs the messages from the input pipelines to the output pipelines.

[0043] The hardware specifications of data plane 402 depend on the communication protocol implemented via the forwarding element 400. Ethernet switches use application-specific integrated circuits (ASICs) designed to process Ethernet frames and the TCP / IP protocol stack. These ASICs are optimized for a wide range of traffic types, including unicast, multicast, and broadcast. Ethernet switch ASICs are typically designed for a balance of cost, power consumption, and performance, although high-end Ethernet switches may also support advanced features such as deep packet inspection and enhanced quality of service (QoS). InfiniBand switches use specialized ASICs designed for extremely low latency and high throughput.These ASICs enable features such as optimized processing of the InfiniBand protocol and offer support for RDMA and other functions that require precise timing and high-speed data processing. However, high-end Ethernet switches can support RoCE (RDMA over Converged Ethernet), which offers similar advantages to InfiniBand but has higher latency compared to native InfiniBand RDMA.

[0044] The 400 forwarding element can also be configured as an NVLink switch (e.g., NVSwitch), which connects multiple GPUs using the NVLink connection protocol. When configured as an NVLink switch, the 400 forwarding element can provide GPU servers with higher GPU-to-GPU bandwidth compared to GPU servers connected via InfiniBand. An NVLink switch can reduce network traffic hotspots that can occur when interconnected GPU-equipped servers perform tasks such as distributed neural network training.

[0045] When data plane 402, in coordination with a program running on data plane 402 (e.g., a program written in the P4 language), performs message or packet forwarding operations on incoming data, control plane 404 generally determines how messages or packets are forwarded. The behavior of a program running on data plane 402 is partly determined by control plane 404, which populates match-action tables with specific forwarding rules. The forwarding rules used by the program running on data plane 402 are independent of the data plane program itself. In a configuration, the control plane can be coupled to an administration port 410, which allows the administrator to configure the forwarding element 400.The data connection established via the management port 410 is separate from the data connections for the input and output data ports. In one configuration, the management ports 410 can be connected to a management level 405, which facilitates administrative access to the device, enables analysis of the device's status and condition, and allows for device reconfiguration. The management level 405 can be part of the control level 404 or communicate directly with it. In one implementation, the administrator does not have direct access to the components of the control level 404. Instead, information is collected by the management level 405, and changes to the control level 404 are made by the management level 405.

[0046] Fig. Figure 4B shows a Network 420 with Switches 432A-432E supporting adaptive routing and telemetry-based congestion control. The Network 420 can be implemented using a variety of the communication protocols described here. In one embodiment, the Network 420 is implemented using the InfiniBand protocol. In another embodiment, the Network 420 is an Ethernet, Converged Ethernet, or Ultra-Ethernet network. The Network 420 can incorporate aspects of Fabric 370. Fig. 3 included. Switches 432A-432E may be an implementation of the forwarding element 400 of Fig. Network 420 handles packet-based communication for multiple nodes (e.g., node 424, node 446), including a source node 422 and a destination node 442, of a data transmission to be carried out over network 420. The packets of a flow are routed through network 420 along a route that traverses the switches (switches 432A-432E) and links (links 426A-426B, 427A-427B, 428, 429A-429B, 430A-430B) of network 420. In an InfiniBand application, the switches and links belong to a specific InfiniBand subnet, which is managed by a subnet manager (SM) that may be located in one of the switches (e.g., switch 432D). Source node 422 and destination node 442 are the source and destination nodes for an example data flow. Depending on the configuration of network 420, packets can flow from any node to any other node via one or more paths.

[0047] The 432A-432E switches comprise a 402 data plane, a 404 control plane, a 405 management plane, and 406 physical ports, as described in the forwarding element 400 of Fig. 4A. A 404 control plane processor can be used to implement adaptive routing techniques to adjust a route between source node 422 and destination node 442 based on the current state of the network. During network operation, the route from source node 422 to destination node 442 may become unsuitable or impaired in its ability to transmit packets due to various events, such as congestion, link failures, or head-of-line blocking. Should such a scenario occur, switches 432A-432E can be configured to dynamically adjust the route of packets flowing over a compromised path.

[0048] An adaptive routing (AR) event can be detected by any switch along a compromised route, for example, when the switch attempts to output packets to a specific output port. For instance, data from source node 422 to destination node 442 might traverse connections across the network's switches. An AR event might be detected by switch 432D for connection 429B, for example, in response to congestion or a connection failure associated with connection 429B. Upon detection of the AR event, switch 432D, as the detecting switch, generates an adaptive routing notification (ARN) with an identifier that distinguishes an ARN packet from other packet types. In various implementations, the ARN includes parameters such as an identifier for the detecting switch, the type of AR event, the source and destination addresses of the flow that triggered the AR event, and / or other appropriate parameters.The detecting switch sends the ARN backward along the route to the preceding switches. The ARN can contain a request to the notified switches to change their route to avoid passing over the detected switch. A notified switch can then check if its routes can be modified to bypass the detected switch. Otherwise, the switch forwards the ARN to the preceding switch along the route. In this scenario, switch 432B is unable to bypass switch 432D and forwards the ARN to switch 432A. Switch 432A can determine that the route to destination node 442 should be adjusted via connection 427A to switch 432C. Switch 432C can reach Switch 432E via connection 429A, so that packets from source node 422 can reach destination node 442, while bypassing the AR event with respect to connection 429B.

[0049] In various configurations, the 420 network can also adapt to congestion scenarios via programmable data planes within switches 432A-432E. These data planes are capable of executing programs to implement network-internal congestion control algorithms (CCAs) for TCP over Ethernet-based fabrics. Using in-band network telemetry (INT), the programmable data planes in switches 432A-432E can detect when a port or link along a route is congested and proactively attempt to reroute packets along alternative paths. For example, switch 432A can balance traffic to destination node 442 between links 427A and 427B based on the degree of congestion on the routes downstream of these links.

[0050] Fig. 4C shows an InfiniBand switch 450, which is an implementation of the forwarding element 400 of Fig. The InfiniBand Switch 450 can handle 4A. It includes a programmable data plane and is configurable to perform adaptive routing and telemetry-based congestion control, as described herein. The InfiniBand Switch 450 comprises the multiport IB interfaces 460A-460D and the core switch logic 480. The multiport IB interfaces 460A-460D comprise multiple ports. In one embodiment, a single physical interface (IB PHY 453) with input and output buffers is provided, connected to a single port. In another embodiment, the ports have separate physical interfaces. The ports can be coupled, for example, to an HCA 452, a TCA 461, or another InfiniBand Switch 432. The multiport IB interfaces 460A-460D can include a coupling field 454 configured to selectively couple input and output port buffers to the local memory 456.The 454 coupling field is a non-blocking coupling field that enables direct switching with low latency and fixed or variable packet size.

[0051] Local memory 456 contains several queues, including an outer receive queue 462, an outer send queue 463, an inner receive queue 464, and an inner send queue 465. The outer queues are used for data received at a specific Multiport IB interface and intended to be sent back over the same Multiport IB interface. The inner queues are used for data that is forwarded over a different Multiport IB interface than the one through which the data was received. Other types of queue configurations can be implemented in local memory 456. For example, different queues can exist to support multiple traffic classes, either based on individual ports, shared ports, or a combination thereof.The multi-port IB interfaces 460A-460D include a power management circuit 455, which can adjust the power state of the circuits within the respective multi-port IB interface. Additionally, power management logic performing similar operations can be implemented as part of the core switch logic.

[0052] The multi-port IB interfaces 460A-460D include the packet processing and switching logic 458, which is generally used to perform aspects of packet processing and / or switching operations that are performed locally across multiple ports, rather than across the IB switch as a whole. Depending on the implementation, the packet processing and switching logic 458 may be configured to perform a subset of the operations of the packet processing and switching logic 478 within the core switching logic 480, or it may be configured with the full functionality of the packet processing and switching logic 478 within the core switch logic 480. The processing functionality of the packet processing and switching logic 458 can vary depending on the complexity of the operations and / or the required speed of execution.The packet processing and switching logic of version 458 can, for example, include processors ranging from microcontrollers to multi-core processors. Different types or architectures of multi-core processors can also be used. Additionally, some of the packet processing operations can be implemented using embedded hardware logic.

[0053] The core switch logic 480 comprises a crosspoint switch 482, a memory 470, a subnet management agent (SMA 476), and packet processing and switching logic 478. The crosspoint switch 482 is a non-blocking, low-latency crosspoint switch that connects the multiport IB interfaces 460A-460D and establishes a connection to the memory 470. The memory 470 comprises receive queues 472 and transmit queues 474. In one embodiment, packets to be routed between the multiport IB interfaces 460A-460D can be received by the crosspoint switch 482, stored in one of the receive queues 472, processed by the packet processing and switching logic 478, and stored in a transmit queue 474 for transmission to the outgoing multiport IB interface.In implementations that do not use the multiport IB interfaces 460A-460D, the core switch logic 480 and the crosspoint switch 482 switch packets directly between I / O buffers with receive queues 472 and transmit queues 474 within the memory 470.

[0054] The Packet Processing and Switching Logic 478 includes programmable functions and can execute data-plane programs across a variety of multi-core processor types and architectures. The Packet Processing and Switching Logic 478 represents the applicable circuitry and logic for implementing switching and packet processing operations beyond those performed at the ports themselves. The processing elements of the Packet Processing and Switching Logic 478 execute software and / or firmware commands configured to implement packet processing and switching operations. This software and / or firmware can be stored in non-volatile memory on the switch itself. Alternatively, the software can be downloaded or updated over a network in conjunction with the initialization of the InfiniBand Switch 450.

[0055] The SMA 476 is configurable for managing, monitoring, and controlling the functions of the InfiniBand 450 switch. The SMA 476 also acts as an agent of the subnet manager (SM) for the subnet connected to the InfiniBand 450 switch and communicates with it. The SM is the entity that discovers the devices within the subnet and performs periodic checks of the subnet to detect changes in the subnet topology. An SMA within a subnet can be designated as the primary SMA for that subnet and act as the SM. Other SMAs within the subnet then communicate with this SMA. Alternatively, the SMA 476 can cooperate with other SMAs in the subnet and function as a distributed SM. In some embodiments, the SMA 476 includes or runs on standalone circuitry and logic, such as a microcontroller, a single-core processor, or a multi-core processor.In other embodiments, SMA 476 is implemented via software and / or firmware instructions executed on a processor core or other processing element that is part of a processor or other processing element used to implement the packet processing and switching logic 478.

[0056] The embodiments are not specifically limited to implementations with IB multiple interfaces 460A-460D. In one embodiment, the ports are connected to their own receive and transmit buffers, with the crosspoint switch 482 configured to connect these buffers to the receive queues 472 and the transmit queues 474 in memory 470. Packet processing and switching is then primarily performed by the packet processing and switching logic 478 of the core switch logic 480.

[0057] Fig. Figures 5A-5B show an example of network interface devices. Fig. 5A shows a network interface device 500 that can be configured as a Smart Ethernet device. Fig. Figure 5B shows a 550 network interface device that can be configured as an InfiniBand channel adapter.

[0058] As in Fig. As shown in Figure 5A, the network interface device 500 can, in one configuration, include a transceiver 502, a transmit queue 507, a receive queue 508, a memory 510, a bus interface 512, and a DMA engine 526. The network interface device 500 can also include a SoC / SiP 545, which comprises processors 505 for implementing the intelligent network interface device functionality, as well as accelerators 506 for various accelerated functions such as NVMe-oF or RDMA. The specific characteristics of the network interface device 500 depend on the protocol implemented through the network interface device 500.

[0059] In various configurations, the Network Interface Device 500 can be configured to connect to networks, including but not limited to Ethernet, including Ultra Ethernet. However, the Network Interface Device 500 can also be configured as an InfiniBand or NVLink interface by modifying various components. For example, the Transceiver 502 can receive and send packets in accordance with the InfiniBand, Ethernet, or NVLink protocols. Other protocols can also be used. The Transceiver 502 can receive and send packets to and from a network over a network medium. The Transceiver 502 can include a PHY circuit 514 and a Media Access Control (MAC) circuit 516.The PHY circuit 514 can include encoding and decoding circuitry to encode and decode data packets according to applicable physical layer specifications or standards. The MAC circuit 516 can be configured to assemble the data to be transmitted into packets containing destination and source addresses along with network control information and error detection hash values.

[0060] The SoC / SiP 545 can contain processors that can be any combination of a CPU, a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other programmable hardware device, enabling programming of the 500 network interface device. For example, an intelligent network interface can provide packet processing capabilities within the network interface using 505 processors. The configuration of the 505 processors' operation, including the programmable data plane processors, can be programmed using protocol-independent packet processors (P4), C, Python, Broadcom Network Programming Language (NPL), x86- or ARM-compatible executable binaries, or other executable binaries.

[0061] The packet distributor 524 can distribute received packets for processing by multiple CPUs or cores using time slot allocation. An interrupt coalescing circuit 522 can perform interrupt moderation, where it waits for multiple packets to arrive or a timeout before generating an interrupt to the host system to process the received packets. Receive segment coalescing (RSC) can be performed by the network interface 500, where portions of incoming packets are grouped into segments of a single packet. The network interface device 500 can then provide this grouped packet to an application.A DMA engine 526 can copy a packet header, packet payload, and / or descriptor directly from host memory to the network interface or vice versa, instead of copying the packet to an intermediate buffer on the host and then performing another copy operation from the intermediate buffer to the destination buffer. Memory 510 can be any type of volatile or non-volatile memory and can store any queue or instructions used to program the network interface device 500. The send queue 507 can contain data or references to data to be transmitted over the network interface. The receive queue 508 can contain data or references to data received from a network over the network interface. The descriptor queues 520 can contain descriptors that reference data or packets in the send queue 507 or the receive queue 508.The 512 bus interface can provide an interface to the host device. For example, the 512 bus interface can be compatible with PCI Express, although other connection standards can also be used.

[0062] As in Fig. As shown in Figure 5B, a network interface device 550 can be configured as an implementation of the network interface device 500 to implement an InfiniBand HCA. The network interface device 550 includes network ports 552A-552B, memory 554A-554B, a PCIe interface 558, and an integrated circuit 556 containing hardware, firmware, and / or software for implementing, managing, and / or controlling the HCA functionality. In one implementation, the integrated circuit includes a hardware transport engine 560, an RDMA engine 562, congestion control logic 563, virtual endpoint logic 564, offload engines 566, QoS logic 568, GSA / SMA logic 569, and a management interface. Different implementations of the network interface device 550 may include additional components or omit some components.A network interface device 550 configured as a TCA contains an implementation-specific subset of the functionality of an HCA. The integrated circuit 556 contains programmable and fixed functional hardware for implementing the described functionality.

[0063] While the illustrated implementation of the Network Interface Device 550 uses a PCIe 558 interface, other implementations may use different interfaces. For example, the Network Interface Device 550 may use an Open Compute Project (OCP) mezzanine connector. Furthermore, the PCIe 558 interface can also be configured with a multi-host solution, allowing multiple compute or storage hosts to connect to the Network Interface Device 550. The PCIe 558 interface can also support a technology that enables direct PCIe access to multiple CPU sockets, eliminating the need for network traffic to traverse the interprocessor bus of a multi-socket server motherboard when the server includes the Network Interface Device 550.

[0064] The 550 network interface device implements endpoint elements of the InfiniBand architecture, which is based on queue pairs and RDMA. InfiniBand offloads traffic control from software by using execution queues (e.g., work queues) that are initiated by a software client and managed in hardware. Communication endpoints comprise a queue pair (QP) with a send queue and a receive queue. A QP is a memory-based abstraction where communication occurs between memory-to-memory transfers between applications or between applications and devices. Communication with the QPs takes place over virtual lanes of network ports 552A-552B, which allow multiple independent data flows to use the same connection, with separate buffering and flow control for each flow.

[0065] Communication occurs via channel I / O, where a virtual channel directly connects two applications residing in separate address spaces. The Hardware Transport Engine 560 contains hardware logic for performing transport-layer operations across the QP for an endpoint. The RDMA Engine 562 leverages the Hardware Transport Engine 560 to perform RDMA operations between endpoints. Implementing RDMA operations in hardware, the RDMA Engine 562 allows an application to read and write the memory of a remote system without operating system kernel intervention or unnecessary data copying by permitting one endpoint of a communication channel to directly place information into the memory of another endpoint. The Virtual Endpoint Logic 564 manages the operation of a virtual endpoint for channel I / O, i.e., a virtual instance of a QP used by an application.The virtual endpoint logic 564 assigns the QPs to the virtual address space of an application connected to a virtual endpoint.

[0066] Congestion control logic 563 performs operations to reduce the occurrence of congestion on a channel. In various implementations, congestion control logic 563 can perform flow control over a channel to limit congestion at the destination of a data transmission. Congestion control logic 563 can perform flow control at the link level to manage source congestion on virtual links of network ports 552A-552B. In some implementations, congestion control logic can perform operations to limit congestion at intermediate points (e.g., IB switches) along a channel.

[0067] The Offload Engines 566 enable the offloading of network tasks that would otherwise be performed in software on the Network Interface Device 550. The Offload Engines 566 can support the offloading of operations, including, but not limited to, offloading receive-side scaling from a device driver or stateless network operations, such as for TCP implementations over InfiniBand, like stateless offloading of TCP / UDP / IP or offloading of Virtual Extensible Local Area Network (VXLAN). The Offload Engines 566 can also offload operations of an interrupt coalescing circuit 522 of the Network Interface Device 500. Fig. Implement 5A. The Offload Engines 566 can also be configured to support offloading NVMe-oF or other storage acceleration operations from a CPU.

[0068] The QoS logic 568 can perform QoS operations, including the QoS functionality built into InfiniBand's basic service provisioning mechanism. The QoS logic 568 can also implement enhanced InfiniBand QoS, such as fine-grained end-to-end QoS. The QoS logic 568 can implement queueing services and management to prioritize traffic flows and ensure service levels or bandwidth according to flow priority. For example, the QoS logic 568 can configure virtual lane arbitration for network ports 552A-552B according to flow priority. The QoS logic 568 can also work in conjunction with the congestion control logic 563.

[0069] The GSA / SMA logic 569 implements General Services Agent (GSA) operations for managing the network interface device 550 and the InfiniBand structure, as well as for performing subnet management agent operations. GSA operations include device-specific management tasks such as querying device attributes, configuring device settings, and controlling device behavior. The GSA / SMA logic 569 can also implement SMA operations, including a subset of those performed by the SMA 476 of the InfiniBand switch 450. Fig. 4C can be executed. For example, the GSA / SMA 569 logic can handle management requests from the subnet manager, including requests to reset the device, update the firmware, or change configuration parameters.

[0070] The 570 management interface provides support for a hardware interface to perform out-of-band management of the 550 network interface device, such as a connection to a Board Management Controller (BMC) or a hardware debug interface.

[0071] Fig. Figure 6 is a block diagram illustrating a Programmable Network Interface 600 and a Data Processing Unit. The Programmable Network Interface 600 is a programmable network engine that can be used to accelerate network-based computing tasks in a distributed environment. The Programmable Network Interface 600 can be connected to a host system via the Host Interface 670. The Programmable Network Interface 600 can be used to accelerate network or memory operations for CPUs or GPUs of the host system. The host system could, for example, be a node of a distributed learning system used, for instance, to perform distributed training, as in Fig. Figure 6 shows the host system. The host system can also be a data center node within a data center.

[0072] In one embodiment, access to remote storage containing model data can be accelerated by the programmable network interface 600. For example, the programmable network interface 600 can be configured to represent remote storage devices as local storage devices to the host system. The programmable network interface 600 can also accelerate RDMA operations between GPUs of the host system and GPUs of remote systems. In one embodiment, the programmable network interface 600 can enable, but is not limited to, storage features such as NVMe-oF. The programmable network interface 600 can also accelerate encryption, data integrity, compression, and other operations for remote storage on behalf of the host system, so that remote storage approaches the latencies of storage devices directly attached to the host system.

[0073] The Programmable Network Interface 600 can also perform resource allocation and management on behalf of the host system. Memory security operations can be offloaded to the Programmable Network Interface 600 and performed in conjunction with the allocation and management of remote storage resources. Network-based operations for managing access to remote storage, which would otherwise be performed by a processor in the host system, can instead be executed by the Programmable Network Interface 600.

[0074] In one embodiment, network and / or data security operations can be offloaded from the host system to the programmable network interface 600. The security policies for a data center node can be managed by the programmable network interface 600 rather than by the host system's processors. For example, the programmable network interface 600 can detect and mitigate an attempted network-based attack (such as a DDoS attack) on the host system, thereby preventing the attack from impacting the host system's availability.

[0075] The programmable network interface 600 can contain a system-on-a-chip (SoC / SiP 620) that runs an operating system across multiple processor cores 622. The processor cores 622 can include general-purpose processor cores (e.g., CPU cores). In one embodiment, the processor cores 622 can also include one or more GPU cores. The SoC / SiP 620 can execute instructions stored in a storage device 640. A storage device 650 can store local operating system data. The storage device 650 and the storage device 640 can also be used to cache remote data for the host system. The network ports 660A-660B provide connectivity to a network or fabric and facilitate network access for the SoC / SiP 620 and, via the host interface 670, for the host system.In one configuration, a first network port 660A can be connected to a first forwarding element, while a second network port 660B can be connected to a second forwarding element. Alternatively, both network ports 660A-660B can be connected to a single forwarding element via a link aggregation protocol (LAG). The programmable network interface 600 can also include an I / O interface 675, such as a USB (Universal Serial Bus) interface. The I / O interface 675 can be used to connect external devices to the programmable network interface 600 or as a debug interface. The programmable network interface 600 also includes a management interface 630, which allows software on the host device to manage and configure the programmable network interface 600 and / or the SoC / SiP 620.In one embodiment, the programmable network interface 600 can also include one or more accelerators or GPUs 645 to accept the offloading of parallel computing tasks from the SoC / SiP 620, host system, or remote systems connected via network ports 660A-660B. For example, the programmable network interface 600 can be configured with a graphics processor and participate in general-purpose or graphics computing operations in a data center environment.

[0076] One or more aspects can be implemented by representative code stored on a machine-readable medium that represents and / or defines logic within an integrated circuit, such as a processor. For example, the machine-readable medium can contain instructions representing different logics within the processor. When read by a machine, the instructions can cause the machine to fabricate the logic to perform the techniques described herein. Such representations, also known as "IP cores," are reusable units of logic for an integrated circuit that can be stored on a physical, machine-readable medium as a hardware model describing the structure of the integrated circuit.The hardware model can be delivered to various customers or manufacturing facilities, which load the hardware model onto manufacturing machines that produce the integrated circuit. The integrated circuit can be manufactured such that the circuit performs operations described in connection with any of the embodiments described herein.

[0077] Fig. Figure 7 is a block diagram showing an IP Core Development System 700. The IP Core Development System 700 can be used to fabricate an integrated circuit to perform the fabric and data center component operations described here. The IP Core Development System 700 can be used to create modular, reusable designs that can be integrated into a larger design or used to build an entire integrated circuit (e.g., an integrated SoC circuit). A Design Facility 730 can generate a software simulation 710 of an IP core design in a higher-level programming language (e.g., C / C++). The software simulation 710 can be used to design, test, and verify the behavior of the IP core using a simulation model 712. The simulation model 712 can include functional, behavioral, and / or temporal simulations.From the simulation model 712, a Register-Transfer-Level Design (RTL Design 715) can then be generated or synthesized. The RTL Design 715 is an abstraction of the integrated circuit's behavior, modeling the flow of digital signals between hardware registers, including the associated logic performed using the modeled digital signals. In addition to an RTL Design 715, subordinate designs at the logic or transistor level can also be created, designed, or synthesized. Thus, the specific details of the initial design and simulation can vary.

[0078] The RTL design 715, or an equivalent design, can further be synthesized by the design facility into a hardware model 720, which may be in a hardware description language (HDL) or another representation of physical design data. The HDL can further be simulated or tested to verify the IP core design. The IP core design can be stored for delivery to a manufacturing facility 765 using non-volatile memory 740 (e.g., hard disk, flash memory, or any non-volatile storage medium). The manufacturing facility 765 can be a third-party manufacturing facility. Alternatively, the IP core design can be transmitted via a wired connection 750 or a wireless connection 760 (e.g., over the Internet). The manufacturing facility 765 can then manufacture an integrated circuit based, at least in part, on the IP core design.The manufactured integrated circuit can be designed to perform operations in accordance with at least one embodiment described herein. Use of resources in a mixture-of-experts process

[0079] Tokens are commonly used in multi-processor processing, such as in a system with a large number of accelerators, like GPUs, across multiple servers. Mixture of Experts (MoE) technology, which places multiple experts on GPUs or other accelerators or units, can enable models to be pre-trained with significantly less computational effort than other models. An expert is a neural network (e.g., a feed-forward network (FFN)), where experts are determined by forwarding tokens to them. Generally, a gate network or router determines which tokens are sent to which experts, according to a specific algorithm implemented in the implementation.

[0080] A router consists of learned parameters and is typically pre-trained concurrently with the rest of the network. In a typical deployment, one or more experts can utilize a single graphics processing unit (GPU), and multiple GPUs across multiple servers can form a cluster with a large number of experts. To maximize the efficiency and accuracy of MoE processing, it is crucial to consider routing, load balancing, and the cross-device provisioning of experts during inference time.

[0081] In some embodiments, routing functions for mixed expert deployments are implemented in separate processing resources, where the processing resources may include hardware accelerators such as IPUs (Infrastructure Processing Units), Data Processing Units (DPUs), SmartNICs (Smart Network Interface Cards), and other units.

[0082] Fig. Figure 8 is a representation of a computer system or computing device with routing functions for mixture-of-experts deployments in processing resources according to some embodiments. In some embodiments, a computer system or computing device comprises 800 hardware components to support high-bandwidth data transmissions in AI processing. As in Fig. Figure 8 shows that the computer system or computing device comprises 800, which is in Fig. 1. The processing system 100 shown may comprise processing resources 805, wherein the processing resources 805 may be one or more central processing units (CPUs) or other general-purpose processors 807, multiple graphics processing units (GPUs) 810 which may be configured to process in sets or clusters, and one or more hardware accelerators 830 which may include infrastructure processing units (IPUs), data processing units (DPUs), or other resources. The one or more processing resources may include elements that are in Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5, Fig. 6, Fig. 7 to Fig. 8 are shown. The computer system or computing device 800 may also include computer memory 840, e.g., high-bandwidth memory (HBM), and computer storage 845, e.g., a solid-state drive (SSD), a hard disk drive (HDD), or other storage technology, to store data for processing.

[0083] In some embodiments, the 830 hardware accelerators include, among other things, a Mixture-of-Experts (MoE) circuit 835 to provide routing functions for Mixture-of-Experts operations. In some embodiments, the circuit includes one or more of the components described in Fig. 13A for a processing resource. A hardware accelerator may also include one or more network ports for connecting to the network and one or more hardware direct memory access (DMA) engines for transferring data over the network.

[0084] Fig. Figure 9 illustrates the application of a mixed expert in AI processing. One component in a MoE topology is a gate network or router that determines which tokens are sent to which expert in a system. Fig. Figure 9 illustrates a specific example where the routers are to determine which experts should receive tokens 1 to 8 for the specified words in an LLM.

[0085] How a token is forwarded to an expert is one of the most important decisions that must be made when working with MoEs in a system. The router itself, as it is configured in Fig. Figure 9 is composed of learned parameters that are pre-trained with the network. In a typical deployment, one or more experts can use a graphics processor (or other accelerator), and multiple graphics processors on multiple servers form a cluster with a large number of experts.

[0086] In such an infrastructure, routing, load balancing, and the provision of experts at inference time across such devices are critical to ensuring efficient operation. In some implementations, these aspects of the MoE are managed through the use of a processing resource that includes expert tracking and management capabilities.

[0087] Fig. Figure 10 illustrates the routing in a mixture-of-experts operation. Fig. Figure 10 illustrates a simplified example 1 of the forwarding of tokens from device n to experts, represented by expert 1 at device 0, expert 2 at device 1 and expert 3 at device 2, with a capacity factor of 1.0.

[0088] However, if too many tokens are forwarded to a single expert, some tokens will be dropped. As shown in Example 1, 3 tokens are forwarded to expert 1, 2 tokens to expert 2, and 1 token to expert 3. In this example, forwarding 3 tokens to expert 1 can result in one of the tokens being dropped because the device's capacity has been exceeded.

[0089] In comparison, Example 2 similarly involves the forwarding of tokens from device n to experts, presented by expert 1 to device 0, expert 2 to device 1, and expert 3 to device 2. In this case, the capacity factor is 1.5, allowing the experts to forward the tokens without dropping them.

[0090] Fig. Figures 11A-11F are illustrations of routing algorithms in a Mixture-of-Experts (MoE) operation. The figures illustrate specific algorithms that employ various routing strategies, which can be used and optimized in MoE operation. However, the implementations are not limited to these examples.

[0091] In the depictions of the Fig. 11A-11F:

[0092] Fig. Figure 11A illustrates an example of top-1 routing, where the highest value is passed on to an expert.

[0093] Fig. Figure 11B illustrates an example of top-2 routing, where the two largest values ​​are forwarded to an expert.

[0094] Fig. 11C illustrates an example of hash routing, where hash functions are used to identify the routing of tokens.

[0095] Fig. Figure 11D illustrates an example of expert token selection, where multiple tokens are provided to multiple routers, with the experts selecting the tokens to be forwarded to one expert.

[0096] Fig. Figure 11E shows an example of BASE (Balanced Assignment of Experts) routing of tokens, where a linear assignment is solved.

[0097] Fig. Figure 11F shows an example of token reinforcement learning routing.

[0098] Since token forwarding plays a crucial role in a mixture-of-experts deployment, the operational efficiency and availability of experts within the group can be considered. For example, an expert might be unavailable at the time of inference due to overload or network congestion. In such a case, the data can be forwarded to another expert, as using a slower expert slows down the rest of the stack and reduces overall GPU utilization, leading to higher costs and lower efficiency.

[0099] In some embodiments, a device, system, or process includes one or more of the following capabilities to enable a scalable, universal provision of a mix of experts for processing in a network: (1) Expert capacity tracking across the network with minimal overhead; (2) Load balancing between experts with minimal overhead; (3) Tracking network traffic across the network with minimal overhead; and (4) Dynamic investigation of the distribution of experts, which can be used, for example, to dynamically migrate or replicate a subset of experts in a cluster.

[0100] The aforementioned capabilities are typically lacking in existing solutions because they are implemented in software, which incurs high operating costs. However, implementing a sophisticated distributed router in one or more GPUs within the network would consume the expensive GPU resources needed and used for data processing. In some implementations, routing support is instead implemented in separate processing resources, such as IPUs, which include hardware circuitry to provide routing support for MoE operations.

[0101] Fig. Figure 12 is an example of a system for providing resources using improved mixture-of-experts processing according to some embodiments. Fig. Section 12 implements System 1200 routing functions for mixture-of-expert deployments in one or more processing resources, which may include infrastructure processing units (IPUs) and other hardware. Hardware implementation, such as in IPUs, offers significantly higher efficiency compared to software implementations because, among other potential advantages, the hardware has better access to telemetry data, network statistics, and other data, and can be used for tracking and advanced processing to achieve improved mixture-of-expert operation.

[0102] As in Fig. In 12, the system 1200 comprises multiple GPUs or other accelerators 1210, wherein at least a subset of the multiple GPUs can operate as experts in a mixture of experts. In some embodiments, the system 1200 also comprises one or more additional processing resources, which may include the illustrated IPUs 1220. The IPUs 1220 contain hardware circuitry that enables a mixture-of-experts operation, including tracking, predicting, and forwarding experts, as well as migrating and replicating specific experts.

[0103] In some implementations, a processing resource maintains a state table that can be used to track token leads on a per-expert basis, along with network congestion statistics for different experts. Additionally, the processing resource tracks the various GPUs / nodes on which an expert is deployed, as well as the utilization of each instance of that expert. The hardware may also include circuitry for predicting expert loads based on historical tracking and for predicting an expert's relevance / weighting when evaluating a given token. If an expert consistently exhibits low relevance, it may be necessary to retrain the expert, and the processing resources can facilitate signaling for this retraining.A processing resource may also include the ability to migrate or replicate an expert as needed.

[0104] Fig. Figure 13A illustrates a resource deployment using enhanced mixture-of-experts processing according to some embodiments. In some embodiments, a hardware accelerator 1300, which may include an IPU, DPU, SmartNIC, or other resource, incorporates a circuit 1305 for mixture-of-experts (MoE) processing. The circuit 1305 includes one or more of the following mechanisms:

[0105] Expert Token Capacity and Network Load Tracking Circuit 1310 - The Expert Token Capacity and Network Load Tracking Circuit 1310 enables the processing resource to track different deployments for an expert within the associated system, along with the GPU ID and network load. This gives the Hardware Accelerator 1300 an overview of the various instances of an expert, as well as the utilization and load of each instance. Combined with network utilization, this provides a dynamic overview of expert deployments and assists processing resources in making advanced routing decisions for expert deployments on a dynamic basis. The data managed by the Tracking Circuit 1310 is represented as a State Table 1350, which is displayed in Fig. 13B will be explained in more detail.

[0106] Expert Load History Tracking Circuit 1315 - In a processing implementation, past processing behavior generally provides a good predictor of future operations, and there is a high correlation between token sequences and a small subset of experts being used. Therefore, the Expert Load History Tracking Circuit 1315 can historically track this correlation by storing hot token sequences (i.e., token sequences that repeat) and the corresponding experts, with the tracking performed in hardware within the Hardware Accelerator 1300.

[0107] Expert Token Load Prediction Circuit 1320 - The Expert Token Load Prediction Circuit 1320 can be used to provide a prediction of future token loads for experts, based at least partially on system knowledge. Since a prediction capacity is available at the time of inference, the prediction circuit can implement various predictors that are adapted based on learning processes.

[0108] Expert Relevance Weighting Tracking Circuit 1325 - The Expert Relevance Weighting Tracking Circuit 1325 tracks the relevance weights for experts handling tokens in a system. Since certain experts may perform better for one or more tokens, the experts' relevance weights are tracked and updated in the Hardware Accelerator 1300 and can be used when routing tokens. Furthermore, if an expert exhibits consistently low relevance, it may be necessary to retrain the expert, and signaling for retraining can be facilitated by the Hardware Accelerator 1300, which is based, at least in part, on the relevance weights tracked by the Expert Relevance Weighting Tracking Circuit 1325.

[0109] Expert Routing Circuit 1330 - The Expert Routing Circuit 1330 manages the forwarding of tokens to experts based on collected expert data, including data from one or more of the Expert Token Capacity and Network Load Tracking Circuit 1310, the Expert Load History Tracking Circuit 1315, the Expert Token Load Prediction Circuit 1320, and the Expert Relevance Weighting Tracking Circuit 1325. Managing token routing may involve adjusting token routing to avoid identified problems in the Mixture of Experts operation, such as overloading one or more experts with tokens.

[0110] Expert Migration Circuit 1330 - The Expert Migration Circuit 1330 can be used to facilitate the migration or replication of expert operations to other hardware resources. A specific expert can be migrated (i.e., moved) from a first GPU to a second GPU, or replicated to the second GPU, if required due to utilization or other considerations. For example, if a particular expert is overloaded, which can occur in a situation where there is significant unevenness in how experts are used for a given token sequence, the migration circuit facilitates the replication or migration of that expert to another resource to relieve at least some of the token-handling load on the expert.

[0111] The Accelerator 1300 includes additional elements, which may include one or more network ports 1360 for connecting to a network and one or more DMA engines 1365 for direct memory access for data transfer. The Accelerator 1300 may also include one or more memory, cache, and data storage devices (which are described in...). Fig. (not shown in 13A) for storing data for processing, including data for MoE processing.

[0112] Fig. Figure 13B shows expert data maintained for mixture-of-experts processing according to some embodiments. In some embodiments, state table 1350 is maintained by one or more processing resources in a system, such as the one in Fig. 13A illustrated hardware accelerator 1300, managed.

[0113] As in Fig. As shown in Figure 13B, the state table 1350, which is maintained by one or more processing resources, can contain, among other things, the following: (a) an identifier for an expert, which may contain an instance identifier. (b) a token capacity for the respective expert. (c) a current token load for the respective expert. (d) an identifier for the GPU hosting the expert, where a GPU may host one or more experts depending on the specific implementation. (e) Identification of related sibling experts and host identifiers. (f) the current network load of the system.

[0114] Fig.Figure 14 is a flowchart illustrating a process for a mixture-of-experts operation in network processing according to some embodiments. In some embodiments, a process 1400 includes receiving data for processing by multiple processing resources in a network 1405, where the processing resources may include multiple GPUs. The processing may include processing using a model, where the model may be a large language model or another AI model.

[0115] In operation, the network involves a mix of experts processing a Model 1410, with the experts being hosted by GPUs or other processing resources. The processing may include routing tokens associated with the model's processing to the experts according to an Algorithm 1415.

[0116] Process 1400 can further include monitoring the operation of the mixture of experts in processing 1420 and maintaining the collected data regarding the experts and the mixture-of-experts operation 1425. The data being tracked can include data on the experts' token capacity and the network's token load. The process can further include tracking the correlation between tokens and the experts used for the tokens 1430 and generating a prediction of the experts' future token loads 1435. In addition, relevance weights can be determined for the experts handling tokens 1440. In certain cases, if it is determined that one or more experts in the operation have low relevance, a signal for retraining one or more experts can be generated 1445.

[0117] In some embodiments, the routing of tokens to experts is managed at least partially based on the collected expert data (1450). Managing token routing may involve adjusting the routing of tokens to avoid identified problems in the mixture-of-experts operation, such as overloading one or more experts with tokens. Furthermore, as needed, the process (1400) may involve migrating one or more experts from a first site to a second site or replicating one or more experts (1455), where the migration or replication may be implemented to offload at least part of the token-handling load for the one or more experts or to address other problems in the mixture-of-experts processing.

[0118] The following examples refer to specific embodiments: In Example 1, a device includes one or more network ports; one or more direct drive access (DMA) engines; and a circuit for processing a mixture of experts (MoE) in the network, wherein the circuit includes at least a circuit for tracking the routing of tokens during MoE processing, a prediction circuit for generating predictions regarding MoE processing, including predictions of future token loads for MoE processing, and a routing management circuit for managing the routing of tokens during MoE processing at least partially based on the predictions regarding MoE processing. In Example 2, for the device of Example 1, the data stored by the circuit with respect to MoE processing includes at least token capacity and token load data for MoE processing. In Example 3, for the device of Example 1 or 2, the circuit also includes a circuit for tracking historical token loads for experts in MoE processing. In Example 4, for the device of one of Examples 1 to 3, the circuit also includes a circuit for tracking the expert relevance weighting for experts in MoE processing. In Example 5, for the device from one of Examples 1 to 4, the circuit is intended to determine whether an expert in MoE processing needs retraining based on the relevance weighting for the expert. In Example 6, for the apparatus of one of Examples 1 to 5, the circuit also includes a circuit to perform one or more of the following steps: migration of one or more experts from a first accelerator to a second accelerator, or replication of one or more experts. In Example 7, for the device according to one of Examples 1 to 6, the device includes an infrastructure processing unit (IPU). In Example 8, a device includes a memory; a plurality of processors, including a plurality of graphics processing units (GPUs); and one or more hardware accelerators, which include a circuit for managing the mixture-of-experts (MoE) processing, wherein the circuit includes at least a circuit for tracking the forwarding of tokens in the MoE processing, a prediction circuit for generating predictions regarding the MoE processing, including predictions of future token loads for the MoE processing, and a routing management circuit for managing the routing of the tokens in the MoE processing at least partially based on the predictions regarding the MoE processing. In Example 9, for the device of Example 8, the data stored by the circuit with respect to MoE processing includes at least token capacity and token load data for MoE processing. In Example 10, for the apparatus of Example 8 or 9, the circuit also includes a circuit for tracking historical token loads for experts in MoE processing. In Example 11, for the apparatus of one of Examples 8 to 10, the circuit also includes a circuit for tracking the expert relevance weighting for experts in MoE processing. In Example 12, for the device from one of Examples 8 to 11, the circuit is intended to determine whether an expert in MoE processing needs retraining, at least partially, based on the relevance weighting for the expert. In Example 13, for the device from one of Examples 8 to 12, the circuit also includes a circuit to perform one or more of the following steps: migration of one or more experts from a first GPU to a second GPU, or replication of one or more experts. In Example 14, for the device from one of Examples 8 to 13, the one or more hardware accelerators comprise one or more infrastructure processing units (IPUs). In Example 15, a procedure involves receiving data for processing a model by a multitude of graphics processing units (GPUs) in a network; routing tokens associated with processing the model; monitoring the operation of the MoE processing in the network; generating predictions regarding the MoE processing, including predicting future token loads in the MoE processing; and managing the routing of tokens in the MoE processing, at least on the basis of the generated predictions. In Example 16, for the procedure of Example 15, the data maintained with respect to MoE processing includes at least token capacity and token load data for one or more experts. In Example 17, for the procedure of Example 15 or 16, the procedure also includes tracking historical token loads for MoE processing. In Example 18, for the procedure of one of Examples 15 to 17, the procedure further includes tracking the expert relevance weighting for experts in MoE processing. In Example 19, for the procedure from one of Examples 15 to 18, the procedure also includes determining whether an expert in MoE processing needs retraining based on the relevance weighting for the expert. In Example 20, for the procedure of one of Examples 15 to 19, the procedure also includes one or more of the following steps: migration of one or more experts from a first GPU to a second GPU; or replication of one or more experts. In Example 21, executable computer program instructions are stored on one or more non-volatile, machine-readable storage media. When executed by one or more processors, these instructions cause the one or more processors to perform operations that include: receiving data for model processing by a multitude of graphics processing units (GPUs) in a network; routing tokens associated with model processing; monitoring the operation of the MoE processing in the network; generating predictions regarding the MoE processing, including predicting future token loads in the MoE processing; and managing the routing of tokens in the MoE processing, at least based on the generated predictions. In Example 22, for the one or more storage media of Example 21, the data retained with respect to MoE processing includes at least token capacity and token load data for one or more experts. In Example 23, for the one or more storage media from Example 21 or 22, the instructions further include instructions for tracking historical token loads for MoE processing. In Example 24, for the one or more storage media from one of Examples 21 to 23, the instructions further include instructions for tracking the expert relevance weighting for experts in MoE processing. In Example 25, for the one or more storage media from one of Examples 21 to 24, the instructions further include instructions to determine whether an expert in MoE processing needs retraining based on the relevance weighting for the expert. In Example 26, for the one or more storage media of one of Examples 21 to 25, the instructions further include instructions for one or more of the following actions: migration of one or more experts from a first GPU to a second GPU; or replication of one or more experts. In Example 27, a device includes a means of receiving data for processing a model by a plurality of graphics processing units (GPUs) in a network; a means of routing tokens associated with the processing of the model; a means of monitoring the operation of the MoE processing in the network; a means of generating predictions regarding the MoE processing, including the prediction of future token loads in the MoE processing; and a means of managing the routing of the tokens in the MoE processing, based at least on the generated predictions. In Example 28, for the device of Example 27, the data retained with respect to MoE processing includes at least token capacity and token load data for one or more experts. In Example 29, for the apparatus of Example 27 or 28, the apparatus also includes means for tracking historical token loads for MoE processing. In Example 30, for the device from one of Examples 27 to 29, the device also includes means for tracking the relevance weighting of the experts during MoE processing. In Example 31, for the device from one of Examples 27 to 30, the device also includes means for determining whether an expert in MoE processing needs retraining based on the relevance weighting for the expert. In Example 32, for the apparatus from one of Examples 27 to 31, the apparatus also includes means for one or more of the following: migration of one or more experts from a first GPU to a second GPU; or replication of one or more experts.

[0119] While the description and illustration of the embodiments provided herein describe certain components, it is known to those skilled in the art that such components can be combined into fewer elements or divided into a larger number of components, depending on the requirements or expediency of a particular implementation. For example, the described component, which is responsible for sorting and temporarily storing inputs, can be implemented as a first sorting component and a second temporary storage component.

[0120] The above description includes numerous specific details for explanatory purposes, to facilitate a comprehensive understanding of the described embodiments. However, it will be clear to a person skilled in the art that the embodiments can also be implemented without some of these specific details. In other cases, known structures and devices are represented in the form of block diagrams. An intermediate structure may exist between the components shown. The components described or shown here may have additional inputs or outputs that are not shown or described.

[0121] Different embodiments can include different methods. These processes can be executed by hardware components or embodied in computer programs or machine-executable instructions that can be used to instruct a general-purpose or specialized processor, or logic circuits programmed with the instructions, to execute the processes. Alternatively, the processes can also be carried out by a combination of hardware and software.

[0122] Components of various embodiments may be provided as a computer program product, which may include a computer-readable medium containing computer program instructions stored thereon. These instructions can be used to program a computer (or other electronic devices) for execution by one or more processors to carry out a process according to certain embodiments. The computer-readable medium may include, but is not limited to, magnetic disks, optical disks, read-only memory (ROM), random-access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or other types of computer-readable media suitable for storing electronic instructions.Furthermore, embodiments can also be downloaded as a computer program product, whereby the program can be transferred from a remote computer to a requesting computer.

[0123] Many of the methods are described in their most basic form, but processes can be added to or removed from any of the methods, and information can be added to or subtracted from any of the described messages without departing from the basic scope of the present embodiments. It will be clear to those skilled in the art that many further modifications and adaptations can be made. The individual embodiments are not intended to limit the concept but to illustrate it. The scope of the embodiments is not determined by the specific examples listed above but only by the claims below.

[0124] When it is stated that an element "A" is coupled to element "B", element A can be coupled directly to element B or indirectly, for example, via element C. If the specification or claims state that a component, feature, structure, process, or characteristic A "causes" a component, feature, structure, process, or characteristic B, this means that "A" is at least a partial cause of "B," but that there may also be at least one other component, feature, structure, process, or characteristic that helps cause "B." If the specification indicates that a component, feature, structure, process, or characteristic "may" or "could" be included, it is not necessary that this particular component, feature, structure, process, or characteristic is included.When the patent specification or claim refers to “one” or “a” element, this does not mean that there is only one of the described elements.

[0125] An embodiment is an implementation or an example. References in the description to "an embodiment," "some embodiments," or "other embodiments" mean that a particular feature, structure, or property described in connection with the embodiments is included in at least some embodiments, but not necessarily in all embodiments. The various appearances of "an embodiment," "some embodiments," or "some embodiments" do not necessarily all refer to the same embodiments. In the preceding description of exemplary embodiments, various features are sometimes grouped together in a single embodiment, figure, or description to simplify the disclosure and facilitate understanding of one or more of the various new aspects.This type of disclosure is not to be understood as meaning that the claimed embodiments require more features than are expressly listed in the individual claims. Rather, as the following claims demonstrate, novel aspects are contained in fewer than all the features of a single embodiment disclosed above. Therefore, the claims are hereby expressly incorporated into this description, each claim constituting a separate embodiment.

[0126] The preceding description and drawings are intended to be illustrative rather than limiting. Those skilled in the art will understand that various modifications and changes can be made to the embodiments described herein without departing from the broader spirit and scope of the features set forth in the appended claims.

Claims

Device comprising: one or more network ports; one or more Direct Drive Access (DMA) engines; and circuitry for processing a Mixture of Experts (MoE) in the network; wherein the circuitry comprises at least: a circuitry for tracking the routing of tokens during MoE processing, a prediction circuitry for generating predictions regarding MoE processing, including predictions of future token loads for MoE processing, and a routing management circuitry for managing the routing of tokens in MoE processing, at least partially, based on the predictions regarding MoE processing. Device according to claim 1, wherein the data stored by the circuit with respect to MoE processing includes at least token capacity and token load data for MoE processing. Device according to claim 1 or 2, wherein the circuit further includes a circuit for tracking historical token loads for experts in MoE processing. Device according to one of claims 1 to 3, wherein the circuit further comprises a circuit for tracking the relevance weighting of experts in MoE processing. Device according to claim 4, wherein the circuit is intended to determine, on the basis of the relevance weighting for the expert, whether an expert in MoE processing requires retraining. Device according to any one of claims 1 to 5, wherein the circuit further comprises a circuit to perform one or more of the following steps: migration of one or more experts from a first accelerator to a second accelerator or replication of one or more experts. Device according to any one of claims 1 to 6, wherein the device includes an infrastructure processing unit (IPU). Device comprising: a memory; a plurality of processors, including a plurality of graphics processing units (GPUs); and one or more hardware accelerators with circuitry for managing the mixture-of-experts (MoE) processing; wherein the circuitry comprises at least: a circuit for tracking the routing of tokens during MoE processing, a prediction circuit for generating predictions regarding the MoE processing, including predictions of future token loads for the MoE processing, and a routing management circuit for managing the routing of tokens in the MoE processing, at least partially, based on the predictions regarding the MoE processing. Device according to claim 8, wherein the data stored by the circuit with respect to MoE processing includes at least token capacity and token load data for MoE processing. Device according to claim 8 or 9, wherein the circuit further includes a circuit for tracking historical token loads for experts in MoE processing. Device according to one of claims 8 to 10, wherein the circuit further comprises a circuit for tracking the relevance weighting of experts in MoE processing. Device according to claim 11, wherein the circuit determines whether an expert in MoE processing requires retraining at least partially based on the relevance weighting for the expert. Device according to any one of claims 8 to 12, wherein the circuit further comprises a circuit to perform one or more of the following steps: migrating one or more experts from a first GPU to a second GPU or replicating one or more experts. Device according to any one of claims 8 to 13, wherein the one or more hardware accelerators comprise one or more infrastructure processing units (IPUs). A method comprising: receiving data for model processing by a multitude of graphics processing units (GPUs) in a network; routing tokens associated with model processing; monitoring the operation of the MoE processing in the network; generating predictions regarding the MoE processing, including predicting future token loads in the MoE processing; and managing the routing of tokens in the MoE processing, at least on the basis of the generated predictions. Method according to claim 15, wherein the data maintained with respect to the MoE processing includes at least token capacity and token load data for one or more experts. The method of claim 15 or 16, further comprising: tracking historical token loads for MoE processing. Method according to one of claims 15 to 17, further comprising: tracking the relevance weighting of experts in MoE processing. The method of claim 18, further comprising: Determining whether an expert for MoE processing needs retraining, based on the relevance weighting for the expert. A method according to any one of claims 15 to 19, further comprising one or more of the following features: migrating one or more experts from a first GPU to a second GPU; or replicating one or more experts. Device comprising: means for receiving data for processing a model by a plurality of graphics processing units (GPUs) in a network; means for routing the tokens associated with the processing of the model; means for monitoring the operation of the MoE processing in the network; means for generating predictions regarding the MoE processing, including the prediction of future token loads in the MoE processing; and means for managing the forwarding of the tokens in the MoE processing, based at least on the generated predictions. Device according to claim 21, wherein the data stored with respect to MoE processing includes at least token capacity and token load data for one or more experts. Device according to claim 21 or 22, further comprising: means for tracking historical token loads for MoE processing. Device according to one of claims 21 to 23, further comprising: means for tracking the relevance weighting of experts in MoE processing; and means for determining whether an expert in MoE processing needs retraining based on the relevance weighting for the expert. Device according to any one of claims 21 to 24, further comprising one or more of the following elements: means for migrating one or more experts from a first GPU to a second GPU; or means for replicating one or more experts.

Images