Intra-network set operation

Abstract

The invention discloses an intra-network set operation. Examples described herein relate to a switch comprising circuitry configured to: for packet communications associated with a set operation for training a machine learning (ML) model: utilize a reliable transport protocol for communications from at least one worker node of the set operation to the switch, wherein utilizing a reliable transport protocol for communication from the at least one worker node of the set operation to the switch comprises storing a packet receipt status for per packet communication from the at least one worker node of the set operation to the switch; and utilizing an unreliable transport protocol by the switch to the device for which the result aggregation is to be performed, wherein the reliable transport protocol includes a protocol different from a protocol of the unreliable transport protocol.

Description

Background Art

[0001] Machine learning (ML) or high performance computing (HPC) clusters utilize a large number of servers and graphics processing units (GPUs) or tensor processing units (TPUs). Collective operations can be performed on data transmitted over the network at different switches. These systems are used to train ML models using iterative algorithms such as stochastic gradient descent. The input data is partitioned across workers and multiple iterations are performed on the training data. At each iteration, the workers calculate updates to the ML model parameters based on a subset of the local data and an intermediate current model. The workers then pass their results to aggregate into the model update, and the aggregated updates are summed for the model parameters at the node of the next iteration. These iterations are performed multiple times (epochs) on the entire dataset.

[0002] A parameter server (PS) can be used for collective operations, whereby worker nodes compute updates and send them to the PS. The PS pushes aggregated data, or pulls data from a PS server. Figure 1 shows an end-to-end solution for machine learning (ML) training using a PS architecture. The PS architecture includes a worker 100 and a parameter server (PS) 120 communicatively coupled using a switch 110. The end-to-end solution of the PS architecture includes reduce-scatter and allgather operators. Figure 1 shows that Worker 1 has three queue pairs (QPs), and each QP is connected to a PS. Worker 2 and Worker 3 also utilize three QPs, and each QP is connected to a PS.

[0003] In the reduce-scatter operator, the workers send partitions of data to the corresponding parameter servers. For example, partition a1 from Worker1, partition a2 from Worker2, and partition a3 from Worker3 are sent to PS1, while partition b1 from worker1, partition b2 from worker2, and partition b3 from worker3 are sent to PS2. A similar pattern applies to PS3. As a result, the data is spread across multiple parameter servers to exploit the parallel computation of the Graphics Processing Units (GPUs) located at the parameter servers. After receiving the data, the PS first performs aggregation on the data from the workers.

[0004] In the Allgather operator, data processed by the GPU is multicast to the workers. The parameter server sends the same copy of the data to the workers. In this process, the bandwidth from one PS is distributed to all the workers, and the network may be the bottleneck. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 shows an end-to-end solution for machine learning (ML) training using a parameter server (PS) architecture.

[0006] Figure 2 Depicted is an example topology.

[0007] Figure 3 An example system is depicted.

[0008] Figure 4 An example of buffer management is depicted.

[0009] Figure 5 An example process is depicted.

[0010] Figure 6 An example network interface device or packet processing device is depicted.

[0011] Figures 7A-7C An example switch is depicted.

[0012] Figure 8 An example system is depicted. DETAILED DESCRIPTION

[0013] In the collective operation of offloading the collective operation for the ML workload to the network interface device (such as a switch) using in-network aggregation (INA), the network interface device can perform the operation of the parameter server (PS) and be connected in a tree, grid or other arrangement. The endpoint switch can communicate with the worker node using a reliable transmission protocol, and the endpoint switch can maintain connection state information for packet communication to or from the worker node. The endpoint switch can aggregate data from the worker node by performing one or more of SUM, SUBTRACT, MIN, MAX, MULTIPLY, etc. The endpoint switch can perform floating point operations as part of the data processing. The endpoint switch can connect to at least one other switch using an unreliable protocol and may not store connection state information for packet communication to or from at least one other switch. The endpoint switch can communicate with at least one other switch using a best effort and may not retransmit packets that are not received by at least one other switch. In some examples, the endpoint switch can communicate with the worker node using a first protocol and communicate with at least one other switch using a second protocol, and the first protocol and the second protocol may be different. The at least one other switch can at least aggregate (such as one or more of SUM, SUBTRACT, MIN, MAX, MULTIPLY, etc.) data from the switch.

[0014] While waiting for the reduce sum response, the endpoint switch need not store connection status (e.g., confirmation of receipt of a packet) or utilize a timer to determine whether to retransmit data to at least one other switch. The at least one other switch can provide processed data from the root node switch to the endpoint switch for forwarding to at least one worker node. Confirmation or indication of receipt of a packet by at least one other switch to the endpoint switch can occur based on receipt of the processed data and an identifier of data sent by the endpoint switch that was used to generate the processed data. Thus, by storing the status of the connection to the worker node rather than the upstream switch, memory and processor usage in the endpoint switch can be reduced.

[0015] Reliable transport protocols may include connection-oriented protocols whereby a receiver acknowledges receipt of data to a data sender and, after a timeout interval, the sender attempts to retransmit undelivered data and / or the sender delays data transmission based on detected network congestion. Non-limiting examples of reliable transport protocols include Remote Direct Memory Access in Reliable Mode (RDMA Reliable Connections (RC)), InfiniBand, Transmission Control Protocol (TCP) (e.g., Internet Engineering Task Force (IETF) RFC 793 (1981), Quick UDP Internet Connections (QUIC), RoCEv2, Amazon's Scalable Reliable Datagram (SRD), Amazon AWS Elastic Fabric Adapter (EFA), Microsoft Azure Distributed Universal Access (DUA) and Lightweight Transport Layer (LTL), Google GCP Snap microkernel Pony Express, High Precision Congestion Control (HPCC) (e.g., Li et al., "HPCC: High Precision Congestion Control" SIGCOMM (2019)), Improved RoCENIC (IRN) (e.g., Mittal et al., "Revisiting network support for RDMA," SIGCOMM 2018), Homa (e.g., Montazeri et al., "Homa: A Receiver-Driven Low-Latency Transport Protocol Using Network Priorities," SIGCOMM 2018), NDP (e.g., Handley et al., “Re-architecting Datacenter Networks and Stacks for Low Latency and HighPerformance,” SIGCOMM 2017), EQDS (e.g., Olteanu et al., “An edge-queued datagramservice for all datacenter traffic,” USENIX 2022), or others.

[0016] An unreliable transport protocol may include a connectionless protocol, whereby the sender tries to send data, but the receiver does not acknowledge receipt of the data to the sender. Non-limiting examples of unreliable transport protocols include the User Datagram Protocol (UDP) (e.g., IETF RFC 768 (1980)), the Internet Protocol (IP), Remote Direct Memory Access in Unreliable Mode (RDMA Unreliable Connections (UC) and Unreliable Datagrams (UD)).

[0017] In some examples, a switch or network interface device may refer to one or more of the following: a network interface controller (NIC), a remote direct memory access (RDMA) enabled NIC, a SmartNIC, a router, a switch, a forwarding element, an infrastructure processing unit (IPU) with a programmable packet processing pipeline, a data processing unit (DPU), or an edge processing unit (EPU). The edge processing unit (EPU) may include a network interface device that utilizes a processor and an accelerator (e.g., a digital signal processor (DSP), a signal processor, or a wireless specific accelerator for a virtualized radio access network (vRAN), cryptographic operations, compression / decompression, etc.).

[0018] Figure 2 An example topology is depicted. System management 200 can determine devices associated with nodes of a logical tree and assign a logical root and an aggregation ID (A-id) to the logical tree. Determining devices associated with nodes of a logical tree can utilize a topology service that uses a Simple Network Management Protocol (SNMP) probe or a Link Layer Detection Protocol (LLDP) to probe devices. The primary value of the A-id can represent a logical root logical address (RA) to which workers send their data. The A-id can include edge nodes or workers marked with E and switches marked with T and S and the root R of the tree. This logical topology can be different from the physical topology because there can be other switches in the network that do not participate in the aggregation tree and forward traffic toward the next upstream node of the aggregation tree. In some examples, system management 200 can be executed on a server or a network interface device.

[0019] System management 200 can configure multicast routing and membership to worker nodes and switches along one or more paths of the multicast tree. The set of worker nodes to participate in collective operations can form a logical tree. System management 200 can determine the switches that form the logical tree and determine the root switch of the logical tree. Tree members can be initialized with rules of the aggregation tree (e.g., applicable protocols, packet forwarding rules, the number of packets aggregated together) and starting values ​​(e.g., synchronizing the initial starting sequence number of data sent by different workers, source Internet Protocol (IP) addresses, and destination IP addresses). In some examples, worker nodes E1 to E7 can be assigned the same source IP address. In some examples, the destination IP address may refer to an aggregation receiver (AR) described herein.

[0020] exist Figure 2 In the example of , worker nodes E1 to E3 can be communicatively coupled to the first hop switch T1. Worker node E4 can be communicatively coupled to the first hop switch T2. Worker nodes E5 to E6 can be communicatively coupled to the first hop switch T3. Worker node E7 can be communicatively coupled to the first hop switch S2. Figure 8 Various examples of circuitry and software in worker nodes are described.

[0021] The worker node can be initialized to connect to the logical root (R1) at the logical root address, but the endpoint switch (e.g., T1, T2, T3, or S2) can form a reliable transmission connection with the worker node. The endpoint switch can simulate the completion of the connection with R1 for the connection establishment handshake packets (e.g., SYN (synchronization) and ACK (acknowledgement) packets) before data transmission, because the established connection can be offloaded to the flow table in the endpoint switch. The worker nodes E1 to E7 can transmit packets with the final Internet Protocol (IP) root address of R1 to the endpoint switch T1, T2, T3, or S2 for forwarding to R1.

[0022] Switch T1 can perform operations on aggregated data from workers E1 to E3. Switch T2 can perform operations on aggregated data from worker E4. Switch T3 can perform operations on aggregated data from workers E5 and E6. Switches T1 and T2 can be communicatively coupled to switch S1. Switch T3 can be communicatively coupled to switch S2. Worker nodes can transmit metadata to the switch in a packet header, where the metadata identifies the data ID, the set of nodes participating in the job (e.g., group identifier 1 (E1-E7), group identifier 2 (E8-E10) (not shown), etc.).

[0023] Switch S1 can perform operations on aggregated data from switches T1 and T2. Switch S2 can perform operations on aggregated data from switch T3 and worker E7. Switches S1 and S2 can be communicatively coupled to root node switch R1. Root switch R1 (logical root) can perform operations on aggregated data from switches S1 and S2. Operations can include one or more of SUM, SUBTRACT, MIN, MAX, MULTIPLY, etc.

[0024] As described herein, switch T1 can utilize aggregate receiver (AR) A1 to execute a reliable transport protocol for communications to and from workers E1-E3. Switch T2 can utilize ARA2 to execute a reliable transport protocol for communications to and from worker E4. Switch T3 can utilize ARA3 to execute a reliable transport protocol for communications to and from workers E5 and E6. Switch S2 can utilize ARA4 to execute a reliable transport protocol for communications to and from worker E7. ARA1-A4 can be implemented as one or more of the following: a virtual machine (VM) or container executed by a host connected via a host interface, a VM or container executed by a processor of a corresponding switch, a circuit in a corresponding switch, or other. In some examples, communications from a network interface device of a worker to an endpoint switch can utilize a reliable transport protocol to provide reliable communication for data from the worker and results provided to the worker.

[0025] For example, one or more of A1-A4 can execute a TCP stack to maintain a TCP connection state for communicating with a worker node. The connection state can include one or more of the following: an initial sequence number, a number of bytes received, a byte sequence number received to keep received packets in order, an n-tuple (e.g., source address, destination address, IP protocol, transport layer source port and / or destination port), a connection flag, or other. The endpoint switch can send a data receipt confirmation (ACK) to the worker node to indicate the receipt of a packet transmitted by the worker node to the endpoint switch. In some examples, the endpoint switch can generate an ACK based on a stored template ACK packet and adjust the indication of the number of bytes received sent to the worker node in the ACK. Using template ACKs can reduce memory usage and potentially allow more connections for a certain amount of available memory.

[0026] In case of utilizing RoCE as a reliable transport protocol, the system management 200 can initialize the QP number and RoCE protocol parameters, and the worker node can respond to the ACK with an ACK flag.

[0027] Switch T1 can transmit aggregate data to switch S1 by using an unreliable transport protocol. Switch T1 can accumulate aggregate data for transmission in one or more packets and include packet header values ​​received from E1 to E3 according to a reliable transport connection, such as a sequence number, the number of bytes received, etc. Similarly, switch T2 can transmit aggregate data to switch S1 and packet header values ​​received from E4 according to a reliable transport connection, such as a sequence number, the number of bytes received, etc. by using an unreliable transport protocol. Switch T3 can transmit aggregate data to switch S2 by using an unreliable transport protocol. S1 and S2 switches can transmit aggregate data to switch R1 by using an unreliable transport protocol. In some cases, a reliable transport protocol can be used instead of an unreliable transport protocol.

[0028] Switches T1, T2, T3, and S2 can perform one or more operations for combining inputs and send the result to the next switch which combines the data again, and so on. Root R1 can perform a combination for obtaining a final result which is a combination of data from workers E1-E7. Thus, one or more of switches T1, T2, T3, or S2 can receive and process data transmitted by a worker node, process data having other worker data from the same group identifier (A-id), and transmit the resulting processed data to an upstream switch (e.g., S1 or S2). For example, T1 can sum data received from E1 to E3, where the data received from E1 to E3 is identified by the same group identifier. For example, T2 can process or forward data received from E4, where the data received from E4 is identified by a group identifier. For example, T3 can sum data received from E5 and E6, where the data received from E5 and E6 is identified by a group identifier. For example, S2 can sum the data received from E7 and T3, where the data received from E7 and T3 are identified by the group identifier. In some examples, the group identifier can include a packet sequence number or a range of sequence numbers.

[0029] Switch T1 can transmit one or more packets with synthesized processed data to the next switch (e.g., S1), and S1 can receive additional data from other members of A-id (e.g., data from T2). Switch T3 can transmit one or more packets with synthesized processed data to the next switch (e.g., S2), and S2 can receive additional data from other members of A-id (e.g., data from E7). Switches S1 and S2 can provide the resulting processed data to root node R1 for processing and aggregation. Node R1 can transmit the results of the aggregation (e.g., the results of the received cumulative sum) to edge workers E1-E7.

[0030] Based on multicast routing and membership, R1 can route packets to A-id member AR via associated switch T. AR can cause transmission of packets to worker endpoints by replicated unicast or by multicast (or broadcast) while utilizing unicast Internet Protocol (IP) addresses. For example, root R1 can perform aggregation and send results to worker nodes using multicast operations or directed per-hop multicast, whereby S1 multicasts or transmits results to T1 and T2. T1 can multicast or transmit results to E1-E3. T2 can transmit results to E4. T3 can multicast or transmit results to E5–E6. S2 can transmit results to E7. For multicast operations, switches can send replica packet payloads as well as headers indicating destination switches. Root switch R1 may include pipelines or circuits programmed to replicate packets with results and send replicated packets to AR, A1-A4 via corresponding switches T1, T2, T3, and S2. System management 200 can initialize the multicast tree when a job is set up. The switch can transmit packets with a per-job multicast address so that the receiver switch performs a multicast of the packets to the receivers. After the worker node receives the packet, the worker node can terminate the connection or initiate another data cycle. When using TCP, the AR can recalculate the protocol checksum before the packet is transmitted to the worker node.

[0031] The root switch can perform aggregation of data associated with a data identifier (ID) provided with the data from the endpoint switch (e.g., T1, T2, T3, or S2). If the endpoint switch does not receive aggregated data with the transmitted data identifier from R1 within a certain amount of time, the AR associated with the endpoint switch can generate an error or cancel the transaction and notify the system management 200 to take corrective action by restarting the transaction. Thus, the endpoint switch can use the data ID to track whether R1 received the packet.

[0032] The following describes example operations based on a TCP connection between an endpoint switch and a worker node. System management 200 can select one of the worker nodes as the primary worker (PW) for A-id. The AR associated with the endpoint switch can form a connection with the PW. The ingress port of the PW can receive a SYN packet to establish a connection. Other workers can receive a SYN with an initial send sequence number (ISS)+1 as the sequence number. The worker node can transmit an ACK, and the receiver switch (e.g., T1, T2, T3, or S2) can discard the ACK, except on the PW port.

[0033] Figure 3An example system is depicted. The system can be part of a distributed ML or deep neural network (DNN) system that is trained to perform inference consistent with SwitchML or utilizes one or more of: message passing interface (MPI), symmetric hierarchical memory access (SHMEM), unified parallel C (UPC), or other. In some examples, the network interface device 300 can perform operations of an endpoint switch coupled to one or more worker nodes, as described herein. In some examples, the network interface device 300 can perform operations of an aggregation receiver, as described herein. In some examples, the network interface device 300 can perform operations of a parameter server, as described herein.

[0034] The network interface device 300 can utilize the communication circuit 310 to communicate with other devices through a network or structure via one or more ports. The communication circuit 310 can be configured to use any one or more communication technologies (e.g., wired or wireless communication) and associated protocols (e.g., Ethernet, 4G LTE, 5G, etc.) to perform such communications. The communication circuit 310 can include one or more network hardware resources, such as ingress queues, egress queues, direct memory access (DMA) circuits, crossbar switches, shared memory switches, media access control (MAC), physical layer interfaces (PHY), Ethernet port logic, and other network hardware resources.

[0035] The network interface device 300 can utilize a packet processor 302 to process received packets and prepare packets for transmission. For example, the packet processor 302 can perform a calculation 304, which includes one or more of the following: summing, multiplying, dividing, minimizing, maximizing, or other data calculation operations related to Allreduce, ReduceScatter, or Allgather of the packet data with other packet data from other workers. The calculation 304 can utilize circuits to perform operations on floating point format data, as described herein. The packet header and / or the packet data can be stored in the memory 320.

[0036] Compute 304 can perform operations on data types and precisions used by ML applications. Compute 304 can perform operations on data of various operand precisions including single-precision floating point (e.g., 32-bit), half-precision floating point (e.g., 16-bit), custom floating point formats (e.g., BF16), integer words (16-bit), bytes (8-bit), and nibbles (4-bit). Depending on the specific implementation, matrix elements can be stored in different precisions, including 16-bit words, 8-bit bytes (e.g., INT8), and 4-bit nibbles (e.g., INT4). To support in-network aggregation of ML applications, compute 304 can perform calculations and arithmetic based on floating point (FP) values ​​and other data formats, as ML workloads can share floating point tensor updates. Compute 304 can utilize floating point calculator 340, which supports FP and integer arithmetic in the data path. Calculation 304 can be implemented as a network accessible FP calculator 340 including an application specific integrated circuit (ASIC) and / or a field programmable gate array (FPGA) that receives data and provides FP formatted data at an output port to the network interface device 300 for forwarding to a target egress port.

[0037] For example, the FP calculator 340 can perform vector floating point in parallel and provide FP data values ​​to the network interface device 300 in packets of a maximum size (MTU). The FP calculator 340 can store the value of the first received packet in a memory, or retrieve the value from the memory, add the value to the value in the packet (e.g., sum), and store the result back to the same memory location. For example, the FP calculator 340 can discard packets except for the last packet of a round of aggregation, and the FP calculator 340 can read the final result from the memory and write the final result to packets that can be sent back to the network interface device 300 to be forwarded to the next node in the aggregation tree. In some examples, the FP calculator 340 can be integrated into the network interface device 300 in the same system on chip (SoC) or die as the packet processor 302.

[0038] For example, the packet processor 302 can perform connection management 306 including managing connections with a worker node or a network interface device (such as a switch). For example, the connection management 306 can generate a reliable connection state 322 and use the reliable connection state 322 to manage communications with the worker node to indicate that the packet has been received. For example, the reliable connection state 322 may include the number of bytes received, the received packet sequence number that keeps the received packets in order, the A-id, or other metadata. For example, the connection management 306 can use the packet data identifier 324 to identify whether to retransmit data to the root switch or the intermediate switch. For example, the packet data identifier 324 can identify the data transmitted to the root switch and whether the calculation from the root switch is based on the transmitted data. Based on the calculation that the data identifier with the data transmitted to the root switch or the intermediate switch is not received from the root switch, the connection management 306 can cause the data associated with the data identifier to be retransmitted to the root switch. In some examples, the packet data identifier 324 can include a packet sequence number or a range of packet sequence numbers.

[0039] For example, the packet processor 302 can perform buffer management 308 to count the data stored in the data slots in the buffer 326 using a counter or register to determine when aggregation is complete and to determine when the calculated data can be sent to the next switch in the tree. The buffer management 308 can release data sets for transmission based on the receipt of packets to fill data gaps that may occur when packets are received out of order. To drain the list of completed slots, the buffer management 308 can cause the transmission of packets with data from the slots and generate (e.g., by packet duplication) a second packet that is recycled through the packet processor 302 until no full slots remain.

[0040] A stream can be a sequence of packets transmitted between two endpoints, generally representing a single session using a known protocol. Thus, a stream can be identified by a defined set of tuples, and for the purpose of routing, a stream is identified by two tuples (e.g., source address and destination address) identifying the endpoints. For content-based services (e.g., load balancers, firewalls, intrusion detection systems, etc.), streams can be distinguished with finer granularity by using N-tuples (e.g., source address, destination address, IP protocol, transport layer source port, and destination port). The packets in the expected stream have the same set of tuples in the packet header. The packet stream to be controlled can be identified by a combination of a tuple (e.g., Ethernet type field, source and / or destination IP address, source and / or destination user datagram protocol (UDP) port, source / destination TCP port, or any other header field) and a unique source and destination queue pair (QP) number or identifier. Packets can be used herein to refer to various formatted bit sets that can be sent across a network, such as Ethernet frames, IP packets, TCP segments, UDP datagrams, etc. Furthermore, as used in this document, references to L2, L3, L4, and L7 layers (Layer 2, Layer 3, Layer 4, and Layer 7) are references to the second data link layer, the third network layer, the fourth transport layer, and the seventh application layer, respectively, of the OSI (Open Systems Interconnection) layer model.

[0041] References to flows may alternatively or additionally refer to tunnels (e.g., Multiprotocol Label Switching (MPLS) Label Distribution Protocol (LDP), Segment Routing over IPv6 Data Plane (SRv6) source routing, VXLAN tunneling services, GENEVE tunneling services, virtual local area network (VLAN)-based network slicing, the techniques described in Mudigonda, Jayaram et al., “Spain: Cots data-center ethernet for multipathing over arbitrary topologies,” NSDI. Vol. 10, 2010 (hereinafter “SPAIN”), and the like.

[0042] Worker nodes can perform aggregate operations on data of the same size, and corresponding packets from different worker nodes (with payloads to be aggregated together) can have the same size. The data slots in the buffer 326 can store intermediate data resulting from payload aggregation, and can store metadata specifying the length of the data so that when the final result is retrieved, the network interface device 300 can determine the actual payload size to be generated.

[0043] For example, after calculation 304, the buffer slots can be filled with data from the worker nodes, and the data in the buffer slots can be summed. Calculation 304 can perform the summing according to the operation associated with the A-id operation (e.g., SUM, MIN, MAX, etc.) defined for the A-id (e.g., defined by system management 200). Each port can have multiple data slots for flow control. Calculation 304 can provide the sum into a packet and send the packet upstream to the upstream switch.

[0044] The operation of the packet processor 302 can be programmed using one or more of the following: configuration files, OneAPI, programming protocol-independent packet processors (P4), software for open networking in the cloud (SONiC), Network Programming Language (NPL), DOCA TM , Data Plane Development Kit (DPDK), OpenDataPlane (ODP), Infrastructure Programmer Development Kit (IPDK), eBPF, x86 compatible executable binary, or other executable binary.

[0045] The server 350 can be coupled to the network interface device 300 using a device interface or a network connection, examples of which are described herein. The server 350 can include a processor 352, a memory 360, and at least one of the embodiments described herein. Figure 8 In some examples, processor 352 can perform connection maintenance 354 to manage connections between workers (not shown). For example, connection maintenance 354 can confirm packet receipt to the sender and identify whether a packet sequence number is not received from the sender based on the connection context 362 stored in memory 360, and request packet retransmission.

[0046] Figure 4 An example of buffer management is depicted. Switch 400 can manage the use of buffer 402 as described herein. Buffer 402 can store headers and / or data from packets received from a worker node (e.g., Ex, where x is an integer), as well as aggregated data from an upstream switch (e.g., Sx, Rx). Connection management 404 can store the state of packets received from a connection with a worker node in connection state 410. Connection state 410 can include one or more of the following: number of bytes received, received packet sequence numbers to keep received packets in order, and the like. The number of slots in buffer 402 can be set for an A-id.

[0047] In some examples, worker nodes can be configured to send packets with the same starting sequence number and increase the sequence numbers for subsequent packets. For example, workers E1 to E3 can send packets with the same sequence number to different ports of T1 for data to be processed and aggregated by T1. However, based on receiving packets with the same sequence number from all E1 to E3, T1 can process and aggregate data from packets with the same sequence number from all E1 to E3, and transmit one or more packets with aggregated data to S1. Sequence numbers from different packets can be used to identify data to be aggregated. Similarly, based on receiving packets with the same sequence number from all E5 to E6, T3 can process and aggregate data from packets with the same sequence number from all E5 to E6, and transmit one or more packets with aggregated data to S2.

[0048] However, based on the non-receipt of packets with sequence numbers, an endpoint switch (e.g., T1, T2, T3, or S2) can reserve buffer entries for packets with sequence numbers that were not received and wait for retransmissions of packets with sequence numbers that were not received before sending data aggregations of packets with the same sequence numbers to the upstream switch. For example, if T1 receives packets with sequence numbers 1 to 4 from E1 to E3, and a packet with sequence number 5 is received from E1 and E2, but not from E3, T1 can reserve one or more buffers for the packet with sequence number 5 from E3. After the packet with sequence number 5 is retransmitted or received from E3 and such retransmitted packets are received at T1, T1 can process and aggregate data from one or more packets with sequence number 5 and transmit the aggregated data associated with sequence number 5 to S1.

[0049] If there is no space in buffer 402 for storing the packet content (e.g., header and / or payload) from the worker node, the received packet may be discarded and no ACK may be sent to the worker node. The worker node may resend the packet for which no ACK was received within the time limit. Connection management 404 may not send ACK to other worker nodes, or otherwise cause other worker nodes to suspend packet transmission until the discarded packet is re-received.

[0050] Connection management 404 can indicate receipt of a packet by transmitting an ACK to the sender worker node, which can trigger the sender worker node to send one or more additional packets. Switch 400 can send ACKs to worker nodes to indicate receipt of multiple packets, up to and including the packet for which receipt is acknowledged.

[0051] For example, the data received tracker 412 can indicate whether the data identifier (e.g., packet sequence number) transmitted to the upstream switch has been returned in the aggregated data from the upstream switch. In some examples, if the AR 420 does not receive a result with the transmitted data identifier from the logical root node within a preset time window, the connection management 404 can cause the packet with the data associated with the transmitted data identifier to be retransmitted. In some examples, if the AR 420 does not receive a result with the transmitted data identifier from the logical root node within a preset time window, the AR 420 can cancel the transaction and notify the worker node to abort the operation and close the connection. System management can initiate a new or another transaction from the worker node.

[0052] In some examples, AR 420 can be implemented as software or circuitry executed by a processor in switch 400, or as software or circuitry executed by a processor in a host connected to switch 400 via a host interface (e.g., Peripheral Component Interconnect Express (PCIe) or Compute Express Link (CXL)).

[0053] Figure 5 An example process is depicted. The process may be performed by a network interface device. At 502, the network interface device can form a communication connection with one or more worker nodes based on a reliable transport protocol. For example, the reliable transport protocol can track the receipt of packets from one or more worker nodes and indicate the receipt of packets to the one or more worker nodes. The one or more worker nodes can retransmit packets that are not indicated as received to the network interface device.

[0054] At 504, the network interface device can form a communication connection with the upstream switch based on the unreliable transport protocol. For example, the unreliable transport protocol can track the receipt of packets by the upstream switch by tracking the transmitted data identifier. Based on the non-receipt of the transmitted data identifier, packets associated with the data identifier that were not sent with the calculation based on the aggregated data to the network interface device can be resent, possibly with an error, or the transaction can be canceled, and system management can take corrective action.

[0055] At 506, the network interface device can process data received from one or more worker nodes. For example, the network interface device can perform an aggregation operation on the received data and forward the aggregated data to an upstream switch. At 508, the network interface device can receive the aggregated data from the root switch. The network interface device can forward the aggregated data to one or more worker nodes.

[0056] Figure 6An example network interface device or packet processing device is depicted. In some examples, the circuitry of the network interface device can be used to perform an aggregation receiver, connect to a worker node or switch, and / or communicate using a reliable or unreliable transport protocol, as described herein. In some examples, the packet processing device 600 can be implemented as a network interface controller, a network interface card, a host structure interface (HFI), or a host bus adapter (HBA), and such examples can be interchangeable. The packet processing device 600 can be coupled to one or more servers using a bus, PCIe, CXL, or double data rate (DDR). The packet processing device 600 can be embodied as part of a system on chip (SoC) including one or more processors, or included in a multi-chip package that also includes one or more processors.

[0057] Some examples of the packet processing device 600 are part of an infrastructure processing unit (IPU) or a data processing unit (DPU), or are utilized by an IPU or DPU. An xPU can refer to at least an IPU, a DPU, a GPU, a GPGPU, or other processing unit (e.g., an accelerator device). An IPU or DPU can include a network interface with one or more programmable or fixed-function processors to perform offloads of operations that might otherwise be performed by a CPU. An IPU or DPU can include one or more memory devices. In some examples, an IPU or DPU can perform virtual switch operations, manage storage transactions (e.g., compression, cryptography, virtualization), and manage operations performed on other IPUs, DPUs, servers, or devices.

[0058] The network interface 600 can include a transceiver 602, a processor 604, a transmit queue 606, a receive queue 608, a memory 610, and a bus interface 612 and a DMA engine 652. The transceiver 602 may be capable of receiving and transmitting packets conforming to an applicable protocol, such as Ethernet described in IEEE 802.3, but other protocols may also be used. The transceiver 602 can receive packets from a network via a network medium (not depicted) and transmit packets to the network. The transceiver 602 can include a PHY circuit 614 and a media access control (MAC) circuit 616. The PHY circuit 614 can include encoding and decoding circuits (not shown) to encode and decode data packets according to applicable physical layer specifications or standards. The MAC circuit 616 can be configured to group the data to be transmitted into packets, the packets including a destination address and a source address together with network control information and an error detection hash value.

[0059] Processor 604 may be any combination of processors, cores, graphics processing units (GPUs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or other programmable hardware devices that allow programming of network interface 600. For example, an "intelligent network interface" can use processor 604 to provide packet processing capabilities in a network interface.

[0060] In some embodiments, the processor 604 can include one or more packet processing pipelines that can be configured to perform matching actions on received packets to identify packet processing rules and next hops using information stored in a ternary content addressable memory (TCAM) table or an exact match table. For example, a matching action table or circuit can be used whereby a hash of a portion of a packet is used as an index to find an entry. The packet processing pipeline can perform one or more of the following: packet parsing (parser), exact match action (e.g., small exact match (SEM) engine or large exact match (LEM)), wildcard match action (WCM), longest prefix match block (LPM), hash block (e.g., receive side scaling (RSS)), packet modifier (modifier) ​​or traffic manager (e.g., transmission rate metering or shaping). For example, the packet processing pipeline can implement access control lists (ACLs) or packet discards due to queue overflow.

[0061] The operational configuration of the processor 604, including its data plane, can be programmed based on one or more of: a protocol independent packet processor (P4), software for open networking in the cloud (SONiC), Network Programming Language (NPL), DOCA TM , Infrastructure Programmer Development Kit (IPDK), etc.

[0062] The packet distributor 624 can use the time slot allocation or RSS described herein to provide distribution of received packets for processing by multiple CPUs or cores. When the packet distributor 624 uses RSS, the packet distributor 624 can calculate a hash or make another determination based on the content of the received packet to determine which CPU or core is to process the packet.

[0063] Interrupt coalesce 622 can perform interrupt moderation, whereby the network interface interrupt coalesce 622 waits for multiple packets to arrive, or waits for a timeout to expire, before generating an interrupt to the host system to process (one or more) received packets. Receive segment coalescing (RSC) can be performed by the network interface 600, whereby portions of incoming packets are combined into segments of a packet. The network interface 600 provides the coalesced packet to the application.

[0064] The direct memory access (DMA) engine 652 can copy packet headers, packet payloads and / or descriptors directly from host memory to a network interface, or vice versa, rather than copying the packets to an intermediate buffer at the host and then using another copy operation from the intermediate buffer to a destination buffer.

[0065] Memory 610 can be any type of volatile or non-volatile memory device and can store any queue or instructions for programming network interface 600. Transmit queue 606 can include data or references to data for transmission by the network interface. Receive queue 608 can include data or references to data received from the network by the network interface. Descriptor queue 620 can include descriptors that reference data or packets in transmit queue 606 or receive queue 608. Bus interface 612 can provide an interface with a host device (not depicted). For example, bus interface 612 can be compatible with PCI, PCI Express, PCI-x, Serial ATA, and / or USB compatible interfaces (although other interconnect standards can be used).

[0066] Fig. 7A An example switch is depicted. Various examples can be used in or with a switch to perform operations of an aggregation receiver, connect to a worker node or switch, and / or communicate using a reliable or unreliable transport protocol, as described herein. Switch 704 can route packets or frames of any format or according to any specification from any port 702-0 to 702-X to any port in ports 706-0 to 706-Y (or vice versa). Any port in ports 702-0 to 702-X can be connected to a network of one or more interconnected devices. Similarly, any port in ports 706-0 to 706-Y can be connected to a network of one or more interconnected devices.

[0067] In some examples, the switch fabric 710 can provide routing of packets from one or more ingress ports for processing before outflow from the switch 704. The switch fabric 710 can be implemented as one or more multi-hop topologies, where example topologies include ring, butterfly, buffered multi-stage, etc., or a shared memory switch fabric (SMSF) and other implementations. An SMSF can be any switch fabric connected to an ingress port and an egress port in a switch, where an ingress subsystem writes (stores) packet segments into the fabric's memory, and an egress subsystem reads (extracts) packet segments from the fabric's memory.

[0068] The memory 708 can be configured to store packets received at the port before flowing out from one or more ports. The packet processing pipeline 712 can include inlet and outlet packet processing circuits to process incoming packets and outgoing packets respectively. The packet processing pipeline 712 can use a table that maps packet characteristics to associated output ports to determine which port to transfer a packet or frame to. In some examples, the packet processing pipeline 712 can be configured to use information stored in a ternary content addressable memory (TCAM) table or an exact match table to perform a matching action on the received packet to identify the packet processing rule and the next jump. For example, a matching action table or circuit can be used, whereby a hash of a portion of the packet is used as an index to find an entry (e.g., forwarding decision based on packet header content). The packet processing pipeline 712 can implement access control lists (ACLs) or packet discards caused by queue overflow. The packet processing pipeline 712 can be configured to perform an aggregation receiver, connect to a worker node or switch, and / or communicate using a reliable or unreliable transport protocol, as described herein. The operational configuration of the packet processing pipeline 712, including its data plane, can be programmed using P4, C, Python, Broadcom Network Programming Language (NPL), or an x86 compatible executable binary or other executable binary. Packet processing or modification can be performed using the processor 716 and FPGA 718.

[0069] Traffic manager 713 can perform hierarchical scheduling and transmission rate shaping and metering of packet transmissions from one or more packet queues. Traffic manager 713 can perform congestion management such as flow control, congestion notification message (CNM) generation and reception, priority flow control (PFC), etc.

[0070] Figure 7B An example network forwarding system that can be used as a network interface device or router is depicted. The forwarding system can perform the operations of an aggregation receiver, connect with a worker node or switch, and / or communicate using a reliable or unreliable transport protocol, as described herein. For example, Figure 7B Illustrated are several ingress pipelines 720, a traffic management unit (referred to as a traffic manager) 750, and several egress pipelines 730. Although shown as separate structures, in some examples, the ingress pipelines 720 and the egress pipelines 730 can use the same circuit resources.

[0071] The pipeline operation can be programmed using a protocol-independent packet processor (P4), C, Python, Broadcom NPL or x86 compatible executable binary or other executable binary. In some examples, the pipeline circuit is configured to synchronously process inlet and / or outlet pipeline packets and non-packet data. In other words, the specific stage of the pipeline can process any combination of inlet packets, outlet packets and non-packet data in the same clock cycle. However, in other examples, the inlet and outlet pipelines are separate circuits. In some examples of these other examples, the inlet pipeline also processes non-packet data.

[0072] In some examples, in response to receiving a packet, the packet is directed to one of the inlet pipelines 720, where the inlet pipelines may correspond to one or more ports of a hardware forwarding element. After passing through the selected inlet pipeline 720, the packet is sent to the service manager 750, where the packet is queued and placed in the output buffer 754. In some examples, the inlet pipeline 720 that processes the packet specifies which queue the packet is to be placed in by the service manager 750 (e.g., based on the destination of the packet or the flow identifier of the packet). The service manager 750 then dispatches the packet to the appropriate egress pipeline 730, where the egress pipeline may correspond to one or more ports of a forwarding element. In some examples, there is no necessary correlation between which inlet pipeline in the inlet pipelines 720 processes the packet and which egress pipeline in the egress pipelines 730 the service manager 750 dispatches the packet to. That is, after a packet is received through a first port, it may initially be processed by the inlet pipeline 720b, and then subsequently sent out from the second port by the egress pipeline 730a, and so on.

[0073] At least one ingress pipeline 720 includes a parser 722, a series of multiple match action units or circuits (MAUs) 724, and a deparser 726. Similarly, the egress pipeline 730 can include a parser 732, a series of MAUs 734, and a deparser 736. In some examples, the parser 722 or 732 receives a packet as a set of bits formatted in a specific order and parses the packet into its constituent header fields. In some examples, the parser starts at the beginning of the packet and assigns the header fields to fields (e.g., data containers) for processing. In some examples, the parser 722 or 732 separates the packet header from the payload of the packet (until a specified point) and sends the payload (or the entire packet, including the header and payload) directly to the deparser without being processed by the MAU. The egress parser 732 can use additional metadata provided by the ingress pipeline to simplify its processing.

[0074] MAU 724 or 734 can perform processing on packet data. In some examples, the MAU includes a sequence of stages, each of which includes one or more matching tables and an action engine. The matching table can include a set of matching entries to which packet header fields are matched (e.g., using a hash table), wherein the matching entries reference action entries. When a packet matches a specific matching entry, the specific matching entry references a specific action entry, which specifies a set of actions to be performed on the packet (e.g., sending the packet to a specific port, modifying one or more packet header field values, discarding the packet, mirroring the packet to a mirror buffer, etc.). The action engine of this stage can perform actions on the packet, which is then sent to the next stage of the MAU. For example, using (one or more) MAUs, packet processing, receiving worker data, forwarding packet headers from a worker to a server, or inserting calculated result data into a packet to be sent to a worker, as described herein.

[0075] The inverse parser 726 or 736 can reconstruct the packet using the PHV modified by the MAU 724 or 734 and the payload received directly from the parser 722 or 732. The inverse parser can construct a packet that can be sent out over a physical network or to a traffic manager 750. In some examples, the inverse parser can construct this packet based on data received with the PHV that specifies the protocol to be included in the packet header, and its own stored list of data container locations for the header fields for each possible protocol.

[0076] Traffic manager (TM) 750 can include a packet duplicator 752 and an output buffer 754. In some examples, traffic manager 750 may include other components, such as a feedback generator for sending signals about output port failures, a series of queues and schedulers for these queues, a queue status analysis component, and additional components. The packet duplicator 752 of some examples performs duplication of broadcast / multicast packets, generating multiple packets to be added to the output buffer (e.g., to be distributed to different egress pipelines).

[0077] In some examples, the output buffer 754 can be part of the queuing and buffering system of the service manager. The service manager 750 can provide a shared buffer that accommodates any queuing delays in the egress pipeline. In some examples, the shared output buffer 754 can store packet data, and for each egress pipeline 730, a reference (e.g., a pointer) to the packet data is kept in a different queue. The egress pipeline can request their corresponding data from the common data buffer using a queuing strategy configurable by the control plane. When the packet data reference reaches the head of its queue and is scheduled for dequeueing, the corresponding packet data can be read from the output buffer 754 and enter the corresponding egress pipeline 730.

[0078] Figure 7C An example switch is depicted. Various examples can be used in or with a switch to perform operations of an aggregation receiver, connect to a worker node or switch, and / or communicate using a reliable or unreliable transport protocol, as described herein. Switch 780 may include a network interface 782 that can provide an Ethernet consistent interface. Network interface 782 can support 25GbE, 50GbE, 100GbE, 200GbE, 400GbE Ethernet port interfaces. Cryptographic circuitry 784 can at least perform media access control security (MACsec) or Internet Protocol security (IPSec) decryption on received packets, or perform encryption on packets to be transmitted.

[0079] The various circuits can perform one or more of: service metering, packet counting, operations, operations and management (OAM), protection engine, instrumentation and telemetry, and clock synchronization (eg, based on IEEE 1588).

[0080] The database 786 can store device profiles to configure the operation of the switch 780. The memory 788 can include a high bandwidth memory (HBM) for packet buffering. The packet processor 790 can perform one or more of the following: determination of the next hop in conjunction with packet forwarding, packet counting, access list operations, bridging, routing, multi-protocol label switching (MPLS), virtual private LAN service (VPLS), L2VPNs, L3VPN, OAM, data center tunneling encapsulation (e.g., VXLAN and NV-GRE), or other. The packet processor 790 can include one or more FPGAs. The buffer 794 can store one or more packets. The traffic manager (TM) 792 can provide bandwidth guarantees per subscriber based on a service level agreement (SLA) and perform layer-level quality of service (QoS). The fabric interface 796 can include a serializer / deserializer (SerDes) and provide an interface to the switch fabric.

[0081] Fig. 7A , 7B and / or the operations of the switch components of the switch example of 7C can be combined, and Fig. 7A , 7B and / or 7C example switch components can be included in Fig. 7A , 7B and / or other examples of switches of the example of 7C. For example, Fig. 7A , 7B The components of the switch examples of and / or 7C can be implemented in a switch system on chip (SoC) that includes at least one interface to other circuits in the switch system. The switch SoC can be coupled to other devices in the switch system, such as ingress or egress ports, memory devices, or host interface circuits.

[0082] Figure 8 A system is depicted. In some examples, the circuit can execute an aggregation receiver, connect to a worker node or switch, and / or communicate using a reliable or unreliable transport protocol, as described herein. System 800 includes a processor 810 that provides processing, operation management, and instruction execution for system 800. Processor 810 can include any type of microprocessor, central processing unit (CPU), graphics processing unit (GPU), XPU, processing core, or other processing hardware for providing processing for system 800, or a combination of processors. XPU can include one or more of the following: CPU, graphics processing unit (GPU), general purpose GPU (GPGPU) and / or other processing units (e.g., accelerator or programmable or fixed function FPGA). Processor 810 controls the overall operation of system 800 and can be or include one or more programmable general or special purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), etc. or a combination of such devices.

[0083] In one example, the system 800 includes an interface 812 coupled to the processor 810, which can represent a higher speed interface or a high throughput interface for system components (such as a memory subsystem 820 or a graphics interface component 840 or an accelerator 842) that require a higher bandwidth connection. The interface 812 represents an interface circuit, which can be an independent component or integrated into the processor die. In the case of existence, the graphics interface 840 is connected to the graphics component through the interface for providing a visual display to a user of the system 800. In one example, the graphics interface 840 can drive a display that provides an output to the user. In one example, the display can include a touch screen display. In one example, the graphics interface 840 generates a display based on data stored in the memory 830 or based on operations performed by the processor 810 or based on both. In one example, the graphics interface 840 generates a display based on data stored in the memory 830 or based on operations performed by the processor 810 or based on both.

[0084] The accelerator 842 may be a programmable or fixed function offload engine that can be accessed or used by the processor 810. For example, an accelerator among the accelerator 842 can provide data compression (DC) capabilities, cryptographic services (such as public key encryption (PKE)), passwords (ciphers), hashing / authentication capabilities, decryption or other capabilities or services. In some cases, the accelerator 842 can be integrated into a CPU socket (e.g., a connector to a mainboard or circuit board that includes a CPU and provides an electrical interface to the CPU). For example, the accelerator 842 can include a single-core or multi-core processor, a graphics processing unit, a logic execution unit, a single-level or multi-level cache, a functional unit that can be used to independently execute a program or thread, an application-specific integrated circuit (ASIC), a neural network processor (NNP), a programmable control logic, and a programmable processing element (such as a field programmable gate array (FPGA)). The accelerator 842 can provide multiple neural networks, CPUs, processor cores, general-purpose graphics processing units, or enable a graphics processing unit to be used for artificial intelligence (AI) or machine learning (ML) models. For example, the AI ​​model can use or include any of the following or a combination thereof: a reinforcement learning scheme, a Q-learning scheme, a deep-Q learning or an asynchronous advantage Actor-Critic (A3C), a combination neural network, a recursive combination neural network, or other AI or ML models. Multiple neural networks, processor cores, or graphics processing units can be made available for use by the AI ​​or ML model to perform learning and / or inference operations.

[0085] The memory subsystem 820 represents the main memory of the system 800 and provides storage for code to be executed by the processor 810 or data values ​​to be used in executing routines. The memory subsystem 820 can include one or more memory devices 830, such as read-only memory (ROM), flash memory, one or more variants of random access memory (RAM) (such as DRAM), or other memory devices or combinations of such devices. The memory 830 stores and hosts an operating system (OS) 832, among other things, to provide a software platform for executing instructions in the system 800. Additionally, applications 834 can be executed on the software platform of the OS 832 from the memory 830. Applications 834 represent programs that have their own operating logic to perform one or more functions. Processes 836 represent agents or routines that provide auxiliary functions to the OS 832 or one or more applications 834 or a combination. The OS 832, applications 834, and processes 836 provide software logic that provides functions for the system 800. In one example, the memory subsystem 820 includes a memory controller 822, which is a memory controller for generating commands and issuing commands to the memory 830. It will be understood that the memory controller 822 can be a physical part of the processor 810 or a physical part of the interface 812. For example, the memory controller 822 can be an integrated memory controller integrated onto a circuit with the processor 810.

[0086] Application 834 and / or process 836 can alternatively or additionally refer to a virtual machine (VM), a container, a microservice, a processor, or other software. The various examples described herein can execute an application composed of microservices, wherein the microservice runs in its own process and communicates using a protocol (e.g., an application program interface (API), a hypertext transfer protocol (HTTP) resource API, a message service, a remote procedure call (RPC), or a Google RPC (gRPC)). Microservices can communicate with each other using a service grid, and can be executed in one or more data centers or edge networks. Microservices can be deployed independently using centralized management of these services. The management system can be written in different programming languages ​​and use different data storage technologies. Microservices can be characterized by one or more of the following: polyglot programming (e.g., code written in multiple languages ​​to capture additional functionality and efficiency that cannot be obtained with a single language), or lightweight container or virtual machine deployment, and decentralized continuous microservice delivery.

[0087] In some examples, OS 832 may be Server or personal computer, VMware vSphere, openSUSE, RHEL, CentOS, Debian, Ubuntu or any other operating system. The OS and drivers can be Texas etc.

[0088] Although not specifically shown, it will be understood that system 800 can include one or more buses or bus systems between devices, such as a memory bus, a graphics bus, an interface bus or other. A bus or other signal line can couple components together communicatively or electrically, or communicatively and electrically. A bus can include a physical communication line, a point-to-point connection, a bridge, an adapter, a controller or other circuit or combination. For example, a bus can include one or more of the following: a system bus, a peripheral component interconnect (PCI) bus, a super transport or an industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB) or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (Firewire).

[0089] In one example, system 800 includes interface 814, which can be coupled to interface 812. In one example, interface 814 represents an interface circuit, which can include independent components and integrated circuits. In one example, multiple user interface components or peripheral components or both are coupled to interface 814. Network interface 850 provides the ability to communicate with remote devices (e.g., servers or other computing devices) through one or more networks to system 800. Network interface 850 can include an Ethernet adapter, a wireless interconnection component, a cellular network interconnection component, a USB (universal serial bus) or other interfaces or proprietary interfaces based on wired or wireless standards. Network interface 850 can transmit data to a device or remote device in the same data center or rack, which can include sending data stored in a memory. Network interface 850 can receive data from a remote device, which can include storing received data in a memory. In some examples, the packet processing device or network interface device 850 can refer to one or more of the following: a network interface controller (NIC), a remote direct memory access (RDMA) enabled NIC, a SmartNIC, a router, a switch, a forwarding element, an infrastructure processing unit (IPU), or a data processing unit (DPU). An example IPU or DPU is described with respect to FIG.

[0090] In one example, the system 800 includes one or more input / output (I / O) interfaces 860. The I / O interfaces 860 can include one or more interface components through which a user interacts with the system 800. The peripheral interfaces 870 can include any hardware interface not specifically mentioned above. A peripheral device generally refers to a device that is connected to the system 800 in a dependent manner.

[0091] In one example, the system 800 includes a storage subsystem 880 that stores data in a non-volatile manner. In one example, in certain system implementations, at least some components of the storage subsystem 880 can overlap with components of the memory subsystem 820. The storage subsystem 880 includes (one or more) storage devices 884, which can be or include any conventional media for storing large amounts of data in a non-volatile manner, such as one or more magnetic disks, solid-state disks, or optical-based disks, or a combination. The storage device 884 saves code or instructions and data 886 in a persistent state (e.g., the value is retained despite the interruption of power to the system 800). The storage device 884 can generally be considered as "memory", although the memory 830 is generally an execution or operating memory that provides instructions to the processor 810. Although the storage device 884 is non-volatile, the memory 830 can include volatile memory (e.g., if the power to the system 800 is interrupted, the value or state of the data is uncertain). In one example, the storage subsystem 880 includes a controller 882 for connecting to the storage device 884 through an interface. In one example, controller 882 is a physical part of interface 814 or processor 810 , or may include circuitry or logic in both processor 810 and interface 814 .

[0092] Volatile memory is memory whose state (and therefore the data stored therein) is indeterminate if power to the device is interrupted. Non-volatile memory (NVM) devices are memory whose state is deterministic even if power to the device is interrupted.

[0093] In an example, the system 800 can be implemented using an interconnected computing sled of processors, memory, storage devices, network interfaces, and other components. The high-speed interconnect can be based on: Ethernet (IEEE 802.3), Remote Direct Memory Access (RDMA), InfiniBand, Internet Wide Area RDMA Protocol (iWARP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Quick UDP Internet Connection (QUIC), RDMA over Converged Ethernet (RoCE), Peripheral Component Interconnect Express (PCIe), Intel Quick Path Interconnect (QPI), Intel Ultra Path Interconnect (UPI), Intel System-on-Chip Fabric (IOSF), Omni-path, Compute Express Link (CXL), HyperTransport, High-Speed ​​Fabric, NVLink, Advanced Microcontroller Bus Architecture (AMBA) Interconnect, OpenCAPI, Gen-Z, Infinity Fabric (IF), Cache Coherent Interconnect for Accelerators (CCIX), 3GPP Long Term Evolution (LTE) (4G), 3GPP 5G, and variants thereof. The data can be copied or stored to a virtualized storage node, or the data can be accessed using a protocol such as NVMe over Fabric (NVMe-oF) or NVMe (e.g., a Non-Volatile Memory Express (NVMe) device can operate in a manner consistent with the Non-Volatile Memory Express (NVMe) Specification Revision 1.3c, dated May 24, 2018 (the "NVMe Specification"), or a derivative or variant thereof).

[0094] Communications between devices can occur using a network that provides: die-to-die communications; chip-to-chip communications; board-to-board communications; and / or package-to-package communications.

[0095] In an example, the system 800 can be implemented using an interconnected computing sled of processors, memory, storage devices, network interfaces, and other components. A high-speed interconnect such as PCIe, Ethernet, or optical interconnect (or a combination thereof) can be used.

[0096] Examples herein may be implemented in various types of computing and networking equipment, such as switches, routers, racks, and blade servers, such as those employed in data center and / or server farm environments. Servers used in data centers and server farms include array server configurations, such as rack-based servers or blade servers. These servers are interconnected in communication via various network supplies, such as partitioning a server set into a local area network (LAN), with appropriate switching and routing facilities between the LANs to form a private intranet. For example, cloud hosting facilities may typically employ large data centers with a large number of servers. A blade includes a separate computing platform configured to perform server-type functions, i.e., a "card server". Thus, a blade includes components common to conventional servers, including a main printed circuit board (motherboard) that provides internal wiring (e.g., a bus) for coupling appropriate integrated circuits (ICs) and other components mounted to the board.

[0097] Various examples can be implemented using hardware elements, software elements or a combination of the two. In some examples, hardware elements may include devices, components, processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, ASICs, PLDs, DSPs, FPGAs, memory cells, logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc. In some examples, software elements may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, processes, software interfaces, APIs, instruction sets, computing codes, computer codes, code segments, computer code segments, words, values, symbols or any combination thereof. Determining whether to use hardware elements and / or software elements to implement examples may vary according to any number of factors, such as desired computing rates, power levels, heat resistance, processing cycle budgets, input data rates, output data rates, memory resources, data bus speeds, and other design or performance constraints, as desired for a given implementation. A processor may be a hardware state machine, digital control logic, a central processing unit, or one or more combinations of any hardware, firmware, and / or software elements.

[0098] Some examples may be implemented using or as an article of manufacture or at least one computer-readable medium. A computer-readable medium may include a non-transitory storage medium for storing logic. In some examples, a non-transitory storage medium may include one or more types of computer-readable storage media capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. In some examples, logic may include various software elements, such as software components, programs, applications, computer programs, applications, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, processes, software interfaces, APIs, instruction sets, computing codes, computer codes, code segments, computer code segments, words, values, symbols, or any combination thereof.

[0099] According to some examples, a computer-readable medium may include a non-transitory storage medium for storing or maintaining instructions, which, when executed by a machine, a computing device, or a system, causes the machine, the computing device, or the system to perform methods and / or operations according to the described examples. The instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, etc. The instructions may be implemented according to a predefined computer language, manner, or syntax for commanding a machine, a computing device, or a system to perform a certain function. The instructions may be implemented using any suitable high-level, low-level, object-oriented, visual, compiled, and / or interpreted programming language.

[0100] One or more aspects of at least one example may be implemented by representative instructions stored on at least one machine-readable medium representing various logic within the processor, which when read by a machine, computing device or system causes the machine, computing device or system to fabricate logic for performing the techniques described herein. Such representations, known as "IP cores," may be stored on a tangible, machine-readable medium and supplied to various customers or manufacturing facilities to load into the fabrication machines that actually fabricate the logic or processor.

[0101] The appearance of the phrase "an example" or "example" does not necessarily refer to the same example or embodiment. Any aspect described herein can be combined with any other aspect or similar aspect described herein, regardless of whether these aspects are described with respect to the same figure or element. The division, omission or inclusion of block functions depicted in the accompanying drawings does not imply that the hardware components, circuits, software and / or elements used to implement these functions will necessarily be divided, omitted or included in the embodiments.

[0102] The expressions "coupled" and "connected" and their derivatives may be used to describe some examples. These terms are not necessarily intended to be synonymous with each other. For example, descriptions using the terms "connected" and / or "coupled" may indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0103] The terms "first", "second", etc. do not refer to any order, quantity or importance herein, but are used to distinguish one element from another element. The terms "a and an" do not refer to the limitation of quantity herein, but refer to the existence of at least one of the referenced items. The term "asserted" used in this article with reference to a signal refers to the state of a signal, wherein the signal is active, and it can be realized by applying any logic level of logic 0 or logic 1 to the signal. The term "following" or "after..." may refer to immediately following or following a certain or certain other events. Other operation sequences may also be performed according to alternative embodiments. In addition, depending on the specific application, additional operations may be added or removed. Any combination of variations may be used, and those of ordinary skill in the art who benefit from the present disclosure will understand its many variations, modifications and alternative embodiments.

[0104] Unless specifically stated otherwise, disjunctive language such as the phrase "at least one of X, Y, or Z" is otherwise understood within the context in which it is generally used to present items, terms, etc., otherwise disjunctive language such as the phrase "at least one of X, Y, or Z" can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require that at least one of X, at least one of Y, or at least one of Z each be present. Additionally, conjunctive language such as the phrase "at least one of X, Y, and Z" should also be understood to mean X, Y, Z, or any combination thereof, including "X, Y, and / or Z," unless specifically stated otherwise.

[0105] Illustrative examples of the apparatus, systems, and methods disclosed herein are provided below. Embodiments of the apparatus, systems, and methods may include any one or more and any combination of the examples described below.

[0106] Example 1 includes one or more examples, and includes a device comprising: an interface and a circuit coupled to the interface, the circuit being configured to: for packet communications associated with a collective operation for training a machine learning (ML) model: utilizing a reliable transmission protocol for communications from at least one worker node of the collective operation to a switch, wherein the utilizing of the reliable transmission protocol for communications from at least one worker node of the collective operation to the switch includes storing a packet receipt status for each packet communication from the at least one worker node of the collective operation to the switch; and utilizing an unreliable transmission protocol from the switch to a device to perform result aggregation, wherein the reliable transmission protocol includes a protocol different from a protocol of the unreliable transmission protocol.

[0107] Example 2 includes one or more examples wherein the reliable transport protocol includes one or more of: remote direct memory access (RDMA), InfiniBand, Transmission Control Protocol (TCP), Quick UDP Internet Connections (QUIC), or RDMA over Converged Ethernet (RoCE).

[0108] Example 3 includes one or more examples, wherein the communication from at least one worker node of the collective operation to the switch utilizing the reliable transport protocol includes: causing transmission of a packet receipt confirmation (ACK) message to the at least one worker node based on receipt of a packet.

[0109] Example 4 includes one or more examples wherein the unreliable transport protocol includes a User Datagram Protocol (UDP).

[0110] Example 5 includes one or more examples wherein utilizing an unreliable transport protocol includes tracking receipt of a packet transmission by the switch to a device to perform result aggregation based on receipt of a result aggregation from a root switch including a content identifier of the packet transmission by the switch to the device.

[0111] Example 6 includes one or more examples wherein the circuit is to cause a packet associated with an identifier not received from the device to be retransmitted by the switch to the device.

[0112] Example 7 includes one or more examples and includes a switch system on chip (SoC), wherein the switch SoC includes the interface and the circuit.

[0113] Example 8 includes one or more examples and includes at least one non-transitory computer-readable medium, including instructions stored thereon, which, if executed by one or more processors, cause the one or more processors to: configure a first network interface device to utilize a reliable transmission protocol for communications from at least one worker node of a collective operation to the first network interface device, wherein the utilization of a reliable transmission protocol for communications from at least one worker node of the collective operation to the first network interface device includes storing a packet receipt status for each packet communication from at least one worker node of the collective operation to the first network interface device; and configure the first network interface device to utilize an unreliable transmission protocol for communications to a second network interface device to perform result aggregation, wherein the reliable transmission protocol includes a protocol different from a protocol of the unreliable transmission protocol.

[0114] Example 9 includes one or more examples wherein the reliable transport protocol includes one or more of: remote direct memory access (RDMA), InfiniBand, Transmission Control Protocol (TCP), Quick UDP Internet Connections (QUIC), or RDMA over Converged Ethernet (RoCE).

[0115] Example 10 includes one or more examples, wherein the communication from at least one worker node of the collective operation to the first network interface device utilizing a reliable transmission protocol includes: causing transmission of a packet receipt confirmation (ACK) message to the at least one worker node based on receipt of a packet.

[0116] Example 11 includes one or more examples wherein the unreliable transport protocol includes a User Datagram Protocol (UDP).

[0117] Example 12 includes one or more examples wherein utilizing an unreliable transport protocol includes tracking receipt of a packet transmission by the first network interface device to the second network interface device based on receipt of a result aggregation from a root switch that includes an identifier of content of the packet transmission by the first network interface device to the second network interface device.

[0118] Example 13 includes one or more examples wherein the first network interface device is to retransmit, to the second network interface device, a packet associated with an identifier not received from the second network interface device.

[0119] Example 14 includes one or more examples and includes instructions stored thereon, which, if executed by one or more processors, cause the one or more processors to: configure a plurality of network interface devices to perform a collective operation for training a machine learning (ML) model by associating a logical node with the plurality of network interface devices.

[0120] Example 15 includes one or more examples, and includes a method comprising: a first network interface device utilizing a reliable transmission protocol for communication from at least one worker node of a collective operation to the first network interface device, wherein the utilizing of the reliable transmission protocol for communication from at least one worker node of the collective operation to the first network interface device includes storing a packet receipt status for each packet communication from the at least one worker node of the collective operation to the first network interface device; and the first network interface device utilizing an unreliable transmission protocol for communication from the first network interface device to a second network interface device to perform result aggregation, wherein the reliable transmission protocol includes a protocol different from a protocol of the unreliable transmission protocol.

[0121] Example 16 includes one or more examples wherein the reliable transport protocol includes one or more of: remote direct memory access (RDMA), InfiniBand, Transmission Control Protocol (TCP), Quick UDP Internet Connections (QUIC), or RDMA over Converged Ethernet (RoCE).

[0122] Example 17 includes one or more examples, wherein the use of a reliable transport protocol for communication from at least one worker node of the collective operation to the first network interface device includes: transmitting a packet receipt confirmation (ACK) message to the at least one worker node based on receipt of the packet.

[0123] Example 18 includes one or more examples wherein the unreliable transport protocol includes a User Datagram Protocol (UDP).

[0124] Example 19 includes one or more examples wherein utilizing an unreliable transport protocol includes tracking receipt of a packet transmission by the first network interface device to the second network interface device based on receipt of a result aggregation from a root switch that includes an identifier of content of the packet transmission by the first network interface device to the second network interface device.

[0125] Example 20 includes one or more examples and includes the first network interface device causing a retransmission of a packet associated with an identifier not received from the second network interface device to the second network interface device.

Claims

1. A device comprising: interface; as well as A circuit coupled to the interface, the circuit being configured to: For packet communications associated with collective operations used to train machine learning (ML) models: Utilizing a reliable transport protocol for communications from at least one worker node of the collective operation to the switch, wherein the utilizing a reliable transport protocol for communications from at least one worker node of the collective operation to the switch comprises: storing a packet receipt status for each packet communication from the at least one worker node of the collective operation to the switch; and An unreliable transmission protocol is utilized by the switch to a device that is to perform result aggregation, wherein the reliable transmission protocol includes a protocol different from a protocol of the unreliable transmission protocol.

2. The device according to claim 1, wherein: The reliable transport protocol includes one or more of: Remote Direct Memory Access (RDMA), InfiniBand, Transmission Control Protocol (TCP), Quick UDP Internet Connections (QUIC), or RDMA over Converged Ethernet (RoCE).

3. The device of claim 1, wherein: The utilizing of the reliable transport protocol for communications from at least one worker node of the collective operation to the switch includes causing transmission of a packet receipt acknowledgement (ACK) message to the at least one worker node based on receipt of a packet.

4. The device of claim 1, wherein: The unreliable transport protocol includes a User Datagram Protocol (UDP).

5. The device of claim 1, wherein: The circuit is configured to cause a packet associated with an identifier not received from the device to be retransmitted by the switch to the device.

6. The device of claim 1, comprising a switch system on chip (SoC), wherein: The switch SoC includes the interface and the circuit.

7. The device according to any one of claims 1 to 6, wherein: The utilizing of the unreliable transport protocol includes tracking receipt of packet transmissions by the switch to a device to perform result aggregation based on receipt of result aggregation from a root switch including a content identifier of the packet transmissions by the switch to the device.

8. At least one non-transitory computer-readable medium comprising instructions stored thereon that, if executed by one or more processors, cause the one or more processors to: configuring the first network interface device to utilize a reliable transport protocol for communications from at least one worker node of the collective operation to the first network interface device, wherein, The utilizing of a reliable transport protocol for communications from the at least one worker node of the collective operation to the first network interface device includes: storing a packet receipt status for each packet communication from the at least one worker node of the collective operation to the first network interface device; and The first network interface device is configured to utilize an unreliable transport protocol for communications to a second network interface device to perform result aggregation, wherein the reliable transport protocol comprises a protocol different than a protocol of the unreliable transport protocol.

9. The non-transitory computer readable medium of claim 8, wherein: The reliable transport protocol includes one or more of: Remote Direct Memory Access (RDMA), InfiniBand, Transmission Control Protocol (TCP), Quick UDP Internet Connections (QUIC), or RDMA over Converged Ethernet (RoCE).

10. The non-transitory computer readable medium of claim 8, wherein: The utilizing of a reliable transport protocol for communications from at least one worker node of the collective operation to the first network interface device includes causing transmission of a packet receipt confirmation (ACK) message to the at least one worker node based on receipt of a packet.

11. The non-transitory computer readable medium of claim 8, wherein: The unreliable transport protocol includes a User Datagram Protocol (UDP).

12. The non-transitory computer readable medium of claim 8, wherein: The utilizing an unreliable transport protocol includes tracking receipt of packet transmissions by the first network interface device to the second network interface device based on receipt of a result aggregation from a root switch including an identifier of content of packet transmissions by the first network interface device to the second network interface device.

13. The non-transitory computer readable medium of claim 8, wherein: The first network interface device is to retransmit to the second network interface device packets associated with the identifier not received from the second network interface device.

14. The non-transitory computer-readable medium of any one of claims 8 to 13, comprising instructions stored thereon, which if executed by one or more processors, cause the one or more processors to: A plurality of network interface devices are configured to perform collective operations for training a machine learning (ML) model by associating a logical node with the plurality of network interface devices.

15. A method comprising: The first network interface device utilizes a reliable transport protocol for communications from at least one worker node of the collective operation to the first network interface device, wherein the utilizing of a reliable transport protocol for communications from at least one worker node of the collective operation to the first network interface device comprises: storing a packet receipt status for each packet communication from the at least one worker node of the collective operation to the first network interface device; and The first network interface device utilizes an unreliable transport protocol for communications from the first network interface device to a second network interface device to perform result aggregation, wherein the reliable transport protocol comprises a protocol different from a protocol of the unreliable transport protocol.

16. The method of claim 15, wherein: The reliable transport protocol includes one or more of: Remote Direct Memory Access (RDMA), InfiniBand, Transmission Control Protocol (TCP), Quick UDP Internet Connections (QUIC), or RDMA over Converged Ethernet (RoCE).

17. The method of claim 15, wherein: The utilizing of a reliable transport protocol for communications from at least one worker node of the collective operation to the first network interface device includes transmitting a packet receipt confirmation (ACK) message to the at least one worker node based on receipt of a packet.

18. The method of claim 15, wherein: The unreliable transport protocol includes a User Datagram Protocol (UDP).

19. The method of claim 15, wherein: The utilizing an unreliable transport protocol includes tracking receipt of packet transmissions by the first network interface device to the second network interface device based on receipt of an aggregate of results from a root switch including an identifier of content of the packet transmissions by the first network interface device to the second network interface device.

20. The method of any one of claims 15 to 19, comprising: The first network interface device causes a retransmission to the second network interface device of packets associated with an identifier not received from the second network interface device.

Images