Collective perception network configuration

Abstract

This disclosure relates to a collective-aware network configuration. A system includes a plurality of processors for communicating with each other via one or more network devices. One or more processors are configured to, for a given type of collective operation to be performed by the system, specify a packet processing configuration to be applied by the network devices when processing packets associated with the given type of collective operation, and, during the performance of the given type of collective operation, mark packets associated with the collective operation using a tag indicating the packet processing configuration, thereby instructing one or more network devices to process the packets associated with the collective operation according to the packet processing configuration in response to the tag.

Description

Technical Field

[0001] This disclosure generally relates to collective operations in computing systems, and more specifically to a collective operation-aware packet processing method and system. Background Technology

[0002] A collective operation is an operation in which multiple processors work together to exchange, aggregate, or transform data. Non-limiting examples of collectives include an "all-gather" where each processor collects data from all other processors, and an "all-reduce" where processors combine data using operations such as summation or averaging. Collective operations ensure efficient communication and consistency across distributed systems, enabling large-scale computation and collaborative processing. For example, collectives are common in artificial intelligence (AI) model training. Summary of the Invention

[0003] One embodiment described herein provides a system including a plurality of processors for communicating with each other via one or more network devices. One or more processors are configured to specify, for a given type of collective operation to be performed by the system, a packet processing configuration to be applied by the network devices when processing packets related to the given type of collective operation; and during the performance of the given type of collective operation, to mark packets related to the collective operation using a flag indicating the packet processing configuration, thereby instructing one or more network devices to process the packets related to the collective operation according to the packet processing configuration in response to the flag.

[0004] In one embodiment, one or more processors are configured to specify a packet processing configuration by assigning a given Quality of Service (QoS) level to the packet. In another embodiment, one or more processors are configured to specify a packet processing configuration by selecting a lossy communication protocol for transmitting the packet. In yet another embodiment, one or more processors are configured to specify a packet processing configuration by selecting a lossless communication protocol for transmitting the packet.

[0005] In one example embodiment, one or more processors are configured to specify a packet processing configuration by specifying a network device for routing packets using adaptive routing (AR). In an alternative embodiment, one or more processors are configured to specify a packet processing configuration by specifying a network device for routing packets using static routing.

[0006] In some embodiments, one or more processors are used to tag a packet by setting the packet's Differential Service Code Point (DSCP) field to a value indicating the packet's processing configuration. In some embodiments, collective operation is part of an artificial intelligence (AI) model training process.

[0007] According to one embodiment described herein, a network device is additionally provided, the network device including a plurality of ports and a switching structure. The switching structure is used to receive packets via one of the ports, the packets being (i) related to a collective operation of a given type, and (ii) marked with a flag indicating a packet processing configuration specified for performing the given type of collective operation; and to process the packets according to the packet processing configuration indicated by the flag.

[0008] According to the embodiments described herein, a method is also provided in a system comprising a plurality of processors that communicate with each other via network devices. The method includes: specifying a packet processing configuration to be applied by network devices when processing packets related to a given type of collective operation to be performed by the system. During the performance of the given type of collective operation, packets related to the collective operation are marked with a tag indicating the packet processing configuration, thereby instructing one or more network devices to process the packets related to the collective operation according to the packet processing configuration in response to the tag.

[0009] According to the embodiments described herein, a method is also provided, the method comprising: receiving a packet in a network device, the packet (i) relating to a collective operation of a given type, and (ii) being marked with a tag indicating a packet processing configuration specified for performing the given type of collective operation. The packet is processed by the network device according to the packet processing configuration indicated by the tag.

[0010] This disclosure will be more fully understood from the following detailed description of embodiments of this disclosure in conjunction with the accompanying drawings. Attached Figure Description

[0011] Figure 1 This is a schematic block diagram illustrating a computing system performing collective operations according to embodiments described herein; Figure 2 This is an illustrative representation of an embodiment described herein for use in... Figure 1 A flowchart of the method for generating packages in the system; Figure 3 This is an illustrative representation of an embodiment described herein for use in... Figure 1 A flowchart of the method for packet processing in the system; and Figure 4 This is a schematic block diagram illustrating a computing system performing collective operations according to embodiments described herein. Detailed Implementation

[0012] Overview A typical system performing collective operations includes multiple processors interconnected via network devices. Processors may include, for example, a graphics processing unit (GPU) and / or a central processing unit (CPU). Network devices may include, for example, packet switches and / or routers.

[0013] Performing collective operations involves transmitting a large number of packets between processors, and thus involves processing a large number of packets within network devices. In practice, different types of collective operations differ from one another in their packet processing characteristics and requirements. Therefore, no single packet processing configuration can be optimal for all types of collectives.

[0014] The embodiments described herein provide methods and systems for improving the performance of collective operations in a computing system. In some embodiments, at least some of the processors in the system run an application that performs collective operations in addition to performing other tasks. A typical example is an application that uses multiple GPUs to train an artificial intelligence (AI) model in a distributed manner. Distributed training typically includes: sending training data to the GPUs; performing collective computations on the data; and collecting the training results from the GPUs.

[0015] In some embodiments, the application specifies a "packet processing configuration" for at least one type of collective operation, which the network device should apply when processing packets related to that type of collective operation. The application can distinguish many different types of collective operations and specify a corresponding packet processing configuration for each type.

[0016] In this context, the term "packet processing configuration" refers to any suitable configuration, settings, definitions, parameter selections, and / or algorithms that affect how a network device processes packets. Examples of packet processing configurations that can be specified for packets of a given type of collective operation include the following: Assign a specific Quality of Service (QoS) level (also known as a service category) to the packet.

[0017] The specified packets should be transmitted using a lossy communication protocol.

[0018] The specified packets should be transmitted using a lossless communication protocol.

[0019] The specified packet should be routed using adaptive routing (AR).

[0020] The specified packet should be routed using static routes.

[0021] Typically, packet processing configurations are selected to best match the specific requirements or characteristics of the corresponding type of collective operation. For example, a high QoS level might be assigned to a collective comprising a relatively small number of processors for faster completion. As another example, lossy communication protocols might be assigned to a collective involving communication between data centers (sometimes referred to as a "DC-to-DC" collective). Lossless communication protocols might be assigned to a collective limited to communication within a single data center.

[0022] In some embodiments, the processor instructs the network device to apply packet processing configuration by marking packets belonging to a collective operation. After generating a packet associated with a given type of collective, the processor generating the packet marks the packet with a tag indicating the packet processing configuration defined for that type of collective.

[0023] Upon receiving a tagged packet, the network device processes the packet according to the packet processing configuration indicated by the tag. The processor can tag packets in various ways; one example is setting the packet's Differential Service Code Point (DSCP) field to a value indicating the packet processing configuration.

[0024] The disclosed technology is highly effective in optimizing the operation of network devices to best match the actual collective operations being performed. Therefore, it can significantly improve overall application performance.

[0025] System Description Figure 1 This is a schematic diagram illustrating a computing system 20 performing collective operations according to embodiments described herein. System 20 may include, for example, a data center (DC), a high-performance computing (HPC) cluster, or any other suitable type of computing system. In one example, although use case system 20 is used to train an AI model, the disclosed techniques are by no means limited to any particular application.

[0026] System 20 includes multiple GPU nodes 24 interconnected via one or more packet switches 28. The GPU nodes 24 and switches 28 communicate via network link 32. Each GPU node 24 in this example includes one or more GPUs 36 and one or more network interface controllers (NICs) 40. Each switch 28 includes multiple ports 44, switching fabric 48, and a CPU 52. System 20 can operate according to any suitable network protocol, such as Ethernet or InfiniBand™ (IB) .

[0027] GPU 36 typically runs at least one application, such as an AI training application. As part of its operation, the application instructs at least some of the GPUs to collectively perform a collective operation, such as a “full collection” or a “full reduction.” The terms “collective operation” and “collective” are used interchangeably herein.

[0028] During collective execution, GPU 36 performs various computations and also generates packets and transmits them to each other via switch 28. In some embodiments, when generating a packet associated with a given type of collective, the generating GPU 36 marks the packet using a tag indicating the packet processing configuration specified for the given type of collective. GPU 36 uses NIC 40 to send the marked packet to one of the switches in switch 28.

[0029] In some implementations, the tagged packets are generated by GPU 36 itself. In other embodiments, the packets are generated by NIC 40 as part of serving GPU 36. Thus, in some embodiments, the tag (indicating the required packet processing configuration) is inserted into the packet by GPU 36, while in other embodiments, the tag is inserted by NIC 40 on instructions from GPU 36.

[0030] All the above implementations, as well as any other suitable interplay between GPU 36 and NIC 40, are considered as “processors for generating and marking packets” in this document. In this context, GPU node 24 as a whole is also considered as a processor for generating and marking packets. More generally, packets can be generated and marked by any other suitable processor. For example, CPU 52 of switch 28 can also participate in collective execution as a processor and can generate and mark packets itself.

[0031] Switch 28 receives tagged packets from GPU node 24 via port 44; processes packets according to the specified packet processing configuration; and forwards packets to other switches 28 or GPU nodes 24 via port 44.

[0032] Typically, the CPU 52 of switch 28 is pre-configured for packet processing configurations defined for various types of collections. In one example embodiment, the CPU 52 of each switch 28 is configured with a mapping that maps tag values ​​to corresponding packet processing configurations. The CPU 52 pre-configures the structure 48 of switch 28 to extract tags from each tagged packet and to process the packet according to the packet processing configuration indicated by the tags.

[0033] like Figure 1 The depicted configuration of system 20 and its components (e.g., GPU node 24 and switch 28) is an example configuration chosen purely for clarity of concept. Any other suitable configuration may be used in the various alternative embodiments.

[0034] For example, GPU 36 does not necessarily need to be packet-based within the GPU node. As another example, NIC 40 can include any suitable type of network interface, such as an IB Host Channel Adapter (HCA), Data Processing Unit (DPU), etc. Processors participating in the collective can include, for example, a CPU, as a complement or replacement for the GPU.

[0035] In various embodiments, system 20 and its components (e.g., GPU node 24 and switch 28) may be implemented using appropriate software, appropriate hardware (such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs)), or a combination of hardware and software.

[0036] In some embodiments, certain elements of GPU node 24 and switch 28 (e.g., portions of GPU 36, CPU 52, and / or NIC 40) are implemented using one or more general-purpose processors, which are programmed in software to perform the techniques described herein. The software may be downloaded to the processor electronically, for example, via a network, or may be alternatively or additionally provided to and / or stored on a non-transitory tangible medium, such as magnetic memory, optical memory, or electronic memory.

[0037] Collective perception packet processing configuration Figure 2 This is a schematic flowchart illustrating a method for packet generation and tagging according to embodiments described herein. Figure 2 The method is typically executed by the processor of system 20 (e.g., GPU 36 or GPU node 24).

[0038] The method begins with the processor generating a packet at packet generation operation 60, intended for transmission to another processor in system 20. At collective check operation 64, the processor checks whether the packet is associated with a collective. If so, at tagging operation 68, the processor tags the packet using a tag indicating the packet processing configuration defined for that type of collective. In a non-limiting example, the processor sets the packet's Differential Service Code Point (DSCP) field to a value indicating the packet processing configuration. Alternatively, any other suitable tagging scheme may be used.

[0039] In various embodiments, the processor can specify any suitable packet processing configuration for packets of any suitable type of collective. Some non-limiting examples include: (i) assigning a specified QoS level to a packet; (ii) specifying that the packet should be transmitted using a lossy communication protocol; (iii) specifying that the packet should be transmitted using a lossless communication protocol; (iv) specifying that the packet should be routed using adaptive routing (AR); and / or specifying that the packet should be routed using static routing.

[0040] If the packet is not related to the group, operation 68 is skipped and the packet is not marked. At send operation 72, the processor sends the packet to one of the switches in switch 28.

[0041] Figure 3 This is a schematic flowchart illustrating a method for performing packet processing according to embodiments described herein. Figure 3 The method is typically performed by switch 28 of system 20.

[0042] At packet reception operation 80, the method begins with switch 28 receiving a packet via one of the ports 44. The packet originates from one of the processors and may arrive at the switch from the processor or via another switch.

[0043] At tagging operation 84, exchange structure 48 checks whether the packet contains a tag indicating the required packet processing configuration. If the packet is tagged, at tagged packet processing operation 88, structure 48 processes the packet using the packet processing configuration indicated by the tag. If the packet is not tagged, structure 48 processes the packet using some default packet processor configuration at untagged packet processing operation 92.

[0044] Example System Use Cases Figure 4 This is a schematic block diagram of a computing system 1000 (e.g., a data center or high-performance computing (HPC) cluster) performing collective operations according to embodiments described herein. According to at least one embodiment, system 1000 includes multiple subsystems, such as multiple processing devices, multiple network devices, and multiple networks coupled to each other. The computing system 1000 is designed with multiple integrated circuits (referred to as processing devices), each of which may include one or more CPUs and GPUs, thereby forming a powerful and flexible architecture.

[0045] Various processing devices are interconnected via NVLink or other high-speed interconnects, enabling high-speed communication between subsystems; and are also connected via NICs or DPUs to ensure efficient data transfer across computing system 1000 and to one or more external networks 1030, 1036. In this example, system 1000 includes packet switch 1048 connecting NIC / DPU 1028 to network 1030 and packet switch 1050 connecting NIC / DPU 1032 to network 1036.

[0046] Seamless data exchange and parallel processing are enabled through NVLink coupling of the processing device, thereby improving overall computing performance. The processing device connects to multiple networks via one or more network interface cards (NICs) or DPUs, enabling the system to handle complex multi-network tasks with high bandwidth and low latency. This configuration is ideal for demanding applications requiring significant processing power, such as artificial intelligence (AI), machine learning (ML), and data-intensive computing, while ensuring robust connectivity and scalability across various networking environments. The integrated circuits of the computing system 1000 may include one or more CPUs and one or more GPUs.

[0047] Figure 4 An example architecture of a multi-GPU architecture is also demonstrated. As illustrated in the accompanying drawings, the computing system 1000 includes a processing device 1002 with a multi-GPU architecture. Specifically, the processing device 1002 may be a system-on-a-chip and includes multiple subsystems such as a CPU 1006, a GPU 1008, and a GPU 1010. The CPU 1006 may be coupled to the GPU 1008 via a bare-chip to bare-chip (D2D) or chip-to-chip (C2C) interconnect 1012 (such as a ground reference signal interconnect (GRS interconnect)). The CPU 1006 may be coupled to the GPU 1010 via a D2D or C2C interconnect 1014. The CPU 1006 may also be coupled to the GPU 1008 and GPU 1010 via a PCIe interconnect.

[0048] The CPU 1006 can be coupled to one or more NICs or DPUs, which in turn are coupled to one or more networks. For example, as Figure 4 As illustrated, CPU 1006 is coupled to a first NIC / DPU 1026, which is coupled to network 1030. CPU 1006 is also coupled to a second NIC / DPU 1028, which is coupled to network 1030 via switch 1048. For example, NIC / DPU 1026 and NIC / DPU 1028 can be coupled to network 1030 via Ethernet (ETH), NVLINK, or InfiniBand (IB) connections.

[0049] The computing system 1000 also includes a processing device 1004 with a multi-GPU architecture. Specifically, the processing device 1004 includes multiple subsystems, including a CPU 1016, a GPU 1018, and a GPU 1020. The CPU 1016 can be coupled to the GPU 1018 via a D2D or C2C interconnect 1022. The CPU 1016 can be coupled to the GPU 1020 via a D2D or C2C interconnect 1024. The CPU 1016 can also be coupled to the GPU 1018 and GPU 1020 via a PCIe interconnect. The CPU 1016 can be coupled to one or more NICs or DPUs, which in turn are coupled to one or more networks. For example, as... Figure 4 As illustrated, CPU 1016 is coupled to a first NIC / DPU 1034, which is coupled to network 1036. CPU 1016 is also coupled to a second NIC / DPU 1032, which is coupled to network 1036 via switch 1050. NIC / DPU 1032 and NIC / DPU 1034 can be coupled to network 1036 via Ethernet (ETH), NVLINK, or InfiniBand (IB) connections.

[0050] In at least one embodiment, processing device 1002 and processing device 1004 can communicate with each other via NIC / DPU 1038 (such as via PCIe interconnect). Processing device 1002 and processing device 1004 can also communicate with each other via high-bandwidth communication interconnect 1040 (such as NVLink interconnect or other high-speed interconnect).

[0051] Figure 4 The packet switch in the figure may include, for example, an Nvidia Quantum-2 switch. The NIC / DPU in the figure may include, for example, an Nvidia Bluefield DPU. In various embodiments, system 1000 may use the disclosed techniques to perform collective operations, for example, as part of AI training.

[0052] It should be understood that the embodiments described above are presented by way of example, and the invention is not limited to what has been specifically shown and described above. Rather, the scope of the invention includes both combinations and sub-combinations of the various features described above, as well as variations and modifications that may occur to those skilled in the art upon reading the foregoing description and which are not disclosed in the prior art. Documents incorporated herein by reference should be considered part of this application, and unless any term in such incorporated documents is defined in a manner that conflicts with the express or implied definitions in this specification, only the definitions in this specification should be considered.

Claims

1. A system comprising a plurality of processors for communicating with each other via one or more network devices, wherein, One or more of the processors are used for: For a given type of collective operation to be performed by the system, specify the packet processing configuration to be applied by the network device when processing packets related to the given type of collective operation; as well as During the execution of the given type of collective operation, the packet associated with the collective operation is marked with a tag indicating the packet processing configuration, thereby instructing one or more network devices to process the packet associated with the collective operation in response to the tag according to the packet processing configuration.

2. The system of claim 1, wherein, The one or more processors in the processors are used to specify the packet processing configuration by assigning a given Quality of Service (QoS) level to the packet.

3. The system of claim 1, wherein, The one or more processors in the processors are used to specify the packet processing configuration by selecting a lossy communication protocol to transmit the packet.

4. The system of claim 1, wherein, The one or more processors in the processors are used to specify the packet processing configuration by selecting a lossless communication protocol to transmit the packet.

5. The system according to claim 1, wherein, The one or more processors in the processors are configured to specify the packet processing configuration by specifying that the network device uses adaptive routing (AR) to route the packets.

6. The system according to claim 1, wherein, The one or more processors in the processors are configured to specify the packet processing configuration by specifying that the network device uses static routing to route the packets.

7. The system according to claim 1, wherein, The one or more processors in the processors are used to mark the packet by setting the differential service code point (DSCP) field of the packet to a value indicating the packet processing configuration.

8. The system according to claim 1, wherein, The collective operation is part of the artificial intelligence (AI) model training process.

9. A network device, the network device comprising: Multiple ports; as well as A switching structure, the switching structure being used for: A packet is received via one of the ports, the packet being (i) associated with a collective operation of a given type, and (ii) marked with a flag indicating the packet processing configuration specified for performing the collective operation of the given type; as well as The packet is processed according to the packet processing configuration indicated by the tag.

10. A method, the method comprising: In a system comprising multiple processors communicating with each other via network devices, for a given type of collective operation to be performed by the system, a packet processing configuration is specified for the network devices to apply when processing packets associated with the given type of collective operation; as well as During the execution of the given type of collective operation, the packet associated with the collective operation is marked with a tag indicating the packet processing configuration, thereby instructing one or more network devices to process the packet associated with the collective operation in response to the tag according to the packet processing configuration.

11. The method according to claim 10, wherein, Specifying the packet processing configuration includes assigning a given Quality of Service (QoS) level to the packet.

12. The method according to claim 10, wherein, Specifying the packet processing configuration includes selecting a lossy communication protocol for transmitting the packet.

13. The method according to claim 10, wherein, Specifying the packet processing configuration includes selecting a lossless communication protocol for transmitting the packet.

14. The method of claim 10, wherein, Specifying the packet processing configuration includes instructing the network device to use adaptive routing (AR) to route the packets.

15. The method according to claim 10, wherein, Specifying the packet processing configuration includes instructing the network device to use static routes to route the packets.

16. The method of claim 10, wherein, Marking the packet includes setting the differential service code point (DSCP) field of the packet to a value indicating the packet processing configuration.

17. The method according to claim 10, wherein, The collective operation is part of the artificial intelligence (AI) model training process.

18. A method, the method comprising: A packet is received in a network device, the packet being (i) associated with a collective operation of a given type, and (ii) marked with a flag indicating a packet processing configuration specified for performing the collective operation of the given type; as well as The network device processes the packet according to the packet processing configuration indicated by the tag.

Images