PCIe-based traffic processing methods and scale-up network architecture

Abstract

This application provides a PCIe-based traffic processing method and a scale-up network architecture. In this embodiment, at least one PCIe switch system is deployed on top of an existing scale-up network architecture. The original N GPUs in this scale-up network architecture are connected to the corresponding PCIe switch system via PCIe links, and the original N GPUs in the scale-up network architecture are also connected via the scale-up network. Based on this, the first PCIe link connecting the first GPU to the PCIe switch system can be used to send the first type of data to the second GPU. This effectively avoids the bandwidth occupation of the original scale-up network links by data such as KV cached traffic, thereby reducing the impact on the transmission latency and stability of data such as TP traffic and EP traffic.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a traffic processing method and a scale-up network architecture based on the Peripheral Component Interconnect Express (PCIE) standard. Background Technology

[0002] With the rapid development of large models, such as inference-based large models, in their ability to handle long contexts, they are widely used in scenarios such as medical image data analysis, surveillance video data analysis, and scientific research data analysis.

[0003] Currently, in large-scale model training and inference scenarios involving long texts, the TP (Tensor Parallel) traffic, EP (Expert Parallel) traffic, and KV (Key-Value) cache traffic generated in this scenario typically share the same network resources, i.e., they are transmitted between GPUs using the same communication link. However, in this approach, the low-priority KV cache traffic preempts the bandwidth of the communication link during transmission, thus affecting the transmission latency and stability of high-priority data such as TP and EP traffic, and consequently impacting the performance of large-scale model training and inference. Here, TP stands for Tensor Parallel, EP for Expert Parallel, KV for Key-Value, and GPU for Graphics Processing Unit. Summary of the Invention

[0004] In view of this, this application provides a PCIe-based traffic processing method and a scale-up network architecture to avoid affecting the transmission latency and stability of high-priority traffic such as TP traffic and EP traffic.

[0005] This application provides a PCIe-based traffic processing method, which adds at least one PCIe switch system to a scale-up network architecture; each PCIe switch system includes at least one PCIe switch device; the original N GPUs in the scale-up network architecture are connected to the corresponding PCIe switch system via PCIe links; one PCIe switch system connects multiple different GPUs, and different PCIe switch systems connect to different GPUs; the original N GPUs in the scale-up network architecture are also connected via a scale-up network; the method includes:

[0006] When the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than a first set bandwidth threshold, it switches the first type of data that the first GPU is sending to the second GPU from the scale-up network link to the first PCIe link that the first GPU is connected to the PCIe Switch system, so as to send the first type of data to the second GPU through the first PCIe link; the first type of data refers to data with a priority lower than a set priority threshold.

[0007] If the first GPU detects that the bandwidth utilization of the second PCIe link connected to the PCIe Switch system is lower than the second set bandwidth threshold, and the second PCIe link is the link involved in the communication between the first GPU and the third GPU, then the first GPU will share the load of the third type of data sent from the first GPU to the third GPU through the scale-up network link between the first GPU and the third GPU and the second PCIe link; the third type of data refers to data with a priority higher than the set priority threshold.

[0008] This application also provides a scale-up network architecture, in which at least one PCIe Switch system is added; each PCIe Switch system includes at least one PCIe Switch device; the original N GPUs in the scale-up network architecture are connected to the corresponding PCIe Switch system via PCIe links; one PCIe Switch system connects multiple different GPUs, and different PCIe Switch systems connect to different GPUs; the original N GPUs in the scale-up network architecture are also connected via the scale-up network.

[0009] When the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than a first set bandwidth threshold, it switches the first type of data that the first GPU is sending to the second GPU from the scale-up network link to the first PCIe link that the first GPU is connected to the PCIe Switch system, so as to send the first type of data to the second GPU through the first PCIe link; the first type of data refers to data with a priority lower than a set priority threshold.

[0010] If the first GPU detects that the bandwidth utilization of the second PCIe link connected to the PCIe Switch system is lower than the second set bandwidth threshold, and the second PCIe link is the link involved in the communication between the first GPU and the third GPU, then the first GPU will share the load of the third type of data sent from the first GPU to the third GPU through the scale-up network link between the first GPU and the third GPU and the second PCIe link; the third type of data refers to data with a priority higher than the set priority threshold.

[0011] As can be seen from the above technical solutions, in this embodiment of the application, by adding at least one PCIE Switch system to the original Scale-up network architecture, the original N GPUs of the Scale-up network architecture are connected to the corresponding PCIE Switch system through PCIE links. One of the PCIE Switch systems connects to multiple different GPUs, and the GPUs connected to different PCIE Switch systems are different. The original N GPUs in the Scale-up network architecture are also connected through the Scale-up network. Based on this, in this scale-up network architecture, when the first GPU detects that the bandwidth occupancy of the scale-up network link between itself and the second GPU is greater than a first set bandwidth threshold, it switches the first type of data (i.e., data with a priority lower than the set priority threshold, such as KV cache traffic) from the scale-up network link to the first PCIe link connected to the PCIe Switch system. This allows the first type of data to be sent to the second GPU via the first PCIe link. This not only effectively reduces the data transmission pressure on the original scale-up network link, but also effectively avoids the bandwidth occupation of the original scale-up network link by data such as KV cache traffic. This reduces the transmission interference of data with a priority higher than the set priority threshold, such as TP traffic and EP traffic, in the original scale-up network link, thereby reducing the impact on the transmission latency and stability of data such as TP traffic and EP traffic, and thus improving the performance of large model training and inference.

[0012] Furthermore, in this embodiment, when the first GPU detects that the bandwidth utilization of the second PCIe link connected to the PCIe Switch system is lower than a second set bandwidth threshold, where the second PCIe link is the link involved in communication between the first GPU and the third GPU, the load of the third type of data sent by the first GPU to the third GPU is shared by the scale-up network link between the first GPU and the third GPU and the second PCIe link. The third type of data refers to data with a priority higher than the set priority threshold. This not only further reduces the data transmission pressure of the original scale-up network link, but also effectively improves the data transmission efficiency of this type of data with a priority higher than the set priority threshold. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this application, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0014] Figure 1 This is a schematic diagram illustrating the implementation of a scale-up network architecture provided in an embodiment of this application.

[0015] Figure 2 This is a flowchart illustrating a PCIe-based traffic processing method provided in an embodiment of this application.

[0016] Figure 3 This is a flowchart illustrating another PCIE-based traffic processing method provided in an embodiment of this application.

[0017] Figure 4 This is a schematic diagram illustrating another implementation of a scale-up network architecture provided in an embodiment of this application.

[0018] Figure 5 This is a schematic diagram of a scale-up network architecture provided in an embodiment of this application. Detailed Implementation

[0019] To facilitate understanding of the technical solutions of the embodiments of this application, the application scenarios of the embodiments of this application are described below:

[0020] In this embodiment, scale-up refers to enhancing computing power by adding computing units (such as GPUs, CPUs, memory, etc.) within a single node, and relying on high-speed communication protocols (such as PCIe, NVLink, UALink, etc.) to ensure that these computing units can work together efficiently. Furthermore, computing units such as GPUs located on different nodes can also be connected via the same high-speed communication protocol (such as PCIe, NVLink, UALink, etc.) to achieve scale-up. Here, NVLink is an abbreviation for NVIDIA Link, UALink is an abbreviation for UltraAccelerator Link, and CPU is an abbreviation for Central Processing Unit.

[0021] As described above, in practical applications, in large-model long-text training and inference scenarios based on scale-up network architectures, the generated TP traffic, EP traffic, and KV cache traffic usually share the same network resources, that is, they are transmitted between GPUs using the same communication link. See also... Figure 1 As shown, taking GPU1 and GPU2 as an example, the TP traffic, EP traffic, and KV cache traffic that need to be transmitted between GPU1 and GPU2 are all transmitted through the high-bandwidth interconnect channel between GPU1 and GPU2 (that is, the communication link based on the above-mentioned high-speed communication protocol, which can also be called the scale-up network link). Figure 1 XLink in this context can refer to a collective term for high-speed communication protocols such as NVLink and UALink. This can lead to situations where data with a priority lower than the set priority threshold (referred to as low-priority data), such as KV cache traffic, preempts the bandwidth of the scale-up network link. This, in turn, affects the transmission latency and stability of data with a priority higher than the set priority threshold (referred to as high-priority data), such as TP traffic and EP traffic, thereby impacting the performance of large model training and inference. Based on this, this application provides a PCIe-based traffic processing method to solve the problem of low-priority data preempting the bandwidth of the scale-up network link, thus affecting the transmission latency and stability of high-priority data.

[0022] In this embodiment, TP and EP are two commonly used model parallelization techniques. TP refers to splitting the model's tensors (such as the weight matrix) into multiple pieces (denoted as tensor slices) and distributing them across different GPUs, with each GPU responsible for processing its corresponding tensor slice. Based on this, TP throughput can refer to the model parameters and computation results generated by each GPU during model training or inference based on its corresponding tensor slice. EP refers to distributing multiple expert sub-models in a Mixture of Experts (MFA) model across different GPUs, with each GPU responsible for processing its corresponding expert sub-model. Based on this, EP throughput can refer to the model parameters and computation results generated by each GPU during model training or inference based on its corresponding expert sub-model.

[0023] KV cache traffic can refer to historical context information (such as previous input and / or output information) generated during model training or inference.

[0024] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0025] See Figure 2 , Figure 2 This is a flowchart illustrating a PCIe-based traffic processing method provided in an embodiment of this application. In this embodiment, as an example, the method can be applied to any GPU in a scale-up network architecture. This embodiment adds at least one PCIe switch system to the scale-up network architecture, each PCIe switch system including at least one PCIe switch device; the original N GPUs in the scale-up network architecture are connected to the corresponding PCIe switch system via PCIe links, wherein one PCIe switch system connects multiple different GPUs, and different PCIe switch systems connect to different GPUs; furthermore, the original N GPUs in the scale-up network architecture are also connected via a scale-up network.

[0026] like Figure 2 As shown, the GPU to which this method is applied can be referred to as the first GPU, and the process may include the following steps:

[0027] Step 201: When the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than the first set bandwidth threshold, the first GPU switches the first type of data sent from the first GPU to the second GPU from the scale-up network link to the first PCIe link connected to the PCIe Switch system, so as to send the first type of data to the second GPU through the first PCIe link; the first type of data refers to data with a priority lower than the set priority threshold.

[0028] In this embodiment, the second GPU is different from the first GPU. As an example, the first GPU can periodically (e.g., every 30 minutes or 1 hour) obtain the bandwidth utilization of the scale-up network link between itself and the second GPU. If the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is detected to be greater than a first preset bandwidth threshold, the first type of data (such as the aforementioned KV cache traffic) sent by the first GPU to the second GPU is switched from the scale-up network link to the first PCIe link connected to the PCIe Switch system, so that the first type of data can be sent to the second GPU via the first PCIe link. Here, the first PCIe link can refer to the PCIe link involved in the communication between the first GPU and the second GPU.

[0029] Optionally, the first set bandwidth threshold can be flexibly set based on actual application needs, such as 80%, 85% or 90%, etc., and is not specifically limited here.

[0030] In this embodiment, as one example, when the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than a first set bandwidth threshold, it can send a second type of data from the first GPU to the second GPU through the scale-up network link. The second type of data can refer to data with a priority higher than the set priority threshold, such as TP traffic, EP traffic, etc.

[0031] As for how to obtain the bandwidth utilization of the scale-up network link between the first GPU and the second GPU, an example will be given below, and will not be elaborated here.

[0032] Step 202: If the first GPU detects that the bandwidth utilization of the second PCIe link connected to the PCIe Switch system is lower than the second set bandwidth threshold, and the second PCIe link is the link involved in the communication between the first GPU and the third GPU, then the load sharing of the third type of data sent by the first GPU to the third GPU is achieved through the scale-up network link between the first GPU and the third GPU and the second PCIe link; the third type of data refers to data with a priority higher than the set priority threshold.

[0033] In this embodiment, the third GPU is different from the first GPU. As an example, after the first GPU detects that the bandwidth utilization of the scale-up network link between itself and the second GPU is greater than a first preset bandwidth threshold, it can further periodically (e.g., every 10 minutes or every half hour) obtain the bandwidth utilization of the second PCIe link connecting itself to the PCIe Switch system. When it detects that the bandwidth utilization of the second PCIe link connecting itself to the PCIe Switch system is lower than a second preset bandwidth threshold, the third type of data sent by the first GPU to the third GPU is load-sharing between the scale-up network link and the second PCIe link between the first GPU and the third GPU. In other words, the third type of data sent by the first GPU to the third GPU is distributed to the scale-up network link and the second PCIe link between the first GPU and the third GPU for transmission, which can effectively reduce the transmission pressure of the scale-up network link.

[0034] In this embodiment, as one example, the second set bandwidth threshold is less than the first set bandwidth threshold. Optionally, the second set bandwidth threshold can be, for example, 20% or 30%, etc., and is not specifically limited here.

[0035] Optionally, in this embodiment, the first GPU, the second GPU, and the third GPU are GPUs connected to the same PCIe Switch system.

[0036] As for how to obtain the bandwidth utilization of the second PCIe link connecting the first GPU to the PCIe Switch system, an example will be given below, and will not be elaborated here.

[0037] This concludes the process. Figure 2 The process is shown below.

[0038] pass Figure 2As can be seen from the process shown, in this embodiment of the application, by adding at least one PCIe Switch system to the original Scale-up network architecture, the original N GPUs of the Scale-up network architecture are connected to the corresponding PCIe Switch system through PCIe links. One of the PCIe Switch systems connects to multiple different GPUs, and the different PCIe Switch systems connect to different GPUs. The original N GPUs in the Scale-up network architecture are also connected through the Scale-up network. Based on this, in this scale-up network architecture, when the first GPU detects that the bandwidth occupancy of the scale-up network link between itself and the second GPU is greater than a first set bandwidth threshold, it switches the first type of data (i.e., data with a priority lower than the set priority threshold, such as KV cache traffic) from the scale-up network link to the first PCIe link connected to the PCIe Switch system. This allows the first type of data to be sent to the second GPU via the first PCIe link. This not only effectively reduces the data transmission pressure on the original scale-up network link, but also effectively avoids the bandwidth occupation of the original scale-up network link by data such as KV cache traffic. This reduces the transmission interference of data with a priority higher than the set priority threshold, such as TP traffic and EP traffic, in the original scale-up network link, thereby reducing the impact on the transmission latency and stability of data such as TP traffic and EP traffic, and thus improving the performance of large model training and inference.

[0039] Furthermore, in this embodiment, when the first GPU detects that the bandwidth utilization of the second PCIe link connected to the PCIe Switch system is lower than a second set bandwidth threshold, where the second PCIe link is the link involved in communication between the first GPU and the third GPU, the load of the third type of data sent by the first GPU to the third GPU is shared by the scale-up network link between the first GPU and the third GPU and the second PCIe link. The third type of data refers to data with a priority higher than the set priority threshold. This not only further reduces the data transmission pressure of the original scale-up network link, but also effectively improves the data transmission efficiency of this type of data with a priority higher than the set priority threshold.

[0040] The following describes how to obtain the bandwidth utilization of the scale-up network link between the first GPU and the second GPU:

[0041] In this embodiment, the designated register of the first GPU records the amount of data sent by the first GPU to other GPUs through the scale-up network link between the first GPU and other GPUs within a certain time period. Based on this, as an embodiment, obtaining the bandwidth utilization rate of the scale-up network link between the first GPU and the second GPU can be implemented as follows: First, obtain the amount of data sent by the first GPU to the second GPU through the scale-up network link between the first GPU and the second GPU within a set time period (such as 1 second or 10 seconds) from the designated register of the first GPU, and calculate a first ratio of the obtained amount of sent data to the set time period; then, based on the first ratio and a second ratio of the total bandwidth of the scale-up network link between the first GPU and the second GPU, determine the bandwidth utilization rate of the scale-up network link between the first GPU and the second GPU. For example, the second ratio can be used as the bandwidth utilization rate of the scale-up network link between the first GPU and the second GPU.

[0042] For example, assuming that the amount of data sent by the first GPU to the second GPU in the second prior to the current moment is 0.8 T / s obtained from the designated register of the first GPU, and the total bandwidth of the scale-up network link between the first GPU and the second GPU is 1 T / s, then the bandwidth utilization rate of the scale-up network link between the first GPU and the second GPU can be (0.8 T / s) / (1 T / s) = 80%. Here, T is an abbreviation for Terabyte.

[0043] As for how to obtain the bandwidth utilization of the second PCIe link connecting the first GPU to the PCIe Switch system, it is similar to how to obtain the bandwidth utilization of the scale-up network link between the first GPU and the second GPU.

[0044] Specifically, as an example, the designated register of the first GPU also records the amount of data sent to other GPUs via the PCIe links involved in communication between the first GPU and other GPUs within a certain time period. Based on this, the above-mentioned acquisition of the bandwidth utilization rate of the second PCIe link connected to the PCIe Switch system by the first GPU can be implemented as follows: First, obtain the amount of data sent by the first GPU to the second GPU via the second PCIe link within a set time period (such as 1 second or 10 seconds) from the designated register of the first GPU, and calculate a third ratio of the obtained amount of sent data to the set time period; then, based on the third ratio and a fourth ratio of the total bandwidth of the second PCIe link, determine the bandwidth utilization rate of the second PCIe link, for example, the fourth ratio can be used as the bandwidth utilization rate of the second PCIe link.

[0045] The following is about the above. Figure 2 The PCIe-based traffic processing method shown will be further described below:

[0046] In this embodiment, as one example, any PCIe switch system is connected to at least one cache disk, wherein any PCIe switch device in any PCIe switch system is connected to at least one cache disk. Optionally, the cache disk here can be, for example, a solid-state drive (SSD) based on the Non-Volatile Memory Express (NVME) protocol, etc., and this embodiment is not specifically limited to this. Based on this, see... Figure 3 As shown, the method further includes:

[0047] Step 301: Before sending the first type of data to the second GPU via the first PCIe link, the first GPU caches the first type of data to the available cache disk connected to the PCIe Switch system corresponding to the first GPU.

[0048] In this embodiment, as described above, any PCIe Switch system may include at least one PCIe Switch device; wherein, any GPU is specifically connected to a PCIe Switch device in the corresponding PCIe Switch system via a PCIe link, and any PCIe Switch device is connected to at least one GPU. As an example, any PCIe Switch device in any PCIe Switch system includes a cache acceleration component based on hardware chip-level acceleration. This cache acceleration component can be used to cache the first type of data from the GPU to an available cache disk. Optionally, the cache acceleration component may be, for example, a data acceleration card, etc., and is not specifically limited here.

[0049] This embodiment introduces a cache disk to cache data such as key-value (KV) cache traffic. Data read and write operations on the cache disk can be performed through direct access based on memory semantics. This effectively alleviates the caching pressure on the designated memory of the GPU. Compared to linearly expanding the designated memory to accommodate more data, the deployment cost of the cache disk is lower. Furthermore, this embodiment also introduces a cache acceleration component based on hardware chip-level acceleration. This component implements data sorting, compression, and index merging operations of the KV cache traffic in hardware and on a chip, accelerating the writing of KV cache traffic. This effectively improves the cache rate, enabling the GPU to quickly offload tens of millions of small packet IOPS corresponding to the KV cache traffic, thus meeting the high-computing-power bandwidth requirements of the GPU for KV cache traffic offloading.

[0050] IOPS stands for Input / Output Operations Per Second.

[0051] Step 302: Before sending the second type of data from the first GPU to the second GPU via the scale-up network link, the first GPU caches the second type of data in the local memory of the first GPU.

[0052] In this embodiment, optionally, the local memory of any GPU can be, for example, the GPU video memory in the GPU or the CPU memory in the electronic device to which the GPU belongs, and is not specifically limited here.

[0053] Step 303: Before the first GPU load-sharing the third type of data sent from the first GPU to the third GPU through the scale-up network link and the second PCIe link between the first GPU and the third GPU, the first GPU caches the third type of data in the local memory of the first GPU.

[0054] It should be noted that in this embodiment, there is no specific execution order between steps 301 and 303. The labels of steps 301 to 303 in this embodiment are only for ease of description.

[0055] In this embodiment, as one example, the method further includes: when the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than a first preset bandwidth threshold, if it is identified that the data to be transmitted to the second GPU currently includes KV cache data (i.e., the aforementioned KV cache traffic), TP data (i.e., the aforementioned TP traffic), and / or EP data (i.e., the aforementioned EP traffic), then the KV cache data sent by the first GPU to the second GPU is switched from the scale-up network link to the PCIe connection of the first GPU. The Switch system uses a first PCIe link to send the KV cache data to the second GPU; and to send TP data and / or EP data to the second GPU via the Scale-up network link. This enables the first GPU to transmit KV cache data, TP data, and / or EP data to the second GPU in parallel. This not only reduces the transmission pressure on the Scale-up network link through the first PCIe link, but also avoids the problem of KV cache data occupying the bandwidth of the Scale-up network link and interfering with the transmission of TP data and / or EP data, thereby reducing the impact on the transmission latency and stability of TP data and / or EP data.

[0056] To facilitate understanding of the specific implementation process of the PCIE-based traffic processing method described above, specific examples are given below.

[0057] This application provides a scale-up network architecture. See also... Figure 4 As shown, at least one PCIe switch system is added to the existing scale-up network architecture. Each PCIe switch system includes at least one PCIe switch device. The original N GPUs in this scale-up network architecture are connected to the corresponding PCIe switch system via PCIe links. Each PCIe switch system connects multiple different GPUs, and different PCIe switch systems connect to different GPUs. Furthermore, the original N GPUs in this scale-up network architecture are also connected via the scale-up network.

[0058] Optionally, the PCIe Switch devices in any PCIe Switch system can be connected in a cascading or fully connected manner, for example, without specific limitations.

[0059] Alternatively, the GPU and the PCIe Switch system can be connected via a physical cable (such as a dedicated high-speed signal transmission cable for PCIe).

[0060] like Figure 4 As shown, this embodiment adds an NVMe disk (i.e., a cache disk based on the NVMe protocol) to the original scale-up network architecture. Each PCIe Switch system connects to at least one NVMe disk. Furthermore, each PCIe Switch device in any PCIe Switch system also includes a cache acceleration component based on hardware chip-level acceleration to accelerate data read and write operations on the NVMe disk.

[0061] In this embodiment, as an example, the user can determine the number of PCIe Switch systems to be deployed and the number of PCIe Switch devices within the PCIe Switch system based on the number of GPUs in the scale-up network architecture and the number of PCIe lanes (Lines) supported by the GPUs.

[0062] Specifically, users can group the N GPUs in the scale-up network architecture into M GPU groups based on actual application needs, where M is greater than or equal to 1, and each GPU group includes multiple GPUs. A PCIE switch system is then deployed for each GPU group. For example, for each GPU group, the number of PCIE switch devices in the corresponding PCIE switch system can be determined based on the number of GPUs in the group and the number of PCIE lines supported by each GPU; wherein the sum of the number of PCIE lines supported by each PCIE switch device in the PCIE switch system corresponding to any GPU group is greater than the sum of the number of PCIE lines supported by each GPU in the group. It should be noted that this is merely an illustrative example, and this embodiment does not specifically limit how the PCIE switch system is deployed.

[0063] The following is based on the above. Figure 4 The scale-up network architecture shown above, using the inference scenario of the vLLM model as an example, describes the above PCIe-based traffic processing method:

[0064] In this embodiment, vLLM is an abbreviation for Very Large Language Model. The inference process of the vLLM model can be divided into the vLLM Prefill stage and the vLLM Decode stage. In the vLLM Prefill stage, the input data (such as user-inputted prompt text) is preprocessed to obtain data that the vLLM model can understand, such as key-value (KV) format data. This KV format data can be considered as KV buffer traffic, preparing for the subsequent inference calculations in the vLLM Decode stage. In the vLLM Decode stage, corresponding inference calculations are performed based on the data output from the vLLM Prefill stage to obtain the inference result and output it.

[0065] Optionally, the GPU used to perform the vLLM Prefill stage is at least one GPU in a scale-up network architecture, and the GPU used to perform the vLLM Decode stage is at least one GPU in a scale-up network architecture. The GPU used to perform the vLLM Prefill stage and the GPU used to perform the vLLM Decode stage are different GPUs in the scale-up network architecture.

[0066] Based on this, assuming the user inputs a prompt text on the front end, the front end will send an inference request to the GPU used to perform the vLLM Prefill stage based on the prompt text. Upon receiving the inference request, the GPU will generate KV cache data, TP data, and / or EP data. If the transmission link for the KV cache data from this GPU to the target GPU has been switched to the PCIe link involved in communication between this GPU and the target GPU, the KV cache data will first be transmitted to the PCIe switch device connected to this GPU via the PCIe link. Then, the cache acceleration component in the PCIe switch device will accelerate the caching of the KV cache data to the NVMe cache disk connected to the PCIe switch device. Data is then read from the NVMe cache disk and transmitted to the target GPU via the PCIe link. Additionally, the TP data and / or EP data will be cached in the local memory of this GPU. Data is then read from the local memory of this GPU and transmitted to the target GPU via the scale-up network link between this GPU and the target GPU.

[0067] In this embodiment, as an example, if the transmission link of the KV cache data sent from this GPU to the target GPU has been switched to the PCIe link involved in the communication between this GPU and the target GPU, if the KV cache data generated based on the current inference request is transmitted and the transmission of TP data and / or EP data generated based on the current inference request is completed, there is no need to wait for the transmission of TP data and / or EP data generated based on the current inference request to be completed. The KV cache data generated by the next inference request based on the current inference request can be processed directly.

[0068] It should be noted that the aforementioned GPU and the target GPU are GPUs connected to the same PCIe Switch system. If the aforementioned GPU and the target GPU are not connected to the same PCIe Switch system, the GPU will transfer the key-value cache data to be transmitted to the target GPU via the scale-up network link between the GPU and the target GPU.

[0069] In this embodiment, by constructing a PCIE link independent of the scale-up network link for the transmission of low-priority data such as KV cache data, it is possible to effectively avoid KV cache data competing with high-priority data such as TP data and EP data for the bandwidth of the original scale-up network link, thereby ensuring the predictability (i.e., delay controllability) of the transmission of high-priority data such as TP data and EP data, as well as the determinism and stability of throughput.

[0070] This concludes the description of the method provided in the embodiments of this application. The apparatus provided in the embodiments of this application will now be described:

[0071] As an example, this embodiment also provides a scale-up network architecture. For example, see... Figure 5 , Figure 5 This is a schematic diagram of a scale-up network architecture provided in an embodiment of this application. This scale-up network architecture corresponds to... Figure 2 The method flow is shown below. Figure 5 As shown, at least one PCIe Switch system 501 is added to the Scale-up network architecture 500; each PCIe Switch system 501 includes at least one PCIe Switch device; the original N GPUs 502 in the Scale-up network architecture 500 are connected to the corresponding PCIe Switch system 501 via PCIe links; one PCIe Switch system 501 connects to multiple different GPUs 502, and different PCIe Switch systems 501 are connected to different GPUs 502; the original N GPUs 502 in the Scale-up network architecture 500 are also connected via the Scale-up network.

[0072] When the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than a first set bandwidth threshold, it switches the first type of data that the first GPU is sending to the second GPU from the scale-up network link to the first PCIe link that the first GPU is connected to the PCIe Switch system, so as to send the first type of data to the second GPU through the first PCIe link; the first type of data refers to data with a priority lower than a set priority threshold.

[0073] If the first GPU detects that the bandwidth utilization of the second PCIe link connected to the PCIe Switch system is lower than the second set bandwidth threshold, and the second PCIe link is the link involved in the communication between the first GPU and the third GPU, then the first GPU will share the load of the third type of data sent from the first GPU to the third GPU through the scale-up network link between the first GPU and the third GPU and the second PCIe link; the third type of data refers to data with a priority higher than the set priority threshold.

[0074] As an example, when the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than a first set bandwidth threshold, the first GPU sends the second type of data that was sent from the first GPU to the second GPU through the scale-up network link. The second type of data refers to data with a priority higher than the set priority threshold.

[0075] As an example, any PCIe Switch system is connected to at least one cache disk;

[0076] Before sending the first type of data to the second GPU via the first PCIe link, the first GPU caches the first type of data to the available cache disk connected to the PCIe Switch system corresponding to the first GPU.

[0077] Before sending the second type of data from the first GPU to the second GPU via the scale-up network link, the first GPU caches the second type of data in the local memory of the first GPU.

[0078] Before the third type of data sent from the first GPU to the third GPU is load-sharing through the scale-up network link between the first GPU and the third GPU and the second PCIe link, the first GPU caches the third type of data in the local memory of the first GPU.

[0079] As an example, any GPU is specifically connected to a PCIe Switch device in the corresponding PCIe Switch system via a PCIe link;

[0080] Any PCIe Switch device in any PCIe Switch system includes a cache acceleration component based on hardware chip-level acceleration; the cache acceleration component is used to accelerate the caching of first-type data from the GPU to an available cache disk.

[0081] As an example, the first GPU, the second GPU, and the third GPU are GPUs connected to the same PCIESwitch system.

[0082] As an example, when the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than a first set bandwidth threshold, if it identifies that the data to be transmitted to the second GPU currently includes KV cache data, TP data and / or EP data, then the first GPU switches the KV cache data sent from the first GPU to the second GPU from the scale-up network link to the first PCIe link connected to the PCIe Switch system, so as to send the KV cache data to the second GPU through the first PCIe link; and sends the TP data and / or EP data to the second GPU through the scale-up network link, so as to realize the parallel transmission of KV cache data, TP data and / or EP data from the first GPU to the second GPU.

[0083] This concludes the process. Figure 5 The structural description of the scale-up network architecture shown.

[0084] For the scale-up network architecture implementation, since it basically corresponds to the method implementation, the relevant parts can be referred to in the description of the method implementation, and will not be repeated here. Those skilled in the art can understand and implement it without creative effort.

[0085] The above are merely preferred embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A traffic processing method based on the high-speed serial computer expansion bus standard PCIe, characterized in that, In a vertically scaled-up network architecture, at least one PCIe switch system is deployed; each PCIe switch system includes at least one PCIe switch device. The original N graphics processing units (GPUs) in the scale-up network architecture are connected to the corresponding PCIe switch system via PCIe links; one PCIe switch system connects to multiple different GPUs, and different PCIe switch systems connect to different GPUs. The original N GPUs in the scale-up network architecture are still connected through the scale-up network; the method includes: When the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than a first set bandwidth threshold, it switches the first type of data that the first GPU is sending to the second GPU from the scale-up network link to the first PCIe link that the first GPU is connected to the PCIe Switch system, so as to send the first type of data to the second GPU through the first PCIe link; the first type of data refers to data with a priority lower than a set priority threshold. If the first GPU detects that the bandwidth utilization of the second PCIe link connected to the PCIe Switch system is lower than the second set bandwidth threshold, and the second PCIe link is the link involved in the communication between the first GPU and the third GPU, then the first GPU will share the load of the third type of data sent from the first GPU to the third GPU through the scale-up network link between the first GPU and the third GPU and the second PCIe link; the third type of data refers to data with a priority higher than the set priority threshold.

2. The method according to claim 1, characterized in that, The method further includes: When the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than a first set bandwidth threshold, it sends the second type of data from the first GPU to the second GPU through the scale-up network link. The second type of data refers to data with a priority higher than the set priority threshold.

3. The method according to claim 2, characterized in that, Each PCIe Switch system is connected to at least one cache disk; The method further includes: Before sending the first type of data to the second GPU via the first PCIe link, the first GPU caches the first type of data to the available cache disk connected to the PCIe Switch system corresponding to the first GPU. Before sending the second type of data from the first GPU to the second GPU via the scale-up network link, the first GPU caches the second type of data in the local memory of the first GPU. Before the third type of data sent from the first GPU to the third GPU is load-sharing through the scale-up network link between the first GPU and the third GPU and the second PCIe link, the first GPU caches the third type of data in the local memory of the first GPU.

4. The method according to claim 1 or 3, characterized in that, Each GPU is specifically connected to a PCIe Switch device in the corresponding PCIe Switch system via a PCIe link; Any PCIe Switch device in any PCIe Switch system includes a cache acceleration component based on hardware chip-level acceleration; the cache acceleration component is used to accelerate the caching of first-type data from the GPU to an available cache disk.

5. The method according to claim 1, characterized in that, The first GPU, the second GPU, and the third GPU are GPUs connected to the same PCIe Switch system.

6. The method according to claim 1, characterized in that, The method further includes: If the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than a first set bandwidth threshold, and if it identifies that the data to be transmitted to the second GPU currently includes KV cache data, TP data and / or EP data, then the first GPU switches the KV cache data sent from the first GPU to the second GPU from the scale-up network link to the first PCIe link connected to the PCIe Switch system, so as to send the KV cache data to the second GPU through the first PCIe link; and sends the TP data and / or EP data to the second GPU through the scale-up network link, so as to realize the parallel transmission of KV cache data, TP data and / or EP data from the first GPU to the second GPU.

7. A vertically scalable scale-up network architecture, characterized in that, In a scale-up network architecture, deploy at least one high-speed serial computer expansion bus standard PCIe switch system; any PCIe switch system includes at least one PCIe switch device. The original N graphics processing units (GPUs) in the scale-up network architecture are connected to the corresponding PCIe switch systems via PCIe links; one PCIe switch system connects multiple different GPUs, and different PCIe switch systems connect to different GPUs; the original N GPUs in the scale-up network architecture are also connected via the scale-up network. When the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than a first set bandwidth threshold, it switches the first type of data that the first GPU is sending to the second GPU from the scale-up network link to the first PCIe link that the first GPU is connected to the PCIe Switch system, so as to send the first type of data to the second GPU through the first PCIe link; the first type of data refers to data with a priority lower than a set priority threshold. If the first GPU detects that the bandwidth utilization of the second PCIe link connected to the PCIe Switch system is lower than the second set bandwidth threshold, and the second PCIe link is the link involved in the communication between the first GPU and the third GPU, then the first GPU will share the load of the third type of data sent from the first GPU to the third GPU through the scale-up network link between the first GPU and the third GPU and the second PCIe link; the third type of data refers to data with a priority higher than the set priority threshold.

8. The architecture according to claim 7, characterized in that, When the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than a first set bandwidth threshold, it sends the second type of data that the first GPU was sending to the second GPU to the second GPU through the scale-up network link. The second type of data refers to data with a priority higher than a set priority threshold. And / or, Each PCIe Switch system is connected to at least one cache disk; Before sending the first type of data to the second GPU via the first PCIe link, the first GPU caches the first type of data to the available cache disk connected to the PCIe Switch system corresponding to the first GPU. Before sending the second type of data from the first GPU to the second GPU via the scale-up network link, the first GPU caches the second type of data in the local memory of the first GPU. Before the first GPU load-sharing the third type of data sent from the first GPU to the third GPU through the scale-up network link between the first GPU and the third GPU and the second PCIe link, the first GPU caches the third type of data in the local memory of the first GPU. And / or, Each GPU is specifically connected to a PCIe Switch device in the corresponding PCIe Switch system via a PCIe link; Any PCIe Switch device in any PCIe Switch system includes a cache acceleration component based on hardware chip-level acceleration; the cache acceleration component is used to accelerate the caching of first-type data from the GPU to an available cache disk.

9. The architecture according to claim 7, characterized in that, The first GPU, the second GPU, and the third GPU are GPUs connected to the same PCIe Switch system.

10. The architecture according to claim 7, characterized in that, If the first GPU detects that the bandwidth utilization of the scale-up network link between the first GPU and the second GPU is greater than a first set bandwidth threshold, and if it identifies that the data to be transmitted to the second GPU currently includes KV cache data, TP data and / or EP data, then the first GPU switches the KV cache data sent from the first GPU to the second GPU from the scale-up network link to the first PCIe link connected to the PCIe Switch system, so as to send the KV cache data to the second GPU through the first PCIe link; and sends the TP data and / or EP data to the second GPU through the scale-up network link, so as to realize the parallel transmission of KV cache data, TP data and / or EP data from the first GPU to the second GPU.

Images