Data transmission method

Abstract

The application provides a data transmission method, which can be applied to the technical field of servers. The data transmission method comprises the following steps: a first graphics processing unit writes to-be-transmitted data according to a first address into a double-rate memory, and triggers a first quick computing link controller to monitor a first interrupt event of the first address being written into the double-rate memory when the data writing is completed; the first quick computing link controller writes the first interrupt event into a first central processing unit after encapsulation, so that the first address is written into the first central processing unit; the first central processing unit triggers a first interrupt processing program to modify the value of a memory monitoring register in the first central processing unit; in the case that an application state polling thread associated with the first graphics processing unit in the first central processing unit monitors that the value of the memory monitoring register is modified, the polling thread is interrupted, and a transmission instruction based on a quick computing link protocol is issued according to a second address which is queried according to the first address.

Description

Technical Field

[0001] This application relates to the field of server technology, and more specifically to a data transmission method. Background Technology

[0002] With the continuous application and promotion of neural networks, technologies based on them, such as deep learning, high-performance computing, and large models, have developed rapidly. Based on this, accelerators deployed on different servers in a distributed system need to communicate frequently. However, in related technologies, for devices deployed on different server nodes, devices supporting the Compute Express Link protocol need to use their network interface cards (NICs) to transmit data across nodes to devices that do not support the Compute Express Link protocol, resulting in significant data transmission latency. Summary of the Invention

[0003] In view of the above problems, this application provides a data transmission method.

[0004] According to one aspect of this application, a data transmission method is provided, applied to a server, the server comprising: a first graphics processing unit and a first central processing unit; the first graphics processing unit includes a first fast computing link controller and double-rate memory; the method comprises: the first graphics processing unit writing data to be transmitted into double-rate memory according to a first address, and triggering the first fast computing link controller to detect a first interrupt event that the first address has been written into double-rate memory when the data writing is completed; the first fast computing link controller encapsulating the first interrupt event and writing it into the first central processing unit, so that the first address is written into the first central processing unit; the first central processing unit triggering a first interrupt handler to modify the value of a memory monitoring register in the first central processing unit; when an application-state polling thread associated with the first graphics processing unit in the first central processing unit detects that the value of the memory monitoring register has been modified, the polling thread is interrupted, and a second address is queried according to the first address, and a transmission instruction based on a fast computing link protocol is issued to transmit the data to be transmitted to another server according to the second address through a fast computing link switch, so that when the other server detects that the data to be transmitted has been written into a memory expansion device in the other server, it calls the second graphics processing unit of the other server to read the data to be transmitted from the memory expansion device.

[0005] According to another aspect of this application, a data transmission method is provided, applied to a server, the server including: a second central processing unit, a second graphics processing unit, and a memory expansion device, the memory expansion device including a second fast computing link controller and extended memory; the method includes: the second central processing unit receiving data to be transmitted from another server and writing the data to be transmitted to the memory expansion device according to a second address; upon completion of data writing, triggering a second controller to detect a second interrupt event where the second address is written to the extended memory; the second fast computing link controller encapsulating the second interrupt event and writing it to the second central processing unit, so that the second address is written to the second central processing unit; the second central processing unit triggering a second interrupt handler to modify the value of a memory monitoring register in the second central processing unit; and, if an application-state polling thread associated with the second graphics processing unit in the second central processing unit detects that the value of the memory monitoring register has been modified, interrupting the polling thread and calling the second graphics processing unit to read the data to be transmitted from the extended memory according to the second address.

[0006] According to an embodiment of this application, when it is detected that a first graphics processing unit supporting a memory interconnect protocol writes data to be transmitted into the first memory of the first graphics processing unit at a first address, a first interrupt event is triggered. Then, the first fast computing link controller can immediately encapsulate the first interrupt event and write it into the first central processing unit, so that the first address is written into the first central processing unit, thereby causing the first central processing unit to trigger a first interrupt handler, and subsequently modify the value of the memory monitoring register in the first central processing unit.

[0007] If the application-state polling thread associated with the first graphics processing unit in the first central processing unit detects that the value of the memory monitoring register has been modified, it can immediately interrupt the polling thread, query the second address based on the first address, and issue a transmission command based on the memory interconnect protocol to transmit the data to be transmitted to the second server according to the second address via a fast-calculated link switch. Subsequently, if the second central processing unit of the second server detects that the data to be transmitted is written to the extended memory of the memory expansion device according to the second address, it can immediately trigger a second interrupt event, encapsulate the second interrupt event, and write it to the first central processing unit so that the second address is written to the second central processing unit. The second central processing unit triggers a second interrupt handler to modify the value of the memory monitoring register in the second central processing unit. When the application-state polling thread associated with the second graphics processing unit in the second central processing unit detects that the value of the memory monitoring register has been modified, the polling thread is interrupted, and the second graphics processing unit, which does not support the memory interconnect protocol, is invoked to read the data to be transmitted from the extended memory of the memory expansion device according to the second address. This application saves the time required for processors on different nodes to use network cards to assemble and unpack transmitted data to achieve cross-node communication, enabling efficient cross-node communication between the first graphics processing unit that supports the memory interconnect protocol and the second graphics processing unit that does not support the memory interconnect protocol. Attached Figure Description

[0008] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0009] Figure 1A A schematic diagram of a distributed system according to an embodiment of this application is shown.

[0010] Figure 1B A flowchart of a data transmission method according to an embodiment of this application is shown.

[0011] Figure 1C A flowchart of a data transmission method according to another embodiment of this application is shown.

[0012] Figure 1D A schematic diagram of a first server and a second server according to an embodiment of this application is shown.

[0013] Figure 2A A schematic diagram of a first server according to another embodiment of this application is shown.

[0014] Figure 2B A schematic diagram of a second server according to another embodiment of this application is shown.

[0015] Figure 2CA schematic diagram of a first server and a second server according to another embodiment of this application is shown.

[0016] Figure 3A A schematic diagram of a first server according to another embodiment of this application is shown.

[0017] Figure 3B A schematic diagram of a second server according to another embodiment of this application is shown. Detailed Implementation

[0018] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0019] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0020] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0021] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0022] With the continuous application and promotion of neural networks, technologies based on them, such as deep learning, high-performance computing, and large models, have developed rapidly. For deep learning applications with increasingly complex large models, massive artificial intelligence (AI) server clusters are needed to provide massive computing power. These AI servers can be deployed with processors, or accelerators, to accelerate model computation. For example, accelerators may include graphics processing units (GPUs).

[0023] Because the computing power of a single AI server is limited, some solutions typically require multiple AI servers to collaborate on computation via network interconnection. However, in distributed AI server cluster applications, while network interconnection can effectively improve the computing power of the AI ​​server cluster, the data transmission capacity between graphics processing units (GPUs) is limited. This limitation still restricts the upper limit of the AI ​​server cluster's computing power. Therefore, when the data transmission capacity of an AI server cluster is restricted, the overall computing performance of the cluster will decrease.

[0024] Therefore, when using distributed AI server clusters to accelerate large models or complex neural networks, even if the number of graphics processing units is sufficient, the computing power of the distributed AI server cluster will still be limited by the large number of parameters of the neural network deployed on it, as well as the data transmission capability of the graphics processing units of the AI ​​server cluster when transmitting parameters.

[0025] For graphics processing units (GPUs) deployed on different servers, cross-node communication between GPUs deployed on different server nodes can be achieved through Remote Direct Memory Access (RDMA) network interface cards (NICs). For example, a server cluster may include a first server and a second server. The first server can be the data sender, and the second server can be the data receiver. Here, data can refer to the computation results obtained by the sending GPU due to performing computational tasks.

[0026] For example, after completing the computation task, the sending graphics processing unit (GPU) can write the computation result into its memory via its internal bus. This memory could be Double Data Rate Random Access Memory (DDR). Then, the GPU needs to send the computation result from its DDR memory to the buffer area of ​​its RDMA network interface card (NIC) via a Peripheral Component Interconnect Express (PCIe) bus. The RDMA NIC can then combine the computation result in its buffer with information such as the receiver's RDMA NIC address and the GPU's buffer address to assemble a data packet. Finally, the RDMA NIC can transmit the data packet to the receiver's RDMA NIC's buffer area via a switch.

[0027] The receiving RDMA network card can parse the received data packet to obtain the calculation results. Then, the receiving RDMA network card can write the calculation results to the receiving double-database memory corresponding to the receiving graphics processing unit (GPU) via the receiving internal high-speed peripheral interconnect bus. Afterwards, the computing unit of the receiving GPU can read the stored calculation results from the receiving double-database memory.

[0028] Therefore, the scheme described above primarily relies on RDMA network cards to facilitate communication between the graphics processing units of different servers. This method requires both the sending and receiving RDMA network cards. The sending graphics processing unit and the sending RDMA network card need to perform data copying and buffering. Furthermore, the sending RDMA network card needs to combine the receiving RDMA network card's Internet Protocol (IP) address and the receiving graphics processing unit's double-rate memory address to reassemble and generate data packets, thus introducing a significant processing delay. Similarly, the receiving RDMA network card needs to parse the received data packets back into their original calculation results, also introducing a considerable processing delay.

[0029] Therefore, it is clear that the time consumed in the aforementioned data transmission process limits the computing power of the distributed AI server cluster. To address this, one solution is to use a third-party device (such as a network interface card or storage device) to directly access the double-speed memory corresponding to the graphics processing unit (GPU), i.e., the GPU's memory, via an RDMA network card. Alternatively, a third-party device can directly access the GPU's memory. Yet another solution involves optimizing the underlying data transmission mechanism described above. For example, a communication library can be used to automatically manage data transmission between GPUs, enabling simultaneous communication and computation.

[0030] However, the above method has problems. Because data transfer in an RDMA network card is asynchronous, the data in the memory area may not be updated before the data transfer is fully completed. Therefore, if a user attempts to access or use the memory area before the RDMA network card has finished transferring data, it may result in reading incomplete or outdated data. Furthermore, to ensure data integrity and consistency, users typically need to explicitly perform synchronization operations or check the RDMA network card's transfer completion status, which is inconvenient.

[0031] Therefore, although the communication library can automatically manage the data transmission operations of the RDMA network card, users still need to pay attention to the synchronization and data consistency issues. Furthermore, the transmission method based on the communication library is similar to the data transmission scheme described above; the additional processing latency introduced by the RDMA network card in packet assembly and decompression still exists.

[0032] In other solutions, data transfer between the Central Processing Unit's (CPU) memory space and the memory of devices interconnected with the CPU can be achieved based on memory interconnect protocols to maintain data consistency. For example, the memory interconnect protocol could be the Fast Compute Link Protocol (CXL). The CXL can include three sub-protocols: CXL.I / O, CXL.cache, and CXL.mem. Based on these three sub-protocols, the host and devices that need to share memory resources (such as accelerators and memory expanders) can access memory with low latency and ensure data consistency. The CXL specification defines three types of devices suitable for the CXL protocol: Type 1 devices, Type 2 devices, and Type 3 devices.

[0033] For example, if the main memory of the central processing unit has a need for caching data, it can be called a first-type device. Therefore, this first-type device only needs to be configured with the CXL.I / O sub-protocol and the CXL.cache sub-protocol.

[0034] For example, a graphics processing unit (GPU) that has memory and needs to interconnect with a central processing unit (CPU) can be called a second-type device. This second-type device can be configured with the CXL.I / O sub-protocol for CPU configuration. Furthermore, it can be configured with the CXL.cache sub-protocol to allow the GPU to access the CPU's memory. Additionally, it can be configured with the CXL.mem sub-protocol for CPU access to its memory.

[0035] For example, the third type of device is compatible with the CXL.I / O sub-protocol to configure memory buffers such as memory expansion cards. The third type of device is also compatible with the CXL.I / O sub-protocol so that the central processing unit can access the stored data in the memory buffer. It should be noted that the memory of the memory buffer can provide two operating modes with the assistance of the fast computation link controller. The fast computation link controller can be used to parse incoming fast computation link data packets, and after parsing the information in the data packets (e.g., address and length), the memory manager can perform load / store operations on the extended memory at the corresponding address.

[0036] In the first operating mode, the Fast Compute Link protocol enables devices (such as memory buffers or graphics processing units) to directly map to the memory of devices like the central processing unit (CPU) for more efficient memory access. This mode allows applications to bypass the operating system's memory management layer (such as the page cache layer) and interact directly, reducing latency and improving performance. Therefore, devices supporting the Fast Compute Link protocol must be able to handle the memory mapping and access mechanisms of the Fast Compute Link to achieve direct access in the first operating mode.

[0037] In the second operating mode, the memory buffer's memory is treated as part of the system memory, and accelerators such as the central processing unit or graphics processing unit that support fast computation link protocols can access the memory buffer's memory just like accessing local random access memory (RAM).

[0038] It should be noted that since the third type of device is mainly used as extended memory, it is difficult to actively trigger the following two types of data transfer operations: 1) It is difficult to actively trigger small-scale data transfer operations such as load / store based on the CXL.mem sub-protocol; 2) It is difficult to actively trigger large-scale data transfer operations such as DMA (direct memory access) based on the CXL.I / O sub-protocol.

[0039] Furthermore, the first and second type devices can interact with the third type device via the Load / Store function, assisted by the central processing unit supporting the three sub-protocols of the fast computing link and its fast computing link root port. In addition, the central processing unit supporting the three sub-protocols of the fast computing link can also support data interaction with the third type device via DMA capabilities.

[0040] Furthermore, for data interaction between two third-type devices (defined as a sending third-type device and a receiving third-type device), there are two methods. For example, in the first method, the central processing unit (or second-type device) supporting the three sub-protocols of the fast calculation link issues a Load instruction to load data from the extended memory of the sending third-type device. Then, the central processing unit (or second-type device) supporting the three sub-protocols continues to execute a Store instruction to store the data in the extended memory of the receiving third-type device. As another example, in the second method, the central processing unit supporting the three sub-protocols of the fast calculation link issues a DMA transfer instruction to transfer data from the extended memory of the sending third-type device to the extended memory of the receiving third-type device using DMA functionality.

[0041] Building upon this, some solutions have proposed prototype systems based on the memory decomposition of the fast computing link (hereinafter collectively referred to as Direct Fast Computing Link). With the server supporting the fast computing link protocol, the interconnection methods between the server and third-type devices in Direct Fast Computing Link technology are divided into the following two types:

[0042] In the first approach, devices supporting the Fast Compute Link protocol communicate directly with third-type devices. For example, a central processing unit (or a second-type device) in a device supporting the Fast Compute Link protocol that supports all three sub-protocols of the Fast Compute Link can directly access the extended memory of the third-type device via the Fast Compute Link bus.

[0043] The second approach involves devices supporting the Fast Compute Link protocol communicating with third-type devices via a Fast Compute Link switch. For example, a central processing unit (or a second-type device) within a device supporting the Fast Compute Link protocol, which supports all three sub-protocols of the Fast Compute Link protocol, interacts with a third-type device (interconnected to the Fast Compute Link switch based on sub-protocols such as CXL.mem and CXL.I / O) using a Load / Store method.

[0044] In the aforementioned Direct fast computing link scheme, the Direct fast computing link technology is not applicable when a graphics processing unit that does not support the fast computing link related protocols communicates with other graphics processing units across hosts.

[0045] Based on the above, it is clear that graphics processing units (GPUs) are widely used in complex deep learning applications such as large models. Consequently, some vendors deploy massive numbers of commercial GPUs on their servers to accelerate applications, resulting in a huge amount of data. This necessitates frequent data transfers between GPUs across server nodes.

[0046] Although various commercial graphics processing unit (GPU) manufacturers have proposed their own GPU interconnection protocols to accelerate data transmission between their GPUs, these protocols are only applicable to GPUs developed by each manufacturer for cloud service providers. This leads to communication difficulties between GPUs from different manufacturers. Because the sending GPU and the sending RDMA network card need to perform data copying and buffering, and the sending RDMA network card needs to combine the receiving RDMA network card's IP address and the receiving GPU's memory address to reassemble data packets, a significant processing delay is introduced before transmission.

[0047] Similarly, the receiving RDMA network card needs to parse the received data packet back into the original calculation result, which also introduces a relatively long processing delay. Although the other technologies described above have also optimized the cross-node communication method of RDMA-assisted graphics processing units, users still need to pay attention to the issues of transmission synchronization and data consistency.

[0048] Furthermore, although the Fast Compute Link Protocol (CXL.mem) is an industry-supported cache coherent interconnect protocol for processors, memory expansion, and accelerators, it is difficult for accelerators such as graphics processing units (GPUs) that do not support CXL.mem to directly use sub-protocols such as CXL.mem to transfer data stored in their double-rate memory to other GPUs or central processing units across server nodes.

[0049] In view of this, this application proposes a server, data transmission method and distributed system for cross-node communication of graphics processing units in distributed deep learning scenarios, so as to enable a second type of device that supports the fast computation link protocol to transmit data to a third type of device that does not support the fast computation link protocol, and reduce data transmission latency.

[0050] Figure 1A A schematic diagram of a distributed system according to an embodiment of this application is shown.

[0051] like Figure 1AAs shown, the distributed system of this embodiment may include multiple nodes. For example, the multiple nodes may include a first server and a second server, but it should be understood that this application is not limited thereto, and the distributed system of this application may also include other nodes. The multiple nodes in the distributed system can perform cross-node communication, specifically, cross-node communication can be performed via a fast computing link switch. For example, the fast computing link switch may implement data transmission based on the fast computing link protocol.

[0052] Furthermore, the following explanation will use the first server and the second server as examples.

[0053] Figure 1B A flowchart of a data transmission method according to an embodiment of this application is shown.

[0054] like Figure 1B As shown, the data transmission method of this embodiment is applied to the first server and may include operations S111~S115.

[0055] In operation S111, the first graphics processing unit writes the data to be transmitted into the double-rate memory according to the first address, and when the data writing is completed, it triggers the first fast computing link controller to detect the first interrupt event of the first address being written into the double-rate memory.

[0056] In operation S112, the first fast computing link controller encapsulates the first interrupt event and writes it into the first central processing unit, so that the first address is written into the first central processing unit.

[0057] In operation S113, the first interrupt handler is triggered to modify the value of the memory monitoring register in the second processor.

[0058] In operation S114, the first central processing unit triggers the first interrupt handler to modify the value of the memory monitoring register in the first central processing unit.

[0059] In operation S115, if the application-state polling thread associated with the first graphics processing unit in the first central processing unit detects that the value of the memory monitoring register has been modified, the polling thread is interrupted, and the second address is queried according to the first address. A transmission instruction based on the fast computing link protocol is issued to transmit the data to be transmitted to another server according to the second address through the fast computing link switch. This enables the other server to call the second graphics processing unit of the other server to read the data to be transmitted from the memory expansion device when it detects that the data to be transmitted has been written to the memory expansion device of the other server.

[0060] Figure 1C A flowchart of a data transmission method according to another embodiment of this application is shown.

[0061] like Figure 1C As shown, the data transmission method of this embodiment is applied to the second server and may include operations S121~S126.

[0062] In operation S121, the second central processing unit receives data to be transmitted from another server and writes the data to be transmitted into the memory expansion device according to the second address.

[0063] During operation S122, when data writing is completed, a second interrupt event is triggered to monitor the second address being written to extended memory.

[0064] In operation S123, the second fast computing link controller encapsulates the second interrupt event and writes it into the second central processing unit, so that the second address is written into the second central processing unit.

[0065] During operation S124, the second central processing unit triggers the second interrupt handler to modify the value of the memory monitoring register in the second central processing unit.

[0066] In operation S125, if the application-state polling thread associated with the second graphics processing unit in the second central processing unit detects that the value of the memory monitoring register has been modified, the polling thread is interrupted, and the second graphics processing unit is invoked to read the data to be transmitted from the extended memory according to the second address.

[0067] The following combination Figure 1D The data transmission method described above in this application will be further explained.

[0068] Figure 1D A schematic diagram of a first server and a second server according to an embodiment of this application is shown.

[0069] like Figure 1D As shown, the first server may include a first graphics processing unit and a first central processing unit. For example, the first graphics processing unit may be used to perform computational tasks to obtain computational results. In some embodiments, the first graphics processing unit may use the computational results as data to be transmitted from the first server to the second server. It should be noted that the first graphics processing unit may be a processor that supports memory interconnect protocols (such as the fast computation link protocol described above), so that the first graphics processing unit can communicate with other devices based on the three fast computation link sub-protocols described above.

[0070] Specifically, the first graphics processing unit may include a first memory. For example, the first graphics processing unit can write the data to be transferred into the first memory at a first address. The first address refers to the address of the data to be transferred in the first memory, which is also a zero-copy address. In this way, the first graphics processing unit can directly write the data to be transferred into the first memory based on the zero-copy channel. This process does not require an additional data forwarding process, reducing the latency of the data writing process.

[0071] The first graphics processing unit may further include a first fast computation link controller and a first memory. The first memory may be implemented based on double-rate memory. The first fast computation link controller can manage the data in the first memory. For example, the first fast computation link controller can monitor the stored data in the first memory. The first fast computation link controller can be connected to the first central processing unit. Thus, when it is detected that data to be transmitted has been written to the first memory, the first fast computation link controller can write the first address to the first central processing unit. Furthermore, when the first graphics processing unit completes the data writing, it can trigger the first fast computation link controller to detect a first interrupt event where the first address is written to the first memory. Then, the first fast computation link controller can encapsulate the first interrupt event and write it to the first central processing unit, so that the first address is written to the first central processing unit. For example, the interrupt event described in this application may be an extended message signaled interrupt (MSI-X) event. Specifically, the first fast computation link controller can encapsulate the MSI-X into a transaction layer packet (TLP) and write the TLP to the first central processing unit.

[0072] The first central processing unit (CPU) can trigger a first interrupt handler to modify the value of the memory monitoring register in the CPU. The CPU may be deployed with an application-mode polling thread supporting three sub-protocols of the fast computation link. This thread, associated with the first graphics processing unit (GPU), interrupts the polling thread when it detects that the memory monitoring register value has been modified. Therefore, since the CPU pre-stores the association between the memory addresses of the first memory and the memory addresses of the memory expansion device of the second server, upon detecting that the first address has been written to the CPU, the CPU can query the second address based on this association; this second address is the memory address of the memory expansion device of the second server. For example, the CPU may be deployed with an interrupt controller. This interrupt controller can immediately trigger a predetermined procedure (e.g., an interrupt handler) to modify the value of the memory monitoring register in the CPU to the first address when the TLP is written to the interrupt controller of the CPU. Specifically, the first central processing unit can be deployed with a polling thread supporting the three sub-protocols of the fast computing link. This thread can immediately interrupt polling upon detecting a change in the value of the memory monitoring register to the first address, and based on the first address, query the second address of the second server supporting the memory interconnect protocol for receiving the data to be transmitted, within the aforementioned association relationships. Furthermore, it issues a transmission command based on the memory interconnect protocol to transmit the data to be transmitted to the second server according to the second address via the fast computing link switch.

[0073] The first central processing unit (CPU) can support a memory interconnect protocol. Thus, the CPU can interconnect with a fast computing link switch based on the memory interconnect protocol, and the fast computing link switch can interconnect with a second server based on the memory interconnect protocol. On this basis, the CPU can transmit data to be transferred to the second server according to a second address via the fast computing link switch. Specifically, the CPU can send a Load / Store instruction based on the CXL.mem sub-protocol (or a DMA instruction based on the CXL.I / O sub-protocol) to the fast computing link switch to transmit data to be transferred from the memory region corresponding to the first address in the first memory supporting the fast computing link protocol to the second server via the fast computing link switch through a cross-node high-speed data transmission channel based on the fast computing link protocol.

[0074] The second server may include a memory expansion device. This memory expansion device can be a memory buffer. Specifically, it can be a memory buffer that supports a fast computation link protocol, such as a memory buffer that supports the CXL.I / O sub-protocol and the CXL.mem sub-protocol, i.e., the third type of device described above. The memory expansion device may include extended memory. For example, data to be transmitted may be written to the extended memory of the memory expansion device at a second address. That is, the second address may be the extended memory address of the memory expansion device, which is also the starting address of the zero-copy memory.

[0075] The second server may further include a second graphics processing unit (GPU). The second GPU can perform computational tasks to obtain computational results. However, this embodiment is not limited to this; the second GPU can also perform computational tasks based on the computational results of processors on other nodes to obtain computational results. The first GPU works similarly and will not be described in detail here. It should be noted that the second GPU may not support memory interconnect protocols, but it can be deployed with a DMA engine to access the memory of the memory extension device based on zero-copy technology. The memory interconnect protocol may include the fast computation link protocol described above. The second GPU may refer to a GPU that does not deploy a protocol stack of the fast computation link protocol. For example, such a GPU may have difficulty parsing transaction information specific to the fast computation link protocol, such as cache consistency requests and memory pooling instructions. However, such a GPU can read and write data to a fixed memory region divided by the corresponding central processing unit based on its own deployed DMA engine.

[0076] Thus, the second graphics processing unit can connect to the memory expansion device and read the data to be transferred from the expanded memory of the memory expansion device. Then, the second graphics processing unit can perform computational tasks based on the data to be transferred to obtain the corresponding computational results. Furthermore, the second graphics processing unit can read the data to be transferred from the expanded memory of the memory expansion device based on the high-speed concurrent read / write characteristics and zero-copy channel of the DMA engine. This process does not require the introduction of an additional forwarding process (such as the data forwarding process of the central processing unit), reducing the latency of the data writing process.

[0077] The second server may also include a second central processing unit (CPU). For example, the second CPU supports a memory interconnect protocol. Thus, the second CPU can interconnect with a high-speed compute link switch based on the memory interconnect protocol, and the second CPU can also connect to a second graphics processing unit (GPU) and a memory expansion device. Based on this, the second CPU can receive data to be transmitted from the first GPU of the first server and write the data to be transmitted to the memory expansion device according to a second address.

[0078] The memory expansion device may further include a second fast computing link controller connected to the second central processing unit. The second fast computing link controller can write a second address to the second central processing unit upon detecting that data to be transmitted has been written to the second memory, so that the second central processing unit can monitor the write status of the data to be transmitted to the expanded memory of the memory expansion device. The second memory can be implemented based on the expanded memory of the memory expansion device. Thus, the second central processing unit can, upon detecting that data to be transmitted has been written to the expanded memory of the memory expansion device, invoke the second graphics processing unit to read the data to be transmitted from the expanded memory of the memory expansion device.

[0079] Specifically, the second fast computing link controller can manage the data in the extended memory of the memory expansion device. For example, the second fast computing link controller can monitor the stored data in the extended memory of the memory expansion device. The second fast computing link controller can be connected to the second central processing unit. Thus, when it is detected that the data to be transmitted has been written to the extended memory of the memory expansion device, the second fast computing link controller can write the second address to the second central processing unit. For example, when the data writing is completed, the second central processing unit can trigger a second interrupt event to detect that the second address has been written to the second memory. The second fast computing link controller can then encapsulate the second interrupt event and write it to the first central processing unit, so that the second address is written to the second central processing unit. The second central processing unit can trigger a second interrupt routine to modify the value of the memory monitoring register in the second central processing unit. When the application-state polling thread associated with the second graphics processing unit in the second central processing unit detects that the value of the memory monitoring register has been modified, the polling thread is interrupted, and the second graphics processing unit is invoked to read the data to be transmitted from the second memory according to the second address.

[0080] Specifically, when the second fast computing link controller receives a write operation instruction via a high-speed data transmission channel based on the fast computing link protocol and zero-copy technology, the second fast computing link controller can encapsulate the MSI-X into a TLP while writing the data to be transmitted into the extended memory of the memory expansion device, and write the TLP into the second central processing unit via the fast peripheral component interconnect bus.

[0081] Furthermore, upon receiving a TLP (Transmission Limit Point) written to the interrupt controller of the second central processing unit, the interrupt controller can immediately trigger a predetermined program (e.g., an interrupt handler) to modify the value of the memory monitoring register in the second central processing unit to the second address. Specifically, the second central processing unit can be deployed with an application-mode polling thread supporting three sub-protocols of the fast computation link. This thread can immediately interrupt polling upon detecting that the value of the memory monitoring register has changed to the second address, and call the second graphics processing unit to read the data to be transmitted from the extended memory of the memory expansion device. This completes the low-latency, high-speed cross-node data transmission process between the first graphics processing unit supporting the memory interconnect protocol and the second graphics processing unit not supporting the memory interconnect protocol.

[0082] Based on this, when it is detected that the first graphics processing unit supporting the memory interconnect protocol writes the data to be transmitted into the first memory of the first graphics processing unit according to the first address, a first interrupt event is triggered. Then, the first fast computing link controller can immediately encapsulate the first interrupt event and write it into the first central processing unit, so that the first address is written into the first central processing unit, thereby causing the first central processing unit to trigger the first interrupt handler, and then modify the value of the memory monitoring register in the first central processing unit.

[0083] If the application-state polling thread associated with the first graphics processing unit in the first central processing unit detects that the value of the memory monitoring register has been modified, it can immediately interrupt the polling thread, query the second address based on the first address, and issue a transmission command based on the memory interconnect protocol to transmit the data to be transmitted to the second server according to the second address via a fast-calculated link switch. Subsequently, if the second central processing unit of the second server detects that the data to be transmitted is written to the extended memory of the memory expansion device according to the second address, it can immediately trigger a second interrupt event, encapsulate the second interrupt event, and write it to the first central processing unit so that the second address is written to the second central processing unit. The second central processing unit triggers a second interrupt handler to modify the value of the memory monitoring register in the second central processing unit. When the application-state polling thread associated with the second graphics processing unit in the second central processing unit detects that the value of the memory monitoring register has been modified, the polling thread is interrupted, and the second graphics processing unit, which does not support the memory interconnect protocol, is invoked to read the data to be transmitted from the extended memory of the memory expansion device according to the second address. This application saves the time required for processors on different nodes to use network cards to assemble and unpack transmitted data to achieve cross-node communication, enabling efficient cross-node communication between the first graphics processing unit that supports the memory interconnect protocol and the second graphics processing unit that does not support the memory interconnect protocol.

[0084] In this embodiment, before executing the data transmission process described above, the first server can partition the double-speed memory to obtain storage space based on the storage space required for the data to be transmitted; and associate the first address of the storage space with the identifier of the data to be transmitted. Thus, the storage space associated with the first address can be used as first memory. Similarly, the second server can partition the extended memory of the memory expansion device to obtain storage space based on the storage space required for the data to be transmitted; and associate the second address of the storage space with the identifier of the data to be transmitted. Thus, the storage space associated with the second address can be used as second memory. The following is combined with... Figure 1D Please provide a detailed explanation.

[0085] Continue to refer to Figure 1D The first server may include a first graphics processing unit and a first central processing unit. The first graphics processing unit may include double-speed memory and a first fast computing link controller. The first fast computing link controller may also divide the memory of the first graphics processing unit into storage space according to the storage space required for the data to be transmitted, and may associate the first address of the storage space with the identifier of the data to be transmitted.

[0086] Specifically, during the distributed deep learning model compilation phase, the first central processing unit (CPU) uses zero-copy technology to call the first graphics processing unit (GPU) to divide the double-speed memory into storage spaces. This storage space is used as the first memory, i.e., zero-copy memory, so that the GPU can write data to it. The physical starting address of this first memory can be recorded as `send_mem_addr`, i.e., the first address. The size of the first memory can be the same as the amount of data to be transmitted by the GPU during the model compilation phase (i.e., the storage space required for the transmitted data). Furthermore, if there are multiple copies of data to be transmitted in the model, the GPU can divide the storage space into the same number of copies. After dividing the storage space, the CPU can use a hash table to associate the first address of the storage space with the identifier of the data to be transmitted. Specifically, the hash table can be a sending information table named `comm_send_mem_table`. The identifier of the data to be transmitted can include the communication process number `comm_tag`. The first address can include the starting address of the zero-copy memory, `send_mem_addr`. In this way, the identifier of the data to be transmitted can be used as the key and the first address as the value, so as to determine the corresponding first address based on the identifier of the data to be transmitted carried by the data to be transmitted, and write the data to be transmitted into the corresponding first memory.

[0087] Continue to refer to Figure 1DThe second server may include a memory expansion device, a second graphics processing unit, and a second central processing unit. The memory expansion device may include extended memory and a second fast computing link controller. The second central processing unit can partition the extended memory of the memory expansion device to obtain second memory based on the storage space required for the data to be transmitted, and can associate the second address of the storage space with the identifier of the data to be transmitted.

[0088] Specifically, during the distributed deep learning model compilation phase, the second central processing unit (CPU), based on zero-copy technology, calls the second graphics processing unit (GPU) to allocate storage space from the extended memory of the memory expansion device. This storage space can be used as second memory, i.e., zero-copy memory, so that the second GPU can write data to the second memory of the memory expansion device. The physical starting address of this second memory can be recorded as recv_mem_addr, i.e., the second address. The size of the second memory can be the same as the amount of data to be transmitted by the second GPU during the model compilation phase (i.e., the storage space required for the transmitted data). Furthermore, if there are multiple copies of data to be transmitted in the model, the second GPU can allocate the same number of copies of the second memory as zero-copy memory. After allocating the second memory, the second CPU can use a hash table to associate and store the second address of the second memory with the identifier of the data to be transmitted. Specifically, this hash table can be a receive information table named comm_recv_mem_table. The identifier of the data to be transmitted can include the communication process number comm_tag. The second address can include the starting address of the zero-copy memory, recv_mem_addr. In this way, the identifier of the data to be transmitted can be used as the key and the second address as the value, so as to determine the corresponding second address based on the identifier of the data to be transmitted and write the data to be transmitted into the corresponding second memory.

[0089] It should be noted that when there are multiple communication processes during the compilation phase of a distributed deep learning model, each communication process in the entire model can be numbered and denoted as comm_tag. The value type is an integer, and the value starts from 0 and increments sequentially. That is, for multiple data to be transmitted generated in the multiple communication processes of the model, sequentially incrementing data identifiers can be generated.

[0090] Furthermore, after partitioning the first memory and the second memory, the second server can also transmit the second address and the data identifier to be transmitted to the first graphics processing unit deployed on the first server via a high-speed computing link switch. Upon detecting that the second address and the data identifier to be transmitted from the memory expansion device have been written into the memory of the first graphics processing unit, the first server can write the second address and the data identifier to be transmitted from the memory expansion device into the first central processing unit, then query the first address based on the data identifier to be transmitted, and associate and store the first address and the second address together.

[0091] For example, the second central processing unit can transmit the second address and the data identifier to be transmitted to the first graphics processing unit deployed on the first server via a fast computing link switch. Specifically, the second central processing unit can send a Load / Store instruction based on the CXL.mem sub-protocol (or a DMA instruction based on the CXL.I / O sub-protocol) to write the data identifier to be transmitted from the received information table and the second address of the second memory corresponding to the data identifier to be transmitted into the memory space (named comm_recv_info_mem) in the extended memory of the memory expansion device used to store information from the second server, via the fast computing link switch. Then, the first fast computing link controller can write the second address and the data identifier to be transmitted of the memory expansion device into the first central processing unit if it detects that the second address and the data identifier to be transmitted of the memory expansion device have been written into the memory of the first graphics processing unit. Specifically, the first fast computing link controller can monitor the MSI-X written to the aforementioned memory space. The MSI-X can record the aforementioned second address and the data identifier to be transmitted. Subsequently, after detecting the aforementioned MSI-X, the first fast computing link controller can encapsulate the second address and the data identifier to be transmitted recorded by the MSI-X into a TLP, and write the TLP to the first central processing unit via the fast peripheral component interconnect bus.

[0092] The first central processing unit (CPU) can query the first address based on the identifier of the data to be transmitted and associate and store the first address and the second address. Specifically, when the interrupt controller of the first CPU detects that the TLP is written to the interrupt controller, it can immediately trigger a predetermined program (e.g., an interrupt handler), query the corresponding first address in the receive information table based on the identifier of the data to be transmitted in the TLP, and associate and store the first address and the second address in an information table (named send_recv_info_table), which can be a hash table. The first address value can be used as the key, and the second address can be used as the value, so as to query the second address based on the first address, thereby facilitating communication between the first CPU and the second CPU. Based on this, by establishing the mapping relationship between the first address and the second address during the pre-compilation stage, dynamic addressing and handshaking operations are avoided during subsequent inference / training. Thus, this application can decouple the data transmission control flow from the graphics processing unit (GPU) execution flow, thereby achieving a low-overhead, high-concurrency asynchronous communication process of "write-to-trigger, automatic forwarding" in the actual communication stage. This achieves asynchronous and efficient execution of the communication process, significantly improving the startup efficiency and system scalability of cross-node GPU communication. The overall solution does not require native support for the Fast Compute Link protocol in the graphics processing unit (GPU). It can leverage the third type of device (i.e., memory expansion device) in the Fast Compute Link ecosystem and the MSI-X interrupt mechanism to establish a low-latency, high-bandwidth heterogeneous communication link with zero software intervention from the central processing unit. This design is compatible with the GPU architectures of some vendors and has good engineering feasibility, making it particularly suitable for accelerating the distributed inference and training of large models in a heterogeneous GPU hybrid deployment environment on cloud platforms.

[0093] The above description of the zero-copy memory partitioning process in this application has been completed. Next, this application will describe the specific structures of the first server and the second server in conjunction with the following figures. First, for the first server of this application, the first central processing unit may include a first composite unit. For example, the first composite unit can be connected to the first graphics processing unit via a first memory interconnect channel. For example, the first memory interconnect channel can be a communication link based on a memory interconnect protocol. Specifically, it can be a communication link based on the CXL.I / O sub-protocol, distinct from a bus connection, or a communication link based on the CXL.mem sub-protocol. The following, in conjunction with… Figure 2A Please provide a detailed explanation.

[0094] Figure 2A A schematic diagram of a first server according to another embodiment of this application is shown.

[0095] like Figure 2AAs shown, the first server in this embodiment may include a first graphics processing unit and a first central processing unit. The first graphics processing unit may include a computing unit, a first memory, and a first fast computing link controller. The first central processing unit may include a first root composite unit.

[0096] The computing unit can be connected to the first fast computing link controller via a second double-rate bus, and the first fast computing link controller can be connected to the first memory via the first double-rate bus. The second and first double-rate buses can be buses based on a memory interconnect protocol. The first graphics processing unit can transmit data to be transmitted from the computing unit to the first fast computing link controller via the second double-rate bus, and then write the data to be transmitted to the first memory via the first double-rate bus. In this way, high-speed writing of data to be transmitted can be achieved based on a memory interconnect protocol. The first fast computing link controller can parse data packets from the computing unit to generate data to be transmitted. The data packets may include the calculation results of the computing unit. Based on this, the first fast computing link controller can parse the data packets to obtain the calculation results. Then, the first fast computing link controller can encapsulate the calculation results to generate data to be transmitted based on a fast computing link protocol.

[0097] Furthermore, the first composite unit may include a first fast computing link root port and a second fast computing link root port. For example, the second fast computing link root port and the fast computing link switch may be connected via the first fast computing link external interconnect bus. The first fast computing link root port and the second fast computing link root port may be connected via the first fast computing link internal interconnect bus. The first fast computing link external interconnect bus and the first fast computing link internal interconnect bus may be buses based on a memory interconnect protocol.

[0098] Based on this, the first central processing unit can read the data to be transmitted from the first memory, transmit the data to the first fast computing link root port via the first memory interconnect channel, transmit it to the second fast computing link root port via the internal interconnect bus of the first fast computing link, and transmit it to the second server via the fast computing link switch via the external interconnect bus of the first fast computing link. Because the first composite unit is a hardware path module, when the first graphics processing unit interacts with the memory expansion device, there is no need to call the first central processing unit for software intervention, thereby saving the time consumed by the first central processing unit in processing the data to be transmitted during data transmission.

[0099] Furthermore, compared to the high-speed peripheral component interconnect bus, the first double-speed bus and the second double-speed bus have larger transmission bandwidth and lower transmission latency. Moreover, the computing unit and the first memory are both located in the first graphics processing unit, which belongs to the second type of device of the high-speed computing link, and do not occupy the data transmission channel other than the first graphics processing unit, thus saving channel resources.

[0100] The specific structure of the first server has been described above. The specific structure of the second server will be described below with reference to the attached diagram.

[0101] Figure 2B A schematic diagram of a second server according to another embodiment of this application is shown.

[0102] like Figure 2B As shown, the second server may include a second graphics processing unit, a second central processing unit, and a memory expansion device. The second graphics processing unit may include a computing unit and corresponding double-speed memory. The second central processing unit may include a second root composite unit. For example, a second memory interconnect channel is configured between the memory expansion device and the second root composite unit. For example, the second memory interconnect channel may be a communication link based on a memory interconnect protocol. Specifically, it may be a communication link based on the CXL.I / O sub-protocol, distinct from a bus connection, or a communication link based on the CXL.mem sub-protocol. The second root composite unit and a fast computing link switch may be connected via a second fast computing link external interconnect bus. For example, the second fast computing link external interconnect bus may be a bus based on a memory interconnect protocol.

[0103] Based on this, the second central processing unit can pre-transmit the second address and the data identifier to be transmitted to the second composite unit via the second memory interconnect channel. Then, the second composite unit can transmit the second address and the data identifier to be transmitted to the fast computing link switch via the second fast computing link external interconnect bus, and then transmit the second address and the data identifier to be transmitted to the first graphics processing unit via the fast computing link switch, so that the first central processing unit can associate and store the first address and the second address according to the data identifier to be transmitted.

[0104] Specifically, the second central processing unit transmits the second address and the identifier of the data to be transmitted to the fourth fast computing link root port via the second memory interconnect channel; and transmits them to the third fast computing link root port via the second fast computing link internal interconnect bus; and transmits them to the fast computing link switch via the second fast computing link external interconnect bus; and then transmits them to another server via the fast computing link switch.

[0105] Then, the first central processing unit can send the data to be transmitted to the second server based on the first address, the second address, and the identifier of the data to be transmitted stored in the association.

[0106] Furthermore, the second central processing unit can transmit the data to be transmitted from the high-speed computing link switch to the second composite unit via the second high-speed computing link external interconnect bus, and write it to the second memory of the memory expansion device according to the second address via the second memory interconnect channel. For example, if the second central processing unit detects that the second address has been written to the second graphics processing unit, it can read the data to be transmitted from the second memory of the memory expansion device according to the second address via the second memory interconnect channel, and transmit the data to be transmitted to the second graphics processing unit via the high-speed peripheral component interconnect bus.

[0107] Furthermore, the second composite unit may include a third fast computing link root port and a fourth fast computing link root port. For example, the third fast computing link root port is connected to the fast computing link switch via a second fast computing link external interconnect bus. The third fast computing link root port and the fourth fast computing link root port are connected via a second fast computing link internal interconnect bus. For example, the second fast computing link internal interconnect bus can be a bus based on a memory interconnect protocol. A second memory interconnect channel is configured between the fourth fast computing link root port and the memory expansion device. In this way, the second central processing unit can transmit the data to be transmitted from the fast computing link switch to the third fast computing link root port via the second fast computing link external interconnect bus, then transmit the data to be transmitted to the fourth fast computing link root port via the second fast computing link internal interconnect bus, and finally write the data to be transmitted to the memory expansion device via the second memory interconnect channel.

[0108] Furthermore, the second root complex unit may also include a Fast Peripheral Component Interconnect (FPCI) root port. For example, the FPCI root port and the fourth Fast Compute Link (CCL) root port are connected via an internal routing bus within the root complex. The second central processing unit (CPU) can read the data to be transferred from the extended memory of the memory expansion device via the second memory interconnect channel, upon detecting that the second address has been written to the second graphics processing unit (GPU). Then, the second CPU can transfer the data to be transferred to the fourth CCL root port, and then transfer it to the FPCI root port via the internal routing bus within the root complex. Afterward, the second CPU can transfer the data to be transferred to the second GPU via the FPCI bus. Thus, the second GPU can perform computational tasks based on the data to be transferred.

[0109] Therefore, since the second composite unit is a hardware path module, when the data to be transmitted is written to the double-speed memory of the second graphics processing unit, there is no need to call the second central processing unit for software intervention, thereby saving the time consumed by the second central processing unit in processing the data to be transmitted during the data transmission process.

[0110] Building upon this, since the DMA engine deployed in the second graphics processing unit has the advantage of parallel high-speed transmission, the central processing unit supporting the three sub-protocols of the fast computing link does not need to participate, and can directly read and write memory, thereby significantly improving data transmission efficiency. Furthermore, compared to the difficulty in data transmission based on the CXL.mem sub-protocol's Load / Store instruction (or the CXL.I / O sub-protocol's DMA instruction) after transferring data to DRAM (Dynamic Random Access Memory) memory modules using zero-copy technology, this application, based on the fast computing link protocol and zero-copy technology, can utilize high-speed CXL.mem and CXL.I / O sub-protocols such as the CXL.mem sub-protocol's Load / Store instruction (or the CXL.I / O sub-protocol's DMA instruction) to write the data to be transmitted into the extended memory of the second server's memory expansion device via the fast computing link switch, achieving high-bandwidth and low-latency cross-server node communication for the graphics processing unit.

[0111] Additionally, after the distributed deep learning model inference / training phase is completed, the first server can call the resource cleanup function to release the resources of the first memory pre-allocated by the first graphics processing unit, the resources of the memory monitoring register of the first central processing unit, and the stop interrupt status, etc., and the second server can call the resource cleanup function to release the resources of the second memory pre-allocated by the memory expansion device, the resources of the memory monitoring register of the second central processing unit, and the stop interrupt status, etc.

[0112] Figure 2C A schematic diagram of a first server and a second server according to another embodiment of this application is shown.

[0113] like Figure 2CAs shown, the cross-node transmission path based on the Fast Computing Link protocol can be as follows: a first graphics processing unit, a first memory interconnect channel (e.g., CXL.I / O channel or CXL.mem channel), a first fast computing link root port, a first fast computing link internal interconnect bus, a second fast computing link root port, a first fast computing link external interconnect bus, a first switch port, a second switch port, a second fast computing link external interconnect bus, a third fast computing link root port, a second fast computing link internal interconnect bus, a fourth fast computing link root port, a second memory interconnect channel, and a memory expansion device. The fast computing link switch can transmit data based on the Fast Computing Link protocol. The first and second switch ports can be fast computing link root ports.

[0114] Although the first composite unit is located within the first central processing unit and the second composite unit is located within the second central processing unit, since the first and second composite units are hardware path modules, there is no need to call the first or second central processing unit for software intervention when the second fast computing link root port interacts with the first switch interface or the third fast computing link root port interacts with the second switch interface. This saves the processing time of the first and second central processing units for the data to be transmitted during cross-node data transmission.

[0115] Furthermore, since both the devices between the first graphics processing unit and the memory expansion device support the CXL.mem and CXL.I / O sub-protocols, small-scale data interaction can be performed using the Load / Store method based on the CXL.mem sub-protocol, and large-scale concurrent data interaction can be performed using the DMA method based on the CXL.I / O sub-protocol. Therefore, this application provides an interaction method based on two sub-protocols, improving interaction flexibility. Moreover, compared to the cross-node computing technology for graphics processing units assisted by RDMA network cards described above, this application's solution saves the time consumed by data packet assembly and depacketization, achieving shorter processing latency and greater data transmission bandwidth across server nodes.

[0116] In this embodiment, after the data to be transmitted is transferred from the first graphics processing unit to the memory expansion device based on the fast computing link protocol, the second graphics processing unit can directly read the data. During this reading process, a high-speed direct data transmission channel is established between the second graphics processing unit and the memory expansion device, which does not require software intervention from the central processing unit (CPU). In this process, the second graphics processing unit's computing unit does not need to call its internal double-speed memory, the CPU does not require software intervention from the CPU supporting the three sub-protocols of the fast computing link during forwarding, and the zero-copy memory of the third type of device (i.e., the memory expansion device) is written directly to the local machine without caching. This improves data transmission efficiency. Furthermore, the second graphics processing unit can directly read the extended memory of the memory expansion device based on the DMA engine; this process also does not require the participation of the CPU supporting the three sub-protocols of the fast computing link, significantly improving data transmission efficiency.

[0117] Figure 3A A schematic diagram of a first server according to another embodiment of this application is shown.

[0118] like Figure 3A As shown, in one embodiment of this application, the path by which the first graphics processing unit writes the data to be transmitted into the first memory can be as follows: the computing unit of the first graphics processing unit, the second double-speed bus (for illustrative purposes only), Figure 3A (not shown in the diagram), first fast computing link controller, first double-rate bus (for illustration purposes, ... ... Figure 3A (Not shown in the image) and a first memory. Thus, the first central processing unit can read data to be transferred from the first memory via a first memory interconnect channel.

[0119] Since the first composite unit is a hardware path module, when the first graphics processing unit interacts with the first memory, there is no need to call the first central processing unit (i.e., the integrated memory controller (IMC)) for software intervention, thereby saving the time consumed by the first central processing unit in processing the data to be transmitted during the data transmission process.

[0120] Since the first graphics processing unit supports the CXL.I / O sub-protocol, after the first graphics processing unit allocates the first memory, based on the characteristic of zero-copy memory that does not require the first central processing unit to perform forwarding intervention through software, a high-speed direct data transmission channel (i.e., the second double-rate bus and the first double-rate bus) can be realized between the first graphics processing unit and the first memory.

[0121] In another embodiment of this application, when the DRAM in the first server is used as zero-copy memory, the transmission path of the data to be transmitted can be as follows: the computing unit of the first graphics processing unit, the first memory interconnect channel, the first fast computing link root port, the internal interconnect bus of the first central processing unit, the memory channel port, the double data rate bus, and the DRAM memory module. Compared with the data transmission path described above, in this embodiment, after the data to be transmitted to the first fast computing link root port of the first composite unit, it can be transmitted to the memory channel port via the internal interconnect bus of the first central processing unit, and then directly written to the zero-copy memory in the DRAM memory module via the double data rate bus. This eliminates the need for conversion between the data packets of the Fast Peripheral Component Interconnect Protocol (FPIP) and the Fast Computing Link Protocol (CCP) via the second fast computing link root port, saving time consumed by protocol conversion. It should be noted that this method has a larger transmission bandwidth than the methods described above. However, this method struggles to transfer data to zero-copy memory in the third type of fast computing link device connected via a fast computing link switch in a receiving server that supports the fast computing link protocol, using the Load / Store instructions of the CXL.mem sub-protocol (or the DMA instructions of the CXL.I / O sub-protocol). Therefore, this application does not allocate the first memory within the DRAM memory module, but instead chooses to allocate the first memory within the memory of the first graphics processing unit, to achieve cross-node data transfer based on the first scheme described above. Based on this, this application overcomes the limitations of traditional communication paths on the protocol compatibility of graphics processing units, thereby achieving efficient cross-node data transfer for graphics processing units.

[0122] Figure 3B A schematic diagram of a second server according to another embodiment of this application is shown.

[0123] like Figure 3B As shown, in one embodiment of this application, the path by which the second graphics processing unit reads the data to be transmitted from the memory expansion device can be as follows: the second memory of the memory expansion device, the fourth fast computing link root port, the fast peripheral component interconnection root port, and the computing unit.

[0124] Since the second composite unit is a hardware path module, when the second graphics processing unit interacts with the memory expansion device, there is no need to call the second central processing unit (i.e., the IMC) for software intervention, thereby saving the time consumed by the second central processing unit in processing the data to be transmitted during the data transmission process.

[0125] In this embodiment, since the memory expansion device supports the CXL.I / O sub-protocol, and the CXL.I / O channel reuses the Fast Peripheral Component Interconnect 5.0 and higher version bus (i.e., Fast Peripheral Component Interconnect bus), this application can realize a high-speed direct data transmission channel (e.g., Fast Peripheral Component Interconnect bus, root complex internal routing bus and second memory interconnect channel) between the second graphics processing unit and the memory expansion device based on the high-speed transmission characteristics of DMA and the zero-copy memory characteristic that does not require the second central processing unit to forward data through software intervention.

[0126] Furthermore, in the RDMA network card-assisted graphics processing unit cross-node computing technology described above, the data is received by the graphics processing unit's double-speed memory. However, in this application, the data to be transmitted is received by the computing unit of the second graphics processing unit. Thus, the second graphics processing unit of this application can directly interact with the second high-speed peripheral interconnect bus and the high-speed peripheral interconnect root port. This saves the time consumed by performing the read operation in the process of "the calculation result is first written to the double-speed memory, and then the computing unit of the graphics processing unit reads the calculation result from the double-speed memory."

[0127] Based on the above, this application provides a high-speed direct data transmission channel for the sender based on fast computation link and zero-copy technology: Within a sender server supporting the fast computation link protocol, an internal high-speed direct transmission channel is constructed for the sender's graphics processing unit, which also supports the fast computation link protocol. This channel, based on the fast computation link protocol, enables the sender's graphics processing unit to directly write data to locally allocated zero-copy memory, eliminating the need for software intervention and explicit caching by the central processing unit supporting the three sub-protocols of the fast computation link, thus improving data writing efficiency.

[0128] Furthermore, for the aforementioned cross-node high-speed data transmission channel based on the Fast Computing Link Protocol, once the first memory completes the data writing, the Fast Computing Link controller triggers an MSI-X interrupt. The local central processing unit supporting the three sub-protocols of the Fast Computing Link completes the lookup mapping from send_addr to recv_addr. Then, through the CXL.mem or CXL.I / O sub-protocol, the data is directly transmitted to the third type of device (i.e., memory extension device) of the receiving server supporting the Fast Computing Link Protocol via the Fast Computing Link switch, achieving high-bandwidth, low-latency cross-node data transfer.

[0129] Furthermore, based on the high-speed direct data transmission channel of the receiver using fast computing link and zero-copy technology, a high-speed DMA read channel is constructed between the graphics processing unit (GPU) that does not support the fast computing link protocol but has a DMA engine and the local fast computing link third-type device. The GPU detects the arrival of data by polling the memory monitoring register at the application layer and directly retrieves the data from the local third-type device, completing the entire communication process without software intervention from the central processing unit. For reasons similar to those described above, this application does not partition the second memory in the DRAM memory module using the integrated memory controller and memory channel port of the second central processing unit. Instead, it partitions the second memory in the extended memory of the memory expansion device to achieve cross-node data transmission based on the first scheme described above. Based on this, this application overcomes the limitations of traditional communication paths on the protocol compatibility of the GPU, thereby achieving efficient cross-node data transmission for the GPU.

[0130] Building upon this, this application proposes an optimized cross-node communication mechanism for graphics processing units (GPUs) based on fast computation link polling, further improving communication efficiency. Compared to traditional cross-node GPU communication schemes assisted by RDMA network cards, this application skips the double-rate relay, data copying, and reassembly operations between the GPU and the RDMA network card. Furthermore, the first GPU triggers an MSI-X interrupt upon writing the data to be transmitted into the first memory. The first central processing unit then automatically issues a remote write command after the interrupt, without polling or software stack intervention. Subsequently, the second GPU can promptly detect the completion of data writing through a register polling mechanism and directly read the data via DMA. The overall process eliminates steps such as cache writing, network card packet assembly / disassembly, and target address mapping in the RDMA path, significantly reducing communication latency. This is particularly suitable for high-frequency communication scenarios such as frequent cross-node transmissions using key-value cache technology in large models. While improving system bandwidth utilization and heterogeneous communication efficiency, it also overcomes the technical barriers to GPU compatibility with fast computation link protocols.

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0132] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0133] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A data transmission method applied to a server, the server comprising: First graphics processing unit and first central processing unit; The first graphics processing unit includes a first fast computation link controller and double-speed memory; characterized in that the method includes: The first graphics processing unit writes the data to be transmitted into the double-rate memory according to the first address, and when the data writing is completed, it triggers the first fast computing link controller to detect the first interrupt event of the first address being written into the double-rate memory; The first fast computing link controller encapsulates the first interrupt event and writes it into the first central processing unit, so that the first address is written into the first central processing unit. The first central processing unit (CPU) triggers a first interrupt handler to modify the value of a memory monitoring register in the CPU. If the application-state polling thread associated with the first graphics processing unit (GPU) in the CPU detects that the value of the memory monitoring register has been modified, the polling thread is interrupted, and a second address is retrieved based on the first address. A transmission instruction based on the fast computing link protocol is then issued. The first CPU is interconnected with a fast computing link switch based on a memory interconnect protocol. The fast computing link switch is interconnected with another server based on the memory interconnect protocol to transmit the data to be transmitted to the other server according to the second address via the fast computing link switch. This allows the other server to detect that the data to be transmitted has been written to a memory expansion device in the other server, and then invoke the second graphics processing unit of the other server to read the data to be transmitted from the memory expansion device. The first CPU supports the memory interconnect protocol.

2. The method according to claim 1, characterized in that, The first fast computing link controller encapsulates the first interrupt event and writes it into the first central processing unit, so that the first address is written into the first central processing unit, including: The first fast computing link controller encapsulates the first interrupt event into a data packet and writes it to the interrupt controller of the first central processing unit via the fast peripheral component interconnect bus, so that the first address is written to the first central processing unit, and the interrupt controller of the first central processing unit triggers the first interrupt handler.

3. The method according to claim 1, characterized in that, The first graphics processing unit writes the data to be transmitted into the double-rate memory according to the first address, including: The first graphics processing unit transmits the data to be transmitted to the first fast computing link controller via the first double-rate bus; then the data to be transmitted is written to the double-rate memory via the second double-rate bus.

4. The method according to claim 3, characterized in that, Before the first graphics processing unit writes the data to be transmitted into the double-rate memory according to the first address, the method further includes: The first fast computing link controller parses the data packets from the computing unit to generate the data to be transmitted.

5. The method according to claim 4, characterized in that, The first central processing unit includes: a first fast computing link root port and a second fast computing link root port; the first fast computing link root port and the second fast computing link root port are connected through a first fast computing link internal interconnect bus; a first memory interconnect channel is configured between the first central processing unit and the first graphics processing unit; The data to be transmitted is transmitted to another server according to the second address by a link switch that performs fast calculation, including: The first central processing unit reads the data to be transmitted from the double-speed memory, transmits the data to be transmitted to the first fast computing link root port via the first memory interconnect channel, transmits it to the second fast computing link port via the first fast computing link internal interconnect bus, and transmits it to the other server via the fast computing link switch via the first external fast computing link memory interconnect bus.

6. The method according to any one of claims 1-5, characterized in that: The method further includes: The first central processing unit divides the double-rate memory into a first memory according to the storage space required for the data to be transmitted; and associates the first address of the first memory with the identifier of the data to be transmitted.

7. The method according to claim 6, characterized in that: The method further includes: When the first fast computing link controller detects that the second address of the memory expansion device and the identifier of the data to be transmitted have been written into the first memory, it writes the second address of the memory expansion device and the identifier of the data to be transmitted into the first central processing unit. The first central processing unit queries the first address based on the identifier of the data to be transmitted, and associates and stores the first address and the second address.

8. A data transmission method applied to a server, the server comprising: A second central processing unit, a second graphics processing unit, and a memory expansion device, wherein the memory expansion device includes a second fast computing link controller and extended memory; characterized in that the method includes: The second central processing unit receives data to be transmitted from another server and writes the data to be transmitted into the memory expansion device according to the second address; upon completion of data writing, the second fast computing link controller is triggered to monitor a second interrupt event in which the second address is written into the expanded memory; wherein the data to be transmitted is transmitted by another server according to the method described in any one of claims 1 to 7; The second fast computing link controller encapsulates the second interrupt event and writes it into the second central processing unit, so that the second address is written into the second central processing unit; The second central processing unit triggers a second interrupt handler to modify the value of the memory monitoring register in the second central processing unit; if the application-state polling thread associated with the second graphics processing unit in the second central processing unit detects that the value of the memory monitoring register has been modified, the polling thread is interrupted, and the second graphics processing unit is invoked to read the data to be transmitted from the extended memory according to the second address.

9. The method according to claim 8, characterized in that, The second central processing unit includes: a third fast computing link root port and a fourth fast computing link root port; the third fast computing link root port is connected to a fast computing link switch via a second fast computing link external interconnect bus; the third fast computing link root port and the fourth fast computing link root port are connected via a second fast computing link internal interconnect bus; a second memory interconnect channel is configured between the fourth fast computing link root port and the memory expansion device; The second central processing unit receives data to be transmitted from another server and writes the data to be transmitted into the memory expansion device according to a second address, including: The second central processing unit transmits the data to be transmitted from another server to the third fast computing link root port via the second fast computing link external interconnect bus, to the fourth fast computing link root port via the second fast computing link internal interconnect bus, and writes it to the memory expansion device via the second memory interconnect channel.

10. The method according to claim 9, characterized in that, The second central processing unit further includes: a Fast Peripheral Component Interconnect (FPCI) root port; the FPCI root port is connected to the second graphics processing unit via a FPCI bus, and the FPCI root port is connected to the fourth Fast Compute Link (CCL) root port via a root complex internal routing bus; Calling the second graphics processing unit to read the data to be transmitted from the extended memory according to the second address includes: When the second central processing unit detects that the second address has been written to the second graphics processing unit, it reads the data to be transmitted from the extended memory via the second memory interconnect channel according to the second address; and transmits the data to be transmitted to the fourth fast computing link root port, transmits it to the fast peripheral component interconnect root port via the root complex internal routing bus; and transmits it to the second graphics processing unit via the fast peripheral component interconnect bus.

Images