server

Abstract

This application provides a server applicable to the field of server technology. The server includes: a first graphics processing unit configured to write first data to be transmitted to a first memory expansion device according to a first address; the first memory expansion device configured to, upon detecting that the first data to be transmitted has been written to the first memory expansion device, write the first address to a central processing unit; and the central processing unit configured to, upon detecting that the first address has been written to the central processing unit, query a second address based on the first address, and transmit the first data to be transmitted to another server according to the second address via a fast computation link switch, so that the other server, upon detecting that the first data to be transmitted has been written to a second memory expansion device in the other server, invokes a second graphics processing unit in the other server to read the first data to be transmitted from the second memory expansion device.

Description

Technical Field

[0001] This application relates to the field of server technology, and more specifically to a server. Background Technology

[0002] With the continuous application and promotion of neural networks, technologies based on them, such as deep learning, high-performance computing, and large models, have developed rapidly. Based on this, accelerators deployed on different servers in a distributed system need to communicate frequently. However, in related technologies, because accelerators deployed on different servers need to communicate across nodes using network interface cards (NICs), this results in relatively long latency. Summary of the Invention

[0003] In view of the above problems, this application provides a server.

[0004] According to one aspect of this application, a server is provided, including a first graphics processing unit configured to write first data to be transmitted to a first memory expansion device according to a first address; the first memory expansion device is configured to write the first address to a central processing unit when it is detected that the first data to be transmitted has been written to the first memory expansion device; the central processing unit is configured to, when it is detected that the first address has been written to the central processing unit, query a second address according to the first address, and transmit the first data to be transmitted to another server according to the second address through a fast calculation link switch, so that when the other server detects that the first data to be transmitted has been written to a second memory expansion device in the other server, it calls a second graphics processing unit in the other server to read the first data to be transmitted from the second memory expansion device.

[0005] According to an embodiment of this application, when it is detected that the first graphics processing unit writes the first data to be transmitted into the first memory according to the first address, the first memory expansion device can immediately write the first address into the corresponding central processing unit. This allows the central processing unit, upon detecting that the first address has been written into it, to immediately query the second address based on the first address and, through a fast calculation link switch, transmit the first data to be transmitted to another server according to the second address. Subsequently, when the other server detects that the first data to be transmitted has been written into the second memory expansion device according to the second address, it can immediately invoke the second graphics processing unit to read the first data to be transmitted from the second memory expansion device according to the second address. Thus, this application can save the time required for different servers to assemble and unpack transmitted data using network cards to achieve cross-node communication, enabling efficient cross-node communication between one server and another. Attached Figure Description

[0006] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, which will be explained in conjunction with the drawings.

[0007] Figure 1A A schematic diagram of a distributed system according to an embodiment of this application is shown.

[0008] Figure 1B A schematic diagram of a first server and a second server according to an embodiment of this application is shown.

[0009] Figure 1C A schematic diagram of a first server and a second server according to another embodiment of this application is shown.

[0010] Figure 1D A schematic diagram of a first server and a second server according to another embodiment of this application is shown.

[0011] Figure 2A It is shown that according to this application Figure 1B A schematic diagram of the first server in another embodiment.

[0012] Figure 2B It is shown that according to this application Figure 1B A schematic diagram of a second server according to another embodiment.

[0013] Figure 2C It is shown that according to this application Figure 1B A schematic diagram of a first server and a second server according to another embodiment.

[0014] Figure 3A It is shown that according to this application Figure 1C A schematic diagram of the first server in another embodiment.

[0015] Figure 3B It is shown that according to this application Figure 1C A schematic diagram of a second server according to another embodiment.

[0016] Figure 3C It is shown that according to this application Figure 1C A schematic diagram of a first server and a second server according to another embodiment.

[0017] Figure 4A It is shown that according to this application Figure 1D A schematic diagram of the first server in another embodiment.

[0018] Figure 4B It is shown that according to this application Figure 1D A schematic diagram of a second server according to another embodiment.

[0019] Figure 4C It is shown that according to this application Figure 1DA schematic diagram of a first server and a second server according to another embodiment.

[0020] Figure 5A It is shown that according to this application Figure 1B A schematic diagram of the first server in another embodiment.

[0021] Figure 5B It is shown that according to this application Figure 1B A schematic diagram of a second server according to another embodiment.

[0022] Figure 6A It is shown that according to this application Figure 1C A schematic diagram of the first server in another embodiment.

[0023] Figure 6B It is shown that according to this application Figure 1C A schematic diagram of a second server according to another embodiment.

[0024] Figure 7A It is shown that according to this application Figure 1D A schematic diagram of the first server in another embodiment.

[0025] Figure 7B It is shown that according to this application Figure 1D A schematic diagram of a second server according to another embodiment.

[0026] Figure 8A It is shown that according to this application Figure 1B A flowchart of the data transmission method in this embodiment.

[0027] Figure 8B It is shown that according to this application Figure 1B A flowchart of another embodiment of a data transmission method.

[0028] Figure 9A It is shown that according to this application Figure 1C A flowchart of the data transmission method in this embodiment.

[0029] Figure 9B It is shown that according to this application Figure 1C A flowchart of another embodiment of a data transmission method.

[0030] Figure 10A It is shown that according to this application Figure 1D A flowchart of the data transmission method in this embodiment.

[0031] Figure 10B It is shown that according to this application Figure 1D A flowchart of another embodiment of a data transmission method. Detailed Implementation

[0032] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0033] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0034] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0035] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0036] With the continuous application and promotion of neural networks, technologies based on them, such as deep learning, high-performance computing, and large models, have developed rapidly. For deep learning applications such as large models with increasingly complex neural network models, massive artificial intelligence (AI) server clusters are needed to provide massive computing power. These AI servers can be deployed with processors, or accelerators, to accelerate model computation. For example, accelerators can include graphics processing units (GPUs).

[0037] Because the computing power of a single AI server is limited, some solutions require multiple AI servers to collaborate on computation through network interconnection. However, in distributed AI server cluster applications, while network interconnection can effectively improve the computing power of the AI ​​server cluster, the data transmission capacity between graphics processing units (GPUs) is limited, thus still restricting the upper limit of the AI ​​server cluster's computing power. Therefore, with limited data transmission capacity, the overall computing performance of the AI ​​server cluster will decrease.

[0038] Therefore, when using distributed AI server clusters to accelerate large models or complex neural network applications, even if the number of graphics processing units is sufficient, the computing power of the distributed AI server cluster will still be limited by the large number of parameters of the neural network model deployed on it, as well as the data transmission capability of the AI ​​server cluster's graphics processing units when transmitting parameters.

[0039] For graphics processing units (GPUs) deployed on different server nodes, cross-node communication between GPUs deployed on different server nodes can be achieved through Remote Direct Memory Access (RDMA) network interface cards (NICs). For example, a server cluster may include a first server and a second server. The first server can be the data sender, and the second server can be the data receiver. Here, data can refer to the computation results obtained by the sending GPU due to performing computational tasks.

[0040] For example, after completing its computation task, the sending graphics processing unit (GPU) thread can write the computation result to the sending GPU's Double Data Rate Random Access Memory (DDR) via the sending GPU's internal bus. Then, the sending GPU needs to send the data from the sending GPU's DDR to the buffer area of ​​the sending GPU's RDMA network interface card (NIC) via the Peripheral Component Interconnect Express (PCIe) bus. The RDMA NIC can then combine the data in its buffer with information such as the receiving GPU's RDMA NIC address and the receiving GPU's buffer address to assemble a data packet. Finally, the sending GPU can transmit the data packet to the receiving GPU's RDMA NIC's buffer area via a switch.

[0041] The receiving RDMA network card can parse the received data packet to obtain the calculation results. Then, the receiving RDMA network card can write the calculation results to the receiving DDR memory corresponding to the receiving graphics processing unit (GPU). Afterwards, the computing unit of the receiving GPU can read the stored calculation results from the receiving DDR memory.

[0042] Therefore, the scheme described above primarily relies on a cross-node communication method for graphics processing units (GPUs) across different nodes, assisted by RDMA, to achieve communication between GPUs on different nodes. However, due to the introduction of the sender's and receiver's RDMA network cards (NICs) between the two communicating nodes, data copying and buffering are required between the sending GPU and the sending RDMA NIC. Furthermore, the sending RDMA NIC needs to reassemble and generate data packets by combining information such as the receiver's RDMA NIC's Internet Protocol (IP) address and the receiver GPU's DDR memory address, thus introducing a significant processing delay before transmission. Similarly, the receiving RDMA NIC needs to parse the received data packets, which also introduces a considerable processing delay.

[0043] Therefore, it is evident that the time consumed in the aforementioned data transmission process limits the computing power of the distributed AI server cluster. To address this, one solution is to directly access the DDR (DDR memory) corresponding to the graphics processing unit (GPU) via an RDMA network card through a third-party device (such as a network interface card or storage device). This method bypasses the RDMA network card's cache area (corresponding to host memory), significantly reducing data transmission latency, improving bandwidth utilization, and reducing the GPU load. Another solution achieves the same effect by directly accessing the GPU's memory through a third-party device. Yet another solution involves optimizing the underlying data transmission mechanism described above to automate cross-node GPU communication. This at least partially eliminates the need for users to manage data transmission at the application layer. For example, a communication library can automatically manage data transmission between GPUs, eliminating the need for users to write code to interconnect GPUs; they only need to operate the distributed communication interface. Furthermore, the communication library can automatically select the optimal communication path and use the direct memory access method described earlier to obtain the computation results from the graphics processing unit, reducing unnecessary steps. Also, since the communication library can integrate with existing machine learning frameworks, users do not need to perform additional configuration when using these frameworks. In addition, the communication library can enable simultaneous communication and computation, thus maximizing the utilization of graphics processing unit resources and network bandwidth, further improving the efficiency of distributed computing.

[0044] However, while the above methods can achieve a seamless user experience, problems may still arise in practical applications. For example, if user code attempts to access or use the target memory region before the RDMA transfer is complete, it may result in reading incomplete or outdated data. This is because RDMA transfers are asynchronous, so the data in the target memory may not be updated before the data transfer is fully completed. Furthermore, to ensure data integrity and consistency, users need to explicitly perform synchronization operations or check the completion status of the RDMA transfer. Specifically, this can be done by polling the completion queue or waiting for event notifications to confirm that the data transfer has been completed. This means that users need to understand the underlying transfer status and explicitly handle these synchronization steps in their code. Moreover, in some high-performance computing or real-time applications, if the RDMA transfer latency is significant, users may directly experience the impact of communication operations on overall performance. In such cases, users need to optimize the transfer strategy or communication algorithm to minimize latency.

[0045] Therefore, although the communication library can automatically manage RDMA operations to minimize the user's need to handle low-level details, users still need to pay attention to transmission synchronization and data consistency. In some application scenarios, users may indeed "perceive" the status of RDMA transmissions, especially when data transmission timing requirements are very strict. It should be noted that, similar to the data transmission scheme described above, the additional processing latency introduced by the RDMA network card in packet assembly and decompression still exists, even with the communication library-based transmission method.

[0046] In other solutions, Compute Express Link (CXL) technology enables data consistency between the memory space of the Central Processing Unit (CPU) and the memory of devices interconnected with the CPU. This allows for resource sharing, resulting in higher AI server performance, reduced software stack complexity, and lower overall system costs. Furthermore, users can focus on the workload of the graphics processing unit (GPU) in a relatively simple way, without being burdened by the accelerator's redundant memory management hardware. CXL is based on the Fast Peripheral Component Interconnect (FPCI) protocol. However, because FPCI devices typically access host memory using Direct Memory Access (DMA), it is difficult for the host to cache data on FPCI devices.

[0047] The Fast Compute Link Protocol (CXL) can comprise three sub-protocols: CXL.I / O, CXL.cache, and CXL.mem. Based on these three sub-protocols, the CXL.I / O Link Protocol provides a low-latency access path for memory access between the host and devices that need to share memory resources (such as accelerators and memory expanders), and guarantees cache coherency. The CXL.I / C protocol specification defines three types of devices suitable for the CXL.I / C protocol.

[0048] For example, a device that needs to store data in a central processing unit can be called a first-type device. This first-type device only needs to be configured with the CXL.I / O sub-protocol and the CXL.cache sub-protocol.

[0049] For example, a device that has memory for the deployed accelerator and requires interconnection between the central processing unit (CPU) and the accelerator can be referred to as a second-type device. This second-type device can be configured with the CXL.I / O sub-protocol for CPU configuration. Furthermore, it can be configured with the CXL.cache sub-protocol to access the CPU's memory. Additionally, it can be configured with the CXL.mem sub-protocol for CPU access to its memory.

[0050] For example, devices that require compatibility with the CXL.I / O sub-protocol for configuration, such as memory expansion cards that support the Fast Compute Link sub-protocol, and devices that require compatibility with the CXL.mem sub-protocol to enable processors such as central processing units (CPUs) to access the memory within the memory buffer, can be referred to as third-type devices. It should be noted that the memory in third-type devices can provide two operating modes with the assistance of the Fast Compute Link controller within the same device. In the first operating mode, the Fast Compute Link protocol allows devices (such as Fast Compute Link memory expansion cards, accelerators, or storage devices) to be directly mapped to the address space of processing units (e.g., CPUs or GPUs), thus providing more efficient memory access. This mode allows applications to bypass the operating system's memory management layer (such as page caches) and interact directly with the memory device, reducing latency and improving performance. Therefore, processors or accelerators supporting the Fast Compute Link protocol must be able to handle the memory mapping and access mechanisms of the Fast Compute Link to achieve direct access in the first operating mode. In the second operating mode, the fast computation link memory is treated as part of the system memory. Processing units that support the fast computation link protocol (such as central processing units or graphics processing units) can access the memory of the memory extension device in the same way as they access local random access memory (RAM).

[0051] It should be noted that since the third type of device is mainly used as a storage medium for extended memory, it is difficult to actively trigger the following two types of data transfer operations: 1) It is difficult to actively trigger small-scale data transfer operations such as load / store based on the CXL.mem sub-protocol; 2) It is difficult to actively trigger large-scale data transfer operations such as DMA based on the CXL.I / O sub-protocol.

[0052] Furthermore, the first and second type devices can interact with interconnected third type devices via the Load / Store function, assisted by the central processing unit supporting the three sub-protocols of the fast computing link and its fast computing link root port. The central processing unit supporting the three sub-protocols of the fast computing link, in addition to the Load / Store method, also supports data interaction with interconnected third type devices via DMA. Moreover, for data interaction between two third type devices (defined as a sending third type device and a receiving third type device), there are two methods. For example, in the first method, the central processing unit (or the second type device) supporting the three sub-protocols of the fast computing link issues a Load instruction to load data from the extended memory of the sending third type device, and then the central processing unit (or the second type device) supporting the three sub-protocols continues to execute the Store instruction to store the data in the extended memory of the receiving third type device. As another example, in the second method, the central processing unit supporting the three sub-protocols of the fast computing link issues a DMA transfer instruction to transfer data from the extended memory of the sending third type device to the extended memory of the receiving third type device based on the DMA function.

[0053] Building upon this, some solutions have proposed prototype systems based on the memory decomposition of the fast computing link (hereinafter collectively referred to as Direct Fast Computing Link). With the server supporting the fast computing link protocol, the interconnection methods between the server and third-type devices in Direct Fast Computing Link technology are divided into the following two types:

[0054] The first approach involves a direct connection between a server supporting the Fast Compute Link protocol and a third-type device. Specifically, the central processing unit (or second-type device) within the server supporting the Fast Compute Link protocol, which supports all three sub-protocols of the Fast Compute Link, directly accesses the extended memory in the third-type device via the Fast Compute Link bus. It's important to note that the third-type device includes a Fast Compute Link controller, which connects to the memory via a DDR interface. The Fast Compute Link controller exposes the extended memory of the third-type device to the central processing unit (or second-type device) supporting the Fast Compute Link protocol through a multiplexed Fast Peripheral Component Interconnect (FPCI) channel. In some architectures, the Fast Compute Link controller in the third-type device can parse incoming Fast Compute Link data packets (referred to as Fast Compute Link flits) and, after parsing the information in the data packets (e.g., address and length), the memory manager can perform load / store operations on the extended memory at the corresponding address.

[0055] The second approach involves servers supporting the Fast Compute Link protocol interconnecting with a third type of device via a Fast Compute Link switch. Specifically, the central processing unit (or second-type device) within the server supporting the Fast Compute Link protocol, which supports all three sub-protocols of the Fast Compute Link protocol, interacts with a third-type device interconnected via a Load / Store method based on sub-protocols such as CXL.mem and CXL.I / O. Compared to the first interconnection method, this significantly increases the number of third-type devices that a central processing unit (or second-type device) supporting all three sub-protocols within the server supporting the Fast Compute Link protocol can interconnect with.

[0056] It should be noted that the Direct fast computing link solution does not cover how the central processing unit supporting the three sub-protocols of the fast computing link interacts with interconnected third-type devices through DMA capabilities, nor does it cover the implementation principle of data interaction between third-type devices through Load / Store and DMA methods.

[0057] In the aforementioned Direct fast computing link scheme, the Direct fast computing link technology is not applicable when a graphics processing unit that does not support the fast computing link related protocols communicates with other graphics processing units across hosts.

[0058] Based on the above, it is clear that graphics processing units (GPUs) are widely used in complex deep learning applications, such as those involving large models. Consequently, some manufacturers deploy massive numbers of commercial GPUs on their servers to accelerate applications, resulting in a large volume of data that needs to be frequently transferred between GPUs across server nodes. While each commercial GPU manufacturer has proposed its own interconnect protocols to accelerate data transfer between its GPUs, for cloud service providers, these bus protocols are only applicable to some high-performance GPUs from each manufacturer, and the hardware compatible with these GPUs is expensive. Furthermore, direct communication between GPUs from different manufacturers is difficult. Therefore, most manufacturers use GPUs as interconnect devices for rapid peripheral components of servers, relying on network devices such as switches to achieve communication between GPUs across server nodes. Although some manufacturers almost universally employ methods similar to RDMA-assisted cross-node communication between GPUs, the two communicating nodes still require separate RDMA network cards for both the sending and receiving nodes. Because data copying and caching are required between the sending graphics processing unit (GPU) and the sending RDMA network interface card (NIC), and the sending RDMA NIC needs to reassemble data packets using information such as the receiving RDMA NIC's IP address and the receiving GPU's DDR memory address, a significant processing delay is introduced before transmission. Similarly, the receiving RDMA NIC needs to parse the received data packets, which also introduces a significant processing delay. Although other technologies have optimized cross-node communication methods for GPUs assisted by RDMA, users still need to pay attention to transmission synchronization and data consistency issues. Although the Fast Compute Link (CXL) protocol is an industry-supported cache coherence interconnect protocol for processors, memory expansion, and accelerators, it is difficult for accelerators such as GPUs that do not support CXL to directly use sub-protocols such as CXL.mem to transfer data stored in their DDR to other GPUs or central processing units across server nodes.

[0059] In view of this, in order to solve the problems of high latency, high communication frequency and lack of support for high-speed data transmission protocols such as fast computation link protocol when graphics processing units on different server nodes communicate in distributed deep learning scenarios, this application proposes a server, data transmission method and distributed system for cross-node communication of graphics processing units in distributed deep learning scenarios, based on the fact that neither of the two graphics processing units on the cross-server node supports the fast computation link protocol.

[0060] Figure 1A A schematic diagram of a distributed system according to an embodiment of this application is shown.

[0061] like Figure 1A As shown, the distributed system of this embodiment may include multiple nodes. These multiple nodes may include a first server and a second server, but it should be understood that this embodiment is not limited to these, and the distributed system of this embodiment may also include other nodes. The multiple nodes in the distributed system can perform cross-node communication, specifically, cross-node communication can be performed via a fast computation link switch. For example, the switch may implement data transmission based on the fast computation link protocol.

[0062] Furthermore, the following description uses a first server and a second server among multiple nodes as examples. It should be understood that both the first server and the second server are servers in this application embodiment. The first server can be the server acting as the sender in this application embodiment. The second server can be the server acting as the receiver in this application embodiment.

[0063] Figure 1B A schematic diagram of a first server and a second server according to an embodiment of this application is shown.

[0064] like Figure 1B As shown, the first server may include a first graphics processing unit, a first central processing unit, and a first memory expansion device.

[0065] The first graphics processing unit can be used to perform computational tasks to obtain computational results. In some embodiments, the first graphics processing unit can use the computational result as the first data to be transmitted from the first server to the second server. It should be noted that the first graphics processing unit may not support memory interconnect protocols, but it can be deployed with a DMA engine to allow it to access the memory cache based on zero-copy technology. The memory interconnect protocol may include the fast computation link protocol described above. For example, a graphics processing unit that does not support memory interconnect protocols in this application may refer to a graphics processing unit that has not deployed a protocol stack for the fast computation link protocol. For example, such a graphics processing unit may have difficulty parsing at least one of the transaction information specific to the fast computation link protocol, such as cache consistency requests or memory pooling instructions. However, such a graphics processing unit can read and write data to a fixed memory region divided by the corresponding central processing unit based on its own deployed DMA engine.

[0066] Specifically, the first graphics processing unit can be connected to a first memory expansion device. For example, the first memory expansion device can be a memory buffer. Specifically, the first memory expansion device can be a memory buffer that supports a fast computation link protocol, such as a memory buffer that supports the CXL.I / O sub-protocol and the CXL.mem sub-protocol, i.e., the third type of device described above. Based on this, the first graphics processing unit can write the first data to be transferred to the first memory expansion device according to a first address. For example, the first address can refer to the memory address of the first memory expansion device, i.e., a zero-copy address. In this way, the first graphics processing unit can write the first data to be transferred to the first memory expansion device based on the high-speed concurrent read / write characteristics of the DMA engine and the zero-copy channel. This process does not require the introduction of an additional forwarding process (such as the data forwarding process of the central processing unit), reducing the latency of the data writing process.

[0067] The first central processing unit may pre-store the association between the memory addresses of the first memory expansion device of the first server and the memory addresses of the second memory expansion device of the second server. Thus, when the first address is detected to have been written to the first central processing unit, the first central processing unit can, based on this association, query the second address, which is the memory address of the second memory expansion device of the second server.

[0068] The first central processing unit (CPU) can support a memory interconnect protocol. Thus, the CPU can interconnect with a fast compute link switch based on the memory interconnect protocol, and the fast compute link switch can interconnect with a second server based on the memory interconnect protocol. Based on this, the CPU can transmit first data to be transmitted to the second server according to a second address via the fast compute link switch. Specifically, the CPU can send a Load / Store command based on the CXL.mem sub-protocol (or a DMA command based on the CXL.I / O sub-protocol) to the fast compute link switch to transmit the first data to be transmitted from the memory region corresponding to the first address in a first memory expansion device supporting the fast compute link protocol, via the fast compute link switch, to a second server that also supports the fast compute link protocol, through a high-speed cross-server data transmission channel based on the fast compute link protocol.

[0069] The structure of the second server is similar to that of the first server. For example, the second server may include a second memory expansion device and a second graphics processing unit.

[0070] The second memory expansion device can be a memory buffer. Specifically, the second memory expansion device can be a memory buffer that supports a fast computation link protocol, such as a memory buffer that supports the CXL.I / O sub-protocol and the CXL.mem sub-protocol, i.e., the third type of device described above. For example, the first data to be transmitted can be written to the second memory expansion device according to a second address. That is, the second address can be the memory address of the second memory expansion device, i.e., the starting address of the zero-copy memory.

[0071] The second server may further include a second graphics processing unit (GPU). The GPU can perform computational tasks to obtain computational results. However, this embodiment is not limited to this; the GPU can also perform computational tasks based on the computational results of processors on other servers to obtain computational results. It should be noted that the GPU may not support memory interconnect protocols, but it may be equipped with a DMA engine to access the memory cache using zero-copy technology. Thus, the GPU can connect to a second memory expansion device and read the aforementioned first data to be transferred from the second memory expansion device. For example, if the second server detects that the first data to be transferred has been written to the second memory expansion device within the second server, it can invoke the second GPU within the second server to read the first data to be transferred from the second memory expansion device. Then, the second GPU can perform computational tasks based on the first data to be transferred to obtain corresponding computational results. Based on this, the second graphics processing unit can read the first data to be transferred from the second memory expansion device based on the high-speed concurrent read and write characteristics of the DMA engine and the zero-copy channel. This process does not require the introduction of an additional forwarding process (such as the data forwarding process of the central processing unit), reducing the latency of the data writing process.

[0072] Based on this, upon detecting that the first graphics processing unit writes the first data to be transmitted into the first memory at the first address, the first memory expansion device can immediately write the first address into the corresponding central processing unit. This allows the central processing unit, upon detecting that the first address has been written, to immediately query the second address based on the first address and, through a fast calculation link switch, transmit the first data to be transmitted to another server according to the second address. Subsequently, upon detecting that the first data to be transmitted has been written into the second memory expansion device at the second address, the other server can immediately invoke the second graphics processing unit to read the first data to be transmitted from the second memory expansion device according to the second address. In this way, this application can save the time required for different servers to assemble and unpack transmitted data using network cards to achieve cross-node communication, enabling efficient cross-node communication between one server and another.

[0073] Furthermore, when the first server acts as the receiver, its function is similar to that of the second server described above. Specifically, the first central processing unit can receive the second data to be transmitted from the second graphics processing unit of the second server, and write the second data to be transmitted to the second memory expansion device according to the third address. The second data to be transmitted is similar to the first data to be transmitted, and will not be described in detail here. When the first memory expansion device detects that the second data to be transmitted has been written to the second memory expansion device, it writes the third address to the central processing unit. The third address is similar to the first address, and will not be described in detail here. Furthermore, when the first central processing unit detects that the third address has been written to the first central processing unit, it can call the first graphics processing unit to read the second data to be transmitted from the second memory expansion device according to the third address.

[0074] Furthermore, the first memory expansion device may include a connected first expansion memory and a first fast computing link controller. The first expansion memory is used to store data. For example, the first expansion memory can be used to store the first data to be transmitted described above.

[0075] The first fast compute link controller can manage data in the first memory expansion device. For example, the first fast compute link controller can monitor the stored data in the first memory expansion device. The first fast compute link controller can be connected to the first central processing unit. Thus, when it is detected that the first data to be transmitted has been written to the first expanded memory, the first fast compute link controller can write the first address to the first central processing unit. For example, when the first graphics processing unit completes the data writing, it can trigger the first fast compute link controller to detect an interrupt event that the first address has been written to the first expanded memory. The first fast compute link controller can encapsulate the interrupt event and write it to the first graphics processing unit so that the first address is written to the first graphics processing unit. Specifically, the first fast compute link controller can encapsulate the Message Signaled Interrupts eXtended (MSI-X) event into a Transaction Layer Packet (TLP) and write the TLP to the first central processing unit via the Fast Peripheral Component Interconnect Bus.

[0076] The first central processing unit (CPU) may be equipped with an interrupt controller. The interrupt controller can trigger an interrupt handler to modify the value of the memory monitoring register in the CPU. Then, if the application-state polling thread associated with the first graphics processing unit (GPU) in the CPU detects that the memory monitoring register value has been modified, the polling thread is interrupted, and a second address is retrieved based on a first address. A transmission instruction based on the fast compute link protocol is then issued to transmit the data to be transmitted to the second server via the fast compute link switch at the second address. For example, if the interrupt controller writes a TLP to the interrupt controller of the CPU, it can immediately trigger a predetermined procedure (e.g., an interrupt handler) to modify the value of the memory monitoring register in the CPU to the first address. Specifically, the CPU may be equipped with a polling thread supporting three sub-protocols of the fast compute link. This thread can immediately exit polling upon detecting that the memory monitoring register value has changed to the first address, and, based on the first address, query the second address of the second server supporting the memory interconnect protocol for receiving the first data to be transmitted, according to the aforementioned association, to transmit the first data to be transmitted to the second server via the fast compute link switch at the second address.

[0077] For example, a second memory expansion device may include a connected second extended memory and a second fast computing link controller.

[0078] The second server may also include a second central processing unit (CPU). For example, the second CPU supports a memory interconnect protocol. Thus, the second CPU can interconnect with a high-speed compute link switch based on the memory interconnect protocol, and the second CPU can also connect to a second graphics processing unit (GPU) and a second memory expansion device. Based on this, the second CPU can receive first data to be transmitted from the first GPU of the first server and write the first data to be transmitted to the second memory expansion device according to a second address. The second high-speed compute link controller can, upon detecting that the first data to be transmitted has been written to the second expanded memory, write the second address to the second CPU, so that the second CPU can monitor the write status of the first data to be transmitted to the second expanded memory. Thus, upon detecting that the first data to be transmitted has been written to the second expanded memory, the second CPU can invoke the second GPU to read the first data to be transmitted from the second expanded memory.

[0079] Specifically, the second fast compute link controller can manage the data in the second extended memory. For example, the second fast compute link controller can monitor the stored data in the second extended memory. The second fast compute link controller can be connected to the second central processing unit. Thus, upon detecting that the first data to be transferred has been written to the second extended memory, the second fast compute link controller can write the second address to the second central processing unit. Specifically, when the second fast compute link controller receives a write operation instruction via a high-speed data transmission channel based on the fast compute link protocol and zero-copy technology, while writing the first data to be transferred to the second extended memory, the second fast compute link controller can encapsulate the MSI-X into a TLP and write the TLP to the second central processing unit via the fast peripheral component interconnect bus.

[0080] Furthermore, the second central processing unit (CPU) may be equipped with an interrupt controller. This interrupt controller can immediately trigger a predetermined program (e.g., an interrupt handler) to modify the value of the memory monitoring register in the second CPU to the second address upon receiving a TLP (Time Limit Point) written to the CPU's interrupt controller. Specifically, the second CPU may be equipped with a polling thread supporting three sub-protocols of the fast computation link. This thread can immediately exit polling upon detecting a change in the memory monitoring register value to the second address and invoke the second graphics processing unit (GPU) to read the first data to be transmitted from the second extended memory. This completes the low-latency, high-speed cross-node data transmission process between the first GPU, which does not support the memory interconnect protocol, and the second GPU, which also does not support the memory interconnect protocol.

[0081] Based on this, when it is detected that the first graphics processing unit, which does not support the memory interconnect protocol, writes the first data to be transmitted to the first extended memory according to the first address, the first fast computing link controller can immediately write the first address to the first central processing unit. This allows the first central processing unit, upon detecting that the first address has been written, to immediately query the second address based on the first address and transmit the first data to be transmitted to the second server according to the second address via the fast computing link switch. Subsequently, when the second server detects that the first data to be transmitted has been written to the second extended memory according to the second address, it can immediately call the second graphics processing unit, which does not support the memory interconnect protocol, to read the first data to be transmitted from the second extended memory according to the second address. Thus, this application saves the time required for processors on different servers to assemble and unpack transmitted data using network cards to achieve cross-node communication, enabling efficient cross-node communication between the first graphics processing unit and the second graphics processing unit that do not support the memory interconnect protocol.

[0082] However, it should be understood that the embodiments of this application are not limited thereto. In other embodiments of this application, the first graphics processing unit of the first server may support a memory interconnect protocol, as described below. Figure 1C Please provide an explanation. Figure 1C A schematic diagram of a first server and a second server according to another embodiment of this application is shown.

[0083] like Figure 1C As shown, the first server may include a first graphics processing unit and a first central processing unit. It should be noted that the first graphics processing unit may be a processor that supports memory interconnect protocols (such as the fast computation link protocol described above), thus enabling the first graphics processing unit to communicate with other devices based on the three fast computation link sub-protocols described above.

[0084] Specifically, the first graphics processing unit may include a first memory. For example, the first graphics processing unit can write the first data to be transferred into the first memory at a first address. The first address refers to the address of the first data to be transferred in the first memory, which is also a zero-copy address. In this way, the first graphics processing unit can directly write the first data to be transferred into the first memory based on the zero-copy channel. This process does not require an additional data forwarding process, reducing the latency of the data writing process.

[0085] The first graphics processing unit may further include a first fast compute link controller. The first fast compute link controller can be connected to the first central processing unit. Thus, upon detecting that first data to be transmitted has been written to the first memory, the first fast compute link controller can write the first address to the first central processing unit. Furthermore, when the first graphics processing unit completes the data writing, it can trigger the first fast compute link controller to detect a first interrupt event where the first address is written to the first memory. Then, the first fast compute link controller can encapsulate the first interrupt event and write it to the first central processing unit, thereby ensuring that the first address is written to the first central processing unit.

[0086] The first central processing unit (CPU) can trigger a first interrupt handler to modify the value of the memory monitoring register in the CPU. When the application-state polling thread associated with the first graphics processing unit (GPU) detects that the value of the memory monitoring register has been modified, the CPU interrupts the polling thread. Therefore, since the CPU pre-stores the association between the memory addresses of the first memory and the memory addresses of the second memory extension device of the second server, when the first address is detected to have been written to the CPU, the CPU can query the second address based on this association. The interrupt controller of the CPU can immediately trigger a predetermined procedure (e.g., an interrupt handler) to modify the value of the memory monitoring register in the CPU to the first address when the TLP is written to the interrupt controller of the CPU. Specifically, the CPU can deploy a polling thread supporting three sub-protocols of the fast computation link. This thread can immediately interrupt polling when it detects that the value of the memory monitoring register has changed to the first address, and query the second address of the second server supporting the memory interconnect protocol for receiving the first data to be transmitted, based on the first address and the aforementioned association. Furthermore, a transmission command based on the memory interconnect protocol is issued to transmit the first data to be transmitted to the second server according to the second address via a fast calculation link switch.

[0087] The second fast computing link controller of the second server can write a second address to the second central processing unit (CPU) upon detecting that the first data to be transmitted has been written to the second extended memory. This allows the CPU to monitor the write status of the first data to be transmitted to the second extended memory. Thus, upon detecting that the first data to be transmitted has been written to the second extended memory, the CPU can invoke the second graphics processing unit to read the first data to be transmitted from the second extended memory.

[0088] For example, upon completion of data writing, the second central processing unit (CPU) can trigger the second fast computation link controller to detect a second interrupt event where the second address is written to the second extended memory. The second fast computation link controller can then encapsulate the second interrupt event and write it to the second CPU, ensuring that the second address is written to the second CPU. The second CPU can then trigger a second interrupt routine to modify the value of the memory monitoring register within the CPU. If the application-state polling thread associated with the second graphics processing unit (GPU) in the second CPU detects that the value of the memory monitoring register has been modified, the polling thread is interrupted, and the second GPU is invoked to read the first data to be transmitted from the second extended memory according to the second address.

[0089] Specifically, when the second fast computing link controller receives a write operation instruction via a high-speed data transmission channel based on the fast computing link protocol and zero-copy technology, while writing the first data to be transmitted into the second extended memory, the second fast computing link controller can encapsulate the MSI-X into a TLP and write the TLP into the second central processing unit via the fast peripheral component interconnect bus.

[0090] Furthermore, the second central processing unit (CPU) may be equipped with an interrupt controller. This interrupt controller can immediately trigger a predetermined program (e.g., an interrupt handler) to modify the value of the memory monitoring register in the second CPU to the second address upon receiving a TLP (Time Limit Point) written to the CPU's interrupt controller. Specifically, the second CPU may be equipped with an application-mode polling thread supporting three sub-protocols of the fast computation link. This thread can immediately interrupt polling upon detecting a change in the memory monitoring register value to the second address and invoke the second graphics processing unit (GPU) to read the first data to be transmitted from the second extended memory. This completes the low-latency, high-speed cross-node data transmission process between the first GPU supporting the memory interconnect protocol and the second GPU not supporting the memory interconnect protocol.

[0091] Based on this, when it is detected that the first graphics processing unit supporting the memory interconnect protocol writes the first data to be transmitted into the first memory of the first graphics processing unit according to the first address, a first interrupt event is triggered. Then, the first fast computing link controller can immediately encapsulate the first interrupt event and write it into the first central processing unit, so that the first address is written into the first central processing unit, thereby causing the first central processing unit to trigger the first interrupt handler, and then modify the value of the memory monitoring register in the first central processing unit.

[0092] If the application-state polling thread associated with the first graphics processing unit in the first central processing unit detects that the value of the memory monitoring register has been modified, it can immediately interrupt the polling thread, query the second address based on the first address, and issue a transmission instruction based on the memory interconnect protocol to transmit the first data to be transmitted to the second server according to the second address via a fast-calculated link switch. Subsequently, if the second central processing unit of the second server detects that the first data to be transmitted has been written to the second extended memory according to the second address, it can immediately trigger a second interrupt event, encapsulate the second interrupt event, and write it to the second central processing unit so that the second address is written to the second central processing unit. The second central processing unit triggers a second interrupt handler to modify the value of the memory monitoring register in the second central processing unit. When the application-state polling thread associated with the second graphics processing unit in the second central processing unit detects that the value of the memory monitoring register has been modified, the polling thread is interrupted, and the second graphics processing unit that does not support the memory interconnect protocol is called to read the first data to be transmitted from the second extended memory according to the second address. Thus, this application saves the time required for processors on different nodes to use network cards to assemble and unpack transmitted data to achieve cross-node communication, enabling the first graphics processing unit that supports the memory interconnect protocol and the second graphics processing unit that does not support the memory interconnect protocol to communicate efficiently across nodes.

[0093] However, it should be understood that the embodiments of this application are not limited thereto. In other embodiments of this application, the second graphics processing unit of the second server may support memory interconnection protocols, as described below. Figure 1D Please provide an explanation. Figure 1D A schematic diagram of a first server and a second server according to another embodiment of this application is shown.

[0094] like Figure 1D As shown, the first server may include a first graphics processing unit, a first central processing unit, and a first memory expansion device.

[0095] The first central processing unit (CPU) can partition the memory of the first memory expansion device to obtain the first extended memory based on the storage space required for the first data to be transmitted in the first graphics processing unit (GPU). It then associates and stores the first address of the first extended memory with the identifier of the first data to be transmitted. Thus, upon receiving the first data to be transmitted carrying the first data identifier, the CPU can store the first data to be transmitted into the first extended memory according to the first address. Furthermore, upon detecting that the second address corresponding to the first data identifier has been written to the first memory expansion device from the second server, the first fast computing link controller can trigger an interrupt event to write the second address into the CPU. The interrupt event can be MSI-X. Based on this, since the CPU stores the first address and the second address, it can associate the first address and the second address to obtain the association relationship between them.

[0096] The first graphics processing unit can write the first data to be transmitted into the first extended memory of the first memory extension device according to the first address.

[0097] Upon detecting that the first data to be transmitted has been written to the first extended memory, the first extended device (e.g., the first fast compute link controller included therein) can write the first address to the first central processing unit. Specifically, the first fast compute link controller can encapsulate an MSI-X into a TLP and write the TLP to the first central processing unit via the fast peripheral component interconnect bus.

[0098] Thus, if the first address is detected to have been written to the first central processing unit, the first central processing unit can query the second address based on the association, and the second address is the memory address of the second server.

[0099] The second graphics processing unit (GPU) can be a processor supporting a memory interconnect protocol, allowing the first data to be transmitted to be directly written into the second GPU's second memory. Thus, the second server (specifically, its second GPU) can read the first data to be transmitted from the GPU upon detecting that it has been written to the GPU, and then perform computational tasks based on this data to obtain the corresponding results. Furthermore, the second GPU can directly read the first data to be transmitted from its memory using a fast computation link protocol and execute computational tasks. This process eliminates the need for additional forwarding procedures (such as data forwarding by the central processing unit), reducing latency in the data writing process.

[0100] The second central processing unit can be interconnected with a high-speed computing link switch based on a memory interconnect protocol, and the second central processing unit can also be connected to the second graphics processing unit.

[0101] The second central processing unit (CPU) can be used to partition the memory of the second graphics processing unit (GPU) to obtain a second memory based on the storage space required for storing the first data to be transmitted from the first server. Based on this, the second GPU may include the second memory and a second fast computing link controller. Furthermore, the second CPU can associate a second address of the second memory with an identifier for the first data to be transmitted, and send the second address and the first data to be transmitted identifier to the first server (e.g., a first memory extension device of the first server) via a second fast computing link switch. This allows the first server to associate the second address with the first address and transmit the first data to be transmitted to the second server according to the second address.

[0102] The second central processing unit (CPU) can receive first data to be transmitted from the first graphics processing unit (GPU) of the first server, and write the first data to be transmitted into the second memory according to a second address, i.e., the second address can be the address of the second memory of the second server. The second fast computing link controller can, upon detecting that the first data to be transmitted has been written into the second memory, write the second address into the second CPU, so that the second CPU can monitor the write status of the first data to be transmitted to the second memory. Thus, the second CPU can, upon detecting that the first data to be transmitted has been written into the second memory, invoke the second graphics processing unit to read the first data to be transmitted from the second memory.

[0103] Specifically, the second fast compute link controller can be the fast compute link controller described above. The second fast compute link controller can manage the data in the memory of the second graphics processing unit. For example, the second fast compute link controller can monitor the stored data in the memory of the second graphics processing unit. The second fast compute link controller can be connected to the second central processing unit. Thus, upon detecting that the first data to be transferred has been written to the memory of the second graphics processing unit, the second fast compute link controller can write the second address to the second central processing unit. Specifically, when the second fast compute link controller receives a write operation instruction via a high-speed data transmission channel based on the fast compute link protocol and zero-copy technology, while writing the first data to be transferred to the second memory, the second fast compute link controller can encapsulate the MSI-X into a TLP and write the TLP to the second central processing unit via the fast peripheral component interconnect bus.

[0104] Furthermore, the second central processing unit (CPU) may be equipped with an interrupt controller. This interrupt controller can immediately trigger a predetermined program (e.g., an interrupt handler) to modify the value of the memory monitoring register in the second CPU to the second address upon receiving a TLP (Time Limit Point) written to the CPU's interrupt controller. Specifically, the second CPU may be equipped with a polling thread supporting three sub-protocols of the fast computation link. This thread can immediately interrupt polling upon detecting a change in the memory monitoring register value to the second address and invoke the second graphics processing unit (GPU) to read the first data to be transmitted from the second memory. This completes the low-latency, high-speed cross-node data transmission process between the first GPU, which does not support the memory interconnect protocol, and the second GPU, which does support the memory interconnect protocol.

[0105] Based on this, the first central processing unit divides the first extended memory according to the storage space required by the first data to be transmitted in the first graphics processing unit, and associates the first address of the first extended memory with the identifier of the first data to be transmitted. Thus, when first data to be transmitted carrying the first identifier is received, the first data to be transmitted can be stored in the first extended memory according to the first address.

[0106] Furthermore, the second central processing unit can partition the memory of the second graphics processing unit to obtain a second memory based on the storage space required for the first data to be transmitted in the second graphics processing unit, and send the second address of the second memory and the identifier of the first data to be transmitted to the first server via a fast computing link switch. In this way, the first fast computing link controller can trigger an interrupt event upon detecting that the second address has been written to the first extended memory, thereby writing the second address to the first central processing unit. Upon detecting that the second address has been written to the first central processing unit, the first central processing unit immediately associates and stores the second address with the first address.

[0107] Based on this, when it is detected that the first graphics processing unit (GPU) that does not support the memory interconnect protocol writes the first data to be transmitted into the first extended memory at the first address, the first fast computing link controller can immediately write the first address into the first central processing unit (CPU). This allows the CPU, upon detecting that the first address has been written, to immediately query the second address based on the first address and transmit the first data to be transmitted to the second server via the fast computing link switch at the second address. Subsequently, when the second server detects that the first data to be transmitted has been written into the second memory of the second GPU that supports the memory interconnect protocol at the second address, it can immediately call the second GPU to read the first data to be transmitted from the second memory at the second address. This application saves the time required for processors on different nodes to assemble and unpack transmitted data using network interface cards (NICs) to achieve cross-node communication, enabling efficient cross-node communication between the first GPU that does not support the memory interconnect protocol and the second GPU that does support it.

[0108] Furthermore, before executing the data transmission process described above, the first server can partition the memory of the first memory expansion device to obtain first extended memory based on the storage space required for the first data to be transmitted; and associate the first address of the first extended memory with the identifier of the first data to be transmitted. Similarly, the second server can partition the memory of the second memory expansion device to obtain second extended memory based on the storage space required for the first data to be transmitted; and associate the second extended memory with the identifier of the first data to be transmitted. The following will first combine... Figure 1B Please provide a detailed explanation.

[0109] Continue to refer to Figure 1BDuring the distributed deep learning model compilation phase, the first central processing unit (CPU) uses zero-copy technology to call the first graphics processing unit (GPU) to allocate at least a portion of the memory of the first memory extension device as zero-copy memory. This allows the GPU to write data to the first extended memory of the first memory extension device, and the physical starting address of this first extended memory can be recorded as `send_mem_addr`, also known as the first address. The size of the first extended memory can be the same as the amount of data to be transmitted by the GPU during the model compilation phase (i.e., the storage space required for the first data to be transmitted). Furthermore, if there are multiple copies of the first data to be transmitted in the model, the GPU can allocate the same number of copies of the first extended memory as zero-copy memory. After allocating the first extended memory, the CPU can use a hash table to associate and store the first address of the first extended memory with the identifier of the first data to be transmitted. Specifically, the hash table can be a sending information table named `comm_send_mem_table`. The identifier of the first data to be transmitted can include the communication process number `comm_tag`. The first address can include the starting address of the zero-copy memory, `send_mem_addr`. In this way, the first data identifier to be transmitted can be used as the key and the first address as the value to determine the corresponding first address based on the first data identifier carried by the first data to be transmitted, and then the first data to be transmitted can be written into the corresponding first extended memory.

[0110] Continue to refer to Figure 1B The second server may include a second memory expansion device, a second graphics processing unit, and a second central processing unit. The second memory expansion device may include its own memory and a second fast computing link controller. The second central processing unit can partition the memory of the second memory expansion device to obtain second extended memory based on the storage space required for the first data to be transmitted, and can associate and store a second address of the second extended memory with the identifier of the first data to be transmitted.

[0111] Specifically, during the distributed deep learning model compilation phase, the second central processing unit (CPU), based on zero-copy technology, calls the second graphics processing unit (GPU) to allocate at least a portion of the memory of the second memory extension device as zero-copy memory. This allows the GPU to write data to the second memory of the second memory extension device, and the physical starting address of this second extended memory can be recorded as recv_mem_addr, i.e., the second address. The size of the second extended memory can be the same as the amount of the first data to be transmitted by the GPU during the model compilation phase (i.e., the storage space required for the first data to be transmitted). Furthermore, if there are multiple copies of the first data to be transmitted in the model, the GPU can allocate the same number of copies of the second extended memory as zero-copy memory. After allocating the second extended memory, the CPU can use a hash table to associate the second address of the second extended memory with the identifier of the first data to be transmitted. Specifically, this hash table can be a receive information table named comm_recv_mem_table. The identifier of the first data to be transmitted can include the communication process number comm_tag. The second address can include the starting address of the zero-copy memory, recv_mem_addr. In this way, the first data identifier to be transmitted can be used as the key and the second address as the value, so as to determine the corresponding second address based on the first data identifier carried by the first data to be transmitted, and write the first data to be transmitted into the corresponding second extended memory.

[0112] It should be noted that when there are multiple communication processes during the compilation phase of a distributed deep learning model, each communication process in the entire model can be numbered and denoted as comm_tag. The value type is an integer, and the value starts from 0 and increments sequentially. That is, for multiple first data to be transmitted generated in multiple communication processes of the model, a first data to be transmitted identifier with sequential increments can be generated.

[0113] Furthermore, after dividing the first extended memory and the second extended memory, the second server can also transmit the second address and the first data identifier to be transmitted to the first memory extension device deployed on the first server via a fast computing link switch. Upon detecting that the second address and the first data identifier to be transmitted of the second memory extension device have been written into the memory of the first memory extension device, the first server can write the second address and the first data identifier to be transmitted of the second memory extension device into the first central processing unit; and query the first address based on the first data identifier to be transmitted, and associate and store the first address and the second address.

[0114] For example, the second central processing unit can transmit the second address and the first data identifier to be transmitted to the first memory expansion device deployed on the first server via a fast computing link switch. Specifically, the second central processing unit can send a Load / Store instruction based on the CXL.mem sub-protocol (or a DMA instruction based on the CXL.I / O sub-protocol) to write the first data identifier to be transmitted and the second address of the second extended memory corresponding to the first data identifier to be transmitted, as well as other information from the received information table, into the memory space (named comm_recv_info_mem) in the memory of the first memory expansion device for storing information from the second server, via the fast computing link switch. Then, the first fast computing link controller can, upon detecting that the second address and the first data identifier to be transmitted of the second memory expansion device have been written into the memory of the first memory expansion device, write the second address and the first data identifier to be transmitted of the second memory expansion device into the first central processing unit. Specifically, the first fast computing link controller can monitor the MSI-X written to the aforementioned memory space. This MSI-X can record the aforementioned second address and the first data identifier to be transmitted. Subsequently, upon detecting the aforementioned MSI-X, the first fast computing link controller can encapsulate the second address and the first data identifier to be transmitted recorded by the MSI-X into a TLP, and write the TLP to the first central processing unit via the fast peripheral component interconnect bus.

[0115] The first central processing unit (CPU) can query the first address based on the first data identifier to be transmitted and associate and store the first address and the second address. Specifically, when the interrupt controller of the CPU detects that the TLP is written to the interrupt controller, it can immediately trigger a predetermined program (e.g., an interrupt handler), query the corresponding first address in the received information table based on the first data identifier to be transmitted in the TLP, and associate and store the first address and the second address in an information table (named send_recv_info_table), which can be a hash table. The first address can be used as the key, and the second address can be used as the value, so that the second address can be queried based on the first address, thereby facilitating communication between the CPU and the CPU. Based on this, by establishing the mapping relationship between the first address and the second address during the pre-compilation stage, the dynamic addressing and handshake process during subsequent inference / training is avoided. Thus, the embodiments of this application can decouple the data transmission control flow from the graphics processing unit (GPU) execution flow, thereby achieving a low-overhead, high-concurrency asynchronous communication process of "write-to-trigger, automatic forwarding" during the actual communication stage. This achieves asynchronous and efficient execution of the communication process, significantly improving the startup efficiency and system scalability of cross-node GPU communication. The overall solution does not require native support for the Fast Compute Link protocol in the graphics processing unit (GPU). It can leverage the third type of device in the Fast Compute Link ecosystem (i.e., the first and second memory expansion devices) and the MSI-X interrupt mechanism to establish a low-latency, zero-intervention-in-the-central-processing-unit (CPU) heterogeneous communication link. This design is compatible with the GPU architectures of some vendors and has good engineering feasibility, making it particularly suitable for accelerating the distributed inference and training of large models in a heterogeneous GPU hybrid deployment environment on cloud platforms.

[0116] Furthermore, the embodiments of this application are not limited to this. In other embodiments of this application, before executing the data transmission process described above, the first server can divide the memory of the first graphics processing unit into first memory according to the storage space required by the first data to be transmitted; and associate and store the first address of the first memory with the identifier of the first data to be transmitted. The following, in conjunction with... Figure 1C Please provide a detailed explanation.

[0117] Continue to refer to Figure 1CDuring the compilation phase of the distributed deep learning model, the first central processing unit (CPU) uses zero-copy technology to divide the memory of the first graphics processing unit (GPU) to obtain storage space. This storage space is used as the first memory, i.e., zero-copy memory, so that the GPU can write data to the first memory. The physical starting address of this first memory can be recorded as send_mem_addr, i.e., the first address. The size of the first memory can be the same as the size of the first data to be transmitted by the GPU during the model compilation phase (i.e., the storage space required for the first data to be transmitted). Furthermore, if there are multiple copies of the first data to be transmitted in the model, the GPU can divide the storage space into the same number of copies. After dividing the storage space, the CPU can use a hash table to associate the first address of the storage space with the identifier of the first data to be transmitted. Specifically, the hash table can be a sending information table named comm_send_mem_table. The identifier of the first data to be transmitted can include the communication process number comm_tag. The first address can include the starting address of the zero-copy memory, send_mem_addr. In this way, the first data identifier to be transmitted can be used as the key and the first address as the value, so as to determine the corresponding first address based on the first data identifier carried by the first data to be transmitted, and write the first data to be transmitted into the corresponding first memory.

[0118] Furthermore, the embodiments of this application are not limited to this. In other embodiments of this application, before executing the data transmission process described above, the second server can divide the memory of the second graphics processing unit into a second memory according to the storage space required by the first data to be transmitted; and associate the second address of the second memory with the identifier of the first data to be transmitted. The following, in conjunction with... Figure 1D Please provide a detailed explanation.

[0119] Continue to refer to Figure 1DDuring the distributed deep learning model compilation phase, the second central processing unit (CPU) uses zero-copy technology to call the second graphics processing unit (GPU) to partition its memory into storage space. This storage space can be used as second memory, i.e., zero-copy memory, to write data to the second GPU's second memory. The physical starting address of this second memory can be recorded as recv_mem_addr, i.e., the second address. It should be understood that since the second GPU's second memory supports three sub-protocols—CXL.mem, CXL.I / O, and CXL.cache—it is essentially fast computation link memory. The size of the second memory can be the same as the amount of the first data to be transmitted by the GPU during the model compilation phase (i.e., the storage space required for the first data to be transmitted). Furthermore, if there are multiple copies of the first data to be transmitted in the model, the GPU can partition the second memory into the same number of copies as zero-copy memory. After partitioning the second memory, the second CPU can use a hash table to associate the second address of the second memory with the identifier of the first data to be transmitted. Specifically, the hash table can be a receive information table named comm_recv_mem_table. The first data identifier to be transmitted may include the communication process number comm_tag. The second address may include the zero-copy memory starting address recv_mem_addr. Thus, the first data identifier to be transmitted can be used as the key and the second address as the value to determine the corresponding second address based on the first data identifier carried by the first data to be transmitted, and then the first data to be transmitted can be written into the corresponding second memory.

[0120] Furthermore, after partitioning the memory, the second server can also transmit the second address and the first data identifier to be transmitted to the first server via a fast computing link switch. Upon detecting that the second address and the first data identifier to be transmitted have been written to the second graphics processing unit, the first server can write the second address and the first data identifier to be transmitted to the first central processing unit, query the first address based on the first data identifier to be transmitted, and associate and store the first address and the second address.

[0121] For example, the second central processing unit can transmit the second address and the first data identifier to be transmitted to the first server via a fast computing link switch. Specifically, the second central processing unit can send a Load / Store instruction based on the CXL.mem sub-protocol (or a DMA instruction based on the CXL.I / O sub-protocol) to write the first data identifier to be transmitted and the second address of the second memory corresponding to the first data identifier to be transmitted from the received information table into the memory space (named comm_recv_info_mem) in the first server used to store information from the second server via the fast computing link switch. Then, the first fast computing link controller can, upon detecting that the second address and the first data identifier to be transmitted have been written to the second graphics processing unit, write the second address and the first data identifier to be transmitted of the second graphics processing unit into the first central processing unit. Specifically, the first fast computing link controller can monitor the MSI-X written to the aforementioned memory space. This MSI-X can record the aforementioned second address and the first data identifier to be transmitted. Subsequently, upon detecting the aforementioned MSI-X, the first fast computing link controller can encapsulate the second address and the first data identifier to be transmitted recorded by the MSI-X into a TLP, and write the TLP to the first central processing unit via the fast peripheral component interconnect bus.

[0122] Based on this, by establishing the association between the first and second addresses during the pre-compilation stage, dynamic addressing and handshaking operations are avoided during subsequent inference / training. Thus, this embodiment decouples the data transmission control flow from the graphics processing unit (GPU) execution flow, enabling a low-overhead, high-concurrency asynchronous communication process of "write-triggered, automatic forwarding" during the actual communication phase. This achieves asynchronous and efficient execution of the communication process, significantly improving the startup efficiency and system scalability of cross-node GPU communication. The overall solution, with the GPU acting as the first server (the sender), does not require native support for the Fast Compute Link (FCL) protocol. It can leverage the third and second types of devices in the FCL ecosystem, along with the MSI-X interrupt mechanism, to establish a low-latency, zero-software-intervention-by-the-central-processing-unit (CPU) heterogeneous communication link. This design is compatible with the GPU architectures of various vendors and has good engineering feasibility, making it particularly suitable for accelerating large-model distributed inference and training in a heterogeneous GPU hybrid deployment environment on cloud platforms.

[0123] The above has completed the description of the zero-copy memory partitioning process in this application. Then, in conjunction with the following figures, this application will first... Figure 1BThe specific structures of the first server and the second server shown will be described below. Firstly, for the first server of this application, the first central processing unit may include a first composite unit. For example, the first composite unit may be connected to the first graphics processing unit via a first fast peripheral component interconnect bus. Furthermore, a first fast computing link interconnect channel is configured between the first composite unit and the first memory expansion device. For example, the first fast computing link interconnect channel may be a communication link based on a memory interconnect protocol. Specifically, it may be a communication link based on the CXL.I / O sub-protocol, distinct from a bus connection, or a communication link based on the CXL.mem sub-protocol. Thus, the first graphics processing unit can transmit first data to be transmitted to the first composite unit via the first fast peripheral component interconnect bus, and then write the first data to be transmitted to the first memory expansion device via the first fast computing link interconnect channel. The following will be combined with… Figure 2A Please provide a detailed explanation.

[0124] Figure 2A It is shown that according to this application Figure 1B A schematic diagram of the first server in another embodiment.

[0125] like Figure 2A As shown, the first server in this embodiment may include a first graphics processing unit, a first central processing unit, and a first memory expansion device. The first graphics processing unit may include a computing unit and a corresponding DDR. The first central processing unit may include a first composite unit.

[0126] Furthermore, the first composite unit may include a first fast peripheral component interconnect root port and a first fast compute link root port. The first fast peripheral component interconnect root port and the first fast compute link root port can be connected via an internal routing bus within the first composite unit. For example, the internal routing bus may be a memory interconnect protocol-based bus within the first composite unit.

[0127] Based on this, the first graphics processing unit can transmit the first data to be transmitted to the first fast peripheral component interconnection root port via the first fast peripheral component interconnection bus, so as to transmit it to the first fast computing link root port via the first composite internal routing bus, and then write it to the first memory expansion device via the first fast computing link interconnection channel.

[0128] Furthermore, the first composite unit may also include a second fast computing link root port. For example, the first fast computing link root port and the second fast computing link root port are connected via an internal interconnect bus of the first fast computing link. For example, the internal interconnect bus of the first fast computing link may be a bus based on a memory interconnect protocol within the first composite unit.

[0129] The first central processing unit (CPU) can read the first data to be transmitted from the first memory expansion device and transmit the first data to be transmitted to the first fast computing link root port via the first fast computing link interconnect channel, and then transmit it to the second fast computing link root port via the first fast computing link internal interconnect bus. The first composite unit and the fast computing link switch are connected via the first fast computing link external interconnect bus. For example, the first fast computing link external interconnect bus can be a bus based on a memory interconnect protocol. Thus, the first composite unit can transmit the first data to be transmitted to the second server via the fast computing link switch through the first fast computing link external interconnect bus. Specifically, the first CPU can transmit the first data to be transmitted from the first composite unit to the fast computing link switch via the first fast computing link external interconnect bus, so that the fast computing link switch transmits the first data to be transmitted to the second server according to the second address.

[0130] The specific structure of the first server has been described above. The specific structure of the second server will be described below with reference to the attached diagram.

[0131] Figure 2B It is shown that according to this application Figure 1B A schematic diagram of a second server according to another embodiment.

[0132] like Figure 2B As shown, the second server may include a second graphics processing unit, a second central processing unit, and a second memory expansion device. The second graphics processing unit may include a computing unit and corresponding DDR. The second central processing unit may include a second composite unit. A second fast computing link interconnect channel is configured between the second memory expansion device and the second composite unit. For example, the second fast computing link interconnect channel may be a communication link based on a memory interconnect protocol. Specifically, it may be a communication link based on the CXL.I / O sub-protocol, distinct from a bus connection, or a communication link based on the CXL.mem sub-protocol. The second composite unit and the fast computing link switch can be connected via a second fast computing link external interconnect bus. For example, the second fast computing link external interconnect bus may be a bus based on a memory interconnect protocol.

[0133] Based on this, the second central processing unit can pre-transmit the second address and the first data identifier to be transmitted to the second composite unit via the second fast computing link interconnection channel. Then, the second composite unit can transmit the second address and the first data identifier to be transmitted to the fast computing link switch via the second fast computing link external interconnection bus, and then transmit the second address and the first data identifier to be transmitted to the first memory expansion device via the fast computing link switch, so that the first central processing unit can associate and store the first address and the second address based on the first data identifier to be transmitted. Then, the first central processing unit can send the first data to be transmitted to the second server based on the associated stored first address, second address, and first data identifier to be transmitted.

[0134] Furthermore, the second central processing unit can transmit the first data to be transmitted from the high-speed computing link switch to the second composite unit via the second high-speed computing link external interconnect bus, and write it to the second memory expansion device according to the second address via the second high-speed computing link interconnect channel. For example, if the second central processing unit detects that the second address has been written to the second graphics processing unit, it can read the first data to be transmitted from the second memory expansion device according to the second address via the second high-speed computing link interconnect channel, and transmit the first data to be transmitted to the second graphics processing unit via the second high-speed peripheral component interconnect bus.

[0135] Furthermore, the second composite unit may include a third fast computing link root port and a fourth fast computing link root port. For example, the third fast computing link root port is connected to the fast computing link switch via a second fast computing link external interconnect bus. The third fast computing link root port and the fourth fast computing link root port are connected via a second fast computing link internal interconnect bus. For example, the second fast computing link internal interconnect bus can be a bus based on a memory interconnect protocol. A second fast computing link interconnect channel is configured between the fourth fast computing link root port and the second memory expansion device. In this way, the second central processing unit can transmit the first data to be transmitted from the fast computing link switch to the third fast computing link root port via the second fast computing link external interconnect bus, then transmit the first data to be transmitted to the fourth fast computing link root port via the second fast computing link internal interconnect bus, and finally write the first data to be transmitted to the second memory expansion device via the second fast computing link interconnect channel.

[0136] Furthermore, the second composite unit may also include a second Fast Peripheral Component Interconnect (FPCI) root port. For example, the second FPCI root port is connected to the fourth Fast Compute Link (CCL) root port via the second composite internal routing bus. The second central processing unit (CPU) can, upon detecting that a second address has been written to the second graphics processing unit (GPU), read the first data to be transmitted from the memory of the second memory expansion device via the second CCL interconnect channel according to the second address. Then, the second CPU can transmit the first data to be transmitted to the fourth CCL root port, and then transmit it to the second FPCI root port via the second composite internal routing bus. Afterward, the second CPU can transmit the first data to be transmitted to the second GPU via the second FPCI bus. Thus, the second GPU can perform computational tasks based on the first data to be transmitted.

[0137] Based on this, since the DMA engines deployed in the first and second graphics processing units have the advantage of parallel high-speed transmission, there is no need for the central processing unit supporting the three sub-protocols of the fast computing link to participate, and memory can be read and written directly, thereby significantly improving data transmission efficiency. In addition, compared with the scheme that is difficult to use Load / Store instructions (or DMA instructions based on the CXL.I / O sub-protocol) after transferring the first data to be transmitted to the DRAM (Dynamic Random Access Memory) memory module based on zero-copy technology, this application embodiment can realize the use of high-speed CXL.mem and CXL.I / O sub-protocols such as Load / Store instructions (or DMA instructions based on the CXL.I / O sub-protocol) to write the first data to be transmitted to the second memory expansion device of the second server through the fast computing link switch, based on the fast computing link protocol and zero-copy technology, thereby realizing high-bandwidth and low-latency cross-server node communication of graphics processing units.

[0138] Additionally, after the distributed deep learning model inference / training phase is completed, the first server can call the resource cleanup function to release the resources of the first extended memory pre-allocated by the first server, the resources of the memory monitoring register of the first central processing unit, and the stop interrupt status, etc., and the second server can call the resource cleanup function to release the resources of the second extended memory pre-allocated by the second server, the resources of the memory monitoring register of the second central processing unit, and the stop interrupt status, etc.

[0139] Furthermore, when the first server is the receiver, the first central processing unit can partition the memory of the first memory expansion device to obtain a second extended memory for storing the second data to be transmitted, based on the storage space required for the second data to be transmitted. Then, the third address of the second extended memory can be associated with the identifier of the second data to be transmitted. The third address and the second data identifier are then transmitted to the second memory expansion device in the second server via a fast computing link switch. Specifically, the first central processing unit can transmit the third address and the second data identifier to the first composite unit via the first fast computing link interconnect channel, then to the fast computing link switch via the first fast computing link external interconnect bus, and finally to the second server via the fast computing link switch, so that the second server can associate and store the third address and the second data identifier.

[0140] Based on this, the first central processing unit (CPU) can transmit the second data to be transmitted to the first composite unit via the first fast computing link external interconnect bus, and write it to the first memory expansion device according to the third address via the first fast computing link interconnect channel. Specifically, the CPU can transmit the second data to be transmitted to the second fast computing link root port via the first fast computing link external interconnect bus, transmit it to the first fast computing link root port via the first fast computing link internal interconnect bus, and write it to the first memory expansion device via the first fast computing link interconnect channel. The first fast computing link controller can write the third address to the first CPU when it detects that the second data to be transmitted has been written to the second extended memory. Further, the CPU can read the second data to be transmitted from the first memory expansion device according to the third address via the first fast computing link interconnect channel when it detects that the third address has been written to the CPU, and transmit it to the first graphics processing unit via the first fast peripheral component interconnect bus. Specifically, upon detecting that the third address has been written to the first graphics processing unit, the first central processing unit can read the second data to be transmitted from the first memory expansion device according to the third address via the first fast computing link interconnection channel, transmit the second data to be transmitted to the first fast computing link root port, transmit it via the first composite internal routing bus to the first fast peripheral component interconnection root port, and transmit it to the first graphics processing unit via the first fast peripheral component interconnection bus. The above is merely an example; it should be understood that when the first server acts as the receiver, the specific functions that can be implemented are similar to those of the second server. Therefore, the description of the second server in this application is sufficient, and will not be elaborated upon here.

[0141] Figure 2C It is shown that according to this application Figure 1B A schematic diagram of a first server and a second server according to another embodiment. (See diagram below.) Figure 2CAs shown, the cross-node transmission path based on the Fast Computing Link protocol can be as follows: a first memory expansion device, a first Fast Computing Link interconnect channel (e.g., CXL.I / O channel or CXL.mem channel), a first Fast Computing Link root port, a first Fast Computing Link internal interconnect bus, a second Fast Computing Link root port, a first Fast Computing Link external interconnect bus, a first switch port, a second switch port, a second Fast Computing Link external interconnect bus, a fourth Fast Computing Link root port, a second Fast Computing Link internal interconnect bus, a third Fast Computing Link root port, and a second memory expansion device. The switch can be used for data transmission based on the Fast Computing Link protocol. The first and second switch ports can be Fast Computing Link root ports.

[0142] Although the first composite unit is located within the first central processing unit and the second composite unit is located within the second central processing unit, since the first and second composite units are hardware path modules, there is no need to call the first or second central processing unit to intervene when the second fast computing link root port interacts with the first switch port or the third fast computing link root port interacts with the second switch port. This saves the processing time of the first and second central processing units on the first data to be transmitted during cross-node data transmission.

[0143] Furthermore, since both the first and second memory expansion devices support the CXL.mem and CXL.I / O sub-protocols, small-scale data interaction can be performed using the Load / Store method based on the CXL.mem sub-protocol, while large-scale concurrent data interaction can be performed using the DMA method based on the CXL.I / O sub-protocol. Therefore, this application provides an interaction method based on two sub-protocols, improving interaction flexibility. Moreover, compared to the cross-node communication technology for graphics processing units assisted by the RDMA network card described above, this application's solution saves the time consumed by data packet assembly and depacketization, achieving shorter processing latency and greater data transmission bandwidth across server nodes.

[0144] Furthermore, in the above-described scheme of this application embodiment, after the first data to be transmitted is transferred from the first memory expansion device to the second memory expansion device based on the Fast Compute Link protocol, the second graphics processing unit can directly read the first data to be transmitted. During this reading process, a high-speed direct data transmission channel is established between the second graphics processing unit and the second memory expansion device without the intervention of the central processing unit for forwarding. In this process, the computing unit of the second graphics processing unit does not need to call its internal DDR, the central processing unit supporting the three sub-protocols of Fast Compute Link does not need to intervene during the forwarding process, and the zero-copy memory of the third type of device (i.e., the second memory expansion device) is directly written to locally without going through the cache. This improves data transmission efficiency. Moreover, the second graphics processing unit can directly read the second memory expansion device based on the DMA engine, and this process also does not require the participation of the central processing unit supporting the three sub-protocols of Fast Compute Link, greatly improving data transmission efficiency.

[0145] Figure 3A It is shown that according to this application Figure 1C A schematic diagram of the first server in another embodiment.

[0146] like Figure 3A As shown, the first server in this embodiment may include a first graphics processing unit and a first central processing unit. The first graphics processing unit may include a computing unit, a first memory, and a first fast computing link controller. The first central processing unit may include a first root composite unit.

[0147] The computing unit can be connected to the first fast computing link controller via a second double-rate bus, and the first fast computing link controller can be connected to the first memory via the first double-rate bus. The second and first double-rate buses can be buses based on a memory interconnect protocol. The first graphics processing unit can transmit the first data to be transmitted from the computing unit to the first fast computing link controller via the second double-rate bus, and then write the first data to be transmitted to the first memory via the first double-rate bus. In this way, high-speed writing of the first data to be transmitted can be achieved based on a memory interconnect protocol. The first fast computing link controller can parse the data packets from the computing unit to generate the first data to be transmitted. The data packets may include the calculation results of the computing unit. Based on this, the first fast computing link controller can parse the data packets to obtain the calculation results. Then, the first fast computing link controller can encapsulate the calculation results to generate the first data to be transmitted using the fast computing link protocol.

[0148] Furthermore, the first composite unit may include a first fast peripheral component interconnect root port and a second fast compute link root port. The first fast compute link root port and the fast compute link switch can be connected via a first fast compute link external interconnect bus. The first fast peripheral component interconnect root port and the first fast compute link root port can be connected via a first fast compute link internal interconnect bus. The first fast compute link external interconnect bus and the first fast compute link internal interconnect bus can be buses based on a memory interconnect protocol.

[0149] Based on this, the first central processing unit can read the first data to be transmitted from the first memory, and transmit the first data to be transmitted to the first fast peripheral component interconnect root port via the first fast computing link interconnect channel, then to the first fast computing link root port via the first fast computing link internal interconnect bus, and finally to the second server via the fast computing link switch via the first fast computing link external interconnect bus. Because the first composite unit is a hardware path module, when the first graphics processing unit interacts with the first memory expansion device, there is no need to call the first central processing unit for software intervention, thereby saving the time consumed by the first central processing unit in processing the first data to be transmitted during data transmission.

[0150] Furthermore, compared to the high-speed peripheral component interconnect bus, the second double-speed bus and the first double-speed bus have larger transmission bandwidth and lower transmission latency. Moreover, the computing unit and the first memory are both located in the first graphics processing unit, which belongs to the second type of device of the high-speed computing link, and do not occupy the data transmission channel other than the first graphics processing unit, thus saving channel resources.

[0151] The specific structure of the first server has been described above. The specific structure of the second server will be described below with reference to the attached diagram.

[0152] Figure 3B It is shown that according to this application Figure 1C A schematic diagram of a second server according to another embodiment.

[0153] like Figure 3BAs shown, the second server may include a second graphics processing unit, a second central processing unit, and a second memory expansion device. The second graphics processing unit may include a computing unit and corresponding DDR. The second central processing unit may include a second composite unit. A second fast computing link interconnect channel is configured between the second memory expansion device and the second composite unit. For example, the second fast computing link interconnect channel may be a communication link based on a memory interconnect protocol. Specifically, it may be a communication link based on the CXL.I / O sub-protocol, distinct from a bus connection, or a communication link based on the CXL.mem sub-protocol. The second composite unit and the fast computing link switch can be connected via a second fast computing link external interconnect bus. For example, the second fast computing link external interconnect bus may be a bus based on a memory interconnect protocol.

[0154] Based on this, the second central processing unit can pre-transmit the second address and the first data identifier to be transmitted to the second composite unit via the second fast computing link interconnection channel. Then, the second composite unit can transmit the second address and the first data identifier to be transmitted to the fast computing link switch via the second fast computing link external interconnection bus, and then transmit the second address and the first data identifier to be transmitted to the first memory expansion device via the fast computing link switch, so that the first central processing unit can associate and store the first address and the second address based on the first data identifier to be transmitted. Then, the first central processing unit can send the first data to be transmitted to the second server based on the associated stored first address, second address, and first data identifier to be transmitted.

[0155] Furthermore, the second central processing unit can transmit the first data to be transmitted from the high-speed computing link switch to the second composite unit via the second high-speed computing link external interconnect bus, and write it to the second memory expansion device according to the second address via the second high-speed computing link interconnect channel. For example, if the second central processing unit detects that the second address has been written to the second graphics processing unit, it can read the first data to be transmitted from the second memory expansion device according to the second address via the second high-speed computing link interconnect channel, and transmit the first data to be transmitted to the second graphics processing unit via the second high-speed peripheral component interconnect channel.

[0156] Furthermore, the second composite unit may include a third fast computing link root port and a fourth fast computing link root port. For example, the third fast computing link root port is connected to the fast computing link switch via a second fast computing link external interconnect bus. The third fast computing link root port and the fourth fast computing link root port are connected via a second fast computing link internal interconnect bus. For example, the second fast computing link internal interconnect bus can be a bus based on a memory interconnect protocol. A second fast computing link interconnect channel is configured between the fourth fast computing link root port and the second memory expansion device. In this way, the second central processing unit can transmit the first data to be transmitted from the fast computing link switch to the third fast computing link root port via the second fast computing link external interconnect bus, then transmit the first data to be transmitted to the fourth fast computing link root port via the second fast computing link internal interconnect bus, and finally write the first data to be transmitted to the second memory expansion device via the second fast computing link interconnect channel.

[0157] Furthermore, the second root complex unit may also include a second Fast Peripheral Component Interconnect (FPCI) root port. For example, the second FPCI root port is connected to the fourth Fast Compute Link (CCL) root port via the root complex internal routing bus. The second central processing unit (CPU) can read the first data to be transmitted from the second memory expansion device according to the second address via the second CCL interconnect channel, upon detecting that the second address has been written to the second graphics processing unit (GPU). Then, the second CPU can transmit the first data to be transmitted to the fourth CCL root port, and then transmit it to the second FPCI root port via the second root complex internal routing bus. Afterward, the second CPU can transmit the first data to be transmitted to the second GPU via the second FPCI interconnect channel. Thus, the second GPU can perform computational tasks based on the first data to be transmitted.

[0158] Based on this, since the second composite unit is a hardware path module, when the first data to be transmitted is written into the second extended memory of the second graphics processing unit, there is no need to call the second central processing unit for software intervention, thereby saving the time consumed by the second central processing unit in processing the first data to be transmitted during the data transmission process.

[0159] Based on this, since the DMA engine deployed in the second graphics processing unit has the advantage of parallel high-speed transmission, the central processing unit supporting the three sub-protocols of the fast computing link does not need to participate, and can directly read and write memory, thereby significantly improving data transmission efficiency. Furthermore, compared to schemes that, after transferring the first data to be transmitted to the DRAM memory module using zero-copy technology, struggle to transmit data using Load / Store instructions of the CXL.mem sub-protocol (or DMA instructions based on the CXL.I / O sub-protocol), this application, based on the fast computing link protocol and zero-copy technology, can utilize high-speed CXL.mem and CXL.I / O sub-protocols such as Load / Store instructions of the CXL.mem sub-protocol (or DMA instructions based on the CXL.I / O sub-protocol) to write the first data to be transmitted to the second memory expansion device of the second server via the fast computing link switch, achieving high-bandwidth and low-latency cross-server node communication between the graphics processing unit and the server node.

[0160] Additionally, after the distributed deep learning model inference / training phase is completed, the first server can call the resource cleanup function to release the resources of the first memory pre-allocated by the first graphics processing unit, the resources of the memory monitoring register of the first central processing unit, and the interrupt status, etc., and the second server can call the resource cleanup function to release the resources of the second extended memory pre-allocated by the second memory extension device, the resources of the memory monitoring register of the second central processing unit, and the interrupt status, etc.

[0161] Figure 3C It is shown that according to this application Figure 1C A schematic diagram of a first server and a second server according to another embodiment.

[0162] like Figure 3C As shown, the cross-node transmission path based on the Fast Computing Link protocol can be as follows: a first graphics processing unit, a first Fast Peripheral Component Interconnect channel (e.g., CXL.I / O channel or CXL.mem channel), a first Fast Peripheral Component Interconnect root port, a first Fast Computing Link internal interconnect bus, a first Fast Computing Link root port, a first Fast Computing Link external interconnect bus, a first switch port, a second switch port, a second Fast Computing Link external interconnect bus, a third Fast Computing Link root port, a second Fast Computing Link internal interconnect bus, a fourth Fast Computing Link root port, a second Fast Computing Link interconnect channel, and a memory expansion device. The Fast Computing Link switch can transmit data based on the Fast Computing Link protocol. The first switch port and the second switch port can be Fast Computing Link root ports.

[0163] Although the first composite unit is located within the first central processing unit and the second composite unit is located within the second central processing unit, since the first and second composite units are hardware path modules, there is no need to call the first or second central processing unit for software intervention when the first fast computing link root port interacts with the first switch port or the third fast computing link root port interacts with the second switch port. This saves the processing time of the first and second central processing units for the first data to be transmitted during cross-node data transmission.

[0164] Furthermore, since both the devices between the first graphics processing unit and the second memory expansion device support the CXL.mem and CXL.I / O sub-protocols, small-scale data interaction can be performed using the Load / Store method based on the CXL.mem sub-protocol, and large-scale concurrent data interaction can be performed using the DMA method based on the CXL.I / O sub-protocol. Therefore, this embodiment provides an interaction method based on two sub-protocols, improving interaction flexibility. Moreover, compared to the cross-node communication technology of the graphics processing unit assisted by the RDMA network card described above, this embodiment saves the time consumed by data packet assembly and depacketization, achieving shorter processing latency and greater data transmission bandwidth across server nodes.

[0165] Furthermore, in the above-described scheme of this application embodiment, after the first data to be transmitted is transferred from the first graphics processing unit to the second memory expansion device based on the fast computing link protocol, the second graphics processing unit can directly read the first data to be transmitted. During this reading process, a high-speed direct data transmission channel is established between the second graphics processing unit and the second memory expansion device, which does not require software intervention from the central processing unit. In this process, the computing unit of the second graphics processing unit does not need to call its internal DDR, the software intervention of the central processing unit supporting the three sub-protocols of the fast computing link is not required during the forwarding process, and the zero-copy memory of the third type of device (i.e., the memory expansion device) is written directly to the local memory without caching. This improves data transmission efficiency. Moreover, the second graphics processing unit can directly read the memory expansion device based on the DMA engine, a process that also does not require the participation of the central processing unit supporting the three sub-protocols of the fast computing link, significantly improving data transmission efficiency.

[0166] Figure 4A It is shown that according to this application Figure 1D A schematic diagram of the first server in another embodiment.

[0167] like Figure 4AAs shown, the first server in this embodiment may include a first graphics processing unit, a first central processing unit, and a first memory expansion device. The first graphics processing unit may include a computing unit and a corresponding DDR. The first central processing unit may include a first composite unit.

[0168] Furthermore, the first composite unit may include a first fast peripheral component interconnect root port and a first fast compute link root port. For example, the first fast peripheral component interconnect root port and the first fast compute link root port may be connected via an internal routing bus within the first composite unit. For example, the internal routing bus may be a memory interconnect protocol-based bus within the first composite unit.

[0169] Based on this, the first graphics processing unit can transmit the first data to be transmitted to the first fast peripheral component interconnect root port via the fast peripheral component interconnect bus, so as to transmit it to the first fast computing link root port via the first composite internal routing bus, and then write it to the first memory expansion device via the first fast computing link interconnect channel.

[0170] Furthermore, the first composite unit may also include a second fast computing link root port. For example, the first fast computing link root port and the second fast computing link root port are connected via an internal interconnect bus of the first fast computing link. For example, the internal interconnect bus of the first fast computing link may be a bus based on a memory interconnect protocol within the first composite unit.

[0171] The first central processing unit (CPU) can read the first data to be transmitted from the first memory expansion device and transmit the first data to be transmitted to the first fast computing link root port via the first fast computing link interconnect channel, and then transmit it to the second fast computing link root port via the first fast computing link internal interconnect bus. The first composite unit and the fast computing link switch are connected via the first fast computing link external interconnect bus. For example, the first fast computing link external interconnect bus can be a bus based on a memory interconnect protocol. Thus, the first composite unit can transmit the first data to be transmitted to the second server via the fast computing link switch through the first fast computing link external interconnect bus. Specifically, the first CPU can transmit the first data to be transmitted from the first composite unit to the fast computing link switch via the first fast computing link external interconnect bus, so that the fast computing link switch transmits the first data to be transmitted to the second server according to the second address.

[0172] Based on this, since the first composite unit is a hardware path module, when the first graphics processing unit interacts with the first memory expansion device, there is no need to call the first central processing unit for software intervention, thereby saving the time consumed by the first central processing unit in processing the first data to be transmitted during the data transmission process.

[0173] The specific structure of the first server has been described above. The specific structure of the second server will be described below with reference to the attached diagram.

[0174] Figure 4B It is shown that according to this application Figure 1D A schematic diagram of a second server according to another embodiment.

[0175] like Figure 4B As shown, the second server may include a second graphics processing unit and a second central processing unit. The second graphics processing unit may include a second fast compute link controller and a second memory. The second central processing unit may include a second root composite unit. For example, the second root composite unit may be a root composite. A second fast compute link interconnect channel is configured between the second graphics processing unit and the second root composite unit. For example, the second fast compute link interconnect channel may be a communication link based on a memory interconnect protocol. Specifically, it may be a communication link based on the CXL.I / O sub-protocol, distinct from a bus connection, or a communication link based on the CXL.mem sub-protocol. The second root composite unit and the fast compute link switch may be connected via a second fast compute link external interconnect bus. For example, the second fast compute link external interconnect bus may be a bus based on a memory interconnect protocol.

[0176] Based on this, the second central processing unit can pre-transmit the second address and the first data identifier to be transmitted to the second composite unit via the second fast computing link interconnection channel. Then, the second composite unit can transmit the second address and the first data identifier to be transmitted to the fast computing link switch via the second fast computing link external interconnection bus, and then transmit the second address and the first data identifier to be transmitted to the first memory expansion device via the fast computing link switch, so that the first central processing unit can associate and store the first address and the second address based on the first data identifier to be transmitted. Then, the first central processing unit can send the first data to be transmitted to the second server based on the associated stored first address, second address, and first data identifier to be transmitted.

[0177] Furthermore, the second central processing unit can transmit the first data to be transmitted from the fast computing link switch to the second composite unit via the second fast computing link external interconnect bus, and write it into the memory of the second graphics processing unit (e.g., the second memory) via the second fast computing link interconnect channel according to the second address.

[0178] Furthermore, the second composite unit may include a third fast computing link root port and a fourth fast computing link root port. For example, the third and fourth fast computing link root ports may be fast computing link root ports. The third fast computing link root port is connected to the fast computing link switch via a second fast computing link external interconnect bus. The third and fourth fast computing link root ports are connected via a third fast computing link internal interconnect bus. For example, the second fast computing link internal interconnect bus may be a bus based on a memory interconnect protocol. A second fast computing link interconnect channel is configured between the fourth fast computing link root port and the second graphics processing unit. In this way, the second central processing unit can transmit the first data to be transmitted from the fast computing link switch to the third fast computing link root port via the second fast computing link external interconnect bus, then transmit the first data to be transmitted to the fourth fast computing link root port via the second fast computing link internal interconnect bus, and finally write the first data to be transmitted to the second graphics processing unit via the second fast computing link interconnect channel.

[0179] Furthermore, the second graphics processing unit may also include a computing unit. The computing unit and the second fast compute link controller can be connected via a first double-rate bus, and the second fast compute link controller and the second memory can be connected via a second double-rate bus. Thus, when the second central processing unit detects that a second address has been written to the second graphics processing unit, it can read the first data to be transmitted from the second memory according to the second address via the second fast compute link interconnection channel, then transmit the first data to be transmitted to the second fast compute link controller via the second double-rate bus, and then transmit it to the computing unit via the first double-rate bus. In this way, the computing unit can perform computational tasks based on the first data to be transmitted.

[0180] Based on this, since the second composite unit is a hardware path module, when the first data to be transmitted is written into the second memory of the second graphics processing unit, there is no need to call the second central processing unit for software intervention, thereby saving the time consumed by the second central processing unit in processing the first data to be transmitted during the data transmission process.

[0181] Furthermore, compared to the high-speed peripheral component interconnect bus, the first double-rate bus and the second double-rate bus have larger transmission bandwidth and lower transmission latency. Moreover, the computing unit and the second memory are both located in the first central processing unit, which belongs to the second type of high-speed computing link device, and do not occupy data transmission channels other than the first central processing unit, thus saving channel resources.

[0182] Based on this, since the DMA engine deployed in the first graphics processing unit has the advantage of parallel high-speed transmission, there is no need for the central processing unit supporting the three sub-protocols of the fast computing link to participate, and memory can be read and written directly, thereby significantly improving data transmission efficiency. In addition, compared with the scheme that is difficult to transmit data based on the Load / Store instruction of the CXL.mem sub-protocol (or the DMA instruction based on the CXL.I / O sub-protocol) after the first data to be transmitted is transferred to the DRAM memory module based on zero-copy technology, this application embodiment can realize the use of high-speed CXL.mem and CXL.I / O sub-protocols such as the Load / Store instruction of the CXL.mem sub-protocol (or the DMA instruction based on the CXL.I / O sub-protocol) to write the first data to be transmitted into the memory of the second graphics processing unit of the second server through the fast computing link switch, based on the fast computing link protocol and zero-copy technology, thereby realizing high-bandwidth and low-latency cross-server node communication of the graphics processing unit.

[0183] Additionally, after the distributed deep learning model inference / training phase is completed, the first server can call the resource cleanup function to release the resources of the first extended memory pre-allocated by the first memory extension device, the resources of the memory monitoring register of the first central processing unit, and the stop interrupt status, etc., and the second server can call the resource cleanup function to release the resources of the second memory pre-allocated by the second graphics processing unit, the resources of the memory monitoring register of the second central processing unit, and the stop interrupt status, etc.

[0184] Figure 4C It is shown that according to this application Figure 1D A schematic diagram of a first server and a second server according to another embodiment.

[0185] like Figure 4C As shown, the cross-node transmission path based on the Fast Computing Link protocol can be as follows: memory expansion device, first Fast Computing Link interconnect channel (e.g., CXL.I / O channel or CXL.mem channel), first Fast Computing Link root port, first Fast Computing Link internal interconnect bus, second Fast Computing Link root port, first Fast Computing Link external interconnect bus, first switch port, second switch port, second Fast Computing Link external interconnect bus, third Fast Computing Link root port, second Fast Computing Link internal interconnect bus, fourth Fast Computing Link root port, second Fast Computing Link interconnect channel, and second graphics processing unit. The Fast Computing Link switch can transmit data based on the Fast Computing Link protocol. The first switch port and the second switch port can be Fast Computing Link root ports.

[0186] Although the first composite unit is located within the first central processing unit and the second composite unit is located within the second central processing unit, since the first and second composite units are hardware path modules, there is no need to call the first or second central processing unit for software intervention when the second fast computing link root port interacts with the first switch port or the third fast computing link root port interacts with the second switch port. This saves the processing time of the first and second central processing units for the first data to be transmitted during cross-node data transmission.

[0187] Furthermore, since both the memory expansion device and the second graphics processing unit support the CXL.mem and CXL.I / O sub-protocols, small-scale data interaction can be performed using the Load / Store method based on the CXL.mem sub-protocol, while large-scale concurrent data interaction can be performed using the DMA method based on the CXL.I / O sub-protocol. Therefore, this application embodiment provides an interaction method based on two sub-protocols, improving interaction flexibility. Moreover, compared to the graphics processing unit cross-node communication technology assisted by the RDMA network card described above, this application embodiment's solution saves the time consumed by data packet assembly and depacketization, achieving shorter processing latency and greater data transmission bandwidth across server nodes.

[0188] Furthermore, in the above-described scheme of this application embodiment, after the first data to be transmitted is transferred from the first memory extension device to the second memory of the second graphics processing unit based on the Fast Compute Link protocol, the second graphics processing unit can directly read the first data to be transmitted. During this reading process, a high-speed direct data transmission channel is established between the second graphics processing unit and the second memory, eliminating the need for software intervention from the central processing unit. In this process, no software intervention from the central processing unit supporting the three sub-protocols of the Fast Compute Link is required, and zero-copy memory (i.e., the second memory) of the second type of device is written directly to the local storage without caching. This improves data transmission efficiency. Moreover, the second graphics processing unit can directly read the first memory, a process that also does not require the participation of the central processing unit supporting the three sub-protocols of the Fast Compute Link, significantly improving data transmission efficiency.

[0189] Figure 5A It is shown that according to this application Figure 1B A schematic diagram of the first server in another embodiment.

[0190] like Figure 5AAs shown, in one embodiment of this application, the path for the first graphics processing unit to write the first data to be transmitted into the first memory expansion device can be as follows: the computing unit of the first graphics processing unit, the first fast peripheral component interconnect root port of the first composite unit, the first fast computing link root port of the first composite unit, and the first memory expansion device. Furthermore, in Figure 5A The diagram also shows the DDR of the first graphics processing unit and the first fast computing link controller of the first memory expansion device, which can be referred to in the previous description and will not be repeated here.

[0191] Since the first composite unit is a hardware path module, when the first graphics processing unit interacts with the first memory expansion device, there is no need to call the first central processing unit (i.e., the integrated memory controller (IMC)) for intervention, thereby saving the time consumed by the first central processing unit in processing the first data to be transmitted during data transmission.

[0192] Since the first memory expansion device supports the CXL.I / O sub-protocol, and the CXL.I / O channel reuses the Fast Peripheral Component Interconnect 5.0 and higher version bus (i.e. the first Fast Peripheral Component Interconnect bus), after the first graphics processing unit partitions the first extended memory, based on the high-speed transfer characteristics of DMA and the zero-copy memory characteristic that does not require the first central processing unit to forward and intervene, a high-speed direct data connection transmission channel between the first graphics processing unit and the first memory expansion device can be realized (i.e. the first Fast Peripheral Component Interconnect bus, the first composite internal routing bus, and the first fast computing link interconnect channel described above).

[0193] Furthermore, in the RDMA network card-assisted cross-node communication technology for graphics processing units described above, the data is sent by the DDR of the graphics processing unit. However, in this embodiment, the first data to be transmitted is sent by the computing unit of the first graphics processing unit. Thus, the first graphics processing unit of this application can directly call the first Fast Peripheral Component Interconnect bus to interact with the first Fast Peripheral Component Interconnect root port. This saves the time consumed by performing the write operation during the process where the computing unit of the first graphics processing unit writes the calculation result to the DDR, and then the DDR sends the first data to be transmitted.

[0194] In another approach, when the DRAM in the first server is used as zero-copy memory, the transmission path of the first data to be transmitted can be as follows: the computing unit of the first graphics processing unit, the first fast peripheral component interconnect bus, the first fast peripheral component interconnect root port, the internal interconnect bus of the first central processing unit, the memory channel port, the DDR bus, and the DRAM memory module. Compared with the data transmission path described above, in this approach, after the first data to be transmitted is transmitted to the first fast peripheral component interconnect root port of the first composite unit, it can be transmitted to the memory channel port via the internal interconnect bus of the first central processing unit, and then directly written into the zero-copy memory in the DRAM memory module via the DDR bus. This eliminates the need for the conversion between the fast peripheral component interconnect protocol data packets and the fast compute link protocol data packets via the first fast compute link root port, saving time consumed by protocol conversion. It should be noted that this method has a larger transmission bandwidth than the methods described above. However, this method struggles to transfer the first data to be transmitted to zero-copy memory in the third type of fast computing link device connected via a fast computing link switch in a receiving server that supports the fast computing link protocol, using the Load / Store instructions of the CXL.mem sub-protocol (or the DMA instructions of the CXL.I / O sub-protocol). Therefore, this embodiment does not allocate the first memory in the DRAM memory module, but instead allocates it in the memory of the first memory extension device to achieve cross-node data transmission based on the first scheme described above. Thus, this embodiment overcomes the limitations of traditional communication paths on the protocol compatibility of graphics processing units, thereby achieving efficient cross-node data transmission for graphics processing units.

[0195] Figure 5B It is shown that according to this application Figure 1B A schematic diagram of a second server according to another embodiment.

[0196] like Figure 5B As shown, in one embodiment of this application, the path by which the second graphics processing unit reads the first data to be transmitted from the second memory expansion device can be sequentially: the second extended memory of the second memory expansion device, the fourth fast computing link root port, the second fast peripheral component interconnect root port, and the computing unit of the second graphics processing unit. Furthermore, in Figure 5B The diagram also shows the DDR of the second graphics processing unit and the second fast computing link controller of the first memory expansion device, which can be referred to in the previous description and will not be repeated here.

[0197] Since the second composite unit is a hardware path module, when the second graphics processing unit interacts with the second memory expansion device, there is no need to call the second central processing unit (i.e., the IMC) for intervention, thereby saving the time consumed by the second central processing unit in processing the first data to be transmitted during the data transmission process.

[0198] Since the second memory expansion device supports the CXL.I / O sub-protocol, and the CXL.I / O channel reuses the Fast Peripheral Component Interconnect 5.0 and higher version bus (i.e., the second Fast Peripheral Component Interconnect bus), after the second graphics processing unit partitions the second extended memory, based on the high-speed transfer characteristics of DMA and the zero-copy memory characteristic that does not require the second central processing unit to forward and intervene, a high-speed direct data transmission channel can be realized between the second graphics processing unit and the second memory expansion device (i.e., the second Fast Peripheral Component Interconnect bus, the root complex internal routing bus, and the second fast compute link interconnect channel described above).

[0199] Furthermore, in the RDMA network card-assisted cross-node communication technology for graphics processing units described above, the data is received by the DDR of the graphics processing unit. However, in this embodiment, the first data to be transmitted is received by the computing unit of the second graphics processing unit. Thus, the second graphics processing unit in this embodiment can directly call the second fast peripheral component interconnect bus to interact with the second fast peripheral component interconnect root port. This saves the time consumed by performing the read operation in the process of "the calculation result is first written to the DDR, and then the computing unit of the first central processing unit reads the calculation result from the DDR".

[0200] Based on the above, this application provides a high-speed direct data transmission channel for the sender based on fast computing link and zero-copy technology: Within a sender server supporting the fast computing link protocol, a high-speed direct transmission channel is constructed between a sender graphics processing unit (GPU) that does not support the fast computing link protocol but has a DMA engine and a local fast computing link type 3 device. This channel achieves physical layer interoperability with the CXL.I / O protocol through the Fast Peripheral Component Interconnect 5.0 and above protocols, enabling the sender graphics processing unit (GPU) that does not support the fast computing link protocol but has a DMA engine to directly write data to the zero-copy memory allocated in the local fast computing link type 3 device. This eliminates the need for intervention from the local central processing unit supporting the three sub-protocols of the fast computing link and explicit caching, thus improving data writing efficiency.

[0201] Furthermore, for the aforementioned cross-node high-speed data transmission channel based on the Fast Computing Link Protocol, once the data writing is completed in the third type of Fast Computing Link device in the server supporting the Fast Computing Link Protocol, the Fast Computing Link controller of the third type of Fast Computing Link device in the local machine triggers an MSI-X interrupt. The central processing unit in the local machine that supports the three sub-protocols of Fast Computing Link completes the lookup mapping from send_addr to recv_addr. Then, through the CXL.mem or CXL.I / O sub-protocol via the Fast Computing Link switch, the data is directly transmitted to the third type of device (i.e., the second memory extension device) of the receiving server that supports the Fast Computing Link Protocol, realizing high-bandwidth, low-latency cross-node data transfer.

[0202] Furthermore, based on the high-speed direct data transmission channel of the receiver using fast computing link and zero-copy technology, a high-speed DMA read channel is constructed between the graphics processing unit (GPU) that does not support the fast computing link protocol but has a DMA engine and the local fast computing link third-type device. The GPU detects the arrival of data by polling the memory monitoring register at the application layer and directly retrieves the data from the local third-type device, completing the entire communication process without intervention from the central processing unit. For reasons similar to those described above, this embodiment does not partition the second memory in the DRAM memory module using the integrated memory controller and memory channel port of the second central processing unit. Instead, it partitions the second extended memory in the memory of the second memory extension device to achieve cross-node data transmission based on the first scheme described above. Therefore, this embodiment overcomes the limitations of traditional communication paths on the protocol compatibility of the GPU, thereby achieving efficient cross-node data transmission for the GPU.

[0203] Based on this, this application proposes an optimized communication mechanism for cross-node graphics processing units (GPUs) based on fast computation link polling, further improving communication efficiency. Compared to traditional cross-node GPU communication schemes assisted by RDMA network cards, this application skips the DDR relay, data copying, and reassembly operations between the GPU and the RDMA network card. Furthermore, in this application, the first central processing unit (CPU) triggers an MSI-X interrupt by writing the first data to be transmitted into the zero-copy memory of the first memory extension device. After the interrupt, the CPU automatically issues a remote write command without polling or software stack intervention. Subsequently, the second GPU can promptly detect the completion of data writing through a register polling mechanism and directly read the data via DMA. The overall process eliminates steps such as cache writing, network card packet assembly / disassembly, and target address mapping in the RDMA path, significantly reducing communication latency. This is particularly suitable for high-frequency communication scenarios such as frequent cross-node transmissions using key-value cache technology in large models. While improving system bandwidth utilization and heterogeneous communication efficiency, it also overcomes the technical barriers to GPU compatibility with fast computation link protocols.

[0204] Figure 6A It is shown that according to this application Figure 1C A schematic diagram of the first server in another embodiment.

[0205] like Figure 6A As shown, in one embodiment of this application, the path for the first graphics processing unit to write the first data to be transmitted into the first memory can be sequentially: the computing unit of the first graphics processing unit, the first fast computing link controller, and the first memory. Thus, the first central processing unit can read the first data to be transmitted from the first memory. Furthermore, Figure 6A It also includes the first fast peripheral component interconnect root port, the first fast computing link port and the first root composite unit, which can be referred to in the previous text and will not be repeated here.

[0206] Since the first composite unit is a hardware path module, when the first graphics processing unit interacts with the first memory, there is no need to call the first central processing unit (i.e., the integrated memory controller therein) for software intervention, thereby saving the time consumed by the first central processing unit in processing the first data to be transmitted during the data transmission process.

[0207] Since the first graphics processing unit supports the CXL.I / O sub-protocol, after the first graphics processing unit allocates the first memory, based on the characteristic of zero-copy memory that does not require the first central processing unit to forward data through software, a high-speed direct data transmission channel (i.e., the first double-rate bus and the second double-rate bus) can be realized between the first graphics processing unit and the first memory.

[0208] In another embodiment of this application, when the DRAM in the first server is used as zero-copy memory, the transmission path of the first data to be transmitted can be as follows: the computing unit of the first graphics processing unit, the first fast computing link interconnect channel, the first fast peripheral component interconnect root port, the internal interconnect bus of the first central processing unit, the memory channel port, the DDR bus, and the DRAM memory module. Compared with the data transmission path described above, in this embodiment, after the first data to be transmitted is transmitted to the first fast peripheral component interconnect root port of the first composite unit, it can be transmitted to the memory channel port via the internal interconnect bus of the first central processing unit, and then directly written into the zero-copy memory in the DRAM memory module via the DDR bus. This eliminates the need for conversion between the fast peripheral component interconnect protocol data packets and the fast computing link protocol data packets via the first fast computing link root port, saving time consumed by protocol conversion. It should be noted that this method has a larger transmission bandwidth than the methods described above. However, this method struggles to transfer the first data to be transmitted to zero-copy memory in the third type of fast computing link device connected via a fast computing link switch in a receiving server that supports the fast computing link protocol, using the Load / Store instructions of the CXL.mem sub-protocol (or the DMA instructions of the CXL.I / O sub-protocol). Therefore, this embodiment does not allocate the first memory in the DRAM memory module, but instead allocates it in the memory of the first graphics processing unit to achieve cross-node data transmission based on the first scheme described above. Based on this, this embodiment overcomes the limitations of traditional communication paths on the protocol compatibility of graphics processing units, thereby achieving efficient cross-node data transmission for graphics processing units.

[0209] Figure 6B It is shown that according to this application Figure 1C A schematic diagram of a second server according to another embodiment.

[0210] like Figure 6B As shown, in one embodiment of this application, the path by which the second graphics processing unit reads the first data to be transmitted from the second memory expansion device can be as follows: the second extended memory of the second memory expansion device, the fourth fast computing link root port, the second fast peripheral component interconnection root port, and the computing unit of the second graphics processing unit.

[0211] Since the second composite unit is a hardware path module, when the second graphics processing unit interacts with the second memory expansion device, there is no need to call the second central processing unit (i.e., the IMC) for software intervention, thereby saving the time consumed by the second central processing unit in processing the first data to be transmitted during data transmission.

[0212] Since the second memory expansion device supports the CXL.I / O sub-protocol, and the CXL.I / O channel reuses the Fast Peripheral Component Interconnect 5.0 and higher version bus (i.e., the second Fast Peripheral Component Interconnect bus), after the second graphics processing unit partitions the second extended memory, based on the high-speed transfer characteristics of DMA and the zero-copy memory feature that does not require the second central processing unit to perform forwarding intervention through software, a high-speed direct data connection transmission channel (i.e., Fast Peripheral Component Interconnect channel, routing bus, and Fast Compute Link Interconnect channel) can be realized between the second graphics processing unit and the second memory expansion device.

[0213] Furthermore, in the RDMA network card-assisted cross-node communication technology for graphics processing units described above, the data is received by the DDR of the graphics processing unit. However, in this embodiment, the first data to be transmitted is received by the computing unit of the second graphics processing unit. Thus, the second graphics processing unit in this embodiment can directly call the second fast peripheral component interconnect channel to interact with the root port of the fourth fast computing link. This saves the time consumed by performing the read operation in the process of "the calculation result is first written to the DDR, and then the computing unit of the first central processing unit reads the calculation result from the DDR".

[0214] Based on the above, this application provides a high-speed direct data transmission channel for the sender based on fast computation link and zero-copy technology: Within a sender server supporting the fast computation link protocol, an internal high-speed direct transmission channel is constructed for the sender's graphics processing unit, which also supports the fast computation link protocol. This channel, based on the fast computation link protocol, enables the sender's graphics processing unit to directly write data to locally allocated zero-copy memory, eliminating the need for software intervention and explicit caching by the central processing unit supporting the three sub-protocols of the fast computation link, thus improving data writing efficiency.

[0215] Furthermore, for the aforementioned cross-node high-speed data transmission channel based on the Fast Computing Link Protocol, once the first memory completes the data writing, the Fast Computing Link controller triggers an MSI-X interrupt. The local central processing unit supporting the three sub-protocols of the Fast Computing Link completes the lookup mapping from send_addr to recv_addr. Then, through the CXL.mem or CXL.I / O sub-protocol, the data is directly transmitted to the third type of device (i.e., memory extension device) of the receiving server supporting the Fast Computing Link Protocol via the Fast Computing Link switch, achieving high-bandwidth, low-latency cross-node data transfer.

[0216] Furthermore, based on the high-speed direct data transmission channel of the receiver using fast computing link and zero-copy technology, a high-speed DMA read channel is constructed between the graphics processing unit (GPU) that does not support the fast computing link protocol but has a DMA engine and the local fast computing link third-type device. The GPU detects the arrival of data by polling the memory monitoring register at the application layer and directly retrieves the data from the local third-type device, completing the entire communication process without software intervention from the central processing unit. For reasons similar to those described above, this embodiment does not partition the second memory in the DRAM memory module using the integrated memory controller and memory channel port of the second central processing unit. Instead, it partitions the second extended memory in the memory of the second memory extension device to achieve cross-node data transmission based on the first scheme described above. Based on this, this embodiment overcomes the limitations of traditional communication paths on the protocol compatibility of the GPU, thereby achieving efficient cross-node data transmission for the GPU.

[0217] Based on this, this application proposes an optimized cross-node communication mechanism for graphics processing units (GPUs) based on fast computation link polling, further improving communication efficiency. Compared to traditional cross-node GPU communication schemes assisted by RDMA network cards, this application skips the DDR relay, data copying, and reassembly operations between the GPU and the RDMA network card. Furthermore, in this application, the first GPU triggers an MSI-X interrupt by writing the first data to be transmitted into the first memory. Then, the first central processing unit automatically issues a remote write command after the interrupt, without polling or software stack intervention. Subsequently, the second GPU can promptly detect the completion of data writing through a register polling mechanism and directly read the data via DMA. The overall process eliminates steps such as cache writing, network card packet assembly / disassembly, and target address mapping in the RDMA path, significantly reducing communication latency. This is particularly suitable for high-frequency communication scenarios such as frequent cross-node transmission of key-value caching technology in large models. While improving system bandwidth utilization and heterogeneous communication efficiency, it also overcomes the technical barriers to the compatibility of fast computation link protocols with the GPU.

[0218] Figure 7A It is shown that according to this application Figure 1D A schematic diagram of the first server in another embodiment.

[0219] like Figure 7A As shown, in one embodiment of this application, the path by which the first graphics processing unit writes the first data to be transmitted into the first memory expansion device can be as follows: the computing unit of the first graphics processing unit, the first fast peripheral component interconnect root port of the first composite unit, the first fast computing link root port of the first composite unit, and the first extended memory of the first memory expansion device.

[0220] Since the first composite unit is a hardware path module, when the first graphics processing unit interacts with the first memory expansion device, there is no need to call the first central processing unit (i.e., the integrated memory controller therein) for software intervention, thereby saving the time consumed by the first central processing unit in processing the first data to be transmitted during the data transmission process.

[0221] Since the first memory expansion device supports the CXL.I / O sub-protocol, and the CXL.I / O channel reuses the Fast Peripheral Component Interconnect 5.0 and higher version bus (i.e., Fast Peripheral Component Interconnect bus), after the first graphics processing unit partitions the first extended memory, based on the high-speed transfer characteristics of DMA and the zero-copy memory characteristic that does not require the first central processing unit to perform forwarding intervention through software, a high-speed direct data connection transmission channel (i.e., Fast Peripheral Component Interconnect bus, first composite internal routing bus, and first fast computing link interconnect channel) can be realized between the first graphics processing unit and the first memory expansion device.

[0222] Furthermore, in the RDMA network card-assisted cross-node communication technology for graphics processing units described above, the data is sent by the DDR of the graphics processing unit. However, in this embodiment, the first data to be transmitted is sent by the computing unit of the first graphics processing unit. Thus, the first graphics processing unit in this embodiment can directly call the Fast Peripheral Component Interconnect (FPCI) bus to interact with the first FPCI root port. This saves the time consumed by performing the write operation in the process where the computing unit of the first graphics processing unit writes the calculation result to the DDR and then the DDR sends the first data to be transmitted.

[0223] In another scheme, when the DRAM in the first server is used as zero-copy memory, the transmission path of the first data to be transmitted can be as follows: the computing unit of the first graphics processing unit, the first fast peripheral component interconnect bus, the first fast peripheral component interconnect root port, the internal interconnect bus of the first central processing unit, the memory channel port, the DDR bus, and the DRAM memory module.

[0224] Compared to the data transmission path described above, in this scheme, the first data to be transmitted, after being transmitted to the first Fast Peripheral Component Interconnect (FPCI) root port of the first composite unit, can be transmitted to the memory channel port via the internal interconnect bus of the first central processing unit, and then directly written into the zero-copy memory in the DRAM memory module via the DDR bus. This eliminates the need for conversion between FPCI data packets and Fast Compute Link (CCL) data packets via the first Fast Compute Link root port, saving time consumed by protocol conversion. It should be noted that this method has a larger transmission bandwidth than the methods described above. However, this method is difficult to use the Load / Store instruction of the CXL.mem sub-protocol (or the DMA instruction of the CXL.I / O sub-protocol) to transmit the first data to be transmitted to the zero-copy memory in the third type of Fast Compute Link device in the receiving server connected via the Fast Compute Link switch that supports the Fast Compute Link protocol. Therefore, this embodiment does not allocate the first memory in the DRAM memory module, but instead allocates the first extended memory in the memory of the first memory extension device to achieve cross-node data transmission based on the first scheme described above. Based on this, the embodiments of this application break through the limitations of traditional communication paths on the protocol compatibility of graphics processing units, thereby realizing efficient data transmission of graphics processing units across nodes.

[0225] Figure 7B It is shown that according to this application Figure 1D A schematic diagram of a second server according to another embodiment.

[0226] like Figure 7B As shown, in one embodiment of this application, the path by which the computing unit reads the first data to be transmitted from the second memory of the second graphics processing unit can be as follows: second memory, second fast computing link controller and computing unit.

[0227] Based on this, compared with the related technologies that require the receiving RDMA network card to transfer and unpack data from the sending RDMA network card before transmitting it to the receiving memory, the computing unit of this application embodiment can directly interact with the second memory through the first memory bus and the second memory bus, thereby saving the time consumed by the RDMA network card in assembling and unpacking packets during data transmission.

[0228] Since the second composite unit is a hardware path module, when the first data to be transmitted is written into the second memory of the second graphics processing unit, there is no need to call the second central processing unit (i.e., the IMC) for software intervention, thereby saving the time consumed by the second central processing unit in processing the first data to be transmitted during the data transmission process.

[0229] Since the second graphics processing unit supports the CXL.I / O sub-protocol, and the CXL.I / O channel reuses the Fast Peripheral Component Interconnect 5.0 and higher version bus (i.e., the second Fast Compute Link Interconnect channel), after the second graphics processing unit partitions the second memory, based on the characteristic of zero-copy memory that does not require the second central processing unit to perform forwarding intervention through software, a high-speed direct data transmission channel (i.e., the first double-rate bus and the second double-rate bus) can be realized between the second graphics processing unit and the second memory.

[0230] Based on the above, this application provides a high-speed direct data transmission channel for the sender based on fast computing link and zero-copy technology: Within a sender server supporting the fast computing link protocol, a high-speed direct transmission channel is constructed between a sender graphics processing unit (GPU) that does not support the fast computing link protocol but has a DMA engine and a local fast computing link type 3 device. This channel achieves physical layer interoperability with the CXL.I / O protocol through the Fast Peripheral Component Interconnect 5.0 and above protocols, enabling the sender graphics processing unit (GPU) that does not support the fast computing link protocol but has a DMA engine to directly write data to the zero-copy memory allocated in the local fast computing link type 3 device. This eliminates the need for software intervention and explicit caching by the local central processing unit supporting the three sub-protocols of the fast computing link, thus improving data writing efficiency.

[0231] Furthermore, for the aforementioned cross-node high-speed data transmission channel based on the Fast Computing Link Protocol, once the data writing is completed in the third type of Fast Computing Link device in the server supporting the Fast Computing Link Protocol, the Fast Computing Link controller of the third type of Fast Computing Link device in the local machine triggers an MSI-X interrupt. The local central processing unit supporting the three sub-protocols of Fast Computing Link completes the lookup mapping from send_addr to recv_addr. Then, through the CXL.mem or CXL.I / O sub-protocol via the Fast Computing Link switch, it directly reaches the second type of device (i.e., the second graphics processing unit) of the receiving server supporting the Fast Computing Link Protocol, realizing high-bandwidth, low-latency cross-node data transfer.

[0232] Furthermore, based on the high-speed data direct transmission channel of the receiver using fast computing links and zero-copy technology, the receiver's graphics processing unit (GPU) directly retrieves data from the second memory after sensing its arrival by polling the memory monitoring register at the application layer, completing the entire communication process without software intervention from the central processing unit. For reasons similar to those described above, this embodiment does not partition the second memory in the DRAM memory module using the integrated memory controller and memory channel port of the second central processing unit. Instead, it partitions the second memory within the memory of the second GPU to achieve cross-node data transmission based on the first scheme described above. On this basis, this embodiment overcomes the limitations of traditional communication paths on GPU protocol compatibility, thereby achieving efficient cross-node data transmission for the GPU.

[0233] Based on this, this application proposes an optimized cross-node communication mechanism for graphics processing units (GPUs) based on fast computation link polling, further improving communication efficiency. Compared to traditional cross-node GPU communication schemes assisted by RDMA network cards, this application skips the DDR relay, data copying, and reassembly operations between the GPU and the RDMA network card. Furthermore, in this application, the first GPU triggers an MSI-X interrupt by writing the first data to be transmitted into the zero-copy memory of the first memory extension device. Then, the first central processing unit automatically issues a remote write command after the interrupt, without polling or software stack intervention. Subsequently, the second GPU can promptly detect the completion of data writing through a register polling mechanism and directly read the data from the second memory. The overall process eliminates steps such as cache writing, network card packet assembly / disassembly, and target address mapping in the RDMA path, significantly reducing communication latency. This is particularly suitable for high-frequency communication scenarios such as frequent cross-node transmission of key-value caching technology in large models. While improving system bandwidth utilization and heterogeneous communication efficiency, it also overcomes the technical barriers to the compatibility of fast computation link protocols with the GPU.

[0234] Figure 8A It is shown that according to this application Figure 1B A flowchart of the data transmission method in this embodiment.

[0235] As shown in Figure 8, the data transmission method of this embodiment can be applied to a first server deploying a distributed system. This data transmission method may include operations S811 to S813.

[0236] During operation S811, the first data to be transmitted is written to the first memory extension device according to the first address.

[0237] In operation S812, if it is detected that the first data to be transmitted has been written to the first memory extension device, the first address is written to the first central processing unit.

[0238] When operating S813, after detecting that the first address has been written to the first central processing unit, the second address is queried based on the first address, and the first data to be transmitted is transmitted to the second server according to the second address through a fast calculation link switch.

[0239] It should be understood that the first server in this application embodiment can also perform other operations as described above, which will not be repeated here.

[0240] Figure 8B It is shown that according to this application Figure 1B A flowchart of another embodiment of a data transmission method.

[0241] like Figure 8B As shown, the data transmission method of this embodiment can be applied to a second server deploying a distributed system. The data transmission method may include operations S821 to S823.

[0242] In operation S821, the system receives first data to be transmitted from the first graphics processing unit of the first server and writes the first data to be transmitted to the second memory extension device according to the second address.

[0243] When operating S822, if it detects that the first data to be transmitted has been written to the second memory extension device, the second address is written to the second central processing unit.

[0244] When operating S823, if it is detected that the second address has been written to the second graphics processing unit, the second graphics processing unit is invoked to read the first data to be transmitted from the second memory extension device according to the second address.

[0245] It should be understood that the second server in this application embodiment can also perform other operations as described above, which will not be repeated here.

[0246] Based on the above, this application proposes a cross-node communication topology for graphics processing units (GPUs) based on zero-copy technology and a fast computation link protocol. In this topology, a high-speed direct connection channel is established between the sending and receiving GPUs (both lacking a fast computation link protocol but equipped with a DMA engine) and the local fast computation link third-type device. Data is transmitted across server nodes via the CXL.mem sub-protocol or the CXL.I / O sub-protocol, bypassing central processing unit intervention and achieving an efficient cross-server communication path without requiring a fast computation link-compatible GPU.

[0247] Secondly, based on the proposed cross-node communication topology, this application further proposes an optimization mechanism for cross-node communication of graphics processing units (GPUs) based on polling triggering of fast computing links, employing a write-to-trigger, trigger-to-transfer, and transfer-to-read process. For example, a sending GPU that "does not support the fast computing link protocol but has a DMA engine" triggers an interrupt by writing to the zero-copy memory of a third type of fast computing link device in its local machine. The central processing unit in its local machine, which supports the three sub-protocols of the fast computing link, controls the cross-node transmission. The receiving GPU that "does not support the fast computing link protocol but has a DMA engine" polls to detect register changes and then pulls the data, achieving low-latency, high-bandwidth heterogeneous communication. Based on the proposed optimization method for cross-node communication of GPUs in distributed deep learning scenarios, this effectively improves the cross-node communication performance and resource coordination efficiency of current non-fast computing link GPUs in distributed deep learning, breaking through the limitations of traditional GPU communication on protocol compatibility.

[0248] Specifically, based on the cross-node communication optimization mechanism for graphics processing units based on fast computation link polling triggered by the embodiments of this application, the central processing unit in the sending server that supports the fast computation link protocol and has "three sub-protocols of fast computation link" cooperates with the sending graphics processing unit that does not support the fast computation link protocol but has a DMA engine to allocate the extended memory of the local third type device as zero-copy memory using zero-copy technology. This constructs a high-speed direct data transmission channel between the sending graphics processing unit that does not support the fast computation link protocol but has a DMA engine and the zero-copy memory in the local third type device, which does not require the central processing unit that supports the three sub-protocols of fast computation link to forward the data without intervention. That is, a high-speed direct data transmission channel for the sending party based on fast computation link and zero-copy technology.

[0249] Furthermore, based on the graphics processing unit cross-node communication optimization mechanism based on fast computing link polling triggered by the embodiments of this application, when the fast computing link controller of the third type of fast computing link device in the sending server that supports the fast computing link protocol receives the write operation instruction of the receiver's high-speed data direct connection transmission channel based on fast computing link and zero-copy technology, it will trigger the monitoring of the corresponding MSI-X event of the zero-copy memory address in its fast computing link controller while completing the zero-copy memory write in the corresponding third type of fast computing link device. The fast computing link controller will encapsulate the corresponding MSI-X event into a TLP packet and write it to the interrupt controller of the central processing unit through the fast peripheral component interconnection channel. The interrupt controller of the central processing unit will immediately trigger the interrupt handler to modify the value of the memory monitoring register. For the polling thread of the sending graphics processing unit (GPU) that "does not support the fast computing link but has a DMA engine" and "runs on the central processing unit side that supports the three sub-protocols of the fast computing link locally," it exits polling upon detecting a change in the value of the memory monitoring register and issues a Load / Store instruction based on the CXL.mem sub-protocol (or a DMA instruction based on the CXL.I / O sub-protocol) to transfer data from the zero-copy memory of the third type device in the sending server that supports the fast computing link protocol to the zero-copy memory of the third type device in the fast computing link of the receiving server that supports the fast computing link protocol and is connected via a fast computing link switch. Based on this, the embodiments of this application realize efficient and high-bandwidth cross-node communication between graphics processing units on different nodes.

[0250] Figure 9A It is shown that according to this application Figure 1C A flowchart of the data transmission method in this embodiment.

[0251] like Figure 9A As shown, the data transmission method of this embodiment can be applied to a first server deploying a distributed system. The data transmission method may include operations S911 to S915.

[0252] During operation S911, the first graphics processing unit writes the data to be transmitted into the double-rate memory according to the first address, and when the data writing is completed, it triggers the first fast computing link controller to detect the first interrupt event of the first address being written into the memory.

[0253] During operation of S912, the first fast computing link controller encapsulates the first interrupt event and writes it into the first central processing unit, so that the first address is written into the first central processing unit.

[0254] When operating S913, the first interrupt handler is triggered to modify the value of the memory monitoring register in the second processor.

[0255] During operation of S914, the first central processing unit triggers the first interrupt handler to modify the value of the memory monitoring register in the first central processing unit.

[0256] When operating S915, if the application-state polling thread associated with the first graphics processing unit in the first central processing unit detects that the value of the memory monitoring register has been modified, the polling thread is interrupted, and the second address is queried according to the first address. A transmission instruction based on the fast computing link protocol is issued to transmit the data to be transmitted to another server according to the second address through the fast computing link switch. This enables the other server to call the second graphics processing unit of the other server to read the data to be transmitted from the memory expansion device when it detects that the data to be transmitted has been written to the memory expansion device of the other server.

[0257] It should be understood that the first server in this application embodiment can also perform other operations as described above, which will not be repeated here.

[0258] Figure 9B It is shown that according to this application Figure 1C A flowchart of another embodiment of a data transmission method.

[0259] like Figure 9B As shown, the data transmission method of this embodiment can be applied to a second server deploying a distributed system. This data transmission method may include operations S921 to S926.

[0260] In operation S921, the second central processing unit receives data to be transmitted from another server and writes the data to be transmitted into the memory extension device according to the second address.

[0261] When operating the S922, upon completion of data writing, a second interrupt event is triggered, which monitors the second address being written to extended memory.

[0262] During operation of S923, the second fast computing link controller encapsulates the second interrupt event and writes it into the second central processing unit, so that the second address is written into the second central processing unit.

[0263] When operating S924, the second central processing unit triggers the second interrupt handler to modify the value of the memory monitoring register in the second central processing unit.

[0264] When operating S925, if the application-state polling thread associated with the second graphics processing unit in the second central processing unit detects that the value of the memory monitoring register has been modified, the polling thread is interrupted, and the second graphics processing unit is invoked to read the data to be transmitted from the extended memory according to the second address.

[0265] When operating the S926, the third processor is invoked to read the data to be transmitted from the second memory according to the second address.

[0266] It should be understood that the second server in this application embodiment can also perform other operations as described above, which will not be repeated here.

[0267] Figure 10A It is shown that according to this application Figure 1D A flowchart of the data transmission method in this embodiment.

[0268] like Figure 10A As shown, the data transmission method of this embodiment can be applied to a first server in a distributed system. The data transmission method may include operations S1011 to S1014.

[0269] In operation S1011, the memory of the memory expansion device is divided into first memory according to the storage space required for the data to be transmitted in the first graphics processing unit.

[0270] In operation S1012, the first address of the first memory is associated with the identifier of the data to be transmitted and stored.

[0271] In operation S1013, if it is detected that the second address corresponding to the identifier of the data to be transmitted has been written to the memory expansion device from the second server, an interrupt event is triggered to write the second address to the first central processing unit.

[0272] In operation S1014, if it is detected that the second address has been written to the first central processing unit, the second address is associated with the first address and stored together. This is so that if the first central processing unit detects that the data to be transmitted has been written to the memory expansion device and the first address has been written to the first central processing unit, the second address is queried according to the first address, and the data to be transmitted is transmitted to the second server according to the second address through the switch. This is so that if the second server detects that the data to be transmitted has been written to the memory of the second graphics processing unit, it can read the data to be transmitted from the memory of the second graphics processing unit.

[0273] It should be understood that the first server in this application embodiment can also perform other operations as described above, which will not be repeated here.

[0274] Figure 10B It is shown that according to this application Figure 1D A flowchart of another embodiment of a data transmission method.

[0275] like Figure 10B As shown, the data transmission method of this embodiment can be applied to a second server deployed in a distributed system. The data transmission method may include operations S1021 to S1025.

[0276] In operation S1021, the memory of the second graphics processing unit is divided into second memory according to the storage space required to store the data to be transmitted from the first server.

[0277] In operation S1022, the second address of the second memory is associated with the identifier of the data to be transmitted and stored together, so that the second address and the identifier of the data to be transmitted are sent to the first server through the interactive device, so that the first server associates the second address with the first address and transmits the data to be transmitted to the second server according to the second address.

[0278] In operation S1023, if it is detected that the data to be transmitted has been written to the second memory, the second address is written to the second central processing unit.

[0279] In operation S1024, the system receives data to be transmitted from the first graphics processing unit of the first server and writes the data to be transmitted into the second memory according to the second address.

[0280] In operation S1025, if it is detected that the second address has been written to the second central processing unit, the second graphics processing unit is invoked to read the data to be transmitted from the second memory according to the second address.

[0281] It should be understood that the second server in this application embodiment can also perform other operations as described above, which will not be repeated here.

[0282] Based on the above, this application proposes a cross-node communication topology for graphics processing units (GPUs) based on zero-copy technology and a fast computation link protocol. In this topology, a high-speed direct connection is established between a second GPU that does not support the fast computation link protocol but has a DMA engine and a local fast computation link third-type device. Data is transmitted across server nodes via the CXL.mem sub-protocol or the CXL.I / O sub-protocol, avoiding software intervention from the central processing unit and achieving an efficient cross-server communication path without requiring a fast computation link-compatible GPU.

[0283] Secondly, based on the proposed cross-node communication topology, this application further proposes an optimized mechanism for cross-node communication of graphics processing units (GPUs) based on polling triggering of fast computation links. This mechanism employs a write-to-trigger, trigger-to-transfer, and transfer-to-read process. For example, a sending GPU (i.e., the first GPU described above) that "does not support the fast computation link protocol but has a DMA engine" triggers an interrupt by writing to zero-copy memory in the third type of fast computation link device locally. The central processing unit (CPU) supporting the three sub-protocols of the fast computation link in the local machine controls the cross-node transmission. A receiving GPU (i.e., the second GPU described above) that "supports the fast computation link protocol" polls to detect register changes and then retrieves the data, achieving low-latency, high-bandwidth heterogeneous communication. Based on this proposed optimization method for cross-node communication of GPUs in distributed deep learning scenarios, the cross-node communication performance and resource coordination efficiency of GPUs without fast computation link protocols in distributed deep learning are effectively improved, overcoming the limitations of traditional GPU communication on protocol compatibility.

[0284] Specifically, based on the cross-node communication optimization mechanism for graphics processing units based on fast computation link polling triggered by the embodiments of this application, the central processing unit in the sending server that supports the fast computation link protocol and has "three sub-protocols of fast computation link" cooperates with the sending graphics processing unit that does not support the fast computation link protocol but has a DMA engine to allocate the extended memory of the local third type device as zero-copy memory using zero-copy technology. This constructs a high-speed direct data transmission channel between the sending graphics processing unit that does not support the fast computation link protocol but has a DMA engine and the zero-copy memory in the local third type device, which does not require the central processing unit that supports the three sub-protocols of fast computation link to forward the data without intervention. That is, a high-speed direct data transmission channel for the sending party based on fast computation link and zero-copy technology.

[0285] Furthermore, based on the graphics processing unit cross-node communication optimization mechanism based on fast computing link polling triggered by the embodiments of this application, when the fast computing link controller (i.e., the first fast computing link controller described above) of the fast computing link third type device in the sending server that supports the fast computing link protocol receives a write operation instruction from the receiver's high-speed data direct connection transmission channel based on fast computing link and zero-copy technology, it will trigger the monitoring of the corresponding MSI-X event for the zero-copy memory address in its fast computing link controller while completing the zero-copy memory write in the corresponding fast computing link third type device. The fast computing link controller will encapsulate the corresponding MSI-X event into a TLP packet and write it to the interrupt controller of the central processing unit through the fast peripheral component interconnection channel. The interrupt controller of the central processing unit will immediately trigger the interrupt handler to modify the value of the memory monitoring register. For the polling thread of the sending graphics processing unit (GPU) that "does not support the fast computing link but has a DMA engine" and "runs on the central processing unit side that supports the three sub-protocols of the fast computing link locally," the polling is interrupted upon detecting a change in the value of the memory monitoring register. A Load / Store instruction based on the CXL.mem sub-protocol (or a DMA instruction based on the CXL.I / O sub-protocol) is then issued to transfer data from the zero-copy memory of the third type device in the sending server that supports the fast computing link protocol to the zero-copy memory (i.e., the second memory of the second GPU) of the receiving server that supports the fast computing link protocol and is connected via a fast computing link switch. Based on this, the embodiments of this application achieve efficient and high-bandwidth cross-node communication between GPUs on different nodes. It should be noted that in the embodiments of this application, when the sending party is a second type device that supports the fast computing link protocol and the receiving party is a third type device that does not support the fast computing link protocol, a similar effect to the above description can be achieved, which will not be elaborated upon here.

[0286] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0287] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0288] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A server, characterized by The server comprises: a first graphic processing unit configured to write first to-be-transmitted data to a first memory expansion device according to a first address; the first memory expansion device is configured to write the first address to a central processing unit in a case where it is monitored that the first to-be-transmitted data has been written to the first memory expansion device; the central processing unit is interconnected with a fast computing link switch based on a memory interconnection protocol, and the fast computing link switch is interconnected with another server based on the memory interconnection protocol; the central processing unit is configured to query a second address according to the first address in a case where it is monitored that the first address has been written to the central processing unit, and transmit the first to-be-transmitted data to another server according to the second address through the fast computing link switch, so that the second graphic processing unit in another server reads the first to-be-transmitted data from a second memory expansion device in another server in a case where it is monitored that the first to-be-transmitted data has been written to the second memory expansion device in another server.

2. The server of claim 1, wherein, The central processing unit comprises a root complex unit; the root complex unit is connected with the first graphic processing unit through a peripheral component interconnect express bus, and a fast computing link interconnection channel is configured between the root complex unit and the first memory expansion device; the central processing unit is further configured to transmit the first to-be-transmitted data to the root complex unit through the peripheral component interconnect express bus, and write the first to-be-transmitted data to the first memory expansion device through the fast computing link interconnection channel.

3. The server of claim 2, wherein, The root complex unit comprises a peripheral component interconnect express root port and a first fast computing link root port; the peripheral component interconnect express root port and the first fast computing link root port are connected through a root complex internal routing bus; the central processing unit is further configured to transmit the first to-be-transmitted data to the peripheral component interconnect express root port through the peripheral component interconnect express bus, to the first fast computing link root port through the root complex internal routing bus, and to the first memory expansion device through the fast computing link interconnection channel.

4. The server of claim 3, wherein, The root complex unit and the fast computing link switch are connected through a fast computing link external interconnection bus; the central processing unit is further configured to transmit the first to-be-transmitted data from the root complex unit to the fast computing link switch through the fast computing link external interconnection bus, so that the fast computing link switch transmits the first to-be-transmitted data to another server according to the second address.

5. The server of claim 4, wherein, The root complex unit further comprises a second fast computing link root port; the first fast computing link root port and the second fast computing link root port are connected through a fast computing link internal interconnection bus; the second fast computing link root port and the fast computing link switch are connected through a fast computing link external interconnection bus; The central processing unit is further configured to read the first to-be-transmitted data from the first memory expansion device, and transmit the first to-be-transmitted data to the first fast computing link root port via the fast computing link interconnection channel, to the second fast computing link root port via the fast computing link internal interconnection bus, and to the other server via the fast computing link switch via the fast computing link external interconnection bus.

6. The server of claim 5, wherein, The first memory expansion device comprises: A first expansion memory configured to store the first to-be-transmitted data; A fast computing link controller configured to write the first address into the central processing unit when it is monitored that the first to-be-transmitted data has been written into the first expansion memory.

7. The server of claim 6, wherein: The central processing unit is further configured to divide the memory of the first memory expansion device to obtain a first expansion memory for storing the first to-be-transmitted data according to the storage space required by the first to-be-transmitted data; and store the first address of the first expansion memory in association with the first to-be-transmitted data identifier.

8. The server of claim 6, wherein: The graphics processing unit is further configured to trigger the fast computing link controller to monitor the interrupt event of the first address being written into the first expansion memory when the data writing is completed.

9. The server of claim 8, wherein: The fast computing link controller is further configured to write the interrupt event into the graphics processing unit after encapsulation, so that the first address is written into the graphics processing unit.

10. The server of claim 9, wherein: The central processing unit is further configured to trigger an interrupt processing program to modify the value of a memory monitoring register in the central processing unit; and interrupt a polling thread in the central processing unit when the value of the memory monitoring register is monitored to be modified by an application state polling thread associated with the graphics processing unit, and according to the second address queried by the first address, issue a transmission instruction based on the fast computing link protocol to transmit the first to-be-transmitted data to the other server according to the second address via the fast computing link switch.

11. The server of claim 8, wherein: The fast computing link controller is further configured to write the second address of the second memory expansion device and the first to-be-transmitted data identifier into the central processing unit when it is monitored that the second address and the first to-be-transmitted data identifier are written into the first expansion memory; The central processing unit is further configured to query the first address according to the first to-be-transmitted data identifier, and store the first address in association with the second address.

12. The server of any one of claims 1-11, wherein: ​ The central processing unit is further configured to receive second to-be-transmitted data of a second graphics processing unit from another server, and write the second to-be-transmitted data to a second memory expansion device according to a third address; The first memory expansion device is further configured to write the third address to the central processing unit when it is monitored that the second to-be-transmitted data has been written to the second memory expansion device; The central processing unit is further configured to call the first graphics processing unit to read the second to-be-transmitted data from the second memory expansion device according to the third address when it is monitored that the third address has been written to the central processing unit.

13. The server of claim 12, wherein: The central processing unit is further configured to transmit the second to-be-transmitted data to a second quick compute link root port via a quick compute link external interconnect bus, to a first quick compute link root port via a quick compute link internal interconnect bus, and to the first memory expansion device via a quick compute link interconnect lane.

14. The server of claim 13, wherein: The central processing unit is further configured to read the second to-be-transmitted data from the first memory expansion device according to the third address via the quick compute link interconnect lane when it is monitored that the third address has been written to the first graphics processing unit, and to transmit the second to-be-transmitted data to the first quick compute link root port, to a quick peripheral component interconnect root port via a root complex internal routing bus, and to the first graphics processing unit via a quick peripheral component interconnect bus.

15. The server of claim 12, wherein: The first memory expansion device further comprises: A second expansion memory configured to store the second to-be-transmitted data; A quick compute link controller configured to write the third address to the central processing unit when it is monitored that the second to-be-transmitted data has been written to the second expansion memory; The central processing unit is further configured to divide the memory of the first memory expansion device according to a storage space required by the second to-be-transmitted data to obtain a second expansion memory for storing the second to-be-transmitted data, and to store a third address of the second expansion memory and a second to-be-transmitted data identifier in association.

Images