Server system

Abstract

The application provides a server system, which can be applied to the technical field of servers. The server system comprises: a first server comprising a first graphics processing unit and a memory expansion device; the first server is configured to write to-be-transmitted data in the first graphics processing unit into the memory expansion device and transmit the to-be-transmitted data to a switch via a first fast compute link external interconnect bus; the switch is configured to transmit the to-be-transmitted data to a second server via a second fast compute link external interconnect bus; and the second server comprises a second graphics processing unit; the second server is configured to transmit the to-be-transmitted data to the second graphics processing unit via a double data rate bus.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of servers, in particular to a server system. BACKGROUND

[0002] With the continuous application and promotion of neural networks, deep learning, high-performance computing, large models and other technologies based on neural networks have developed rapidly. On this basis, frequent communication is needed between accelerators deployed in different servers of a distributed system. However, due to the differences in transmission protocols supported by different servers, the communication transmission efficiency between different servers in the distributed system is affected. SUMMARY

[0003] In view of the above problems, the present application provides a server system.

[0004] According to an aspect of the present application, a server system is provided, comprising: a first server comprising a first graphics processing unit and a memory expansion device; the first server is configured to write to-be-transmitted data in the first graphics processing unit into the memory expansion device and transmit to a switch via a first fast compute link external interconnect bus; the switch is configured to transmit the to-be-transmitted data to a second server via a second fast compute link external interconnect bus; and the second server comprises a second graphics processing unit; the second server is configured to transmit the to-be-transmitted data to the second graphics processing unit via a double data rate bus.

[0005] According to an embodiment of the present application, by writing the to-be-transmitted data in the first graphics processing unit into the memory expansion device, the to-be-transmitted data can be transmitted to the switch via the first fast compute link external interconnect bus based on the fast compute link protocol, and then transmitted to the second server via the second fast compute link external interconnect bus, so as to be directly written into the second graphics processing unit of the second server via the double data rate bus of the second server. In this process, since the to-be-transmitted data is directly transmitted between the first graphics processing unit and the second graphics processing unit based on the fast compute link protocol, at least part of the problem of low communication efficiency of different servers due to the differences in transmission protocols supported by different servers is solved, and the transmission efficiency of the to-be-transmitted data from the first graphics processing unit to the second graphics processing unit is improved. BRIEF DESCRIPTION OF DRAWINGS

[0006] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application, taken in conjunction with the accompanying drawings, in which:

[0007] Figure 1A A schematic diagram of a distributed system according to an embodiment of the present application is shown.

[0008] Figure 1A A schematic diagram of a first server according to an embodiment of the application is shown.

[0009] Figure 2A A schematic diagram of a first server according to another embodiment of the application is shown.

[0010] Figure 2A A schematic diagram of a second server according to another embodiment of the application is shown.

[0011] Figure 2A A schematic diagram of a first server and a second server according to another embodiment of the application is shown.

[0012] Figure 3A A schematic diagram of a first server according to another embodiment of the application is shown.

[0013] Figure 3A A schematic diagram of a second server according to another embodiment of the application is shown.

[0014] Figure 4 A flow chart of a data transmission method according to an embodiment of the application is shown.

[0015] Figure 4 A flow chart of a data transmission method according to an embodiment of the application is shown. DETAILED DESCRIPTION

[0016] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It is to be understood, however, the drawings are designed for a description only, and are not intended to limit the scope of the present application. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to one skilled in the art that one or more embodiments can be practiced without these specific details. In other instances, well-known structures and

[0017] The terms used herein are merely used to describe specific embodiments, and are not intended to limit the present application. The terms "include" and "have" and the like used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0018] All terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present description, and should not be interpreted in an idealized or overly formal manner.

[0019] In the case of using expressions such as "at least one of A, B, and C", it generally should be interpreted to include any of A, B, or C alone, a combination of at least two of the items A, B, and C, or a combination of all of the items A, B, and C, in a manner that is as generally understood by those of ordinary skill in the art (e.g., "a system having at least one of A, B, and C" should include, but not be limited to, a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C together, etc.).

[0020] With the continuous application and promotion of neural networks, deep learning, high-performance computing, large models, and other technologies based thereon have developed rapidly. For deep learning applications such as large models with increasing complexity, a large amount of artificial intelligence (AI) server clusters are needed to provide massive computing power. Among them, the AI server can be deployed with a processor for implementing model acceleration computing, that is, an accelerator. For example, the accelerator can include a graphics processing unit (GPU).

[0021] Since the computing power of a single AI server is limited, in some schemes, multiple AI servers are generally needed to implement collaborative computing through network interconnection. However, in the application scenario of a distributed AI server cluster, although the above network interconnection can effectively improve the computing power of the AI server cluster, the data transmission capability between the GPUs and the GPUs is limited, and thus this condition still limits the upper limit of the computing power of the AI server cluster. Based on this, when the data transmission capability of the AI server cluster is limited, the computing performance of the entire AI server cluster will also decrease.

[0022] Based on this, when a distributed AI server cluster is used for acceleration of a large model or a complex neural network, even if the number of GPUs is sufficient, the computing power of the distributed AI server cluster will still be limited by the large number of parameters of the neural network deployed thereon, and the data transmission capability of the GPUs of the AI server cluster when transmitting parameters.

[0023] For GPUs deployed on different servers, cross-node communication between GPUs deployed on different server nodes can be achieved through a Remote Direct Memory Access (RDMA) network card. For example, the server cluster can include a first server and a second server. The first server can be a sender of data, and the second server can be a receiver of data. Among them, the data can be the calculation result obtained by the GPU of the sender due to the execution of the calculation task.

[0024] For example, after the sender GPU finishes the computing task, it can write the computing result into the memory of the sender GPU, e.g. Double Data Rage Random Access Memory (DDR), via the internal bus of the sender. Then, the sender GPU needs to send the computing result from the sender DDR to the cache area of the sender RDMA card via the Peripheral Component Interconnect express (PCIe) bus. Then, the sender RDMA card can package the computing result in the cache area together with the address of the receiver RDMA card, the cache address of the receiver GPU, etc. to generate a corresponding data packet. Then, the sender RDMA card can transmit the data packet to the cache area of the receiver RDMA card via the switch.

[0025] After receiving the data packet, the receiver RDMA card can analyze the data packet to obtain the computing result included therein. Then, the receiver RDMA card can write the computing result to the receiver DDR corresponding to the receiver GPU via the internal PCIe bus of the receiver. Then, the computing unit of the receiver GPU can read the computing result stored in the receiver DDR.

[0026] Based on the above, in the scheme described above, the communication between the GPUs of different servers is mainly assisted by the RDMA card. On this basis, this method needs to be implemented by means of the sender RDMA card and the receiver RDMA card, and the sender GPU and the sender RDMA card need to perform data copy and caching, and the sender RDMA card also needs to package the data packet again by combining the Internet Protocol (IP) address of the receiver RDMA card, the DDR memory address of the receiver GPU, etc., thereby introducing a long processing delay. Similarly, the receiver RDMA card needs to analyze the data packet to obtain the original computing result after receiving the data packet, which also introduces a long processing delay.

[0027] Based on this, the time consumption of the data transmission process can limit the computing power of the distributed AI server cluster. To this end, in one scheme, a third-party device (such as a network interface card or a storage device) can be used to directly access the DDR corresponding to the graphics processing unit, that is, the memory of the graphics processing unit, through the RDMA network card. Alternatively, in another scheme, the memory of the graphics processing unit can be directly accessed through the third-party device. Alternatively, in yet another scheme, the underlying data transmission mechanism described above can be optimized. For example, a communication library can be used to automatically manage data transmission between graphics processing units, enabling communication and computation to be performed simultaneously.

[0028] However, the above method has problems. Since the data transmission process of the RDMA network card is asynchronous, the data in the memory region may not be updated when the data transmission process is not yet completed. On this basis, if the user attempts to access or use the memory region when the RDMA network card has not completed the data transmission, it may result in reading incomplete or old data. Moreover, in order to ensure the integrity and consistency of the data, the user usually needs to perform a synchronization operation explicitly, or check the transmission completion status of the RDMA network card, which is low in convenience.

[0029] Based on this, although the communication library can automatically manage the data transmission operation of the RDMA network card, the user still needs to pay attention to the synchronization and data consistency of the transmission. Moreover, based on the transmission mode of the communication library, the additional processing delay introduced by the RDMA network card on packet assembly and unpacking is still present.

[0030] In other schemes, the data transmission between the memory space of a central processing unit (CPU) and the memory of a device interconnected with the central processing unit can be implemented based on a compute express link (CXL) protocol to maintain the consistency of the transmitted data. For example, the compute express link protocol can include three sub-protocols: a CXL.I / O sub-protocol, a CXL.cache sub-protocol, and a CXL.mem sub-protocol. Based on the three sub-protocols, the host and devices (such as accelerators and memory extenders, etc.) that need to share memory resources can perform memory access with low latency and ensure the consistency of the stored data. In the compute express link specification, three types of devices suitable for the compute express link protocol are defined, namely a first type of device, a second type of device, and a third type of device.

[0031] For example, if the main memory of the central processing unit has a need for cache data, it can be referred to as a first type of device. Thus, the first type of device only needs to be configured with the CXL.I / O sub-protocol and the CXL.cache sub-protocol.

[0032] For example, a graphics processing unit has a memory, and the graphics processing unit needs to interconnect with a central processing unit, which can be referred to as a second type of device. Thus, the second type of device can be configured with a CXL.I / O sub-protocol so that the central processing unit can configure the device. Also, the second type of device can also be configured with a CXL.cache sub-protocol so that the graphics processing unit can access the memory of the central processing unit. In addition, the second type of device can also be configured with a CXL.mem sub-protocol so that the central processing unit can access the memory of the second type of device.

[0033] For example, a third type of device is compatible with the CXL.I / O sub-protocol to configure a memory buffer such as a memory expansion card. The third type of device can also be compatible with the CXL.I / O sub-protocol so that the central processing unit can access the stored data of the memory buffer. It should be noted that the memory of the memory buffer can provide two working modes with the assistance of a fast computing link controller. The fast computing link controller can be used to parse incoming fast computing link data packets, and can perform Load / Store operations on the extended memory at the corresponding address by the memory manager after parsing the information (e.g., address and length) in the data packet.

[0034] In the first working mode, the fast computing link protocol enables a device (e.g., a memory buffer or a graphics processing unit) to be directly mapped to the memory of a device (e.g., a central processing unit) for more efficient memory access. This mode enables an application to bypass the memory management layer (e.g., a page cache layer) of the operating system and directly interact, reducing latency and improving performance. Thus, a device supporting the fast computing link protocol must be able to handle the memory mapping and access mechanism of the fast computing link to enable direct access in the first working mode.

[0035] In the second working mode, the memory of the memory buffer is considered as part of the system memory, and the central processing unit or accelerator (e.g., a graphics processing unit) supporting the fast computing link protocol can access the memory of the memory buffer as if it were accessing a local random access memory (RAM).

[0036] It should be noted that since the third type of device is mainly used as an extended memory, it is difficult to actively trigger the following two data transmission operations: 1) it is difficult to actively trigger small-scale data transmission operations such as Load / Store based on the CXL.mem sub-protocol; 2) it is difficult to actively trigger large-scale data transmission operations such as DMA (direct memory access) based on the CXL.I / O sub-protocol.

[0037] Further, the first type device and the second type device can interact with the third type device through Load / Store function with the assistance of the central processing unit supporting the three sub-protocols of the fast computing link and the root port of the fast computing link thereof. In addition, the central processing unit supporting the three sub-protocols of the fast computing link can also support the ability to interact with the third type device through DMA.

[0038] In addition, for data interaction between two third type devices (defined as a sender third type device and a receiver third type device), there are two methods as follows. For example, in the first method, the central processing unit supporting the three sub-protocols of the fast computing link (or the second type device) issues a Load instruction to load data in the extended memory of the sender third type device, and then the central processing unit supporting the three sub-protocols (or the second type device) continues to execute a Store instruction to store the data into the extended memory of the receiver third type device. For another example, in the second method, the central processing unit supporting the three sub-protocols of the fast computing link issues a DMA transfer instruction to transfer the data in the extended memory of the sender third type device to the extended memory of the receiver third type device based on the DMA function.

[0039] On this basis, some schemes propose a fast computing link memory decomposition prototype system technology (hereinafter collectively referred to as Direct fast computing link). On the basis of the server supporting the fast computing link protocol, in the Direct fast computing link technology, the interconnection mode of the server and the third type device is divided into two types as follows:

[0040] The first mode is that the device supporting the fast computing link protocol directly communicates with the third type device. For example, the central processing unit (or the second type device) supporting the three sub-protocols of the fast computing link in the device supporting the fast computing link protocol directly accesses the extended memory of the third type device through the fast computing link bus.

[0041] The second mode is that the device supporting the fast computing link protocol communicates with the third type device through a switch. For example, the central processing unit (or the second type device) supporting the three sub-protocols of the fast computing link in the device supporting the fast computing link protocol interacts with the third type device based on the Load / Store mode based on the CXL.mem and CXL.I / O sub-protocols and the interconnection of the switch.

[0042] In the above Direct fast computing link scheme, when the graphics processing unit not supporting the fast computing link related protocol communicates with other graphics processing units across the host, the Direct fast computing link technology cannot be applied.

[0043] Based on the above, in complex deep learning application scenarios such as large models, graphics processing units are widely used. Further, some manufacturers deploy a large number of commercial graphics processing units on their server side to achieve application acceleration, which generates a large amount of data, and thus frequent data transmission is required between graphics processing units across server nodes.

[0044] Although each commercial graphics processing unit manufacturer has proposed its own graphics processing unit interconnection protocol to accelerate inter-graphics processing unit data transmission, for cloud service providers, these protocols are only applicable to graphics processing units developed by each graphics processing unit manufacturer. Therefore, there is a problem that graphics processing units between different graphics processing unit manufacturers cannot communicate. Since data duplication and caching are required between the sender graphics processing unit and the sender RDMA network card, and the sender RDMA network card also needs to repackage and generate a data packet in combination with the IP address of the receiver RDMA network card and the memory address of the receiver graphics processing unit, a long processing delay is introduced before transmission.

[0045] Similarly, the receiver RDMA network card needs to analyze the data packet received to obtain the original calculation result, which also introduces a long processing delay. Although the other technologies described in the foregoing also optimize the RDMA-assisted graphics processing unit cross-node communication method, users still need to pay attention to the synchronization and data consistency of transmission.

[0046] In addition, although the fast compute link protocol is a high-speed cache coherence interconnection protocol supported by the industry for processors, memory expansion, and accelerators, third-type devices that do not support the fast compute link protocol cannot directly use the CXL.mem sub-protocol to transmit data stored in their memories to second-type devices that support the fast compute link protocol.

[0047] Therefore, the present application proposes a server for distributed deep learning scenario graphics processing unit cross-node communication, a data transmission method, and a distributed system to enable third-type devices that do not support the fast compute link protocol to transmit data to second-type devices based on the fast compute link protocol and reduce data transmission latency. Figure 1A A schematic diagram of a distributed system according to an embodiment of the present application is shown.

[0048] As Figure 1AAs shown, the distributed system of the embodiment can include a plurality of nodes. For example, the plurality of nodes can include a first server and a second server, but it should be understood that the present application is not limited thereto, and other nodes can also be included in the distributed system of the present application. The plurality of nodes in the distributed system can communicate across nodes, specifically, can communicate across nodes via a switch. For example, the switch can be based on a fast computing link protocol to implement data transmission.

[0049] The present application will be described below with the first server and the second server as examples.

[0050] Figure 1B A schematic diagram of the first server and the second server according to an embodiment of the present application is shown. As shown, the first server can include a first graphics processing unit, a first central processing unit, and a memory extension device. For example, the first graphics processing unit can be used to perform a computing task to obtain a computing result. In some embodiments, the first graphics processing unit can transmit the computing result as to-be-transmitted data to be transmitted by the first server to the second server. Figure 1B

[0051] It should be noted that the first graphics processing unit can be a processor that does not support the fast computing link protocol, but the first graphics processing unit can be deployed with a DMA engine to enable the first graphics processing unit to access the memory of the memory buffer based on the zero-copy technology. The first graphics processing unit can refer to a graphics processing unit that does not deploy a protocol stack of the fast computing link protocol. For example, such a graphics processing unit can be difficult to parse transaction information such as cache consistency requests, memory pooling instructions, and the like specific to the fast computing link protocol. However, such a graphics processing unit can read and write data based on the DMA engine deployed by itself to the fixed memory region divided by the corresponding central processing unit.

[0052] Specifically, the first graphics processing unit can be connected to the memory extension device. For example, the memory extension device can be a memory buffer. Specifically, the memory extension device can be a memory buffer that supports the fast computing link protocol, for example, can be a memory buffer that supports the CXL.I / O sub-protocol and the CXL.mem sub-protocol, i.e., the third type of device described above. The memory extension device can include a memory. On this basis, the first graphics processing unit can write the to-be-transmitted data to the first memory of the memory extension device according to a first address. For example, the first address can be the address of the first memory of the memory extension device, i.e., the zero-copy address. In this way, the first graphics processing unit can write the to-be-transmitted data to the first memory of the memory extension device based on the high-speed and concurrent read-write characteristics of the DMA engine and the zero-copy channel, and this process does not need to introduce an additional forwarding process (such as a data forwarding process of the central processing unit), thereby reducing the delay of the data writing process.

[0053] ​Subsequently, the first server can transmit the data to be transmitted in the memory expansion device to the switch via the first Quick Compute Link external interconnect bus. The switch can transmit the data to be transmitted to the second server via the second Quick Compute Link external interconnect bus. The second server can transmit the data to be transmitted to the second graphics processing unit of the second server via the double rate bus.

[0054] On this basis, by writing the data to be transmitted in the first graphics processing unit into the memory expansion device, the data to be transmitted can be transmitted to the switch via the first Quick Compute Link external interconnect bus based on the Quick Compute Link protocol, and then transmitted to the second server via the second Quick Compute Link external interconnect bus, so as to be directly written into the second graphics processing unit of the second server via the double rate bus of the second server. In this process, since the data to be transmitted is directly transmitted between the first graphics processing unit and the second graphics processing unit based on the Quick Compute Link protocol, the problem of low communication efficiency of different servers due to the difference between the transmission protocols supported by different servers is at least partially solved, and the transmission efficiency of the data to be transmitted from the first graphics processing unit to the second graphics processing unit is improved.

[0055] Further, the memory expansion device can further include a first Quick Compute Link controller connected with the first memory. The first Quick Compute Link controller can manage the data in the first memory of the memory expansion device. For example, the first Quick Compute Link controller can monitor the stored data in the first memory of the memory expansion device. The first Quick Compute Link controller can be connected to the first central processing unit. In this way, in an embodiment of the present application, in the case where it is monitored that the second address corresponding to the identifier of the data to be transmitted from the second server has been written into the memory expansion device, the first Quick Compute Link controller can trigger an interrupt event so as to write the second address into the first central processing unit. The interrupt event can be a Message Signaled Interrupts eXtended (MSI-X). In the case where it is monitored that the data to be transmitted has been written into the first memory of the memory expansion device, the first Quick Compute Link controller can write the first address into the first central processing unit. Specifically, the first Quick Compute Link controller can encapsulate the MSI-X into a Transaction Layer Packet (TLP), and write the TLP into the first central processing unit via the Peripheral Component Interconnect Express bus.

[0056] Before performing the data transmission, the first central processing unit can divide the memory of the memory expansion device according to the storage space required by the data to be transmitted in the first graphics processing unit, to obtain a first memory, and store the first address of the first memory in association with the data to be transmitted identifier. In this way, the first central processing unit can store the data to be transmitted in the first memory according to the first address when receiving the data to be transmitted carrying the data to be transmitted identifier. In addition, since the first central processing unit stores the first address and the second address, the first central processing unit can associate the first address and the second address to obtain the association relationship between the first address and the second address.

[0057] In this way, when it is monitored that the first address has been written into the first central processing unit, the first central processing unit can query the second address according to the first address based on the association relationship, and the second address is the memory address of the second server. For example, the first central processing unit can be deployed with an interrupt controller. The interrupt controller can immediately trigger a predetermined program (for example, an interrupt handling program, etc.) to modify the value of the memory monitoring register in the first central processing unit to the first address when the TLP is written into the interrupt controller of the first central processing unit. Specifically, the first central processing unit can be deployed with a polling thread supporting three CXL sub-protocols. When it is monitored that the value of the memory monitoring register is changed to the first address, the thread can immediately interrupt the polling, and according to the first address, query the second address of the second server supporting the CXL protocol for receiving the data to be transmitted in the above association relationship.

[0058] The first central processing unit can support the CXL protocol. In this way, the first central processing unit can communicate with the switch based on the CXL protocol, and the switch can communicate with the second server based on the CXL protocol. On this basis, the first central processing unit can transmit the data to be transmitted to the second server according to the second address through the switch. Specifically, the first central processing unit can send a Load / Store instruction based on the CXL.mem sub-protocol (or a DMA instruction based on the CXL.I / O sub-protocol) to the switch, to transmit the data to be transmitted from the memory region corresponding to the first address in the first memory of the memory expansion device supporting the CXL protocol to the second server also supporting the CXL protocol via the switch through the cross-node high-speed data transmission channel based on the CXL protocol.

[0059] The second graphics processing unit of the second server can be configured to perform a computing task to obtain a computing result. However, the embodiments of the present application are not limited thereto, and the second graphics processing unit can also perform a computing task based on the computing result of the processor of the other node to obtain a computing result. The first graphics processing unit is similar to this, and details are not repeated here. It should be noted that the second graphics processing unit can be a processor supporting the fast compute link protocol, so that the to-be-transmitted data can be directly written into the memory of the second graphics processing unit. In this way, the second server (specifically, the second graphics processing unit of the second server) can read the to-be-transmitted data from the second graphics processing unit when it is monitored that the to-be-transmitted data has been written into the second graphics processing unit, so as to perform a computing task based on the to-be-transmitted data to obtain a corresponding computing result. On this basis, the second graphics processing unit can directly read the to-be-transmitted data from the memory of the second graphics processing unit based on the fast compute link protocol and perform a computing task. This process does not need to introduce an additional forwarding process (such as the data forwarding process of the central processing unit), reducing the delay of the data writing process.

[0060] The second server can also include a second central processing unit. For example, the second central processing unit can be a central processing unit or the like. The second central processing unit supports the fast compute link protocol. In this way, the second central processing unit can be interconnected with the switch based on the fast compute link protocol, and the second central processing unit can also be connected to the second graphics processing unit. For example, the second graphics processing unit can include a second memory and a second fast compute link controller. For example, the second memory can be implemented based on DDR.

[0061] On this basis, the second central processing unit can be configured to divide the memory of the second graphics processing unit to obtain the second memory according to the storage space required for storing the to-be-transmitted data from the first server. And the second address of the second memory is stored in association with the to-be-transmitted data identifier, so as to send the second address and the to-be-transmitted data identifier to the first server (such as the memory expansion device of the first server) through the switch, so that the first server stores the second address in association with the first address, and transmits the to-be-transmitted data to the second server according to the second address.

[0062] The second central processing unit can receive the to-be-transmitted data from the first graphics processing unit of the first server, and write the to-be-transmitted data to the second memory according to a second address, that is, the second address can be an address of the second memory of the second server. The second fast computing link controller can write the second address to the second central processing unit when it is monitored that the to-be-transmitted data has been written to the second memory, so that the second central processing unit monitors the writing condition of the to-be-transmitted data to the second memory. In this way, the second central processing unit can call the second graphics processing unit to read the to-be-transmitted data from the second memory when it is monitored that the to-be-transmitted data has been written to the second memory.

[0063] Specifically, the second fast computing link controller can manage the data in the memory of the second graphics processing unit. For example, the second fast computing link controller can monitor the stored data in the memory of the second graphics processing unit. The second fast computing link controller can be connected to the second central processing unit. In this way, the second fast computing link controller can write the second address to the second central processing unit when it is monitored that the to-be-transmitted data has been written to the memory of the second graphics processing unit. Specifically, when the second fast computing link controller receives a write operation instruction via the high-speed data transmission channel based on the fast computing link protocol and the zero-copy technology, the second fast computing link controller can encapsulate the MSI-X into a TLP while writing the to-be-transmitted data to the second memory, and write the TLP to the second central processing unit via the fast peripheral component interconnect bus.

[0064] Further, the second central processing unit can be deployed with an interrupt controller. The interrupt controller can trigger a predetermined program (for example, an interrupt processing program) to modify the value of the memory monitoring register in the second central processing unit to the second address immediately when the TLP is written to the interrupt controller of the second central processing unit. Specifically, the second central processing unit can be deployed with a polling thread supporting the three sub-protocols of the fast computing link. The thread can interrupt the polling immediately and call the second graphics processing unit to read the to-be-transmitted data from the second memory when it is monitored that the value of the memory monitoring register is changed to the second address. In this way, the low-latency high-speed cross-node data transmission process between the first graphics processing unit not supporting the fast computing link protocol and the second graphics processing unit supporting the fast computing link protocol is completed.

[0065] Based on this, the first central processing unit divides the memory of the memory expansion device according to the storage space required by the to-be-transmitted data in the first graphics processing unit to obtain the first memory, and stores the first address of the first memory and the to-be-transmitted data identifier in association. In this way, the to-be-transmitted data can be stored to the first memory according to the first address when the to-be-transmitted data carrying the to-be-transmitted data identifier is received.

[0066] Further, the second central processing unit can divide the memory of the second graphic processing unit according to the storage space required by the data to be transmitted in the first graphic processing unit to obtain a second memory, and send the second address of the second memory and the identification of the data to be transmitted to the first server via the switch. In this way, the first fast computing link controller can trigger an interrupt event to write the second address to the first central processing unit when it is monitored that the second address has been written to the memory expansion device. The first central processing unit can immediately store the second address in association with the first address when it is monitored that the second address has been written to the first central processing unit.

[0067] On this basis, when it is monitored that the first graphic processing unit not supporting the fast computing link protocol writes the data to be transmitted to the first memory according to the first address, the first fast computing link controller can immediately write the first address to the first central processing unit, so that the first central processing unit can immediately query the second address according to the first address and transmit the data to be transmitted to the second server according to the second address via the switch when it is monitored that the first address has been written to the first central processing unit. Then, the second server can immediately call the second graphic processing unit to read the data to be transmitted from the second memory according to the second address when it is monitored that the data to be transmitted is written to the second memory of the second graphic processing unit supporting the fast computing link protocol according to the second address. Thus, the present application saves the time consumed by the processors of different nodes for packetizing and unpacking the transmission data by means of the network card to realize cross-node communication, so that the first graphic processing unit not supporting the fast computing link protocol and the second graphic processing unit supporting the fast computing link protocol can efficiently communicate across nodes.

[0068] In the embodiments of the present application, before the data transmission process described above is performed, the first server can divide the memory of the memory expansion device according to the storage space required by the data to be transmitted to obtain a storage space, and store the first address of the storage space in association with the identification of the data to be transmitted. In this way, the storage space associated with the first address can be used as the first memory. Similarly, the second server can divide the memory of the second graphic processing unit according to the storage space required by the data to be transmitted to obtain a storage space, and store the second address of the storage space in association with the identification of the data to be transmitted. In this way, the storage space associated with the second address can be used as the second memory. The following will be described in detail in combination with the Figure 1B

[0069] ​With reference back to FIG. 1B, the first server can include a memory expansion device, a first graphics processing unit, and a first central processing unit. On this basis, the memory of the memory expansion device can be divided to obtain a storage space according to the storage space required by the to-be-transmitted data, and a first address of the storage space can be stored in association with the to-be-transmitted data identifier.

[0070] Specifically, in the distributed deep learning model compilation phase, the first central processing unit invokes the first graphics processing unit to divide the memory of the memory expansion device to obtain a storage space based on the zero-copy technology, and a part of the storage space is used as a first memory, that is, a zero-copy memory, so that the first graphics processing unit writes data to the first memory of the memory expansion device, and the memory physical first address of the first memory can be recorded as send_mem_addr, that is, the first address. The size of the first memory can be the same as the data amount of the to-be-transmitted data of the first graphics processing unit in the model compilation phase, that is, the storage space required by the to-be-transmitted data. Moreover, if there are multiple to-be-transmitted data in the model, the first graphics processing unit can be invoked to divide the same number of storage spaces. After dividing the storage space, the first central processing unit can use a hash table to store the first address of the storage space in association with the to-be-transmitted data identifier. Specifically, the hash table can be a sending information table named comm_send_mem_table. The to-be-transmitted data identifier can include a communication process number comm_tag. The first address can include a zero-copy memory first address send_mem_addr. In this way, the to-be-transmitted data identifier can be used as a key and the first address as a value to determine the corresponding first address according to the to-be-transmitted data identifier carried by the to-be-transmitted data, and the to-be-transmitted data is written into the corresponding first memory.

[0071] With reference back to Figure 1B , the second server can include a second graphics processing unit and a second central processing unit. The second central processing unit can divide the memory of the second graphics processing unit to obtain a storage space according to the storage space required by the to-be-transmitted data, and a second address of the storage space can be stored in association with the to-be-transmitted data identifier.

[0072] Specifically, in the distributed deep learning model compilation stage, the second central processing unit calls the second graphics processing unit to divide the memory of the second graphics processing unit based on the zero-copy technology to obtain a storage space, which can be used as a second memory, i.e., a zero-copy memory, so as to write data to the second memory of the second graphics processing unit, and the physical first address of the second memory can be recorded as recv_mem_addr, i.e., a second address. It should be understood that since the second memory of the second graphics processing unit supports three sub-protocols of CXL.mem, CXL.io and CXL.cache, the second memory is actually a fast compute link memory. The size of the second memory can be the same as the data size of the data to be transmitted by the second graphics processing unit in the model compilation stage (i.e., the storage space required by the data to be transmitted). If there are multiple data to be transmitted in the model, the second graphics processing unit can be called to divide the second memory into the same number of parts as the zero-copy memory. After dividing the second memory, the second central processing unit can use a hash table to store the second address of the second memory and the data to be transmitted identifier in association. Specifically, the hash table can be a receiving information table named comm_recv_mem_table. The data to be transmitted identifier can include a communication process number comm_tag. The second address can include a zero-copy memory first address recv_mem_addr. In this way, the data to be transmitted identifier can be used as a key and the second address can be used as a value to determine the corresponding second address according to the data to be transmitted identifier carried by the data to be transmitted, and the data to be transmitted is written into the corresponding second memory.

[0073] It is further explained that when there are multiple communication processes in the distributed deep learning model compilation stage, the multiple communication processes of the entire model can be numbered respectively, denoted as comm_tag, and the numerical type is integer, and the numerical value starts from 0 and increases sequentially, i.e., for multiple data to be transmitted generated in the multiple communication processes of the model, multiple data to be transmitted identifiers that increase sequentially can be generated.

[0074] Further, after dividing the first memory and the second memory, the second server can also transmit the second address and the data to be transmitted identifier to the memory expansion device deployed on the first server through the switch. When the first server monitors that the second address of the second graphics processing unit and the data to be transmitted identifier are written into the first memory of the memory expansion device, the first server writes the second address of the second graphics processing unit and the data to be transmitted identifier to the first central processing unit, queries the first address according to the data to be transmitted identifier, and stores the first address and the second address in association.

[0075] For example, the second central processing unit can transmit the second address and the to-be-transmitted data identifier to the memory expansion device deployed at the first server through the switch. Specifically, the second central processing unit can send a Load / Store instruction based on the CXL.mem sub-protocol (or a DMA instruction based on the CXL.I / O sub-protocol) to write the to-be-transmitted data identifier in the received information table and the second address of the second memory corresponding to the to-be-transmitted data identifier and the like information into a memory space (the name can be comm_recv_info_mem) in the first memory of the memory expansion device for storing information from the second server via the switch. Then, the first fast computing link controller can write the second address and the to-be-transmitted data identifier of the second graphics processing unit to the first central processing unit in the case of monitoring that the second address and the to-be-transmitted data identifier of the second graphics processing unit are written in the memory of the memory expansion device. Specifically, the first fast computing link controller can monitor the MSI-X written into the above memory space. The MSI-X can record the above second address and to-be-transmitted data identifier. Then, after monitoring the above MSI-X, the first fast computing link controller can encapsulate the second address and the to-be-transmitted data identifier recorded in the MSI-X into a TLP and write the TLP to the first central processing unit via the fast peripheral component interconnect bus.

[0076] The first central processing unit can query the first address according to the to-be-transmitted data identifier, and store the first address and the second address in association. Specifically, when the interrupt controller of the first central processing unit monitors that a TLP is written to the interrupt controller, the first central processing unit can immediately trigger a predetermined program (for example, an interrupt handling program), query the corresponding first address in the receive information table according to the to-be-transmitted data identifier in the TLP, and store the first address and the second address in association in the information table (the name can be send_recv_info_table). The information table can be a hash table. The first address value can be used as a key, and the second address can be used as a value, so as to query the second address according to the first address, thereby facilitating communication between the first central processing unit and the second central processing unit. Based on this, by establishing a mapping relationship between the first address and the second address in the pre-compilation stage, dynamic addressing and handshake operations in subsequent reasoning / training are avoided. In this way, the application can decouple the data transmission control flow from the graphics processing unit execution flow, so as to realize a low-overhead, high-concurrency asynchronous communication flow of "write triggering and automatic forwarding" in the actual communication stage, realize asynchronous and efficient execution of the communication process, and significantly improve the startup efficiency and system scalability of cross-node graphics processing unit communication. The overall scheme does not require the graphics processing unit of the first server as the sender to support the rapid computing link protocol natively, and can establish a low-latency, central processing unit zero-software-intervention, high-bandwidth heterogeneous communication link by using the third type of device (that is, the memory expansion device), the second type of device (that is, the second graphics processing unit), and the MSI-X interrupt mechanism in the rapid computing link ecosystem. The design adapts to the graphics processing unit architecture of some manufacturers and has good engineering feasibility, and is especially suitable for large model distributed reasoning and training acceleration requirements in a heterogeneous graphics processing unit mixed deployment environment on a cloud platform.

[0077] The above has completed the description of the partitioning process of the zero-copy memory in the application. Then, the specific structure of the first server and the second server will be described in combination with the following drawings. First, for the first server of the application, the first central processing unit therein can include a first root complex unit. For example, the first root complex unit can be connected with the first graphics processing unit through a first peripheral component interconnect bus. And a first rapid computing link interconnection channel is configured between the first root complex unit and the memory expansion device. For example, the first rapid computing link interconnection channel can be a communication link based on the rapid computing link protocol. Specifically, it can be a communication link based on the CXL.I / O sub-protocol or a communication link based on the CXL.mem sub-protocol, which is different from the bus connection. In this way, the first graphics processing unit can transmit the to-be-transmitted data to the first root complex unit via the first peripheral component interconnect bus, and then write the to-be-transmitted data to the first memory via the first rapid computing link interconnection channel. The following will be described in combination with Figure 2AA specific description is made.

[0078] Figure 2A A schematic diagram of a first server according to another embodiment of the present application is shown.

[0079] As Figure 2A shown, the first server of this embodiment can include a first graphics processing unit, a first central processing unit and a memory expansion device. The first graphics processing unit can include a computing unit and a corresponding DDR. The first central processing unit can include a first root complex unit.

[0080] Further, the first root complex unit can include a peripheral component interconnect express root port and a first fast compute link root port. For example, the peripheral component interconnect express root port and the first fast compute link root port can be connected through a first root complex internal routing bus. For example, the first root complex internal routing bus can be a bus based on the fast compute link protocol inside the first root complex unit.

[0081] On this basis, the first graphics processing unit can transmit the to-be-transmitted data to the peripheral component interconnect express root port via the first peripheral component interconnect express bus, so as to be transmitted to the first fast compute link root port via the first root complex internal routing bus, and then be written into the first memory via the first fast compute link interconnection channel.

[0082] In addition, the first root complex unit can further include a second fast compute link root port. For example, the second fast compute link root port can be a fast compute link root port. The first fast compute link root port and the second fast compute link root port are connected through a first fast compute link internal interconnection bus. For example, the first fast compute link internal interconnection bus can be a bus based on the fast compute link protocol inside the first root complex unit.

[0083] The first central processing unit can read the to-be-transmitted data from the memory (for example, the first memory) of the memory expansion device, and transmit the to-be-transmitted data to the first fast compute link root port via the first fast compute link interconnection channel, and then to the second fast compute link root port via the first fast compute link internal interconnection bus. The first root complex unit and the switch are connected through a first fast compute link external interconnection bus. For example, the first fast compute link external interconnection bus can be a bus based on the fast compute link protocol. In this way, the first root complex unit can transmit the to-be-transmitted data to the second server through the switch via the first fast compute link external interconnection bus. Specifically, the first central processing unit can transmit the to-be-transmitted data from the first root complex unit to the switch via the first fast compute link external interconnection bus, so that the switch transmits the to-be-transmitted data to the second server according to a second address.

[0084] Based on this, since the first root complex unit is a hardware path module, when the first graphics processing unit interacts with the memory expansion device, the first central processing unit does not need to be called for software intervention, thereby saving the time consumed by the first central processing unit in processing the data to be transmitted during the data transmission process.

[0085] The specific structure of the first server has been described above, and the specific structure of the second server will be described below in combination with the drawings.

[0086] Figure 2B A schematic diagram of the second server according to another embodiment of the application is shown.

[0087] As Figure 2B shown, the second server can include a second graphics processing unit and a second central processing unit. The second graphics processing unit can include a second fast compute link controller and a second memory. The second central processing unit can include a second root complex unit. For example, the second root complex unit can be a root complex. A second fast compute link interconnection channel is configured between the second graphics processing unit and the second root complex unit. For example, the second fast compute link interconnection channel can be a communication link based on the fast compute link protocol. Specifically, it can be a communication link based on the CXL.I / O sub-protocol or a communication link based on the CXL.mem sub-protocol, which is distinguished from the bus connection. The second root complex unit and the switch can be connected by a second fast compute link external interconnection bus. For example, the second fast compute link external interconnection bus can be a bus based on the fast compute link protocol.

[0088] On this basis, the second central processing unit can transmit the second address and the to-be-transmitted data identifier to the second root complex unit via the second fast compute link interconnection channel in advance. Then, the second root complex unit can transmit the second address and the to-be-transmitted data identifier to the switch via the second fast compute link external interconnection bus, and then transmit the second address and the to-be-transmitted data identifier to the memory expansion device via the switch, so that the first central processing unit stores the first address and the second address in association according to the to-be-transmitted data identifier. Then, the first central processing unit can send the to-be-transmitted data to the second server based on the first address, the second address and the to-be-transmitted data identifier stored in association.

[0089] Further, the second central processing unit can transmit the data to be transmitted from the switch to the second root complex unit via the second fast compute link external interconnect bus, and write the data to be transmitted into the memory (e.g., the second memory) of the second graphics processing unit via the second fast compute link interconnect channel according to the second address. For example, the second central processing unit can read the data to be transmitted from the memory (e.g., the second memory) of the second graphics processing unit via the second fast compute link interconnect channel according to the second address in case that the second central processing unit monitors that the second address has been written into the second graphics processing unit, and transmit the data to be transmitted to the second graphics processing unit via the second fast compute link interconnect channel.

[0090] Further, the second root complex unit can include a fifth fast compute link root port and a sixth fast compute link root port. For example, the fifth fast compute link root port is connected with the switch via the second fast compute link external interconnect bus. The fifth fast compute link root port is connected with the sixth fast compute link root port via a third fast compute link internal interconnect bus. For example, the third fast compute link internal interconnect bus can be a bus based on the fast compute link protocol. The sixth fast compute link root port is configured with the second fast compute link interconnect channel with the second graphics processing unit. In this way, the second central processing unit can transmit the data to be transmitted from the switch to the fifth fast compute link root port via the second fast compute link external interconnect bus, and then transmit the data to be transmitted to the sixth fast compute link root port via the third fast compute link internal interconnect bus, and then write the data to be transmitted into the second graphics processing unit via the second fast compute link interconnect channel.

[0091] In addition, the second graphics processing unit can further include a compute unit. The compute unit can be connected with the second fast compute link controller via a first double data rate bus, and the second fast compute link controller can be connected with the second memory via a second double data rate bus. In this way, the second central processing unit can read the data to be transmitted from the second memory via the second fast compute link interconnect channel according to the second address in case that the second central processing unit monitors that the second address has been written into the second graphics processing unit, and then transmit the data to be transmitted to the second fast compute link controller via the second double data rate bus, and then transmit the data to be transmitted to the compute unit via the first double data rate bus. In this way, the compute unit can perform a computing task based on the data to be transmitted.

[0092] Based on this, since the second root complex unit is a hardware path module, when the data to be transmitted is written into the second memory of the second graphics processing unit, the second central processing unit does not need to be called for software intervention, thereby saving the time consumed by the second central processing unit in processing the data to be transmitted in the data transmission process.

[0093] And, compared with the Peripheral Component Interconnect Express bus, the transmission bandwidth of the DDR bus (i.e., the first double-rate bus and the second double-rate bus) is larger and the transmission delay is lower, and the computing unit and the second memory are both in the first central processing unit belonging to the second type device of the fast computing link, without occupying the data transmission channel other than the first central processing unit, thereby saving the channel resources.

[0094] On this basis, since the DMA engine deployed by the first graphics processing unit has the advantage of parallel high-speed transmission, the central processing unit supporting the three sub-protocols of the fast computing link does not need to be involved, and the memory is directly read and written, thereby greatly improving the data transmission efficiency. In addition, compared with the scheme that after the to-be-transmitted data is transmitted to the DRAM (Dynamic Random Access Memory, dynamic random access memory) memory bar based on the zero-copy technology, it is difficult to perform data transmission based on the Load / Store instruction of the CXL.mem sub-protocol (or the DMA instruction based on the CXL.I / O sub-protocol), the present application can realize the use of the high-speed CXL.mem and CXL.I / O sub-protocols such as the Load / Store instruction of the CXL.mem sub-protocol (or the DMA instruction based on the CXL.I / O sub-protocol) to write the to-be-transmitted data into the memory of the second graphics processing unit of the second server through the switch, thereby realizing the high-bandwidth and low-latency communication of the graphics processing unit across the server nodes.

[0095] It is additionally explained that after the distributed deep learning model inference / training phase is completed, the first server can call a resource cleaning function to release the resources of the first memory pre-divided by the memory expansion device, the resources of the memory monitoring register of the first central processing unit, and stop the interrupt state, and the second server can call a resource cleaning function to release the resources of the second memory pre-divided by the second graphics processing unit, the resources of the memory monitoring register of the second central processing unit, and stop the interrupt state.

[0096] Figure 2C A schematic diagram of the first server and the second server according to another embodiment of the present application is shown.

[0097] As Figure 2CAs shown, the cross-node transmission path based on the fast computing link protocol can be in turn: a first memory of the memory expansion device, a first fast computing link interconnection channel (such as a CXL.I / O channel or a CXL.mem channel), a first fast computing link root port, a first fast computing link internal interconnection bus, a second fast computing link root port, a third fast computing link root port, a second fast computing link internal interconnection bus, a fourth fast computing link root port, a second fast computing link external interconnection bus, a fifth fast computing link root port, a third fast computing link internal interconnection bus, a sixth fast computing link root port, a second fast computing link interconnection channel, and a second graphics processing unit. The switch can be based on the fast computing link protocol for data transmission.

[0098] Although the first root complex unit is located in the first central processing unit and the second root complex unit is located in the second central processing unit, since the first root complex unit and the second root complex unit are hardware path modules, in the case that the second fast computing link root port interacts with the third fast computing link root port, or the sixth fast computing link root port interacts with the fourth fast computing link root port, no software intervention of the first central processing unit or the second central processing unit is required, thereby saving the processing time of the first central processing unit and the second central processing unit for the to-be-transmitted data in the cross-node data transmission process.

[0099] Moreover, since the devices between the memory expansion device and the second graphics processing unit all support the CXL.mem sub-protocol and the CXL.I / O sub-protocol, small-scale data interaction can be performed using the Load / Store mode based on the CXL.mem sub-protocol, and large-scale concurrent data interaction can be performed using the DMA mode based on the CXL.I / O sub-protocol. Thus, the application provides the interaction modes based on the two sub-protocols, and improves the interaction flexibility. Moreover, compared with the graphics processing unit cross-node communication technology assisted by the RDMA network card described above, the scheme of the application saves the time consumed due to packetizing and unpacking data, and can achieve shorter processing delay and greater cross-server node data transmission bandwidth.

[0100] In the above scheme of the present application, after the to-be-transmitted data is transmitted from the memory expansion device to the first memory of the second graphics processing unit based on the fast computing link protocol, the second graphics processing unit can directly read the to-be-transmitted data. In this reading process, a high-speed data direct connection transmission channel is established between the second graphics processing unit and the second memory without the intervention of the central processing unit through software forwarding. In this process, no software intervention of the central processing unit supporting the three sub-protocols of the fast computing link is required, and no cache is required to write directly to the zero-copy memory (i.e., the second memory) of the local second type device. Thus, the data transmission efficiency is improved. Moreover, the second graphics processing unit can directly read the first memory, and this process also does not require the participation of the central processing unit supporting the three sub-protocols of the fast computing link, greatly improving the data transmission efficiency.

[0101] Figure 3A A schematic diagram of a first server according to another embodiment of the present application is shown.

[0102] As Figure 3A shown, in one scheme of the present application, the path of the first graphics processing unit writing the to-be-transmitted data to the memory expansion device can be in turn: the computing unit of the first graphics processing unit, the first peripheral component interconnect bus, the fast peripheral component interconnect root port of the first root complex unit, the first root complex internal routing bus, the first fast computing link root port of the first root complex unit, and the memory of the memory expansion device.

[0103] Since the first root complex unit is a hardware path module, when the first graphics processing unit interacts with the memory expansion device, no software intervention of the first central processing unit (i.e., the integrated memory controller (IMC) therein) is required, thereby saving the time consumed by the first central processing unit in processing the to-be-transmitted data during data transmission.

[0104] Since the memory expansion device supports the CXL.I / O sub-protocol, and the CXL.I / O channel multiplexes the bus of the fast peripheral component interconnect 5.0 and higher versions (i.e., the first peripheral component interconnect bus), after the first memory is divided, based on the high-speed transmission characteristics of DMA and the characteristics of zero-copy memory without the intervention of the first central processing unit through software forwarding, a high-speed data direct connection transmission channel (i.e., based on the first peripheral component interconnect bus, the first root complex internal routing bus, and the first fast computing link interconnection channel) between the first graphics processing unit and the memory expansion device can be established.

[0105] And, in the RDMA network card assisted graphics processing unit cross-node communication technology described in the foregoing, the data is sent by the DDR of the graphics processing unit, while in the present application, the to-be-transmitted data is sent by the computing unit of the first graphics processing unit, so that the first graphics processing unit of the present application can directly call the first peripheral component interconnect bus and the peripheral component interconnect root port to interact. In this way, the time consumed due to the execution of the write operation in the process of writing the calculation result of the computing unit of the first graphics processing unit into the DDR and then sending the to-be-transmitted data by the DDR is saved.

[0106] In another scheme, when the DRAM in the first server is used as zero-copy memory, the transmission path of the to-be-transmitted data can be in turn: the computing unit of the first graphics processing unit, the first peripheral component interconnect bus, the peripheral component interconnect root port, the internal interconnect bus of the first central processing unit, the memory channel port, the DDR bus, and the DRAM memory bank.

[0107] Compared with the data transmission path described in the foregoing, after the to-be-transmitted data in this scheme is transmitted to the peripheral component interconnect root port of the first root complex unit, it can be transmitted to the memory channel port via the internal interconnect bus of the first central processing unit, and then directly written into the zero-copy memory in the DRAM memory bank via the DDR bus, without the conversion of the data packet of the peripheral component interconnect protocol to the data packet of the fast computing link protocol by the first fast computing link root port, thereby saving the time consumed due to the conversion of the protocol of the data packet. It should be noted that the transmission bandwidth of this mode is larger than that of the above-mentioned mode. However, this mode is difficult to transmit the to-be-transmitted data to the zero-copy memory in the fast computing link third type device in the receiving server supporting the fast computing link protocol connected through the switch based on the Load / Store instruction of the CXL.mem sub-protocol (or the DMA instruction based on the CXL.I / O sub-protocol). Therefore, the present application does not divide the first memory in the DRAM memory bank, but divides the first memory in the memory of the memory expansion device, so as to realize the cross-node data transmission based on the first scheme described in the foregoing. On this basis, the present application breaks through the limitation of the compatibility of the protocol of the graphics processing unit on the traditional communication path, thereby realizing the efficient cross-node data transmission of the graphics processing unit.

[0108] Figure 3B A schematic diagram of the second server according to another embodiment of the present application is shown.

[0109] As Figure 3B shown, in one scheme of the present application, the path of the computing unit reading the to-be-transmitted data from the second memory of the second graphics processing unit can be in turn: the second memory, the second double-rate bus, the second fast computing link controller, and the computing unit.

[0110] On this basis, compared with the technical solution in the related art that needs to be transferred and unpacked by the receiver RDMA network card after the data from the sender RDMA network card is stored, and then transmitted into the memory of the receiver, the computing unit can directly interact with the second memory through the first double-rate bus and the second double-rate bus, thereby saving the time consumed by the RDMA network card for packetizing and unpacking in the data transmission process.

[0111] Since the second complex unit is a hardware path module, when the to-be-transmitted data is written into the second memory of the second graphics processing unit, the second central processing unit (i.e., the IMC therein) does not need to be called for software intervention, thereby saving the time consumed by the second central processing unit for processing the to-be-transmitted data in the data transmission process.

[0112] Since the second graphics processing unit supports the CXL.I / O sub-protocol, and the CXL.I / O channel multiplexes the bus of Peripheral Component Interconnect Express 5.0 and higher versions (i.e., the second fast computing link interconnection channel), after the second graphics processing unit divides the second memory, based on the feature that the zero-copy memory does not need the second central processing unit to intervene by software for forwarding, a high-speed data direct connection transmission channel (i.e., the first double-rate bus and the second double-rate bus) between the second graphics processing unit and the second memory can be realized.

[0113] Based on the above, the application provides a sender high-speed data direct connection transmission channel based on a fast computing link and a zero-copy technology: in a sender server supporting a fast computing link protocol, a high-speed direct connection transmission channel between a sender graphics processing unit not supporting the fast computing link protocol and having a DMA engine and a local fast computing link third-type device is constructed. Through the fact that the physical layer of the Peripheral Component Interconnect Express 5.0 and above protocol and the CXL.I / O protocol interwork, the sender graphics processing unit not supporting the fast computing link protocol and having the DMA engine directly writes data into the zero-copy memory opened in the local fast computing link third-type device, without the software intervention and explicit cache transfer of the central processing unit supporting the three sub-protocols of the fast computing link, and the data writing efficiency is improved.

[0114] And, for the above cross-node high-speed data transmission channel based on the fast computing link protocol, once the data writing in the fast computing link third type device in the server supporting the fast computing link protocol is completed, the fast computing link controller of the fast computing link third type device in the local triggers the MSI-X interrupt, and the local fast computing link third type device completes the table lookup mapping from send_addr to recv_addr by the fast computing link third type device supporting the fast computing link protocol central processing unit, and then realizes the high-bandwidth and low-latency cross-node data transfer through the CXL.mem or CXL.I / O sub-protocol through the switch to the second type device (i.e., the second graphics processing unit) of the receiving server supporting the fast computing link protocol.

[0115] Further, based on the fast computing link and zero-copy technology, the receiving graphics processing unit directly pulls data from the second memory after sensing the data arrival through the application layer polling memory monitoring register, and completes the whole-process communication without the software intervention of the central processing unit. Based on the similar reasons described above, the second memory is not divided in the DRAM memory bar via the integrated memory controller and memory channel port of the second central processing unit, but is selected to be divided in the memory of the second graphics processing unit, so as to realize the cross-node data transmission based on the first scheme described above. On this basis, the application breaks through the limitation of the traditional communication path on the protocol compatibility of the graphics processing unit, thereby realizing the efficient cross-node data transmission of the graphics processing unit.

[0116] On this basis, the application proposes a graphics processing unit cross-node communication optimization mechanism based on fast computing link polling triggering, which further improves the communication efficiency. Compared with the traditional cross-node graphics processing unit communication scheme assisted by the RDMA network card, the application skips the DDR transfer, data duplication and reorganization operation of the graphics processing unit and the RDMA network card. And the first graphics processing unit of the application writes the to-be-transmitted data into the zero-copy memory in the memory expansion device, that is, triggers the MSI-X interrupt, and then the first central processing unit automatically issues a remote write instruction after the interrupt, without polling and waiting or software stack intervention. After that, the second graphics processing unit can timely sense the data writing completion through the register polling mechanism, and directly read the data in the second memory. The whole process eliminates the steps of cache writing, network card packet packaging / unpacking and target address mapping in the RDMA path, and the communication delay is significantly reduced, which is especially suitable for high-frequency communication scenarios such as frequent cross-node transmission of key-value cache technology in large models, and breaks through the technical barrier of the fast computing link protocol on the compatibility of the graphics processing unit end.

[0117] Figure 4 A flowchart of a data transmission method according to an embodiment of the application is shown.

[0118] As Figure 4 shown, the data transmission method of this embodiment can be applied to a first server deployed in a distributed system. The data transmission method can include operations S410-S440.

[0119] In operation S410, the memory of the memory extension device is divided to obtain a first memory according to a storage space required by to-be-transmitted data in a first graphics processing unit.

[0120] In operation S420, a first address of the first memory is stored in association with a to-be-transmitted data identifier.

[0121] In operation S430, an interrupt event is triggered to write a second address to the first central processing unit in a case where it is monitored that the second address corresponding to the to-be-transmitted data identifier has been written to the memory extension device.

[0122] In operation S440, the second address is stored in association with the first address in a case where it is monitored that the second address has been written to the first central processing unit, so that in a case where it is monitored that the to-be-transmitted data has been written to the memory extension device and the first address has been written to the first central processing unit, the second address is queried according to the first address, and the to-be-transmitted data is transmitted to the second server according to the second address through the switch, so that the second server reads the to-be-transmitted data from the memory of the second graphics processing unit in a case where it is monitored that the to-be-transmitted data has been written to the memory of the second graphics processing unit.

[0123] It should be understood that the first server of the embodiments of the present application can also perform other operations as described above, which will not be repeated here.

[0124] Figure 5 A flowchart of a data transmission method according to an embodiment of the present application is shown.

[0125] As Figure 5 shown, the data transmission method of this embodiment can be applied to a second server deployed in a distributed system. The data transmission method can include operations S510-S550.

[0126] In operation S510, the memory of the second graphics processing unit is divided to obtain a second memory according to a storage space required for storing to-be-transmitted data from the first server.

[0127] In operation S520, the second address of the second memory is stored in association with the to-be-transmitted data identifier, so as to send the second address and the to-be-transmitted data identifier to the first server through the interworking machine, so that the first server stores the second address in association with the first address, and transmits the to-be-transmitted data to the second server according to the second address.

[0128] In operation S530, in a case where it is monitored that the to-be-transmitted data has been written into the second memory, the second address is written into the second central processing unit.

[0129] In operation S540, the to-be-transmitted data from the first server is received, and the to-be-transmitted data is written into the second memory according to the second address.

[0130] In operation S550, in a case where it is monitored that the second address has been written into the second central processing unit, the second graphics processing unit is called to read the to-be-transmitted data from the second memory according to the second address.

[0131] It should be understood that the second server of the embodiments of the present application can also perform other operations as described above, which will not be repeated here.

[0132] Based on the above, the present application proposes a graphics processing unit cross-node communication topology based on zero-copy technology and fast computing link protocol. In this topology, a high-speed direct connection channel is established between the second graphics processing unit which does not support the fast computing link protocol and has a DMA engine and the local fast computing link third type device, data is transmitted across server nodes through CXL.mem sub-protocol or CXL.I / O sub-protocol, software intervention of the central processing unit is avoided, and an efficient cross-server communication path without fast computing link compatible graphics processing unit is realized.

[0133] Secondly, based on the proposed cross-node communication topology, the application further proposes a graphics processing unit cross-node communication optimization mechanism based on fast computing link polling trigger, adopting the process of write-trigger, trigger-move, and move-read. For example, the sender graphics processing unit (i.e. the first graphics processing unit described above) that does not support the fast computing link protocol and has a DMA engine writes into the zero-copy memory in the local fast computing link third-type device to trigger an interrupt, and the central processing unit in the local that supports the fast computing link three sub-protocols controls the cross-node transmission. The receiver graphics processing unit (i.e. the second graphics processing unit described above) that supports the fast computing link protocol polls the change of the sense register and then pulls the data, thereby realizing low-delay and high-bandwidth heterogeneous communication. Based on the proposed graphics processing unit cross-node communication optimization method for distributed deep learning scenarios, the cross-node communication performance and resource coordination efficiency of the current graphics processing unit that does not support the fast computing link protocol in distributed deep learning are effectively improved, and the limitation of traditional graphics processing unit communication on protocol compatibility is broken through.

[0134] Specifically, based on the proposed graphics processing unit cross-node communication optimization mechanism based on fast computing link polling trigger, the central processing unit in the sender server that supports the fast computing link three sub-protocols cooperates with the sender graphics processing unit that does not support the fast computing link protocol and has a DMA engine to use the zero-copy technology to open the extended memory of the local third-type device as zero-copy memory, thereby constructing a high-speed data direct transmission channel between the sender graphics processing unit that does not support the fast computing link protocol and has a DMA engine and the zero-copy memory in the local third-type device without the forwarding intervention of the central processing unit that supports the fast computing link three sub-protocols, which is referred to as a sender high-speed data direct transmission channel based on the fast computing link and the zero-copy technology.

[0135] Also, based on the cross-node communication optimization mechanism of the graphics processing unit based on the fast computing link polling trigger proposed in the present application, when the fast computing link controller of the third type of device supporting the fast computing link protocol (i.e., the first fast computing link controller described above) in the sender server supporting the fast computing link protocol receives the write operation instruction of the receiver high-speed data direct transmission channel based on the fast computing link and zero-copy technology, the extended message signaled interrupts (MSI-X) event of the corresponding write zero-copy memory address in the fast computing link controller is triggered while completing the corresponding zero-copy memory write in the third type of device, and the fast computing link controller encapsulates the corresponding MSI-X event as a transaction layer packet (TLP) packet and writes it to the interrupt controller of the central processing unit through the fast peripheral component interconnect channel, and the interrupt controller of the central processing unit immediately triggers the interrupt processing program to modify the value of the memory monitoring register. For the sender graphics processing unit application state polling thread "not supporting the fast computing link and having a DMA engine" running on the "central processing unit side supporting the three sub-protocols of the fast computing link locally", the polling is interrupted after the value of the memory monitoring register is monitored to change, and the Load / Store instruction based on the CXL.mem sub-protocol (or the DMA instruction based on the CXL.I / O sub-protocol) is issued to transmit data from the zero-copy memory in the third type of device in the sender server supporting the fast computing link protocol to the zero-copy memory (i.e., the second memory of the second graphics processing unit described above) in the second type of device of the fast computing link (i.e., the second graphics processing unit described above) in the receiver server supporting the fast computing link protocol connected through the switch. Based on this, the present application realizes efficient and high-bandwidth cross-node communication of graphics processing units of different nodes.

[0136] The computer program product of the first aspect can include a computer readable storage medium. The computer readable storage medium can be a tangible computer readable storage medium storing the program code. The computer readable storage medium can include, but is not limited to, one or more types of computer readable storage media capable of storing the computer program code including solid state memory, hard disks, optical disk, magnetic disk, magnetic tape, or any like suitable device, etc.

[0137] Those skilled in the art will understand that features recited in the various embodiments of the present application can be combined and / or integrated in various ways, even if such combinations or integrations are not expressly noted in the present application. In particular, features recited in the various embodiments of the present application can be combined and / or integrated in ways that do not depart from the spirit and teachings of the present application. All such combinations and / or integrations are within the scope of the present application.

[0138] The above describes the embodiments of the present application. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present application. Although the above describes the embodiments respectively, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present application, and these substitutions and modifications shall fall within the scope of the present application.

Claims

1. A server system, characterized by The server system comprises: a first server comprising a first graphics processing unit, a first central processing unit and a memory expansion device; the first server is configured to write to-be-transmitted data in the first graphics processing unit into the memory expansion device and transmit to a switch via a first rapid computing link external interconnection bus; the first central processing unit is configured to, in a case where it is monitored that the to-be-transmitted data has been written into the memory expansion device and a first address of the to-be-transmitted data in the memory expansion device is written into the first central processing unit, query a second address according to the first address and transmit the to-be-transmitted data to a second server according to the second address via the switch; the switch is configured to transmit the to-be-transmitted data to the second server via a second rapid computing link external interconnection bus; and the second server comprises a second graphics processing unit and a second central processing unit; the second server is configured to transmit the to-be-transmitted data to the second graphics processing unit via a double-rate bus; the second central processing unit is configured to receive the to-be-transmitted data from the first server and write the to-be-transmitted data into a second memory of the second graphics processing unit according to the second address; and in a case where it is monitored that the second address has been written into the second central processing unit, call the second graphics processing unit to read the to-be-transmitted data from the second memory according to the second address.

2. The server system of claim 1, wherein, The first server further comprises: the first central processing unit is configured to divide a memory of the memory expansion device to obtain a first memory according to a storage space required by the to-be-transmitted data in the first graphics processing unit; and store a first address of the first memory and a to-be-transmitted data identifier in association; the memory expansion device comprises a first rapid computing link controller; the first rapid computing link controller is configured to, in a case where it is monitored that a second address corresponding to the to-be-transmitted data identifier from the second server has been written into the memory expansion device, trigger an interrupt event to write the second address into the first central processing unit.

3. The server system of claim 2, wherein, The first central processing unit comprises a peripheral component interconnect root port and a first rapid computing link root port; the peripheral component interconnect root port and the first rapid computing link root port are connected through a first root complex internal routing bus; and a first rapid computing link interconnection channel is configured between the first rapid computing link root port and the memory expansion device; the first central processing unit is configured to transmit the to-be-transmitted data to the peripheral component interconnect root port via a first peripheral component interconnect bus, to the first rapid computing link root port via the first root complex internal routing bus, and to the memory expansion device via the first rapid computing link interconnection channel.

4. The server system of claim 3, wherein, The first central processing unit further comprises a second fast computing link root port; the first fast computing link root port is connected with the second fast computing link root port through a first fast computing link internal interconnection bus; the second fast computing link root port is connected with the switch through a first fast computing link external interconnection bus; The first central processing unit is configured to read the to-be-transmitted data from the memory expansion device, transmit the to-be-transmitted data to the first fast computing link root port via the first fast computing link interconnection channel, transmit the to-be-transmitted data to the second fast computing link root port via the first fast computing link internal interconnection bus, and transmit the to-be-transmitted data to the second server via the switch through the first fast computing link external interconnection bus.

5. The server system of claim 4, wherein, The switch comprises a third fast computing link root port and a fourth fast computing link root port; the second fast computing link root port is connected with the third fast computing link root port through the first fast computing link external interconnection bus; the third fast computing link root port is connected with the fourth fast computing link root port through a second fast computing link internal interconnection bus; and the fourth fast computing link root port is connected with the second server through a second fast computing link external interconnection bus; The switch is configured to transmit the to-be-transmitted data transmitted to the third fast computing link root port via the first fast computing link external interconnection bus to the fourth fast computing link root port via the second fast computing link internal interconnection bus, and transmit the to-be-transmitted data to the second server via the second fast computing link external interconnection bus.

6. The server system of claim 5, wherein, The second server comprises: The second central processing unit is configured to divide the memory of the second graphics processing unit according to the storage space required for storing the to-be-transmitted data from the first server to obtain a second memory; and store a second address of the second memory in association with an identification of the to-be-transmitted data, so as to send the second address and the identification of the to-be-transmitted data to the first server through the switch, so that the first server stores the second address in association with the first address, and transmits the to-be-transmitted data to the second server according to the second address; The second graphics processing unit comprises a second fast computing link controller; The second fast computing link controller is configured to write the second address to the second central processing unit when it is monitored that the to-be-transmitted data has been written into the second memory.

7. The server system of claim 6, wherein, The second central processing unit comprises a root complex unit; a second fast computing link interconnection channel is arranged between the root complex unit and the second graphics processing unit; and the root complex unit is connected with the switch through a second fast computing link external interconnection bus; The second central processing unit is further configured to transmit the data to be transmitted to the root complex unit via the second external interconnection bus of the fast computing link, and write the data to be transmitted into the memory of the second graphics processing unit according to the second address via the second interconnection channel of the fast computing link.

8. The server system of claim 7, wherein: The second central processing unit is further configured to read the data to be transmitted from the memory of the second graphics processing unit according to the second address when it is monitored that the second address has been written into the memory of the second graphics processing unit.

9. The server system of claim 8, wherein, The root complex unit comprises a fifth fast computing link root port and a sixth fast computing link root port; the fifth fast computing link root port is connected with the switch via the second external interconnection bus of the fast computing link; the fifth fast computing link root port is connected with the sixth fast computing link root port via a third internal interconnection bus of the fast computing link; and the second interconnection channel of the fast computing link is configured between the sixth fast computing link root port and the second graphics processing unit. The second central processing unit is further configured to transmit the data to be transmitted to the fifth fast computing link root port via the second external interconnection bus of the fast computing link, to the sixth fast computing link root port via the third internal interconnection bus of the fast computing link, and to write the data to be transmitted into the second graphics processing unit via the second interconnection channel of the fast computing link.

10. The server system of claim 9, wherein, The second graphics processing unit comprises a computing unit, which is connected with the second fast computing link controller via a first double-rate bus; and the second fast computing link controller is connected with the second memory via a second double-rate bus. The second central processing unit is further configured to read the data to be transmitted from the second memory according to the second address via the second interconnection channel of the fast computing link when it is monitored that the second address has been written into the second graphics processing unit; and to transmit the data to be transmitted to the second fast computing link controller via the second double-rate bus, and to the computing unit via the first double-rate bus.

Images