P2P communication method and system between IO devices based on PCIe-NTB

Abstract

The invention provides a PCIe-NTB (Peripheral Component Interconnect Express-Network Temporary Block)-based P2P (Peer-to-Peer) communication method and system between IO (Input / Output) devices, the system comprises a plurality of computing nodes, and the computing nodes comprise a PCIe Switch, an NTB and a plurality of PCIe IO devices; the plurality of PCIe IO devices are connected with the CPU through the PCIe Switch; the NTB is connected with the PCIe Switch; the PCIe IO equipment at least comprises two different types of IO equipment, namely an IO equipment type 1 and an IO equipment type 2; the plurality of computing nodes are connected through the NTB; the system supports P2P communication between IO device types 1 or between IO device types 1 and IO device types 2 of different computing nodes. In the scheme, P2P communication can be directly carried out among IO devices of a plurality of computing nodes on the basis of PCIe-NTB, protocol conversion overhead and main memory and IO device memory moving overhead required by traditional NIC RDMA communication are avoided, and the characteristics of low delay and high throughput of a PCIe network can be fully utilized.

Description

Technical Field

[0001] The present invention relates to the fields of integrated circuit design, data transmission and data communication, and in particular to a P2P communication method and system between IO devices based on PCIe-NTB. Background Art

[0002] High-performance computing workloads often have high requirements for IO resources. For example, many computing clusters rely on computing accelerators such as graphics processing units (GPUs) and field-programmable gate arrays (FPGAs) to increase processing speed. Efficiently moving data between networked nodes and on such computing accelerators has been a research challenge for decades. In recent years, we have also seen the convergence of the research fields of high-performance computing, big data, and machine learning. This has led to new demands for IO performance, with distributed large-capacity storage officially becoming a requirement for high-performance computing, while low-latency networking and facilitating access to computing accelerators have become cloud computing issues. If IO resources (devices) are sparsely distributed among hosts, cluster nodes with IO resources may become bottlenecks when workloads require heavy computation on GPUs or fast access to storage. Conversely, if the workload's needs are more dispersed, over-provisioning node resources may result in underutilization of devices. Heterogeneous workloads may even require very different device compositions from node to node.

[0003] Many network adapters support zero-copying of application memory from one system to another via Remote Direct Memory Access (RDMA). RDMA is not only used in many distributed shared memory cluster applications, but is also often used to implement resource disaggregation. Low-latency storage devices such as non-volatile memory express (NVMe) can be shared at the block level in a cluster. This is the case with NVMe architectures, where RDMA is used to provide direct access and avoid going through the block layer of the operating system on the server. Similarly, the results of GPU computations can be copied directly from GPU memory to the network using RDMA without first being copied to system memory and going through the network stack. RDMA disaggregation is usually implemented as middleware for specific applications. While this usually requires the application software to use specific programming models and semantics, the benefit is that several hosts in the network can share resources. However, while RDMA allows data to be efficiently transferred over the network, translation between network protocols and local IO buses is unavoidable. This protocol translation results in non-negligible latency overhead compared to accessing local devices. Summary of the invention

[0004] In order to solve at least some of the problems existing in the above-mentioned prior art and further reduce the delay overhead of communication between IO devices in multiple computing nodes, the present invention provides a P2P communication method and system between IO devices based on PCIe-NTB. Specifically, the present invention provides the following technical solutions: Multiple computing nodes are interconnected through NTB. Through NTB address conversion, IO devices of different nodes can perform P2P communication between devices based on remote IO memory reading and writing or directly operate remote IO memory based on Queue Pair queue (i.e. QP queue).

[0005] On the one hand, the present invention provides a P2P communication system between IO devices based on PCIe-NTB, the system includes multiple computing nodes, the computing nodes include PCIe Switch, NTB and multiple PCIe IO devices; The multiple PCIe IO devices are connected to the CPU via a PCIe Switch; the NTB is connected to the PCIe Switch; The PCIe IO device includes at least two different types of IO devices, namely, IO device type 1 and IO device type 2; The multiple computing nodes are connected via NTB; The IO device type 1 is connected to an IO memory, and the IO memory is arranged in the IO device; The IO device type 1 is a device that supports IO memory and DMA functions; the IO device type 2 is a device that supports queue pairs and DMA functions; The system supports P2P communication between IO device type 1 in different computing nodes and between IO device type 1 and IO device type 2 in different computing nodes.

[0006] Preferably, the system further comprises a communication subsystem, and the communication subsystem comprises: an NTB resource management module, a computing node communication module, a remote device management module and an IO device communication management module; The NTB resource management module is used to manage the resources in the NTB, and the resources in the NTB include a memory conversion window and a doorbell interrupt; The computing node communication module performs P2P communication between multiple computing nodes based on the memory translation window and doorbell interrupt provided by the NTB resource management module; The remote device management module is used to manage remote PCIe IO devices to complete the configuration of remote PCIe IO devices, remote IO memory management and queue pair configuration; The IO device communication management module is used for P2P communication management between IO devices in different computing nodes.

[0007] Preferably, bidirectional P2P communication is achieved between IO device types 1 in different computing nodes through IO memory sharing, wherein the local computing node maps the IO memory of the remote computing node to the local PCIe domain through a memory conversion window; and / or the remote computing node maps the IO memory of the local computing node to the remote PCIe domain through a memory conversion window.

[0008] Preferably, the IO device type 1 and the IO device type 2 in different computing nodes perform P2P communication based on the queue pair, and after the local computing node initializes the queue pair, the base address of the queue pair is written into the queue control register in the remote computing node; The queue pair stores multiple descriptors, each of which describes data to be written or read from IO device type 2 of a remote computing node; the destination address of the data to be written or read from IO device type 2 of a remote computing node is the IO memory address connected to IO device type 1 of a local computing node.

[0009] Preferably, the P2P communication process between IO device type 1 in different computing nodes is: S11, the IO device A corresponding to the IO device type 1 in the local computing node obtains the IO memory address Y of the IO device B corresponding to the IO device type 1 in the remote computing node through the communication subsystem in the local computing node, and obtains the NTB mapping address M of the address Y in the PCIe domain of the local computing node; S12, IO device A initiates a PCIe transaction message to write a PCIe transaction to the mapping address M through the DMA controller; S13, the PCIe transaction message is routed to the NTB of the local computing node, and the NTB replaces the mapping address M in the PCIe transaction message with the address Y through the address conversion mechanism to form a replaced PCIe transaction message, and sends it to the NTB of the remote computing node; S14, after the replacement, the PCIe transaction message is routed from the NTB of the remote computing node to the IO device B and written to the address Y; S15. After IO device A completes the DMA operation, the communication subsystem of the local computing node notifies the remote computing node that the message transmission is completed, thereby completing the P2P communication.

[0010] Preferably, the P2P communication process between IO device type 1 and IO device type 2 in different computing nodes is: S21, the remote device management modules of the local computing node and the remote computing node respectively complete the initialization of their respective queue pairs and the IO memory initialization of their respective IO devices; S22, the local computing node completes the descriptor initialization of the submission queue in the IO memory of IO device A through the IO device communication management module, and writes the queue base address into the queue control register of IO device B; the IO device B is set at the remote computing node and is an IO device type 2; the IO device A is set at the local computing node and is an IO device type 1; the written or read data comes from IO device B, and the destination address of the written or read data is IO device A; S23, IO device A initiates a request to write or read data to IO device B by updating the descriptor information in the submission queue; S24, IO device B completes writing or reading data through the DMA controller, and writes a completion status in the completion queue after completion; S25. IO device A checks the status in the completion queue to confirm that data writing or reading is completed, thereby completing the P2P communication.

[0011] Preferably, in said S23, the information contained in the descriptor includes at least the operation to be completed, the transmission length, the source address and the destination address; The updating also includes updating a tail pointer of the submission queue.

[0012] Preferably, when performing P2P communication, the remote computing node adds the remote available IO devices and device resources to the remote device management module of the remote computing node based on the user configuration, and then notifies the remote available IO devices and device resources to the computing node communication module in the local computing node through the remote computing node communication module, and then the computing node communication module sends them to the local remote device management module; The information of available IO devices and device resources transmitted between the computing node communication modules is performed through the NTB; The available IO devices are based on user configuration settings.

[0013] Preferably, in the user-based configuration setting, the configuration work includes queue pair application, queue status synchronization, communication data source address, communication data destination address and NTB conversion window configuration.

[0014] On the other hand, the present invention also provides a P2P communication method between IO devices based on PCIe-NTB, the method is applied to the system as described above, and the method includes: S1. After each computing node is started, a communication connection is established through the NTB of each computing node; S2. Obtain available IO devices and device resources of the remote computing node; S3. Configure the IO device for P2P communication and complete the configuration of the NTB address translation window; S4. Determine the type of IO device for P2P communication; if it is P2P communication between IO device type 1, initialize the IO shared memory and communicate through the DMA controller; if it is P2P communication between IO device type 1 and IO device type 2, complete the queue pair initialization and IO shared memory initialization, and communicate based on the queue pair mode.

[0015] Compared with the prior art, this solution has at least the following beneficial effects: In this solution, the IO devices of multiple computing nodes can directly perform P2P communication based on PCIe-NTB, eliminating the protocol conversion overhead and main memory and IO device memory movement overhead required for traditional NIC RDMA-based communication. Based on this communication method, the IO devices between computing nodes can fully enjoy the benefits of low-latency, high-throughput PCIe networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A block diagram of a P2P communication system according to an embodiment of the present invention; Figure 2 A block diagram of the composition of a P2P communication subsystem in a multi-computing node case according to an embodiment of the present invention; Figure 3 A schematic diagram of system module configuration according to an embodiment of the present invention; Figure 4 A schematic diagram of a P2P communication method flow chart according to an embodiment of the present invention; Figure 5 A schematic diagram of a P2P communication data flow according to an embodiment of the present invention; Figure 6 The figure is a schematic diagram of the existing NIC RDMA-based communication data flow between IO devices. DETAILED DESCRIPTION

[0018] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Those skilled in the art should know that the following specific embodiments or specific implementations are a series of optimized settings listed by the present invention to further explain the specific content of the invention, and these settings can be combined or used in association with each other, unless the present invention clearly states that some or a specific embodiment or implementation cannot be associated or used together with other embodiments or implementations. At the same time, the following specific embodiments or implementations are only used as the most optimized settings, and are not to be understood as limiting the scope of protection of the present invention.

[0020] This solution provides a P2P communication method and system between IO devices based on PCIe-NTB, which can effectively reduce the latency overhead of communication between IO devices in multiple computing nodes. PCIe is the most widely used standard for connecting devices to computer systems. Although the PCIe bus was originally designed as a local IO bus connecting devices and the central processing unit (CPU) on the motherboard, this embodiment extends the bus from a single computer to several systems by using a non-transparent bridge (NTB).

[0021] In this embodiment, Figure 1 As shown, the solution can be implemented in a system manner, the system includes multiple computing nodes (the number of nodes is greater than or equal to 2), Figure 1 In the example, two computing nodes are used for explanation; multiple PCIe IO devices are set in the computing node, such as GPU, FPGA, NVMe, etc. Multiple PCIe IO devices can be directly connected to the CPU through the PCIe slot provided by the CPU, or connected to the CPU through PCIe switching; IO devices in the computing node can be divided into two types, namely IO device type 1 and IO device type 2. Multiple computing nodes in the system are connected through PCIe-NTB, and a communication subsystem is set in the computing node to cooperate with the above modules to realize P2P communication between IO devices. P2P communication can be carried out between IO device type 1 and IO device type 1, and IO device type 1 and IO device type 2 between multiple computing nodes through PCIe-NTB.

[0022] The computing node is also provided with a main memory, the processor is respectively connected to the main memory and the PCIe Switch module, the PCIe Switch module is respectively connected to the IO device type 1, the IO device type 2 and the NTB module; in addition, preferably, the IO device type 1 is also provided with an IO memory, that is, the IO memory is provided in the IO device.

[0023] In this embodiment, IO device type 1 is a device that supports IO memory and DMA functions, such as GPU and FPGA computing cards. This type of device can expose part of the IO memory in the device to the PCIe Bar space, and other IO devices directly move the messages to be communicated to the IO memory. IO device type 2 is a device that supports the queue pair (QP, queue pair) mechanism and DMA function, such as NVMe cards. This type of device provides a mechanism for accessing resources in the device to the outside world through submission queues SQ and completion queues CQ, which is called the QP mechanism.

[0024] In this embodiment, the NTB device is a non-transparent bridge device, which is usually provided by a PCIe switching chip, a processor, etc., and is used to connect two PCIe switching domains. The NTB device provides multiple memory conversion windows and doorbell interrupts, etc. Based on the memory conversion windows and doorbell interrupts of the NTB device, communication between two or more PCIe switching domains can be achieved.

[0025] like Figure 2 As shown, it is a block diagram of the P2P communication subsystem between IO devices based on PCIe-NTB of multiple computing nodes provided by an embodiment of the present invention. It should be noted here that the communication subsystem refers to the software system in the communication system proposed in this solution. For the sake of distinction, it is called a communication subsystem. This embodiment takes the system including computing node 1, computing node 2, computing node 3, and computing node 4 as an example. The four computing nodes connect the PCIe-NTB device to the PCIe Switch. It should be understood that the number of computing nodes can be adjusted according to user requirements, such as 2 nodes or more. When there is only one computing node, the computing node can also be set to the method of this solution to cope with possible P2P communication needs in the future. The number of computing nodes should not be understood as a limitation on the protection scope of this solution.

[0026] like Figure 3As shown, in this embodiment, the P2P communication software part between IO devices based on PCIe-NTB, that is, the communication subsystem part, is composed of an NTB resource management module, a computing node communication module, a remote device management module and an IO device communication management module. The NTB resource management module is responsible for managing resources such as memory conversion windows and doorbell interrupts in the NTB; the computing node communication module realizes P2P communication of multiple computing nodes through the memory conversion window and doorbell interrupt provided by the NTB resource management module. More preferably, each computing node communicates based on a memory message queue; the remote device management module is used to manage remote IO devices, and the local and remote ends complete the remote IO device configuration, the IO memory management exposed in the PCIe bar space, and the QP queue configuration of the IO device through the communication mechanism provided by the computing node communication module. The above configuration and management need to be performed at both the local and remote ends; the IO device communication management module realizes P2P communication management between IO devices based on the memory conversion window and doorbell interrupt provided by the NTB resource management module. The main process of P2P communication management here is: 1. Synchronize QP queue information or communication IO memory information; 2. Complete the NTB address translation configuration for communication.

[0027] like Figure 4 As shown, in this embodiment, the P2P communication process between IO devices based on PCIe-NTB is as follows: Multiple computing nodes connected via PCIe-NTB and their PCIe IO devices are powered on and started normally, and the P2P communication subsystem between IO devices based on PCIe-NTB is started normally; After the NTB resource management module is started, the NTB memory conversion window and doorbell interrupt for node communication are configured. The configuration information here can be obtained from the device driver, mainly obtaining queue information and IO memory address information; the communication submodule of the computing node establishes a communication connection through the communication channel provided by NTB.

[0028] The computing node obtains the available IO devices and their device resources of the remote computing node through the computing node communication module, and adds the available IO device type 1 and IO device type 2 to the remote device management module. Here, in a more specific implementation, the remote device management module first adds its device and device resources to the remote device management module based on the user's configuration, and then notifies the local IO device and device information to the corresponding computing node communication module of the other end computing node through the computing node communication module. After receiving the message, the computing node communication module of the other end adds the device and device resources to the remote device management module of the computing node at that end. It should be noted here that the other end and the remote end in this embodiment are used for the convenience of expression and are used to distinguish the two computing stage ends in the communication. Therefore, the two are interchangeable. Here, the judgment of the available IO devices is set according to the user's needs. Which devices of the user need to perform P2P communication across NTB, that is, the IO device type 1 or IO device type 2 that the user needs to perform P2P communication, can be managed by the above-mentioned method of this system.

[0029] In another more specific implementation, the computing node communication submodule obtains remote node information in the following manner: each computing node collects IO resource information within its own domain, and then transmits the information to its own peer through the NTB channel to complete resource information acquisition at the local end.

[0030] The user selects the device for P2P communication between IO devices through configuration, etc., and completes the configuration of device communication according to the P2P communication type through the IO device communication management module, which mainly includes the configuration of device memory conversion window. The configuration of device communication here mainly includes the application of QP (queue pair) queue, synchronization of queue status, source address and destination address of communication data and configuration of NTB conversion window.

[0031] The communication between IO device type 1 is based on IO shared memory. The local computing node maps the IO memory of the remote computing node to the local PCIe domain through the memory conversion window of NTB, so that the local IO device can directly read and write the mapped remote IO memory through DMA; similarly, the remote node maps the local IO memory to the PCIe domain of the remote node through the memory conversion window of NTB, and the remote IO device can directly read and write the mapped local IO memory through DMA; after the memory reading and writing is completed, the node is notified by writing the completion flag or doorbell interrupt to the specified IO memory address. Bidirectional P2P communication can be carried out between IO device type 1.

[0032] In a more specific implementation, the process of notifying the node can be set as follows: if device A writes a completion flag to device B, and device B polls the completion flag, when it finds that the write completion flag is set, it can be known that the data transferred from device A to device B has been transferred. Device A of node 1 wants to write data to device B of node 2, then after the writing is completed, node 1 can trigger a PCIe interrupt to node 2 through the write doorbell interrupt, thereby executing the doorbell interrupt processing function of node 2, and notifying the communication system that the data transmission has been completed.

[0033] The communication between IO device type 1 and IO device type 2 is based on the QP queue (i.e., queue pair). The following is an example of P2P communication between IO device type 1 in compute node 1 (corresponding to device A) and IO device type 2 in compute node 2 (corresponding to device B). The communication process is as follows: (1) The remote device management modules of computing node 1 and computing node 2 complete the initialization of the IO memory and QP queue required for their respective communications; after initialization, since the PCIe network is based on address routing, after initialization according to the method of this embodiment, based on the initialized address, computing node 1 and device A can access the Bar space register of device B through the memory translation window of NTB, and computing node 2 and device B can access the IO memory of device A through the memory translation window of NTB; wherein, the memory translation window method can use the setting method described above; (2) Computing node 1 completes the descriptor initialization of the SQ queue (i.e., submission queue) in the IO memory through the IO device communication management module, and writes the queue base address into the queue control register of device B; the SQ queue stores multiple descriptors, each of which describes the data to be written or read from device B. The destination address for writing or reading data is the IO memory address of device A. Initially, these descriptors are empty; the descriptor initialization operation mainly sets the descriptor content to 0 and initializes the queue head and tail pointers.

[0034] (3) Device A initiates a request to write or read data to device B by updating the descriptor information in the SQ queue (i.e., submission queue). In a specific implementation, this step can be set as follows: The SQ queue is composed of several descriptors, which contain information such as the operation to be completed, the transmission length, the source address of the operation, the destination address, etc. For example, if device A wants to write data to device B, then device A operates the SQ queue provided by device B and modifies the descriptor at the end of the queue: the operation is "write", the source address of the descriptor is the IO memory address X of device A, the destination address of the descriptor is the IO address Y of device B, and the transmission length is L. After writing the descriptor, the tail pointer of the submission queue is updated to complete the request to write or read data to device B. It can be understood that here only the example of device A writing data to device B is used. The same method can also be used to complete the operation of device A reading data from device B. In the same way, the descriptor information is sealed and the tail pointer is modified to initiate and complete the reading of data. Of course, it can be understood here that the source and destination roles of the above-mentioned devices A and B are interchangeable.

[0035] (4) Device B completes the reading or writing of data through the DMA controller, and writes the completion status in the CQ queue (i.e., the completion queue) after completion; writing the completion status can be achieved by, for example, writing 1 to the flag bit in the queue; (5) Device A checks the status in the CQ queue to confirm that data reading and writing have been completed.

[0036] In addition, in another scenario, Figure 5 As shown, for the case where data is transmitted between IO device type 1 and another IO device type 1, in this embodiment, it is taken that IO device A needs to transmit data with address X in its IO memory to IO device B as an example, and the transmission process is as follows: (1) IO device A corresponding to IO device type 1 in computing node 1 obtains IO memory address Y used by IO device B to receive data through the communication subsystem, and obtains the NTB mapping address M of address Y in the PCIe domain of computing node 1; obtains data such as IO memory address Y of device B through the communication subsystems of computing nodes 1 and 2, and the communication subsystems communicate using memory message queues; (2) IO device A initiates a PCIe write transaction to the memory at address M through the DMA controller, forming a PCIe transaction message. From the perspective of IO device A, this is equivalent to writing data to the NTB device of compute node 1. (3) The PCIe transaction message (i.e., PCIe standard TLP) reaches the NTB in computing node 1 after passing through the PCIe routing. The NTB replaces the address M in the PCIe transaction message with the address Y through the address conversion mechanism, thus forming a replaced PCIe transaction message. (4) After the replacement, the PCIe transaction message goes out from the NTB of computing node 2 and arrives at IO device B after passing through the PCIe router. The data is finally written to the IO memory address Y of IO device B. (5) After IO device A completes the DMA operation, the communication subsystem notifies computing node 2 through a message that the message transmission has been completed, and the communication between the entire IO devices is completed.

[0037] As a comparison, combined Figure 6 Explain the traditional NIC transmission method. Figure 6 In the figure, the traditional IO device completes the data transmission process through the RDMA mechanism of the NIC (network interface card device). The IO device first transmits the data to the NIC through the PCIe message. The NIC converts the data into a packaged message through the RDMA protocol and transmits it to the network. After the data reaches the NIC in the target node, it needs to be decapsulated by the protocol, then converted into a PCIe message, and finally written to the peer IO device. Figure 6 In the IO device B in the IO device B. Compared with the communication method in the present invention, the traditional IO device uses the RDMA mechanism of the NIC to communicate between devices, which requires complex network protocol conversion, which will bring additional communication delay and performance overhead. The communication method provided by the present invention has a communication overhead that is basically consistent with the overhead of the local PCIe IO, and NTB hardly introduces additional delay overhead.

[0038] In another embodiment, the present solution can be implemented by a large electronic device, which may include one or more modules in the system modules in the above embodiments. The module may be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0039] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A P2P communication system between IO devices based on PCIe-NTB, characterized in that: The system includes multiple computing nodes, each of which includes a PCIe Switch, an NTB, and multiple PCIe IO devices; The multiple PCIe IO devices are connected to the CPU via a PCIe Switch; the NTB is connected to the PCIe Switch; The PCIe IO device includes at least two different types of IO devices, namely, IO device type 1 and IO device type 2; The multiple computing nodes are connected via NTB; The IO device type 1 is connected to an IO memory, and the IO memory is arranged in the IO device; The IO device type 1 is a device that supports IO memory and DMA functions; the IO device type 2 is a device that supports queue pairs and DMA functions; The system supports P2P communication between IO device type 1 in different computing nodes and between IO device type 1 and IO device type 2 in different computing nodes.

2. The system according to claim 1, characterized in that The system also includes a communication subsystem, which includes: an NTB resource management module, a computing node communication module, a remote device management module and an IO device communication management module; The NTB resource management module is used to manage the resources in the NTB, and the resources in the NTB include a memory conversion window and a doorbell interrupt; The computing node communication module performs P2P communication between multiple computing nodes based on the memory translation window and doorbell interrupt provided by the NTB resource management module; The remote device management module is used to manage remote PCIe IO devices to complete the configuration of remote PCIe IO devices, remote IO memory management and queue pair configuration; The IO device communication management module is used for P2P communication management between IO devices in different computing nodes.

3. The system according to claim 2, characterized in that Bidirectional P2P communication is achieved between IO device types 1 in different computing nodes through IO memory sharing, wherein the local computing node maps the IO memory of the remote computing node to the local PCIe domain through a memory translation window; and / or the remote computing node maps the IO memory of the local computing node to the remote PCIe domain through a memory translation window.

4. The system according to claim 2, characterized in that The IO device type 1 and the IO device type 2 in different computing nodes perform P2P communication based on the queue pair. After the local computing node initializes the queue pair, the base address of the queue pair is written into the queue control register in the remote computing node. The queue pair stores a plurality of descriptors, wherein the descriptors describe data to be written into or read from an IO device type 2 of a remote computing node; The destination address of the data in the IO device type 2 of the remote computing node to be written or read is the IO memory address connected to the IO device type 1 of the local computing node.

5. The system according to claim 3, characterized in that The P2P communication process between IO device type 1 in different computing nodes is as follows: S11, the IO device A corresponding to the IO device type 1 in the local computing node obtains the IO memory address Y of the IO device B corresponding to the IO device type 1 in the remote computing node through the communication subsystem in the local computing node, and obtains the NTB mapping address M of the address Y in the PCIe domain of the local computing node; S12, IO device A initiates a PCIe transaction message to write a PCIe transaction to the mapping address M through the DMA controller; S13, the PCIe transaction message is routed to the NTB of the local computing node, and the NTB replaces the mapping address M in the PCIe transaction message with the address Y through the address conversion mechanism to form a replaced PCIe transaction message, and sends it to the NTB of the remote computing node; S14, after the replacement, the PCIe transaction message is routed from the NTB of the remote computing node to the IO device B and written to the address Y; S15. After IO device A completes the DMA operation, the communication subsystem of the local computing node notifies the remote computing node that the message transmission is completed, thereby completing the P2P communication.

6. The system according to claim 4, characterized in that The P2P communication process between IO device type 1 and IO device type 2 in different computing nodes is as follows: S21, the remote device management modules of the local computing node and the remote computing node respectively complete the initialization of their respective queue pairs and the IO memory initialization of their respective IO devices; S22, the local computing node completes the descriptor initialization of the submission queue in the IO memory of IO device A through the IO device communication management module, and writes the queue base address into the queue control register of IO device B; the IO device B is set at the remote computing node and is an IO device type 2; the IO device A is set at the local computing node and is an IO device type 1; the written or read data comes from IO device B, and the destination address of the written or read data is IO device A; S23, IO device A initiates a request to write or read data to IO device B by updating the descriptor information in the submission queue; S24, IO device B completes writing or reading data through the DMA controller, and writes a completion status in the completion queue after completion; S25. IO device A checks the status in the completion queue to confirm that data writing or reading is completed, thereby completing the P2P communication.

7. The system according to claim 6, characterized in that In said S23, the information contained in the descriptor includes at least the operation to be completed, the transmission length, the source address and the destination address; The updating also includes updating a tail pointer of the submission queue.

8. The system according to claim 2, characterized in that When performing P2P communication, the remote computing node adds the remote available IO devices and device resources to the remote device management module of the remote computing node based on the user configuration, and then notifies the remote available IO devices and device resources to the computing node communication module in the local computing node through the remote computing node communication module, and then the computing node communication module sends it to the local remote device management module; The information of available IO devices and device resources transmitted between the computing node communication modules is performed through the NTB; The available IO devices are based on user configuration settings.

9. The system according to claim 8, characterized in that In the user-based configuration setting, the configuration work includes queue pair application, queue status synchronization, communication data source address, communication data destination address and NTB conversion window configuration.

10. A P2P communication method between IO devices based on PCIe-NTB, characterized in that: The method is applied to the system according to any one of claims 1 to 9, and the method comprises: S1. After each computing node is started, a communication connection is established through the NTB of each computing node; S2. Obtain available IO devices and device resources of the remote computing node; S3. Configure the IO device for P2P communication and complete the configuration of the NTB address translation window; S4. Determine the type of IO device for P2P communication; if it is P2P communication between IO device type 1, initialize the IO shared memory and communicate through the DMA controller; if it is P2P communication between IO device type 1 and IO device type 2, complete the queue pair initialization and IO shared memory initialization, and communicate based on the queue pair mode.

Images