Inter-node communication method and apparatus, electronic device, and storage medium

Direct communication between nodes is realized in the cluster management tool through the RDMA module, which solves the inefficiency problem caused by CPU participation in communication, and improves communication efficiency and GPU utilization.

WO2024244557A1PCT designated stage expired Publication Date: 2024-12-05INSPUR SUZHOU INTELLIGENT TECH CO LTD

Patent Information

Application Number
PCT/CN2024/078825
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-01
Filing Date
2024-02-27
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

When existing cluster management tools handle complex tasks, the CPU needs to participate in network communication between multiple virtual graphics processing units, resulting in long-term communication and poor communication processing capabilities, and serious waste of GPU computing power.

Method used

Communication between nodes is realized through the remote direct memory access (RDMA) module, avoiding CPU participation, and directly communicating between nodes of general computing graphics processing unit. Using field programmable gate array virtualization nodes, generate application descriptors for memory reading and writing, and realize Zero copy communication.

Benefits of technology

It improves communication efficiency between nodes, avoids the problem of long-term use of CPU, improves communication processing capabilities, and saves server resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024078825_05122024_PF_FP_ABST
    Figure CN2024078825_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present application provides an inter-node communication method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring a list of a field-programmable gate array; virtualizing the field-programmable gate array according to the list to obtain a first source node and a first target node; acquiring an application descriptor between the first source node and the first target node, wherein the application descriptor comprises a memory address of the first source node and a memory address of the first target node; acquiring identity information of the first source node according to the memory address of the first source node, and acquiring identity information of the first target node according to the memory address of the first target node; and executing a communication action on the first source node and the first target node by means of a remote direct memory access (RDMA) module according to the identity information of the first source node and the identity information of the first target node. The inter-node communication method provided by the present application depends on the RDMA module to execute the communication action between the first source node and the first target node, and a CPU does not need to participate, thereby improving the inter-node communication efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Inter-node communication method, device, electronic device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on June 1, 2023, with application number 202310640711.4 and application name “Inter-node communication method, device, electronic device and storage medium”, all contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, and non-volatile readable storage medium for inter-node communication. Background Art

[0004] The difficulty of cluster management has increased exponentially with the emergence of large and ultra-large clusters. More and more users are turning to cluster management tools for cluster management. Existing cluster management tools often utilize container technology for deployment. However, with this deployment approach, a single graphics processing unit (GPU) can only be occupied by one container, wasting the GPU's computing power.

[0005] To address this issue, existing technologies often use virtualization to share GPU resources between multiple containers. However, multiple virtual graphics processing unit (VGPU) containers also require multiple networks for communication, and these network communications require the participation of the central processing unit (CPU). When processing complex tasks, the CPU is heavily occupied by communication between different containers, resulting in poor communication processing capabilities.

[0006] Summary of the Invention

[0007] The present application provides an inter-node communication method, device, electronic device and non-volatile readable storage medium. In the inter-node communication method provided by the present application, the inter-node communication method determines the identity information of the first source node and the identity information of the first target node by carrying the memory information of the first source node and the memory information of the first target node, and performs communication actions on the first source node and the first target node through a remote direct memory access (RDMA) module according to the identity information of the first source node and the identity information of the first target node. Since the inter-node communication method provided by the present application relies on the RDMA module to execute the communication action between the first source node and the first target node, the CPU does not need to be involved, which avoids the problem of the CPU being occupied by inter-node communication for a long time and improves the efficiency of inter-node communication.

[0008] In a first aspect, the present application provides a method for inter-node communication, the method comprising:

[0009] Get a list of FPGAs;

[0010] virtualizing the field programmable gate array according to the list of field programmable gate arrays to obtain a first source node and a first target node, wherein the first source node and the first target node are general computing graphics processing unit nodes;

[0011] Obtaining an application descriptor between a first source node and a first target node, wherein the application descriptor includes a memory address of the first source node and a memory address of the first target node;

[0012] Obtaining identity information of the first source node according to the memory address of the first source node, and obtaining identity information of the first target node according to the memory address of the first target node;

[0013] According to the identity information of the first source node and the identity information of the first target node, a communication action is performed on the first source node and the first target node through a remote direct memory access module.

[0014] In some embodiments, the method provided herein further comprises:

[0015] Assigning identity information to a plurality of nodes, wherein the plurality of nodes includes a first source node and a first target node;

[0016] The memory addresses of multiple nodes are edited uniformly according to the identity information allocation order of the multiple nodes.

[0017] In some embodiments, the method provided herein further comprises:

[0018] Perform cluster dynamic allocation actions on multiple nodes and obtain corresponding identity information.

[0019] In some embodiments, the method provided herein further comprises:

[0020] Constructing an application description table of multiple nodes, wherein the application description table carries application descriptors between the multiple nodes;

[0021] The application descriptor between the first source node and the first target node is obtained from the application description table according to the first-in-first-out principle.

[0022] In some embodiments, the method provided herein further comprises:

[0023] Performing address resolution on the memory address of the first source node to obtain the local address of the first source node and the identity information of the first source node;

[0024] Perform address resolution on the memory address of the first target node to obtain the local address of the first target node and the identity information of the first target node.

[0025] In some embodiments, the method provided herein further comprises:

[0026] Generate mappings and de-mappings between memory addresses of multiple nodes and corresponding local addresses.

[0027] In some embodiments, the method provided herein further comprises:

[0028] Acquire Internet Protocol address information of the first source node according to the identity information of the first source node;

[0029] Acquire Internet Protocol address information of the first target node according to the identity information of the first target node;

[0030] According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node, a communication action is performed on the first source node and the first target node through a protocol preset in the remote direct memory access module.

[0031] In some embodiments, the method provided herein further comprises:

[0032] Generate a mapping between multiple node identity information and corresponding Internet Protocol address information.

[0033] In some embodiments, the method provided herein further comprises:

[0034] According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node, a zero copy operation is performed on the first source node and the first target node through a protocol preset in the remote direct memory access module.

[0035] In some embodiments, the method provided herein further comprises:

[0036] According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node, memory read and write operations are performed on the first source node and the first target node through the transmission control protocol preset in the remote direct memory access module.

[0037] In some embodiments, the method provided herein further comprises:

[0038] According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node, memory read and write actions are performed on the first source node and the first target node through the user datagram protocol preset in the remote direct memory access module.

[0039] In some embodiments, the method provided herein further comprises:

[0040] calling a communication mode preset in the remote direct memory access module according to the copy type;

[0041] According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first destination node, a communication action is performed on the first source node and the first destination node through a communication mode.

[0042] In some embodiments, the method provided herein further comprises:

[0043] According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node, a communication action is performed on the first source node and the first target node through a remote communication mode.

[0044] In some embodiments, the method provided herein further comprises:

[0045] According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node, a communication action is performed on the first source node and the first target node through a local communication mode.

[0046] In some embodiments, the method provided herein further comprises:

[0047] A read synchronization process is performed on the first source node and the first target node.

[0048] In some embodiments, the method provided herein further comprises:

[0049] A write synchronization process is performed on the first source node and the first target node.

[0050] In some embodiments, the method provided herein further comprises:

[0051] Generate a communication completion signal according to the result of the read synchronization process and the result of the write synchronization process;

[0052] After the communication completion signal is obtained, an application descriptor between the second source node and the second destination node is obtained.

[0053] In a second aspect, the present application further provides a device for use in a Kubernetes (an open source containerized application for managing multiple hosts in a cloud platform) cluster management system, comprising:

[0054] A field programmable gate array list acquisition module, used to acquire a list of field programmable gate arrays;

[0055] a node virtualization module, configured to virtualize the field programmable gate array according to the field programmable gate array list to obtain a first source node and a first target node;

[0056] a first application description acquisition module, configured to acquire an application descriptor between a first source node and a first target node, wherein the application descriptor includes a memory address of the first source node and a memory address of the first target node, and the first source node and the first target node are general-purpose computing graphics processing unit nodes;

[0057] An identity information acquisition module, configured to acquire the identity information of the first source node according to the memory address of the first source node, and acquire the identity information of the first target node according to the memory address of the first target node;

[0058] The communication action module is used to perform a communication action on the first source node and the first target node through the remote direct memory access module according to the identity information of the first source node and the identity information of the first target node.

[0059] In a third aspect, the present application also provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the inter-node communication method of the first aspect are implemented.

[0060] In a fourth aspect, an embodiment of the present application provides a non-volatile readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the inter-node communication method of the first aspect are implemented.

[0061] In the inter-node communication method provided in this application, the inter-node communication method determines the identity information of the first source node and the identity information of the first target node by carrying the memory information of the first source node and the memory information of the first target node, and performs a communication action on the first source node and the first target node through the RDMA module based on the identity information of the first source node and the identity information of the first target node. Because the inter-node communication method provided in this application relies on the RDMA module to perform the communication action between the first source node and the first target node, it does not require CPU participation, thus avoiding the problem of the CPU being occupied for a long time by inter-node communication and improving the efficiency of inter-node communication.

[0062] The above description is only an overview of the technical solution provided by this application. In order to more clearly understand the technical means of this application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of this application more obvious and easy to understand, the specific implementation methods of this application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0064] FIG1 is a schematic diagram of a method for inter-node communication provided by some embodiments of the present application;

[0065] FIG2 is an example of a hardware framework of an inter-node communication system provided by the present application;

[0066] FIG3 is a second schematic diagram of an inter-node communication method provided in some embodiments of the present application;

[0067] FIG4 is a third schematic diagram of an inter-node communication method provided in some embodiments of the present application;

[0068] FIG5 is a fourth schematic diagram of an inter-node communication method provided by some embodiments of the present application;

[0069] FIG6 is a fifth schematic diagram of an inter-node communication method provided in some embodiments of the present application;

[0070] FIG7 is a sixth schematic diagram of an inter-node communication method provided in some embodiments of the present application;

[0071] FIG8 is a seventh schematic diagram of an inter-node communication method provided by some embodiments of the present application;

[0072] FIG9 is an eighth schematic diagram of an inter-node communication method provided by some embodiments of the present application;

[0073] FIG10 is a ninth schematic diagram of an inter-node communication method provided in some embodiments of the present application;

[0074] FIG11 is a tenth schematic diagram of an inter-node communication method provided by some embodiments of the present application;

[0075] FIG12 is a schematic diagram of an inter-node communication method according to some embodiments of the present application;

[0076] FIG13 is a twelfth schematic diagram of an inter-node communication method provided by some embodiments of the present application;

[0077] FIG14 is a thirteenth schematic diagram of an inter-node communication method provided in some embodiments of the present application;

[0078] FIG15 is a fourteenth schematic diagram of an inter-node communication method provided by some embodiments of the present application;

[0079] FIG16 is a fifteenth schematic diagram of an inter-node communication method provided by some embodiments of the present application;

[0080] FIG17 is a sixteenth schematic diagram of an inter-node communication method provided in some embodiments of the present application;

[0081] FIG18 is a seventeenth schematic diagram of an inter-node communication method provided by some embodiments of the present application;

[0082] FIG19 is a schematic diagram of a distributed cluster construction method for inter-node communication provided by some embodiments of the present application;

[0083] FIG20 is a second schematic diagram of a distributed cluster establishment method for inter-node communication provided by some embodiments of the present application;

[0084] FIG21 is a schematic diagram of an inter-node communication device provided in some embodiments of the present application;

[0085] FIG22 is a schematic structural diagram of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0086] The following describes exemplary embodiments of the present application in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0087] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that some embodiments of the present application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0088] The inter-node communication method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0089] Some embodiments of the present application relate to an inter-node communication method, which is applied to a Kubernetes cluster management system, as shown in FIG1 , and includes:

[0090] Step 113: Obtain a list of field programmable gate arrays;

[0091] Step 114: virtualize the field programmable gate array according to the field programmable gate array list to obtain a first source node and a first target node, wherein the first source node and the first target node are general computing graphics processing unit nodes;

[0092] Step 101: Acquire an application descriptor between a first source node and a first target node, wherein the application descriptor includes a memory address of the first source node and a memory address of the first target node;

[0093] Step 102: Acquire identity information of the first source node according to the memory address of the first source node, and acquire identity information of the first target node according to the memory address of the first target node;

[0094] Step 103: Perform a communication action on the first source node and the first target node through a remote direct memory access module according to the identity information of the first source node and the identity information of the first target node.

[0095] Specifically, the inter-node communication method provided in this application requires building a Kubernetes management system in a Field-Programmable Gate Array (FPGA) hardware cluster. First, a list of FPGAs must be obtained. For example, the host nodes and FPGA devices in the cluster network can be scanned to generate a table of host and network GPU allocation information. Virtual network nodes are then created based on the FPGA list. When a user initiates a VGPU resource request, a new VGPU node is created based on the VGPU resources and the required number of VGPUs in the resource request.

[0096] Subsequently, an application descriptor between the first source node and the first target node is selected from a descriptor table storing application descriptors, wherein the application descriptor between the first source node and the first target node includes memory address information of the first source node and memory address information of the first target node. Subsequently, the memory address of the first target node and the memory address of the first target node in the application descriptor are respectively resolved to obtain the identity information of the first source node and the identity information of the first target node corresponding to the memory addresses. A communication action is performed on the first source node and the first target node based on the identity information of the first source node and the identity information of the first target node, thereby realizing information exchange from the first source node to the first target node.

[0097] It should be emphasized that when traditional cluster management systems, such as Kubernetes systems, apply RDMA devices, they often use a device plugin (a mechanism in Kubernetes for device resource management) plug-in mechanism to facilitate the Kubernetes system to identify and use RDMA devices. After identifying the RDMA device, the Kubernetes CNI (CNI, Container Network Interface, a specification and interface for container networks, which defines a set of standard APIs and plug-ins for creating, configuring and managing networks when containers are running) plug-in is used to reuse a single RDMA network card to build a complete container network. The first source node and the first target node involved in this application are both general purpose computing graphics processing unit nodes (General Purpose Computing on GPU, GPGPU). Compared with the existing form of inserting GPUs and RDMA network cards on servers, the GPGPU provided by this application is independent of the server. With this design, this application does not need to use the existing device plugin plug-in mechanism to identify RDMA devices, avoids the participation of the CPU, and saves a lot of server resources. On this basis, the RDMA communication mechanism based on GPGPU nodes provided by this application, such as the RDMA communication mechanism based on GPGPU unified addressing, realizes RDMA high-speed communication that supports network GPGPU virtualization.

[0098] Based on some implementation methods, the present application also provides an embodiment of a communication system that applies the inter-node communication method of the present application.

[0099] The present application provides an inter-node communication system based on a Field-Programmable Gate Array (FPGA). The hardware framework of the system is shown in Figure 2, including: an RDMA module, a Direct Memory Access (DMA) engine module, and a node module. The DMA engine module includes an address resolution module for obtaining identity information based on a memory address.

[0100] The inter-node communication method provided in this application is applied in a distributed cluster to realize the node communication function between multiple containers in the distributed cluster. Before inter-node communication is performed, the node upper layer application of the container calls the node communication request and keeps the application layer in a blocked state.

[0101] When a node has multiple GPUs, this application also provides an example of a node communication request: the VGPU upper-layer application in the container calls the GPU_remote_memcopy remote communication interface, which triggers the communication action of the RDMA module. For example, the GPU_remote_memcopy function is declared as follows:

[0102] / *param 1: source GPU memory address, unified memory address

[0103] Param 2: destination GPU memory address, unified memory address

[0104] param 3: copy length* /

[0105] int GPU_remote_memcopy(void*src, void*dst, u32t length).

[0106] The DMA engine module is used to select an application descriptor between a first source node and a first target node from a descriptor table storing application descriptors, wherein the application descriptor between the first source node and the first target node includes memory address information of the first source node and memory address information of the first target node. Subsequently, the address resolution module in the DMA engine module performs address resolution on the memory address of the first target node and the memory address of the first target node in the application descriptor, respectively, to obtain the identity information of the first source node and the identity information of the first target node corresponding to the memory address. The RDMA module performs communication actions on the first source node and the first target node based on the identity information of the first source node and the identity information of the first target node, thereby realizing information exchange from the first source node to the first target node.

[0107] After the communication action between the first source node and the first target node is completed, the application layer ends the blocking state and performs subsequent processing based on the data obtained by the first target node.

[0108] In the inter-node communication method provided in this application, the inter-node communication method determines the identity information of the first source node and the identity information of the first target node by carrying the memory information of the first source node and the memory information of the first target node, and performs a communication action on the first source node and the first target node through the RDMA module based on the identity information of the first source node and the identity information of the first target node. Because the inter-node communication method provided in this application relies on the RDMA module to perform the communication action between the first source node and the first target node, it does not require CPU participation, thus avoiding the problem of the CPU being occupied for a long time by inter-node communication and improving the efficiency of inter-node communication.

[0109] In addition, in the inter-node communication method provided by the present application, data transmission between containers only requires one data copy, which effectively improves the communication efficiency of inter-node communication.

[0110] Based on some implementations, as shown in FIG3 , the inter-node communication method provided by the present application further includes, before step 101:

[0111] Step 104: assign identity information to a plurality of nodes, wherein the plurality of nodes include a first source node and a first target node;

[0112] Step 105: Edit the memory addresses of the multiple nodes in a unified manner according to the identity information allocation order of the multiple nodes.

[0113] Specifically, the inter-node communication method provided in this application can be applied to, but not limited to, a global memory management based on a node cluster. Before communicating between multiple nodes, the inter-node communication method provided in this application also edits the memory addresses of these nodes. First, identity information (ID) is assigned to these nodes in the cluster, and a unique physical ID is assigned to each node. After the physical IDs of all nodes are assigned, the memory addresses of these nodes are edited in a unified manner according to the ID order. The editing example is shown in Table 1:

[0114] Table 1 Node and physical memory addressing information table

[0115] Based on some embodiments, since the inter-node communication method provided by the present application pre-assigns identity information to multiple nodes, when executing a communication action between a first source node and a first target node, the user does not need to care about the first source node and the first target node, such as the topology structure and ID information of the GPGPU, but only needs to consider the address of the first source node and the address of the first target node of the memory copy, thereby improving the transparency of remote memory operations and facilitating users to monitor the execution of node communication actions.

[0116] In some embodiments, as shown in FIG4 , multiple nodes are located in a cluster, and step 104 includes:

[0117] Step 141: Perform cluster dynamic allocation actions on multiple nodes to obtain corresponding identity information.

[0118] Specifically, the multiple nodes provided in this application are located in a cluster, and the physical IDs of the multiple nodes are obtained by dynamic allocation of the cluster, and the physical IDs change as the cluster expands and shrinks.

[0119] In some implementations, since the identities of multiple nodes in the inter-node communication method provided herein are dynamically assigned by the cluster, users do not need to monitor the specific nodes that are communicating and interacting; they can determine the specific nodes based on the same real-time memory address. This allows for transparent operation and conversion without modifying the source code.

[0120] Based on some implementations, as shown in FIG5 , in the inter-node communication method provided by this application, step 101 includes:

[0121] Step 111: construct an application description table for multiple nodes, wherein the application description table carries application descriptors between multiple nodes;

[0122] Step 112: Obtain the application descriptor between the first source node and the first target node from the application description table according to the first-in-first-out principle.

[0123] Specifically, in the inter-node communication method provided by this application, the RDMA module is shared by nodes in multiple containers, and multiple inter-node communications need to be completed. Therefore, the RDMA module needs to be scheduled, for example, using a queue method to queue applications, and the inter-node communication actions requested first are executed first.

[0124] Therefore, a request descriptor table is first constructed to meet the First Input First Output (FIFO) principle. Each entry in the request descriptor table is an request descriptor between multiple nodes. Each request descriptor carries the memory address of the source node, the memory address of the target node, and the copy length. When the VGPU upper-layer application in the container calls the GPU_remote_memcopy remote communication interface, the earliest request descriptor in the request descriptor table is retrieved according to the first-in-first-out principle. The source node and target node of this request descriptor are respectively designated as the first source node and the first target node.

[0125] Based on some implementation methods, since the inter-node communication method provided by the present application constructs an application description table based on the first-in-first-out principle for multiple application descriptors, when an application descriptor is obtained again, the newly arrived application descriptor is placed at the end of the queue in the application description table, ensuring that the communication action applied for first is executed first, avoiding the situation where RDMA processes multiple different communication actions at the same time, and improving communication efficiency.

[0126] Based on some implementations, as shown in FIG6 , in the inter-node communication method provided by this application, step 102 includes:

[0127] Step 121: Perform address resolution on the memory address of the first source node to obtain the local address of the first source node and the identity information of the first source node;

[0128] Step 122: Perform address resolution on the memory address of the first target node to obtain the local address of the first target node and the identity information of the first target node.

[0129] Specifically, the DMA engine module performs address resolution on the memory addresses of the first source node and the first target node, and obtains the local addresses and identity information of the first source node and the first target node, respectively, for the RDMA module to determine the identity information address and hardware address of the first source node and the first target node based on the identity information of the first source node and the identity information of the first target node.

[0130] Based on some implementation methods, since the inter-node communication method provided by the present application can obtain the identity information of the first source node and the identity information of the first target node based on the memory address resolution, it is convenient for the RDMA module to obtain the IP address information of the first source node and the first target node based on the identity information of the first source node and the first target node, and assign the address resolution action to a DMA engine module outside the RDMA module for implementation, thereby reducing the workload of the RDMA module, improving the working efficiency of the RDMA module, and thereby improving the communication efficiency of the inter-node communication method.

[0131] Based on some implementations, as shown in FIG7 , the inter-node communication method provided by the present application further includes, before step 121:

[0132] Step 123: Generate mappings and de-mappings between memory addresses of multiple nodes and corresponding local addresses.

[0133] Specifically, the address resolution module in the DMA engine module pre-stores the mapping and inverse mapping of the memory addresses and local addresses of multiple nodes in the cluster. When the address resolution module in the DMA engine module performs address resolution on the first source node and the first target node, it can convert the local address and the memory address, such as the unified memory address, based on the description information in the application descriptor as to whether local communication or remote communication is required, to meet the communication requirements in the application descriptor. In addition, an application program interface (API) for converting the local address and the same memory address can be provided to the upper-level application for the user to perform conversion according to needs.

[0134] Based on some implementation methods, since the inter-node communication method provided in this application pre-sets a mapping between the memory addresses of multiple nodes and the corresponding local addresses, it can quickly obtain the corresponding local address according to the memory address according to user needs, thereby meeting the different needs of users for remote communication or local communication.

[0135] Based on some implementations, as shown in FIG8 , in the inter-node communication method provided by this application, step 103 includes:

[0136] Step 131: Obtain Internet Protocol address information of the first source node according to the identity information of the first source node;

[0137] Step 132: Obtain Internet Protocol address information of the first target node according to the identity information of the first target node;

[0138] Step 133: Perform communication on the first source node and the first target node through a protocol preset in the remote direct memory access module according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0139] Specifically, after the address resolution module in the DMA engine module obtains the identity information of the first source node and the first destination node based on the memory address, the RDMA module obtains the Internet Protocol (IP) address and hardware address information of the first source node based on the identity information of the first source node, and obtains the IP address and hardware address information of the first destination node based on the identity information of the first destination node. The RDMA module then reads the memory of the remote node using a pre-set protocol, thereby achieving communication with the first source node and the first destination node.

[0140] Based on some implementation methods, since the inter-node communication method provided in this application can execute communication actions according to the IP address information of the first source node and the first target node through the protocol pre-set in the RDMA module, it ensures that RDMA can successfully complete communication actions in different nodes and eliminates the CPU's restrictions on inter-node communication.

[0141] Based on some implementations, as shown in FIG9 , the inter-node communication method provided by the present application, before step 131, further includes:

[0142] Step 134: Generate a mapping between multiple node identity information and corresponding Internet Protocol address information.

[0143] Specifically, the RDMA module pre-stores a mapping between the identity information of multiple nodes in the cluster and Internet protocol information, such as an IP-ID table. When communication is performed between the first source node and the first target node, the RDMA module can query the IP-ID table to obtain the IP address information and hardware address information of the first source node and the first target node.

[0144] Based on some embodiments, since the inter-node communication method provided by this application pre-sets a mapping between the identity information of multiple nodes and the corresponding IP address information, when inter-node communication is performed, the RDMA module can quickly obtain the Internet protocol information corresponding to the identity information according to the pre-set mapping, thereby improving the communication efficiency of the inter-node communication method.

[0145] In some embodiments, as shown in FIG10 , the remote direct memory access module shares memory with a graphics processing unit (GPU) corresponding to a general-purpose computing graphics processing unit node. In the inter-node communication method provided herein, step 133 includes:

[0146] Step 135: Perform a zero-copy operation on the first source node and the first target node through a protocol preset in the remote direct memory access module according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0147] Specifically, in some implementations, the RDMA module and the GPU share memory. After the RDMA module obtains the IP address information of the first source node and the IP address information of the first target node, it can directly access the GPU memory corresponding to the first source node and the first target node to achieve zero copy.

[0148] Based on some implementations, since the RDMA module and the GPU share memory in the inter-node communication method provided by this application, the RDMA module can directly access the GPU memory, achieving zero copy, shortening the information interaction steps, and improving the efficiency of inter-node communication.

[0149] Based on some embodiments, as shown in FIG11 , the pre-set protocol includes a transmission control protocol. In the inter-node communication method provided in this application, step 135 includes:

[0150] Step 136: Execute memory read and write operations on the first source node and the first target node through the transmission control protocol preset in the remote direct memory access module according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0151] Specifically, the pre-set protocol in the RDMA module includes the Transmission Control Protocol (TCP). After the RDMA module obtains the IP address information of the first source node and the first destination node, it reads and writes the GPU memory corresponding to the first source node and the first destination node through the TCP protocol, thereby implementing memory read and write operations between the first source node and the first destination node.

[0152] Based on some implementation methods, since the protocol provided in this application includes the TCP protocol, an example of a communication method that can specifically implement this application is provided, thereby improving the versatility of the inter-node communication method provided in this application.

[0153] Based on some implementations, as shown in FIG12 , the pre-set protocol includes the User Datagram Protocol. In the inter-node communication method provided in this application, step 135 further includes:

[0154] Step 137: Execute memory read and write operations on the first source node and the first target node through the user datagram protocol preset in the remote direct memory access module according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0155] Specifically, the protocol pre-set in the RDMA module includes the Transmission Control Protocol (User Datagram Protocol, UDP). After the RDMA module obtains the IP address information of the first source node and the first target node, it reads and writes the GPU memory corresponding to the first source node and the first target node through the UDP protocol, thereby realizing the memory read and write operations between the first source node and the first target node.

[0156] Based on some implementation methods, since the protocol provided in this application also includes the UDP protocol, an example of a specific communication method that can implement this application is provided, and multiple different protocols can be applied, further improving the versatility of the inter-node communication method provided in this application.

[0157] Based on some implementations, as shown in FIG13 , the application descriptor further includes a copy type. In the inter-node communication method provided by this application, step 133 further includes:

[0158] Step 138: Call the communication mode preset in the remote direct memory access module according to the copy type;

[0159] Step 139: Perform a communication action on the first source node and the first target node through a communication mode according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0160] Specifically, the application descriptor in the inter-node communication method provided by this application carries not only the memory address of the first source node, the memory address of the first target node, and the copy length, but also the copy type information. The inter-node communication method provided by this application can invoke the corresponding communication mode pre-set in the RDMA module based on the copy type and execute the corresponding inter-node communication action.

[0161] Based on some implementation methods, since the application descriptor in the inter-node communication method provided by this application also carries a copy type, the RDMA module is pre-set with a corresponding communication mode, which can start different communication modes according to user needs and communication needs, thereby expanding the scope of use of this application.

[0162] Based on some implementations, as shown in FIG14 , the copy type includes remote copy, and the communication mode includes remote communication mode. In the inter-node communication method provided in this application, step 139 includes:

[0163] Step 140: Perform communication actions on the first source node and the first target node through a remote communication mode according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0164] Specifically, copy types include, but are not limited to, remote copy. A remote communication mode is pre-configured in the RDMA module. When the copy type in the application descriptor is remote copy, the memory address of the first source node and the memory address of the first target node in the application descriptor are unified, and the remote communication mode of the RDMA module is invoked for communication.

[0165] Based on some implementation methods, since the RDMA module provided by the present application is provided with a remote communication mode, the inter-node communication method provided by the present application can realize remote communication between different nodes, thereby expanding the scope of application of the present application.

[0166] Based on some implementations, as shown in FIG15 , the copy type further includes a local copy, and the communication mode further includes a local communication mode. In the inter-node communication method provided in this application, step 139 further includes:

[0167] Step 142: Perform communication actions on the first source node and the first target node through a local communication mode according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0168] Specifically, copy types include, but are not limited to, local copies. The RDMA module is pre-configured with a local communication mode. When the copy type in the application descriptor is local copy, the memory address of the first source node and the memory address of the first destination node in the application descriptor are local memory addresses, and the RDMA module's local communication mode is invoked for communication.

[0169] It should be emphasized that the inter-node communication method provided by the present application can only perform remote communication according to the copy type carried by the application descriptor, can only perform local communication according to the copy type carried by the application descriptor, or can perform remote communication and local communication according to the copy type carried by the application descriptor, wherein the order of remote communication and local communication is executed in sequence according to the order in which the application descriptors are obtained, and this application does not limit it.

[0170] Based on some implementation methods, since the RDMA module provided by this application is provided with a local communication mode, the inter-node communication method provided by this application can realize local communication between different nodes, further expanding the scope of application of this application.

[0171] Based on some implementations, as shown in FIG16 , the communication action includes a read action. In the inter-node communication method provided in this application, after step 103, the following steps are further included:

[0172] Step 107: Perform read synchronization processing on the first source node and the first target node.

[0173] Specifically, in the inter-node communication method provided herein, the communication action includes a read action. The RDMA module and GPU shared memory directly access the GPU memory to complete the read communication action between the corresponding GPUs of the first source node and the first target node. Subsequently, a communication completion signal is sent to the first source node, thereby achieving a synchronous processing effect for the read communication.

[0174] Based on some implementations, since the inter-node communication method provided by this application can perform read synchronization processing after completing the communication between the first source node and the first target node, the node reading communication results are saved, thereby enhancing the usability of the inter-node communication method provided by this application.

[0175] Based on some implementations, as shown in FIG17 , the communication action further includes a write action. In the inter-node communication method provided in this application, after step 107, the following steps are further included:

[0176] Step 108: Perform write synchronization processing on the first source node and the first target node.

[0177] Specifically, in the inter-node communication method provided herein, the communication action includes a write action. The RDMA module completes the write communication action between the corresponding GPUs of the first source node and the first destination node by directly accessing GPU memory. Subsequently, the first source node is controlled to send a communication completion signal to the first destination node, thereby achieving a synchronous processing effect for the write communication.

[0178] Based on some implementations, since the inter-node communication method provided by this application can perform write synchronization processing after completing the communication between the first source node and the first target node, the node write communication results are saved, thereby enhancing the usability of the inter-node communication method provided by this application.

[0179] Based on some implementations, as shown in FIG18 , the inter-node communication method provided by the present application further includes, after step 108:

[0180] Step 109: Generate a communication completion signal according to the result of the read synchronization process and the result of the write synchronization process;

[0181] Step 110: After obtaining the communication completion signal, obtain the application descriptor between the second source node and the second target node.

[0182] Specifically, after the read communication action is completed, the first source node writes 1 to a synchronization register pre-set on the first source node after receiving a communication completion signal, and sends a transmission completion request to the driver of the inter-node communication system to interrupt the application layer blocking state. After the write communication action is completed, the first destination node writes 1 to a synchronization register pre-set on the first destination node after receiving a communication completion signal, and sends a transmission completion request to the driver of the inter-node communication system to interrupt the application layer blocking state. After receiving the transmission completion request sent by the first source node or the first destination node, the driver of the inter-node communication system writes 0 to the synchronization register of the node corresponding to the transmission completion request and sends an instruction to the DMA engine module to obtain the application descriptor between the second source node and the second destination node and execute the communication action between the second source node and the second destination node.

[0183] Based on some implementations, the inter-node communication method provided by this application can continue to execute the communication action between the second source node and the second target node based on the communication completion signal, ensuring that the RDMA module can execute communication actions in sequence for communication requests between multiple different nodes, providing container-based one-to-many communication capabilities. Specifically, after executing the communication action between the first source node and the first target node based on the first application descriptor, a communication completion signal is generated, and then the second application descriptor is obtained from the application descriptor table constructed according to the first-in-first-out principle, and then a second communication action is performed. This ensures that multiple different application descriptors used for communication are executed in sequence and do not interfere with each other, thereby improving the stability of the inter-node communication method.

[0184] Based on some implementation methods, the present application also provides a complete set of inter-node communication methods.

[0185] First, the host calls the communication request of the first source node and the first target node, and the system generates an application descriptor. The newly generated application descriptor is filled in the last row of the application description table and enters the queue to wait for the DMA engine module to call. After the DMA engine module takes this application descriptor from the queue, the address resolution module in the DMA engine module converts the unified memory address of the first source node and the first target node into the ID information of the first source node and the first target node based on the application descriptor. The RDMA module obtains the IP address information and hardware address information by querying the pre-set ID-IP table, and reads and writes the remote node memory through the TCP protocol or UDP protocol. For read communication actions, when the RDMA module completes the communication, it sends a completion signal to the first source node to complete the synchronization operation; for write communication actions, when the RDMA module completes the communication, it controls the first source node to send a completion signal to the first target node to complete the synchronization operation. When the first source node or the first target node receives the completion signal, it writes 1 to the synchronization register corresponding to the node and sends a transmission completion signal to the system driver. After the driver receives the transmission completion signal, it writes 0 to the synchronization register corresponding to the node and sends a signal to the DMA engine module. After receiving the signal, the DMA engine module calls the application descriptor between the second source node and the second target node in the application description table to continue the inter-node communication operation.

[0186] Based on the above implementation, the present application also provides a specific example of building a distributed cluster that can implement the inter-node communication method of the present application, as shown in Figures 19-20:

[0187] As shown in Figure 19, using FPGA hardware as an example, multiple open-source instruction set architectures based on reduced instruction set principles, such as RISC-V (an open-source instruction set architecture based on reduced instruction set principles), are built within the FPGA. A parallel scheduling module is added to expand the many-core GPU. Unlike existing CPU-FPGA host binding methods, the FPGA device used in this application is a network computing node that does not rely on a CPU, and a custom RDMA protocol module is added to the many-core RISC-V.

[0188] Based on the aforementioned FPGA hardware cluster, a management system based on open-source application container engines, such as Docker (an open-source application container engine), and Kubernetes was built. This management system includes the Kube Admin module, the VGPU Daemon module, and the VGPU plugin module.

[0189] Kube Admin is a monitoring and management module for GPU clusters. It is deployed on the cluster control (master) node and is used to query and schedule GPU resource lists.

[0190] VGPU Deamon is a GPU resource virtualization module deployed on each worker node. It receives the FPGA list sent by KubeAdmin and virtualizes the FPGAs in the network into GPU local device nodes based on the list.

[0191] The VGPU plugin is a Kubernetes-based device plugin that is deployed on all Kubernetes worker nodes in the form of deamonsets to report the GPU virtual node status of the node.

[0192] Kube Admin is also used to manage the FPGA board information in the cluster: including the FPGA ID, FPGA IP address, FPGA MAC address (MAC address, Media Access Control Address, Media Access Control Address), the host information where the FPGA virtualized GPU is located, VGPU pod (pod, the most basic deployment and management unit in the Kubernetes cluster) information, etc. VGPU Deamon is also used to create virtual GPU nodes and is responsible for creating new VGPU nodes based on user needs. It includes drivers and user-mode programs and runs on all worker nodes. VGPU Plugin runs on the worker node and communicates with Kubelet (an agent component on the Kubernetes worker node) through the GRPC service. It is responsible for informing the status of the VGPU Node (network connection endpoint) in the pod. The communication process between VGPU Plugin and Kubelet is shown in Figure 20.

[0193] The management system based on Docker and Kubernetes can be implemented in the following ways, but not limited to:

[0194] The first step is to build a network GPU cluster: FPGA Admin scans the host nodes and FPGA devices in the cluster network and generates an allocation information table for worker hosts and network GPUs.

[0195] The next step is to deploy the VGPU deamon: it sends a request to the VGPU Admin to obtain the FPGA list, and then completes the creation of the virtual network GPU node based on the FPGA list.

[0196] The VGPU plugin is then deployed through deamonset: the worker host where the network GPU node is located registers the GRPC service with the kubelet, reports the network GPU device list, and the kubelet feeds back to the API server.

[0197] When a user initiates a VGPU resource request, they declare the VGPU resources and the desired quantity in the limits field of the Kubernetes configuration file. The minimum unit of a VGPU is a single physical core of RISC-V. The corresponding VGPU memory (Vmemory) must also be requested, with the minimum unit of memory requested being GPU memory / RISC-V cores. The Kubernetes scheduler module traverses all target nodes and selects candidate worker nodes that meet the requirements. The target node's kubelet selects a GPU from its local GPU list that meets the requested number of cores and the corresponding memory. The kubelet then initiates an Allocate request to the local VGPU plugin, specifying the VGPU node to be assigned to the container. The VGPU plugin receives the Allocate Request and returns an Allocate Response to the kubelet. Based on the Allocate Response, the kubelet creates and initializes a container containing the specified VGPU resources. Once the distributed cluster is established, it can be used in the inter-node communication method provided in this application.

[0198] Some embodiments of the present application relate to an inter-node communication device, which is applied to a Kubernetes cluster management system, as shown in FIG21 , including:

[0199] A field programmable gate array list acquisition module 213, configured to acquire a list of field programmable gate arrays;

[0200] A node virtualization module 214 is configured to virtualize the field programmable gate array according to the field programmable gate array list to obtain a first source node and a first target node;

[0201] A first application description acquisition module 201 is configured to acquire an application descriptor between a first source node and a first target node, wherein the application descriptor includes a memory address of the first source node and a memory address of the first target node, and the first source node and the first target node are general-purpose computing graphics processing unit nodes;

[0202] An identity information acquisition module 202 is configured to acquire the identity information of the first source node according to the memory address of the first source node, and acquire the identity information of the first target node according to the memory address of the first target node;

[0203] The communication action module 203 is configured to execute a communication action on the first source node and the first target node through the remote direct memory access module according to the identity information of the first source node and the identity information of the first target node.

[0204] Based on some implementation methods, the inter-node communication device provided by this application further includes:

[0205] An identity information allocation module 204 is configured to allocate identity information to a plurality of nodes, wherein the plurality of nodes include a first source node and a first target node;

[0206] The unified memory address editing module 205 is configured to edit the unified memory addresses of the multiple nodes according to the identity information allocation order of the multiple nodes.

[0207] In some implementations, multiple nodes are located in a cluster. The identity information allocation module 204 provided in this application includes:

[0208] The cluster dynamic allocation unit 241 is configured to perform a cluster dynamic allocation action on multiple nodes and obtain corresponding identity information.

[0209] Based on some implementation methods, the first application description acquisition module 201 provided in this application includes:

[0210] An application description table construction unit 211 is configured to construct an application description table for multiple nodes, wherein the application description table carries application descriptors between the multiple nodes;

[0211] The application descriptor obtaining unit 212 is configured to obtain the application descriptor between the first source node and the first target node from the application description table according to a first-in-first-out principle.

[0212] Based on some implementations, the identity information acquisition module 202 provided in this application includes:

[0213] A source address resolution unit 221 is configured to perform address resolution on the memory address of the first source node to obtain the local address of the first source node and the identity information of the first source node;

[0214] The target address resolution unit 222 is configured to perform address resolution on the memory address of the first target node to obtain the local address of the first target node and identity information corresponding to the local address.

[0215] Based on some implementation methods, the identity information acquisition module 202 provided in this application further includes:

[0216] The first mapping unit 223 is configured to generate mappings and de-mappings between memory addresses of multiple nodes and corresponding local addresses.

[0217] Based on some implementations, the communication action module 203 provided in this application includes:

[0218] A source Internet protocol acquisition unit 231 is configured to acquire Internet protocol address information of the first source node according to the identity information of the first source node;

[0219] a target Internet protocol acquisition unit 232, configured to acquire Internet protocol address information of the first target node according to the identity information of the first target node;

[0220] The communication unit 233 is configured to perform communication actions on the first source node and the first target node through a protocol preset in the remote direct memory access module according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0221] Based on some implementation methods, the communication action module 203 provided by this application further includes:

[0222] The second mapping unit 234 is configured to generate a mapping between multiple node identity information and corresponding Internet Protocol address information.

[0223] In some embodiments, the node is a GPU, the remote direct memory access module shares memory with the GPU, and the communication unit 233 provided in this application includes:

[0224] The zero copy subunit 235 is used to perform a zero copy operation on the first source node and the first target node through a protocol preset in the remote direct memory access module according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0225] Based on some embodiments, the pre-set protocol includes the Transmission Control Protocol, and the zero-copy subunit 235 provided in this application includes:

[0226] The first read / write subunit 236 is used to perform memory read / write operations on the first source node and the first target node through the transmission control protocol preset in the remote direct memory access module according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0227] Based on some implementations, the pre-set protocol includes the User Datagram Protocol, and the zero-copy subunit 235 provided by the present application further includes:

[0228] The second read / write subunit 237 is used to perform memory read / write operations on the first source node and the first target node through the user datagram protocol pre-set in the remote direct memory access module according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0229] Based on some implementations, the communication unit 233 provided in this application includes:

[0230] The communication mode calling subunit 238 is used to call the communication mode preset in the remote direct memory access module according to the copy type;

[0231] The communication action execution subunit 239 is configured to execute a communication action on the first source node and the first target node through a communication mode according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0232] Based on some implementations, the copy type includes remote copy, the communication mode includes remote communication mode, and the communication action execution subunit 239 provided in this application includes:

[0233] The remote copy subunit 240 is configured to perform communication actions on the first source node and the first target node through a remote communication mode according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0234] Based on some implementations, the copy type further includes local copy, and the communication mode further includes local communication mode. The communication action execution subunit 239 provided in this application further includes:

[0235] The local copy subunit 242 is configured to perform communication actions on the first source node and the first target node through a local communication mode according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node.

[0236] Based on some implementation methods, the inter-node communication device provided by the present application further includes:

[0237] The read synchronization processing module 207 is configured to perform read synchronization processing on the first source node and the first target node.

[0238] Based on some implementation methods, the inter-node communication device provided by the present application further includes:

[0239] The write synchronization processing module 208 is configured to perform write synchronization processing on the first source node and the first target node.

[0240] Based on some implementation methods, the inter-node communication device provided by the present application further includes:

[0241] The communication completion signal module 209 is used to generate a communication completion signal according to the result of the read synchronization process and the result of the write synchronization process;

[0242] The second application description obtaining module 210 is configured to obtain the application descriptor between the second source node and the second target node after obtaining the communication completion signal.

[0243] Some embodiments of the present application relate to a mobile terminal, as shown in FIG22 , including:

[0244] at least one processor 161; and,

[0245] A memory 162 in communication with at least one processor 161; wherein,

[0246] The memory 162 stores instructions that can be executed by at least one processor. The instructions are executed by the at least one processor 161 to enable the at least one processor 161 to implement the inter-node communication method of some embodiments of the present application.

[0247] The memory and processor are connected using a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to the processor.

[0248] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0249] Some embodiments of the present application relate to a non-volatile readable storage medium storing a computer program. When the computer program is executed by a processor, the inter-node communication method of some embodiments of the present application is implemented.

[0250] That is, those skilled in the art will understand that all or part of the steps in some embodiments of the method can be implemented by instructing related hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps in some embodiments of the method of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0251] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0252] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A node-to-node communication method, applied to a Kubernetes cluster management system, characterized in that: The method comprises: Get a list of FPGAs; According to the list of the field programmable gate arrays, the field programmable gate array is virtualized to obtain a first source node and a first target node, wherein the first source node and the first target node are general-purpose computing graphics processing unit nodes; Acquire an application descriptor between the first source node and the first target node, wherein the application descriptor includes a memory address of the first source node and a memory address of the first target node; Acquire the identity information of the first source node according to the memory address of the first source node, and acquire the identity information of the first target node according to the memory address of the first target node; According to the identity information of the first source node and the identity information of the first target node, a communication action is performed on the first source node and the first target node through a remote direct memory access module.

2. The method according to claim 1, characterized in that Before obtaining the application descriptor between the first source node and the first target node, the method further includes: Allocating identity information to a plurality of nodes, wherein the plurality of nodes include a first source node and a first target node; The memory addresses of the multiple nodes are uniformly edited according to the identity information allocation order of the multiple nodes.

3. The method according to claim 2, characterized in that The multiple nodes are located in a cluster, and the assigning of identity information to the multiple nodes includes: A cluster dynamic allocation action is performed on the multiple nodes to obtain corresponding identity information.

4. The method according to claim 2, characterized in that: The obtaining of the application descriptor between the first source node and the first target node comprises: Constructing an application description table of the multiple nodes, wherein the application description table carries application descriptors between the multiple nodes; The application descriptor between the first source node and the first target node is obtained from the application description table according to the first-in-first-out principle.

5. The method according to claim 1, characterized in that The acquiring the identity information of the first source node according to the memory address of the first source node and the acquiring the identity information of the first target node according to the memory address of the first target node includes: Performing address resolution on the memory address of the first source node to obtain the local address of the first source node and the identity information of the first source node; Perform address resolution on the memory address of the first target node to obtain the local address of the first target node and the identity information of the first target node.

6. The method according to claim 5, characterized in that Before performing address resolution on the memory address of the first source node to obtain the local address of the first source node and the identity information of the first source node, the method further includes: Generate mapping and de-mapping between memory addresses of multiple nodes and corresponding local addresses.

7. The method according to claim 1, characterized in that The performing a communication action on the first source node and the first target node through a remote direct memory access module according to the identity information of the first source node and the identity information of the first target node comprises: Acquire Internet Protocol address information of the first source node according to the identity information of the first source node; Acquire Internet Protocol address information of the first target node according to the identity information of the first target node; According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node, communication actions are performed on the first source node and the first target node through a protocol preset in the remote direct memory access module.

8. The method according to claim 7, characterized in that Before acquiring the Internet Protocol address information of the first source node according to the identity information of the first source node, the method further includes: A mapping of the plurality of node identity information and corresponding Internet Protocol address information is generated.

9. The method according to claim 7, characterized in that: The remote direct memory access module shares a memory with a graphics processing unit corresponding to the general computing graphics processing unit node, and the performing of a communication action on the first source node and the first target node through a protocol preset in the remote direct memory access module according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node comprises: According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node, a zero copy action is performed on the first source node and the first target node through a protocol preset in the remote direct memory access module.

10. The method according to claim 9, characterized in that The preset protocol includes a transmission control protocol, and the performing a zero copy operation on the first source node and the first target node through the protocol preset in the remote direct memory access module according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node includes: According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node, memory read and write operations are performed on the first source node and the first target node through the transmission control protocol pre-set in the remote direct memory access module.

11. The method according to claim 9, characterized in that The preset protocol includes a user datagram protocol, and the performing a zero copy action on the first source node and the first target node through the protocol preset in the remote direct memory access module according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node includes: According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node, memory read and write operations are performed on the first source node and the first target node through the user datagram protocol preset in the remote direct memory access module.

12. The method according to claim 7, characterized in that The application descriptor further includes a copy type, and the performing of a communication action on the first source node and the first target node through a protocol preset in the remote direct memory access module according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node includes: calling a communication mode preset in the remote direct memory access module according to the copy type; According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node, communication actions are performed on the first source node and the first target node through the communication mode.

13. The method according to claim 12, characterized in that The copy type includes remote copy, the communication mode includes remote communication mode, and the performing communication actions on the first source node and the first target node through the communication mode according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node includes: According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node, communication actions are performed on the first source node and the first target node through the remote communication mode.

14. The method according to claim 12, characterized in that The copy type further includes a local copy, the communication mode further includes a local communication mode, and the performing a communication action on the first source node and the first target node through the communication mode according to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node includes: According to the Internet Protocol address information of the first source node and the Internet Protocol address information of the first target node, communication actions are performed on the first source node and the first target node through the local communication mode.

15. The method according to claim 1, characterized in that The communication action includes a read action, and after performing the communication action on the first source node and the first target node through the remote direct memory access module according to the identity information of the first source node and the identity information of the first target node, the further comprising: A read synchronization process is performed on the first source node and the first target node.

16. The method according to claim 15, characterized in that The communication action further includes a write action, and after performing the communication action on the first source node and the first target node through the remote direct memory access module according to the identity information of the first source node and the identity information of the first target node, the communication action further includes: A write synchronization process is performed on the first source node and the first target node.

17. The method according to claim 16, characterized in that After performing write synchronization processing on the first source node and the first target node, the method further includes: generating a communication completion signal according to a result of the read synchronization process and a result of the write synchronization process; After the communication completion signal is obtained, an application descriptor between the second source node and the second target node is obtained.

18. An inter-node communication device, characterized in that: Applied to a Kubernetes cluster management system, including: A field programmable gate array list acquisition module, used to acquire a list of field programmable gate arrays; A node virtualization module, configured to virtualize the field programmable gate array according to the list of the field programmable gate array to obtain a first source node and a first target node; A first application description acquisition module, used to acquire an application descriptor between a first source node and a first target node, wherein the application descriptor includes a memory address of the first source node and a memory address of the first target node, and the first source node and the first target node are general computing graphics processing unit nodes; an identity information acquisition module, configured to acquire the identity information of the first source node according to the memory address of the first source node, and acquire the identity information of the first target node according to the memory address of the first target node; A communication action module is used to perform a communication action on the first source node and the first target node through a remote direct memory access module according to the identity information of the first source node and the identity information of the first target node.

19. An electronic device, characterized in that: It includes a processor and a memory, the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the inter-node communication method as described in any one of claims 1 to 17 are implemented.

20. A non-volatile readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the inter-node communication method according to any one of claims 1 to 17 is implemented.

Citation Information

Patent Citations

  • GPU cluster shared video memory system, method, device and equipment

    CN113674133A

  • Method and device for realizing remote direct memory access, electronic equipment and medium

    CN114050998A

  • Virtualized GPU (Graphics Processing Unit) scheduling method and device of container system and medium

    CN115904675A

  • Inter-node communication method and device, electronic equipment and storage medium

    CN116383127A

  • High-availability memory replication in one or more network devices

    US20210294702A1

Cited By

  • Resource management method, electronic equipment and storage medium

    CN121037187A