Data transmission method and apparatus, electronic device, and storage medium

Abstract

The application discloses a data transmission method and device, electronic equipment and storage medium, comprising: for each target actual thread running on a target computing node, selecting one from a plurality of proxy threads enabled on a first storage node as a first peer thread corresponding to the target actual thread; the proxy threads corresponding to different target actual threads are different; the first storage node is any one storage node in a cluster that enables a proxy thread; a remote direct memory access (RDMA) connection is established between each target actual thread and the corresponding first peer thread on the first storage node; and data transmission is performed between the target actual thread on the target computing node and a plurality of actual threads running on the first storage node according to the RDMA connection between each target actual thread and the corresponding first peer thread, with the first peer thread corresponding to each target actual thread as a relay. The scheme can greatly reduce the number of RDMA connections established.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data transmission method, apparatus, electronic device, and storage medium. Background Technology

[0002] In a cluster consisting of multiple compute nodes and multiple storage nodes, threads running on compute nodes and threads running on storage nodes typically establish RDMA (Remote Direct Memory Access) connections. These RDMA connections are established in a fully connected mode, meaning that each thread on a compute node needs to establish an RDMA connection with each thread running on each storage node. Establishing an RDMA connection requires creating corresponding queue pairs. These queue pairs, as the basic unit of RDMA communication, also require maintaining queue pair contexts on both compute and storage nodes. With a large number of RDMA connections, the number of queue pairs is also large, and the memory resources occupied by maintaining these queue pair contexts are also significant, potentially leading to a decrease in overall computational performance. Summary of the Invention

[0003] In view of the above problems, this application proposes a data transmission method, apparatus, electronic device and storage medium to solve the problem of a large number of RDMA connections in the cluster in the related art.

[0004] According to one aspect of the embodiments of this application, a data transmission method is provided, applied to a target computing node in a cluster, wherein the target computing node refers to any computing node in the cluster, the method comprising:

[0005] For each target actual thread running on the target computing node, one of the multiple proxy threads enabled on the first storage node is selected as the first peer thread corresponding to the target actual thread; different target actual threads correspond to different proxy threads; the first storage node is any storage node in the cluster that enables proxy threads;

[0006] Establish a remote direct memory access (RDMA) connection between each of the target actual threads and the corresponding first peer thread on the first storage node;

[0007] Using the first peer thread corresponding to each of the target actual threads as a relay, data is transmitted between the target actual thread on the target computing node and the multiple actual threads running on the first storage node based on the RDMA connection between each of the target actual threads and the corresponding first peer thread.

[0008] According to one aspect of the embodiments of this application, a data transmission apparatus is provided, applied to a target computing node in a cluster, wherein the target computing node refers to any computing node in the cluster, and the data transmission apparatus includes:

[0009] The selection module is used to select one of multiple proxy threads enabled on the first storage node for each target actual thread running on the target computing node, as the first peer thread corresponding to the target actual thread; different target actual threads correspond to different proxy threads; the first storage node is any storage node in the cluster that enables proxy threads;

[0010] The connection establishment module is used to establish a remote direct memory access (RDMA) connection between each of the target actual threads and the corresponding first end thread on the first storage node.

[0011] The data transmission module is used to transmit data between the target actual thread on the target computing node and multiple actual threads running on the first storage node, based on the RDMA connection between each target actual thread and the corresponding first end thread, using the first end thread corresponding to each target actual thread as a relay.

[0012] In some embodiments, the connection establishment module is configured to perform the following processing for each of the target actual threads:

[0013] The target actual thread registers the memory region and creates a first queue pair, obtaining the first memory region information of the registered first memory region and the first queue pair information of the first queue pair.

[0014] Based on the first memory region information and the first queue pair information, information is exchanged with the first end thread corresponding to the target actual thread to obtain the second queue pair information and the second memory region information of the second memory region; the second queue pair is a queue pair created by the first end thread corresponding to the target actual thread, and the second memory region is a memory region registered by the first end thread corresponding to the target actual thread on the first storage node;

[0015] Based on the second queue pair information and the second memory region information, the first queue pair and the second queue pair are associated to establish an RDMA connection between the target actual thread and the corresponding first end thread on the first storage node.

[0016] In some embodiments, the data transmission module includes:

[0017] The first partitioning unit is used to partition a first buffer in the corresponding first memory region by the target actual thread and obtain the first access information of the first buffer;

[0018] The first acquisition unit is configured to acquire a first message sent by a first peer thread corresponding to the target actual thread; the first message includes second access information of the first target data in the second memory area; the first target data is data to be transmitted from the first actual thread on the first storage node to the target actual thread; the first target data is stored in the second memory area by the first actual thread through the first peer thread corresponding to the target actual thread;

[0019] The first sending unit is used to send a data reading task to the sending queue in the first queue pair by the target actual thread according to the first access information and the second access information.

[0020] The read operation unit is used to execute an RDMA read operation after the network card in the target computing node polls for a data read request in the sending queue, so as to temporarily store the first target data in the first buffer.

[0021] The first acquisition unit is used to acquire the first target data from the first buffer by the target actual thread.

[0022] In some embodiments, the data transmission device further includes a task execution module, configured to have the target actual thread perform a computational task using the first target data.

[0023] In other embodiments, the data transmission module includes:

[0024] The first temporary storage unit is used to temporarily store the second target data in the first memory area by the actual target thread, and to obtain the third access information of the second target data in the first memory area;

[0025] The first receiving unit is configured to receive a second message sent by a first peer thread corresponding to the target actual thread, the second message including fourth access information of a second buffer; the second buffer is a buffer partitioned by the first peer thread corresponding to the target actual thread in the second memory area;

[0026] The second sending unit is used to send data to the sending queue in the first queue pair by the target actual thread according to the third access information and the fourth access information.

[0027] The write operation unit is used to execute an RDMA write operation after the network card in the target computing node polls the data write task in the sending queue, so as to write the second target data into the second buffer, so that the first peer thread corresponding to the target actual thread sends the second target data to the second actual thread.

[0028] In other embodiments, the data transmission module includes:

[0029] The second temporary storage unit is used to temporarily store the third target data into the first memory area by the actual target thread, and obtain the fifth access information of the third target data in the first memory area;

[0030] The third sending unit is used to send the first sending task to the sending queue in the first queue pair by the target actual thread according to the fifth access information;

[0031] The transmission operation unit is configured to execute an RDMA transmission operation after the network interface card (NIC) in the target computing node polls the first transmission task in the transmission queue, so that after the NIC in the first storage node receives the third target data, it consumes the first reception task in the reception queue of the second queue pair and caches the third target data in the third buffer corresponding to the sixth access information indicated by the reception task, so that the first peer thread corresponding to the target actual thread sends the third target data to the third actual thread; the third buffer is a buffer partitioned by the first peer thread corresponding to the target actual thread in the second memory area, and the first reception task is generated by the first peer thread corresponding to the target actual thread according to the sixth access information of the third buffer.

[0032] In other embodiments, the data transmission module includes:

[0033] The second partitioning unit is used to partition a fourth buffer in the first memory region by the target actual thread and obtain the seventh access information of the fourth buffer.

[0034] The third sending unit is used to send a second receiving task to the receiving queue in the first queue pair by the target actual thread according to the seventh access information;

[0035] The consumption unit is configured to, if the network card in the target computing node receives the fourth target data sent from the network card in the first storage node, consume the second receiving task in the receiving queue of the first queue pair, and cache the fourth target data in the fourth buffer according to the seventh access information, so that the target actual thread can obtain the fourth target data from the fourth buffer;

[0036] Wherein, after the network card in the first storage node polls the second sending task in the sending queue of the second queue pair, it reads the fourth target data from the fifth buffer and sends the fourth target data to the network card in the target computing node; the second sending task is generated by the first end thread corresponding to the target actual thread according to the eighth access information of the fifth buffer in the second memory area, and the fourth target data is temporarily stored in the fifth buffer by the first end thread corresponding to the target actual thread after obtaining the fourth target data from the fourth actual thread.

[0037] In some embodiments, the data transmission apparatus further includes: a proxy thread information acquisition module, configured to acquire proxy thread information allocated by the first storage node to the target computing node, the proxy thread information including proxy thread identifiers of each proxy thread allocated by the first storage node to the target computing node; an establishment module, configured to establish a first mapping relationship based on the proxy thread identifiers of each proxy thread in the proxy thread information and the thread identifiers of each target actual thread on the target computing node; in the first mapping relationship, one target actual thread corresponds to one proxy thread on the first storage node, and different target actual threads correspond to different proxy threads on the first storage node; correspondingly, the selection module is configured to: determine the first peer thread corresponding to the target actual thread based on the first mapping relationship.

[0038] In some embodiments, the cluster has a second storage node with the agent thread not enabled; the data transmission device further includes:

[0039] The determination module is used to determine one target actual thread for the second storage node among multiple target actual threads running on the target computing node, as the second peer thread corresponding to the second storage node;

[0040] The first connection establishment module is used to establish RDMA connections between the second peer thread corresponding to the second storage node and each actual thread on the second storage node.

[0041] The first transmission module is used to perform data transmission between the target computing node and each actual thread on the second storage node through an RDMA connection between the second peer thread corresponding to the second storage node and the actual thread on the second storage node.

[0042] In some embodiments, the determining module includes: a node information acquisition unit, configured to acquire node information of the second storage node; the node information includes node identification information of the second storage node;

[0043] The hash calculation unit is used to perform hash calculation based on the node identification information of the second storage node to obtain the target hash value;

[0044] The second peer thread determination unit is used to determine the target actual thread corresponding to the target hash value on the target computing node as the second peer thread corresponding to the second storage node, based on the mapping relationship between the hash value and the actual thread.

[0045] In some embodiments, the first transmission module is configured as follows:

[0046] The following processing is performed on the first target actual thread on the target storage node:

[0047] If the first target actual thread is not the second peer thread corresponding to the second storage node, the fifth target data is sent to the second peer thread corresponding to the second storage node; the fifth target data is the data transmitted by the first target actual thread to the reference actual thread in the second storage node.

[0048] Based on the RDMA connection between the second peer thread corresponding to the second storage node and the reference actual thread, the fifth target data is transmitted to the reference actual thread in the second storage node.

[0049] According to one aspect of the embodiments of this application, an electronic device is provided, including: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the data transmission method described above.

[0050] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a processor, implement the data transmission method described above.

[0051] According to one aspect of the embodiments of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the data transmission method described above.

[0052] In this application, by utilizing the proxy thread on the first storage node in the cluster, one of the multiple proxy threads enabled on the first storage node is selected as the first end thread corresponding to the target actual thread on the target compute node. Moreover, the proxy threads corresponding to different target actual threads on the target compute node are different. Then, RDMA connections are established between each target actual thread on the target compute node and its corresponding first end thread. In this way, the number of RDMA connections established between a target compute node and a first storage node is equal to the number of target actual threads running on the target compute node. Compared with the fully connected mode, the number of RDMA connections established is greatly reduced. Correspondingly, the number of queue pairs that need to be created is also greatly reduced, as is the number of queue pair contexts that need to be maintained. This solves the problem of memory space occupation on nodes, especially compute nodes, caused by the need to maintain a large number of queue pair contexts.

[0053] Moreover, the proxy threads enabled in the first storage node and the actual threads running on the first storage node are both internal threads of the first storage node. Each proxy thread on the first storage node can communicate with the actual threads running on the first storage node. Thus, using the first peer thread corresponding to each target actual thread as a relay, and based on the RDMA connection between each target actual thread and its corresponding first peer thread, data transmission is performed between the target actual thread on the target computing node and the multiple actual threads running on the first storage node. This achieves the goal of ensuring data transmission between the target actual thread on the target computing node and each actual thread on the first storage node using fewer RDMA connections. Attached Figure Description

[0054] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0055] Figure 1 This is a schematic diagram illustrating an application scenario of this application according to an embodiment of this application.

[0056] Figure 2 This is a flowchart illustrating a data transmission method according to an embodiment of this application.

[0057] Figure 3 This is a flowchart illustrating step 220 according to an embodiment of this application.

[0058] Figure 4This is a schematic diagram illustrating the communication relationship between computing nodes and storage nodes in a cluster according to an embodiment of this application.

[0059] Figure 5 This is a schematic diagram illustrating the communication relationship between compute nodes and storage nodes in a cluster, according to another embodiment of this application.

[0060] Figure 6 This is a diagram illustrating the establishment of an RDMA connection between a compute node and a storage node in a fully connected mode.

[0061] Figure 7 This example illustrates a schematic diagram of the communication relationship between compute nodes and storage nodes in a cluster in a shared mode.

[0062] Figure 8 This is a flowchart illustrating step 230 according to an embodiment of this application.

[0063] Figure 9 This is a flowchart illustrating step 230 according to another embodiment of this application.

[0064] Figure 10 This is a flowchart illustrating step 230 according to another embodiment of this application.

[0065] Figure 11 This is a flowchart illustrating step 230 according to another embodiment of this application.

[0066] Figure 12 This is a flowchart illustrating a data transmission method according to another embodiment of this application.

[0067] Figure 13 This is a schematic diagram illustrating the communication relationship between a computing node and a second storage node in a cluster, according to an embodiment of this application.

[0068] Figure 14 This is a block diagram of a data transmission apparatus according to an embodiment of this application.

[0069] Figure 15 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0070] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0071] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0072] In the following description, the terms "first" and "second" are used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0073] In this document, "multiple" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. In the following description, references to "some embodiments or some embodiment methods" describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0074] Figure 1 This is a schematic diagram illustrating an application scenario of this application according to an embodiment of this application. For example... Figure 1 As shown, this application scenario includes compute nodes and a first storage node located in the cluster. The number of compute nodes and the first storage node in the cluster is unlimited. Figure 1 The example illustrates a compute node and a first storage node. The compute node in the cluster is used to perform computational tasks, such as floating-point operations, image processing, text recognition, video encoding, and audio encoding. The first storage node is used to store data. The compute node can obtain data from the first storage node for performing computational tasks. Furthermore, the compute node can send new data generated during the execution of computational tasks to the first storage node for storage.

[0075] In this application, multiple actual threads can run on both the compute node and the first storage node, for example... Figure 1The actual threads t1-1, t1-2, and t1-3 run on the intermediate computing node; the actual threads t2-1, t2-2, t2-3, and t2-4 run on the first storage node. Furthermore, in this application, proxy threads are also enabled on the first storage node, for example... Figure 1 The proxy threads d1, d2, and d3 are shown in the figure.

[0076] Each compute node in the cluster can establish an RDMA (Remote Direct Memory Access) connection with the proxy thread on each first storage node according to the method of this application. Subsequently, data transfer between the actual threads on the compute nodes and the actual threads on the first storage nodes can be achieved using the established RDMA connection. Taking the target compute node as an example, it can be processed as follows to achieve data transfer: For each target actual thread running on the target compute node, one of the multiple proxy threads enabled on the first storage node is selected as the first peer thread corresponding to the target actual thread; different target actual threads correspond to different proxy threads; the first storage node is any storage node in the cluster that enables proxy threads; a remote direct memory access (RDMA) connection is established between each target actual thread and its corresponding first peer thread on the first storage node; using the first peer thread corresponding to each target actual thread as a relay, data transfer is performed between the target actual thread on the target compute node and the multiple actual threads running on the first storage node based on the RDMA connection between each target actual thread and its corresponding first peer thread.

[0077] Figure 1 The example illustrates the establishment of RDMA connections between the actual thread t1-1 on the compute node and the proxy thread d1 on the first storage node, between the actual thread t1-2 on the compute node and the proxy thread d2 on the first storage node, and between the actual thread t1-3 on the compute node and the proxy thread d3 on the first storage node.

[0078] The computing nodes and the first storage node can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0079] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0080] Figure 2 This is a flowchart illustrating a data transmission method according to an embodiment of this application, applied to a target computing node in a cluster, where the target computing node refers to any computing node in the cluster. (Refer to...) Figure 2 As shown, the method includes at least steps 210 to 230, which are described in detail below:

[0081] Step 210: For each target actual thread running on the target computing node, select one of the multiple proxy threads enabled on the first storage node as the first peer thread corresponding to the target actual thread; the proxy threads corresponding to different target actual threads are different; the first storage node is any storage node in the cluster that enables proxy threads.

[0082] The cluster comprises multiple compute nodes and multiple storage nodes. Compute nodes execute computational tasks, such as model inference, image processing (e.g., image recognition, image enhancement), text recognition, and data format conversion. Storage nodes store data. Compute nodes can read data needed for computational tasks from storage nodes and also send new data generated during computation to storage nodes for persistent storage. Each compute node in the cluster can serve as a target compute node in this application, transmitting data with the storage nodes according to the method provided in this application.

[0083] A proxy thread, also known as a dummy thread, is a thread used to handle proxy tasks. Its core function is to act as an intermediary layer. A proxy thread is defined in contrast to the thread that is the actual running entity, referred to as the actual thread. In other words, the actual thread is the thread that is actually used for computation scheduling and is the smallest unit for computation scheduling.

[0084] In this application, the storage node in the cluster that has enabled the proxy thread is referred to as the first storage node. It is understood that all storage nodes in the cluster may have enabled the proxy thread; in this case, all storage nodes in the cluster are considered first storage nodes. Alternatively, only some storage nodes in the cluster may have enabled the proxy thread, while others may not. The target compute node can perform data transfer with the actual thread running on each storage node according to this application.

[0085] Since for a target actual thread on a target computing node, one of the multiple proxy threads enabled on the first storage node is selected as the first peer thread corresponding to that target actual thread, and different target actual threads correspond to different proxy threads, in other words, one target actual thread corresponds to one first peer thread on one first storage node, and different target actual threads correspond to different first peer threads on the same first storage node, thus ensuring a one-to-one correspondence between multiple target actual threads on one computing node and multiple proxy threads on one first storage node.

[0086] In some embodiments, considering that the cluster includes multiple compute nodes, and that multiple compute nodes may all involve data transmission with the same first storage node, each first storage node may enable proxy threads for each compute node, the same number as the actual number of threads running on that compute node. In other words, if the actual number of threads running on compute node X1 is k1, then the first storage node Y1 enables k1 proxy threads for compute node X1; if the actual number of threads running on another compute node X2 is k2, then the first storage node Y1 enables k2 proxy threads for compute node X2.

[0087] In some embodiments, the proxy threads enabled on the same first storage node for different compute nodes can be different. That is, a proxy thread on a first storage node is only directed to one actual thread on one compute node, serving as the first-end thread of an actual thread on one compute node, and not as the first-end thread of actual threads on other compute nodes. For example, if the cluster includes N compute nodes, and each compute node runs m actual threads, then for a first storage node, the total number of proxy threads enabled on that first storage node is m*N. Of course, the number of actual threads running on different compute nodes may also be different. In this case, the total number of proxy threads enabled on a first storage node is equal to the sum of the number of actual threads running on all compute nodes in the cluster.

[0088] In some embodiments, proxy threads enabled for different compute nodes on the same first storage node may have the same proxy thread. That is, a proxy thread enabled on a first storage node can be used as the first peer thread corresponding to an actual thread on different compute nodes, but the proxy thread will not be reused by different actual threads on the same compute node. For example, if a proxy thread t1 on the first storage node Y1 can be used as the first peer thread corresponding to an actual thread x1-1 on compute node X1, or as the first peer thread corresponding to an actual thread x2-1 on compute node X2 (different from compute node X1), the proxy thread t1 on the first storage node Y1 will not be used as the first peer thread of any other actual thread on compute node X1 besides actual thread x1-1.

[0089] Assuming the cluster includes N compute nodes, and the maximum number of actual threads running on a single compute node is m1, then the total number of proxy threads enabled on a first storage node ∈ [m1, Z), where Z is the total number of actual threads running on the N compute nodes. It can be understood that if the number of actual threads running on different compute nodes among the N compute nodes is the same, which is m, i.e., m1 = m, then the minimum number of proxy threads enabled on a first storage node is m.

[0090] In some embodiments, each first storage node in the cluster can obtain the number of actual threads running on each compute node in the cluster. Then, each first storage node determines the number of proxy threads it needs to activate based on the number of actual threads running on each compute node in the cluster. In some embodiments, a maximum number of times a proxy thread on a storage node can be reused can be preset. If a proxy thread is used as the first end thread of actual threads located on different first storage nodes, it is said that the proxy thread is reused. Based on this, each first storage node can determine the number of proxy threads activated on the first storage node based on the number of actual threads running on each compute node in the cluster, combined with the preset maximum number of times a single proxy thread can be reused, to ensure that the proxy threads activated on a first storage node can be allocated to all actual threads on all compute nodes in the cluster, and to ensure that the number of times a proxy thread on a first storage node is reused does not exceed the preset maximum number of times.

[0091] It is understandable that if the maximum number of times a single proxy thread can be reused is set to 0, it means that a proxy thread enabled on a first storage node is only used as the first end thread of an actual thread on a compute node. Correspondingly, the number of proxy threads enabled on a first storage node is equal to the sum of the number of actual threads on all compute nodes in the cluster.

[0092] In some embodiments, prior to step 210, the method further includes steps A1 and A2 as follows:

[0093] Step A1: Obtain the proxy thread information allocated by the first storage node to the target computing node. The proxy thread information includes the proxy thread identifier of each proxy thread allocated by the first storage node to the target computing node.

[0094] The proxy thread identifier indicates information used to uniquely identify the proxy thread; different proxy threads have different proxy thread identifiers. The number of proxy threads allocated by the first storage node to the target compute node is equal to the number of actual target threads running on the target compute node.

[0095] It is understood that the first storage node allocates proxy threads to the target compute node based on the actual number of threads running on the target compute node, thereby ensuring that the number of proxy threads allocated to each compute node equals the number of actual threads running on that compute node. In some embodiments, the first storage node can obtain the number of actual threads running on the target compute node and allocate a corresponding number of proxy threads to the first storage node accordingly. If the number of proxy threads enabled on the first storage node is insufficient to support the allocation of proxy threads to the actual threads on the target compute node, new proxy threads can be enabled on the first storage node.

[0096] Step A2: Establish a first mapping relationship based on the agent thread identifier of each agent thread in the agent thread information and the thread identifier of each target actual thread on the target computing node; in the first mapping relationship, one target actual thread corresponds to one agent thread on the first storage node, and different target actual threads correspond to different agent threads on the first storage node.

[0097] Establish the first mapping relationship, that is, associate the proxy thread identifier of a proxy thread in the proxy thread information with the thread identifier of a target actual thread on the target computing node, so that the proxy thread identifiers of different proxy threads in the proxy thread information are associated with the thread identifiers of different target actual threads.

[0098] Correspondingly, step 210 includes: determining the first peer thread corresponding to the target actual thread based on the first mapping relationship.

[0099] For a target actual thread, in the first mapping relationship, the proxy thread indicated by a proxy thread identifier associated with the thread identifier of the target actual thread can be determined as the first peer thread corresponding to the target actual thread.

[0100] In the above embodiment, since the first storage node pre-allocates a corresponding number of proxy threads to the target computing node according to the number of actual threads running on the target computing node, the target computing node then establishes a one-to-one correspondence between the actual threads and the proxy threads, i.e., the first mapping relationship, based on the thread identifier of its own running actual thread and the thread identifier of the proxy thread allocated by the first storage node to the target computing node. In this way, it can be ensured that the first end thread corresponding to each target actual thread can be quickly determined according to the first mapping relationship.

[0101] In other embodiments, the target computing node may send actual thread information to the first storage node. The actual thread information includes the thread identifier of the actual thread running on the target computing node. Then, the first storage node establishes a first mapping relationship between the proxy thread on the first storage node and the actual thread on the target computing node based on the thread identifier of the proxy thread it has enabled and the thread identifier of the actual thread running on the target computing node. The first mapping relationship is then sent to the target computing node so that the target computing node can determine the first peer thread corresponding to different target actual threads according to the first mapping relationship.

[0102] Step 220: Establish a remote direct memory access (RDMA) connection between each target actual thread and the corresponding first end thread on the first storage node.

[0103] Remote Direct Memory Access (RDMA) is a technology that enables direct data transfer between the memory of different nodes without the intervention of the operating system kernels of both nodes. This reduces the processing burden on the central processing unit, lowers data transfer latency, and improves data transfer speed and efficiency.

[0104] The process of establishing an RDMA connection between a target actual thread and the first peer thread corresponding to that target actual thread on the first storage node can be as follows: Figure 3 As shown, it includes:

[0105] Step 310: The target actual thread registers the memory region and creates the first queue pair, obtaining the first memory region information of the registered first memory region and the first queue pair information of the first queue pair.

[0106] A queue pair (QP) is the basic unit of remote direct memory access communication. A queue pair consists of a send queue and a receive queue, which can be collectively referred to as work queues. The send queue stores send tasks, and the receive queue stores receive tasks. A queue pair represents a communication channel, allowing data transmission between two nodes via the corresponding queue pair.

[0107] The first queue pair refers to the queue pair created by the actual target thread. The actual target thread is registered in a memory region, and the registered memory region is called the first memory region. This registered memory region is used to store data from the receiving end or data to be transmitted to other nodes during subsequent data transmission using the first queue pair. Memory registration ensures that the network interface card (NIC) on the target compute node can access the data in the first memory region.

[0108] The first memory region information includes the virtual address and access key of the first memory region. Obtaining the access key is equivalent to having permission to access the first memory region. Furthermore, through memory registration, the software layer running on the target compute node creates a mapping table between virtual addresses and physical memory addresses in the target compute node's memory. This allows subsequent data transmission via the network interface card (NIC) to translate the virtual address of the registered memory region into a physical memory address based on this mapping table, enabling access to data in the memory space pointed to by the virtual address (i.e., the memory region represented by the mapped physical memory address). In other words, during the memory region registration process, the metadata of the registered first memory region is recorded, such as its starting address, size, access permissions, and physical memory address.

[0109] The first queue pair information includes the queue pair context (QPC) of the first queue pair. The queue pair context is used to define the attribute information of the corresponding queue pair, such as the queue pair identifier, the address information of the sending queue (which is a contiguous address space), the queue identifier of the sending queue, the address information of the receiving queue, the queue identifier of the receiving queue, and the access permission information of the memory region corresponding to the queue pair. After the first queue pair is created, the first queue pair information is stored in the memory of the target computing node.

[0110] Step 320: Based on the first memory region information and the first queue pair information, exchange information with the first end thread corresponding to the target actual thread to obtain the second queue pair information and the second memory region information of the second memory region; the second queue pair is a queue pair created by the first end thread corresponding to the target actual thread, and the second memory region is a memory region registered on the first storage node by the first end thread corresponding to the target actual thread.

[0111] By exchanging information with the first paired thread corresponding to the target actual thread, on the one hand, the first queue pair information and the first memory region information on the side of the target actual thread are sent to the first paired thread corresponding to the target actual thread; on the other hand, the second queue pair information and the second memory region information are received from the first paired thread corresponding to the target actual thread. The first paired thread corresponding to the target actual thread registers the memory region and creates queue pairs on the first storage node side according to a similar process to step 310 above.

[0112] Step 330: Associate the first queue pair and the second queue pair with the second queue pair information and the second memory region information to establish an RDMA connection between the target actual thread and the corresponding first end thread on the first storage node.

[0113] Associating the first queue pair and the second queue pair can be achieved by maintaining the second queue pair information and the second memory region information in the queue pair context of the first queue pair on the target computing node side. This represents associating the first queue pair and the second queue pair. Similarly, on the first storage node side, the first queue pair information of the first queue pair and the second memory region information of the second memory region are also maintained in the queue pair context of the second queue pair on the first storage node side.

[0114] Associating the first queue pair with the second queue pair signifies the completion of an RDMA connection between the target actual thread and the first end thread corresponding to that target actual thread on the first storage node. Subsequently, the target actual thread and the first end thread corresponding to that target actual thread on the first storage node can perform data transfer using the first queue pair and the second queue pair. For example, the target actual thread can write data from the target actual thread to the first storage node through the first end thread corresponding to that target actual thread, and the target actual thread can obtain data from the first end thread corresponding to that target actual thread for performing computational tasks.

[0115] Step 230: Using the first paired thread corresponding to each target actual thread as a relay, data is transferred between the target actual thread on the target computing node and the multiple actual threads running on the first storage node based on the RDMA connection between each target actual thread and the corresponding first paired thread.

[0116] For the first storage node, both the proxy threads enabled within it and the actual threads running on it are internal threads. Each proxy thread on the first storage node can communicate with each other individually with the actual threads running on it. Therefore, for a target actual thread on a target computing node, its corresponding first peer thread on the first storage node can act as an intermediary to enable data transfer between the target actual thread and multiple actual threads running on the first storage node.

[0117] For example, such as Figure 4 As shown, if compute node P1 is taken as the target compute node, storage node Q1 is a first storage node, and four actual threads run on compute node P1, namely actual threads t1-1 to t1-4, then QP1, QP2, QP3, and QP4 are the first queue pairs created by actual threads t1-1 to t1-4, respectively. It is worth mentioning that... Figure 4 The proxy thread shown for storage node Q1 is merely an example. The cluster includes other compute nodes; therefore, the proxy threads enabled on storage node Q1 are not limited to... Figure 4 As shown in the image.

[0118] Eight actual threads run on storage node Q1, namely actual threads t2-1 to t2-8. Additionally, proxy threads enabled on compute node P1 include proxy threads d1 to d4. According to the method of this application, proxy thread d1 serves as the first peer thread corresponding to actual thread t1-1 on compute node P1, proxy thread d2 serves as the first peer thread corresponding to actual thread t1-2 on compute node P1, proxy thread d3 serves as the first peer thread corresponding to actual thread t1-3 on compute node P1, and proxy thread d4 serves as the first peer thread corresponding to actual thread t1-4 on compute node P1.

[0119] After establishing an RDMA connection between proxy thread d1 and the actual thread t1-1, proxy thread d1 can act as a relay between actual thread t1-1 and actual threads t2-1 to t2-8, respectively, enabling data transfer between actual thread t1-1 and these threads. For example, actual thread t1-1 can read data from actual threads t2-1 to t2-8 through proxy thread d1. Similarly, proxy thread d2 can act as a relay between actual thread t1-2 and actual threads t2-1 to t2-8, proxy thread d3 can act as a relay between actual thread t1-3 and these threads, and proxy thread d4 can act as a relay between actual thread t1-4 and these threads.

[0120] Figure 5 This is a schematic diagram illustrating the communication relationship between compute nodes and storage nodes in a cluster, according to another embodiment of this application. Figure 5 The diagram illustrates the communication relationship between two storage nodes and the same compute node, compared to... Figure 4 , Figure 5 The newly added illustration shows the communication relationship between storage node Q2 and computing node P1. Storage node Q2 is also the first storage node in this application. The actual threads running on storage node Q2 include actual threads t3-1 to t3-8, and the proxy threads enabled on storage node Q2 include proxy threads d5 to d8. It is worth mentioning that... Figure 5 The proxy threads shown for storage node Q2 are merely illustrative examples. The cluster includes other compute nodes; therefore, the proxy threads enabled on storage node Q2 are not limited to those shown in the example. Figure 5 As shown in the image.

[0121] According to the method of this application, proxy thread d5 can be used as the first peer thread corresponding to actual thread t1-1 on computing node P1, proxy thread d6 can be used as the first peer thread corresponding to actual thread t1-2 on computing node P1, proxy thread d7 can be used as the first peer thread corresponding to actual thread t1-3 on computing node P1, and proxy thread d8 can be used as the first peer thread corresponding to actual thread t1-4 on computing node P1.

[0122] Correspondingly, after establishing the RDMA connection between the proxy thread d5 and the actual thread t1-1, the proxy thread d5 can act as a relay between the actual thread t1-1 and actual threads t3-1 to t3-8, respectively, enabling data transmission between the actual threads t1-1 and t3-1 to t3-8. For example, the actual thread t1-1 can read data from actual threads t3-1 to t3-8 through the proxy thread d5. Similarly, the proxy thread d6 can act as a relay between the actual threads t1-2 and t3-1 to t3-8, the proxy thread d7 can act as a relay between the actual threads t1-3 and t3-1 to t3-8, and the proxy thread d8 can act as a relay between the actual threads t1-4 and t3-1 to t3-8.

[0123] In related technologies, in a cluster, compute nodes and storage nodes establish RDMA connections in a full mesh mode. That is, an actual thread on a compute node will establish an RDMA connection with each actual thread on each storage node. Figure 6This is a diagram illustrating the establishment of an RDMA connection between a compute node and a storage node in a fully connected mode, as shown below. Figure 6 As shown, the actual thread t1-1 on compute node P1 needs to establish RDMA connections with the actual threads t2-1 to t2-y (where y is a positive integer greater than 1 and represents the total number of actual threads running on storage node Q1) on storage node Q1. Similarly, the actual thread t1-2 on compute node P1 needs to establish RDMA connections with the actual threads t2-1 to t2-y on storage node Q1, and the actual thread t1-2x (where x is a positive integer greater than 1 and represents the total number of actual threads running on compute node P1) on compute node P1 needs to establish RDMA connections with the actual threads t2-1 to t2-y on storage node Q1.

[0124] Assuming there are 120 storage nodes in the cluster, with 12 threads running on each storage node and 8 threads running on each compute node, then in extreme cases, a compute node needs to establish a total of 8 × 12 × 120 = 11520 RDMA connections in the cluster. Each RDMA connection on a compute node requires the creation of a queue pair. Therefore, for a compute node, in full-connection mode, it needs to create a maximum of 8 × 12 × 120 = 11520 queue pairs.

[0125] However, if the method of this application is followed, in extreme cases, the total number of RDMA connections that a computing node needs to establish in the cluster is 8 × 120 = 960. For a computing node, a maximum of 8 × 120 = 960 queue pairs need to be created.

[0126] Therefore, it can be seen that, according to the method provided in this application, the number of RDMA connections established between a computing node and a storage node can be equal to the number of actual threads running on the computing node. In this way, compared with the fully connected mode, the number of RDMA connections established is greatly reduced, and correspondingly, the number of queue pairs that need to be created is also greatly reduced. Since creating a queue pair requires maintaining the queue pair context in memory, the method of this application can also greatly reduce the number of queue pair contexts maintained, thereby avoiding excessive queue pair contexts occupying too much memory space.

[0127] In this application, by utilizing the proxy thread on the first storage node in the cluster, one of the multiple proxy threads enabled on the first storage node is selected as the first end thread corresponding to the target actual thread on the target compute node. Moreover, the proxy threads corresponding to different target actual threads on the target compute node are different. Then, RDMA connections are established between each target actual thread on the target compute node and its corresponding first end thread. In this way, the number of RDMA connections established between a target compute node and a first storage node is equal to the number of target actual threads running on the target compute node. Compared with the fully connected mode, the number of RDMA connections established is greatly reduced. Correspondingly, the number of queue pairs that need to be created is also greatly reduced, as is the number of queue pair contexts that need to be maintained. This solves the problem of memory space occupation on nodes, especially compute nodes, caused by the need to maintain a large number of queue pair contexts.

[0128] In particular, as the cluster size expands and the number of compute and storage nodes increases further, the solution proposed in this application can reduce the rate at which the number of RDMA connections grows with the increase in cluster size, thereby saving hardware resources on compute and storage nodes while maintaining the overall computing performance of the cluster.

[0129] Moreover, the proxy threads enabled in the first storage node and the actual threads running on the first storage node are both internal threads of the first storage node. Each proxy thread on the first storage node can communicate with the actual threads running on the first storage node. Thus, using the first peer thread corresponding to each target actual thread as a relay, and based on the RDMA connection between each target actual thread and its corresponding first peer thread, data transmission is performed between the target actual thread on the target computing node and the multiple actual threads running on the first storage node. This achieves the goal of ensuring data transmission between the target actual thread on the target computing node and each actual thread on the first storage node using fewer RDMA connections.

[0130] In related technologies, to address the issue of a large number of RDMA connections when establishing RDMA connections between compute nodes and storage nodes in a fully connected mode within a cluster, a shared mode, namely shared RDMA connections, has been proposed. In the shared mode, for a compute node, only one actual thread on the compute node accesses an actual thread on a storage node. In other words, all actual threads on a storage node share the RDMA connection with one actual thread on a compute node.

[0131] Figure 7 An exemplary diagram illustrates the communication relationship between compute nodes and storage nodes in a cluster in shared mode, such as... Figure 7As shown, an RDMA connection is established between storage node Q1 and the actual thread t1-1 on compute node P1, an RDMA connection is established between storage node Q2 and the actual thread t1-2 on compute node P1, and an RDMA connection is established between storage node Q3 and the actual thread t1-3 on compute node P1. This mode significantly reduces the number of established RDMA connections and the number of queue pairs created.

[0132] Taking a cluster with a total of 120 storage nodes, each storage node running 12 actual threads and each compute node running 8 actual threads as an example, for a compute node, in extreme cases, the total number of RDMA connections that it needs to establish in the cluster is: 8 × 120 = 960.

[0133] However, in shared mode, since multiple actual threads on a storage node share an RDMA connection with a single actual thread on a compute node, a special protocol field is needed to identify which actual thread on the storage node is connecting to which actual thread on the compute node. This approach is opaque to the RDMA library, requiring the library to parse this special protocol field and then perform different processing based on its value. Furthermore, locks are needed to control contention for shared resources. This approach introduces business complexity and performance overhead.

[0134] In the solution provided in this application, an RDMA connection is established between a proxy process and an actual thread on the compute node by enabling a proxy thread on the storage node. The proxy process then acts as a communication intermediary between the actual threads on the storage node and the actual threads on the compute node. This eliminates the need for locks to control resource contention and avoids the need for additional protocol fields. Furthermore, since both the proxy process and the actual threads on the storage node reside on the storage node, the switching between them involves different threads within the node. This switching time is minimal, typically on the order of microseconds (µs). Therefore, the method described in this application still ensures high-speed data transmission between the storage node and the compute node.

[0135] As described above, the compute nodes in the cluster are primarily responsible for executing computation tasks, while the storage nodes are primarily responsible for data storage. When a compute node needs to execute a computation task, it needs to retrieve the data required for that task from the storage nodes. For example, if a target thread on a target compute node needs to retrieve first target data from a first thread on a first storage node (this first thread can be any thread on the first storage node), the target thread on the target compute node can proceed as follows: Figure 8 The process shown obtains the first target data, including:

[0136] Step 810: The target thread allocates a first buffer in the corresponding first memory region and obtains the first access information of the first buffer.

[0137] The first memory region corresponding to the target actual thread refers to the first memory region registered and obtained by the target actual thread during the establishment of the RDMA connection between the target actual thread and the first peer thread corresponding to the target actual thread on the first storage node. The first buffer is a sub-memory region within this first memory region. The first buffer is used to temporarily store the first target data from the first actual thread on the first storage node. In other words, the first buffer is a sub-region within the first memory region pre-registered by the first actual thread with the target actual thread on the target computing node. Similarly, the first access information of the first buffer includes the virtual address of the first buffer and the access key of the first buffer.

[0138] Step 820: Obtain the first message sent by the first peer thread corresponding to the target actual thread; the first message includes the second access information of the first target data in the second memory area; the first target data is the data to be transmitted from the first actual thread on the first storage node to the target actual thread; the first target data is stored in the second memory area by the first actual thread through the first peer thread corresponding to the target actual thread.

[0139] When the first actual thread on the first storage node needs to transmit the first target data to the target actual thread (assumed to be the target actual thread A1) on the target computing node, the first actual thread needs to send the first target data to the proxy thread that has created an RDMA connection with the target actual thread A1 (i.e., the first peer thread corresponding to the target actual thread A1, assumed to be the proxy thread B1). The proxy thread B1 allocates a buffer in the second memory area that has been registered and obtained, and temporarily stores the first target data in the allocated buffer.

[0140] Similarly, the second access information includes the virtual address of the buffer in the second memory region where the first target data is located and the access key of the buffer in the second memory region where the first target data is located.

[0141] The first message sent by the first peer thread corresponding to the target actual thread to the target computing node (or to the target actual thread on the target computing node) is equivalent to informing the target computing node of the location of the first target data requested to be read in the second memory area, and granting access authorization to the target computing node so that RDMA read operations can be performed subsequently.

[0142] Step 830: The target actual thread sends the data reading task to the sending queue in the first queue pair based on the first access information and the second access information.

[0143] The target thread can encapsulate the first and second access information into a data reading task, meaning the data reading task carries both the first and second access information. This data reading task is then sent to the send queue in the first queue pair to await execution.

[0144] Step 840: After the network card in the target computing node polls for a data read request in the transmission queue, it performs an RDMA read operation to temporarily store the first target data in the first buffer.

[0145] The network interface card (NIC) in the target compute node supports communication with NICs in other nodes using the RDMA protocol. The NIC in the target compute node can poll the send queue in the first queue pair. When a data read task is found in the send queue of the first queue pair, the NIC can parse the data read task. In this way, the virtual address and access key in the second access information carried by the data read task can be obtained. Based on the virtual address and access key carried by the second access information obtained by parsing, the NIC sends a data read request to the NIC in the first storage node.

[0146] After receiving a data read request, the network interface card (NIC) in the first storage node obtains the virtual address and access key from the data read request, and converts the virtual address in the data read request into a physical address on the first storage node side. The parsed physical address is the physical address of the first target data in the second memory area. In this way, the NIC in the first storage node can read the first target data from the second memory area according to the converted physical address and the corresponding access key, encapsulate the first target data into a reply data packet, and send the reply data packet to the NIC in the target computing node through the physical link.

[0147] When the network interface card (NIC) in the target compute node receives the reply data packet, it parses the reply data packet to obtain the first target data, and reads the virtual address and access key from the first access information in the task according to the data polled from the sending queue in the first queue pair. The first target data is then stored in the first buffer. Through this process, the RDMA read operation is implemented, transferring the first target data, originally stored in the memory of the first storage node, to the target compute node and storing it in the target compute node's memory.

[0148] Step 850: The target actual thread retrieves the first target data from the first buffer.

[0149] After temporarily storing the first target data in the first buffer, the target actual thread can retrieve the first target data from the first buffer accordingly. For example, if the target actual thread can use the first target data to perform a computation task. In this case, after step 850, the method further includes:

[0150] Step 860: The target actual thread performs the computation task using the first target data.

[0151] Through the above process, an RDMA connection is established between the target actual thread on the target computing node and the first pair of threads on the first storage node, or a queue pair created for the RDMA connection, and the data stored on the first storage node is transferred to the target actual thread on the target computing node by means of RDMA operations.

[0152] In addition to transferring the data stored on the first storage node to the target actual thread on the target computing node, the target computing node may need to send the data to be stored to the actual thread of the first storage node. In this case, RDMA write operations can be used to transfer data from the target actual thread of the target computing node to the actual thread of the first storage node.

[0153] Taking the example of a target actual thread on a target computing node requesting a second actual thread on a first storage node to store second target data, this second actual thread can be any actual thread on the first storage node. For example, it could be the first actual thread mentioned above, or it could be another actual thread different from the first actual thread. Figure 9 As shown, the process includes:

[0154] Step 910: The target actual thread temporarily stores the second target data in the first memory area and obtains the third access information of the second target data in the first memory area.

[0155] The first memory region refers to the memory region registered by the target actual thread during the process of establishing an RDMA connection between the target actual thread and the first paired thread corresponding to the target actual thread.

[0156] The third access information includes the virtual address of the second target data in the first memory region, and an access key for the memory space where the second target data resides in the first memory region. The second target data may be data generated by the target computing node during the execution of a computing task, such as the result of the task execution, but is not limited to this.

[0157] Step 920: Receive a second message sent by the first peer thread corresponding to the target actual thread. The second message includes fourth access information of the second buffer. The second buffer is a buffer partitioned by the first peer thread corresponding to the target actual thread in the second memory area.

[0158] The second buffer is essentially a sub-memory region within the registered second memory area, allocated by the first counterpart thread corresponding to the actual target thread, to temporarily store the second target data from the target computing node side. Similarly, the fourth access information includes the virtual address of the second buffer and the access key for the second buffer.

[0159] Step 930: The target thread sends data to the send queue in the first queue pair based on the third and fourth access information.

[0160] The target thread can encapsulate the third and fourth access information into a data write task, meaning the data write task carries the third and fourth access information. This data write task is then sent to the send queue in the first queue pair to await execution.

[0161] Step 940: After the network card in the target computing node polls for the data write task in the sending queue, it performs an RDMA write operation to write the second target data into the second buffer, so that the first peer thread corresponding to the target actual thread sends the second target data to the second actual thread.

[0162] When the network interface card (NIC) in the target compute node detects a data write task in the send queue of the first queue pair, it can parse the data write task. This allows it to obtain the virtual address and access key from the third access information carried by the data write task. Based on this information, the NIC converts the virtual address to the corresponding physical address (i.e., the physical address of the memory space where the second target data resides in the first memory region). Then, according to the parsed physical address, it retrieves the second target data from the first memory region and encapsulates the second target data and the fourth access information carried by the data write task into a data packet. The NIC in the target compute node then sends this data packet to the first storage node via a physical link.

[0163] After the network card in the first storage node receives the data packet, it parses the data packet to obtain the virtual address and access key in the fourth access information carried in the data packet, converts the virtual address in the fourth access information into the corresponding physical address (i.e. the physical address of the second buffer), and temporarily stores the second target data parsed from the data packet into the second buffer.

[0164] After the first end thread corresponding to the target actual thread obtains the second target data from the second buffer, the first end thread corresponding to the target actual thread can send the second target data to the second actual thread. After that, the second actual thread can persist the second target data.

[0165] Through the above process, the second target data, originally located in the memory of the target computing node, is transferred to the first storage node and stored in the memory of the first storage node via RDMA write operation.

[0166] In addition to transferring the data to be stored to the first storage node via RDMA write operations, the target thread on the target compute node can also do so as follows: Figure 10 The process shown involves transmitting the data to be stored to the first storage node. Taking the example of a target thread on the target storage node needing to transmit third target data to be stored to a third thread on the first storage node, the third thread can be any thread on the first storage node, and can be the same as or different from the first or second thread mentioned above. For example... Figure 10 As shown, it includes:

[0167] Step 1010: The target actual thread temporarily stores the third target data in the first memory area, and obtains the fifth access information of the third target data in the first memory area.

[0168] The third target data refers to the data that the target actual thread on the target storage node requests to be written by the third actual thread on the first storage node. The third target data can be data generated by the target actual thread on the target storage node during the execution of a computation task, such as the execution result of the computation task. Similarly, the fifth access information includes the virtual address of the memory space where the third target data resides in the first memory region and the corresponding access key.

[0169] Step 1020: The target thread issues the first sending task to the sending queue in the first queue pair based on the fifth access information.

[0170] The target thread can package the virtual address and corresponding access key in the fifth access information into a first sending task, and send the first sending task to the sending queue in the first queue pair.

[0171] Step 1030: After the network card in the target computing node polls the first transmission task in the transmission queue, it performs an RDMA transmission operation so that after the network card in the first storage node receives the third target data, it consumes the first reception task in the reception queue of the second queue pair and caches the third target data in the third buffer corresponding to the sixth access information indicated by the reception task. This allows the first peer thread corresponding to the target actual thread to send the third target data to the third actual thread. The third buffer is a buffer partitioned by the first peer thread corresponding to the target actual thread in the second memory area. The first reception task is generated by the first peer thread corresponding to the target actual thread based on the sixth access information of the third buffer.

[0172] During the data transmission preparation phase, in addition to the target thread issuing a first transmission task to the transmission queue in the first queue pair based on the fifth access information, the first peer thread corresponding to the target thread on the first storage node side allocates a third buffer in the second memory area. This third buffer is used to temporarily store the third target data from the target thread. Furthermore, the first peer thread corresponding to the target thread on the first storage node side can generate a first receiving task based on the sixth access information of the third buffer (similarly, this sixth access information includes the virtual address and access key of the third buffer), and issue the first receiving task to the receiving queue in the second queue pair.

[0173] After the network interface card (NIC) in the target compute node polls the first transmission task in the transmission queue, it parses the fifth access information in the first transmission task to obtain the virtual address carried by the fifth access information. It then converts the virtual address into a corresponding physical address (i.e., the physical address of the third target data in the memory space of the first memory region). Based on the converted physical address, it retrieves the third target data from the first memory region. Afterward, the NIC in the target compute node encapsulates the third target data into a first transmission data packet and sends this first transmission data packet to the first storage node via the physical link.

[0174] After the network card in the first storage node receives the first transmit data packet, the network card in the first storage node consumes the first receive task in the receive queue of the second queue pair, obtains the virtual address and access key of the third buffer from the sixth access information carried by the first receive task, converts the virtual address of the third buffer to obtain the physical address of the third buffer, and stores the third target data in the received first transmit data packet into the third buffer according to the physical address and access key of the third buffer.

[0175] Afterwards, the first end thread corresponding to the target actual thread can obtain the third target data from the third buffer and send the third target data to the third actual thread. Upon receiving the third target data, the third actual thread can persist the third target data.

[0176] Through the above process, based on the establishment of an RDMA connection between the target actual thread on the target computing node and the first pair of threads corresponding to the target actual thread on the first storage node, the third target data from the target actual thread on the target computing node is transmitted to the third actual thread on the first storage node by means of the RDMA transmission operation, according to the first queue pair created by the target actual thread and the second queue pair created by the first pair of threads corresponding to the target actual thread on the first storage node.

[0177] In addition to obtaining the necessary data from the actual thread on the first storage node via RDMA read operations, the target actual thread on the target compute node can also do so as follows: Figure 11 The process shown involves receiving data from the actual thread on the first storage node. Taking the example that the target actual thread on the target storage node needs to receive fourth target data from the fourth actual thread on the first storage node, the fourth actual thread can be any actual thread on the first storage node, and can be the same as or different from one of the first, second, or third actual threads mentioned above. For example... Figure 11 As shown, it includes:

[0178] Step 1110: The target thread allocates a fourth buffer in the first memory area and obtains the seventh access information of the fourth buffer.

[0179] The fourth buffer is a sub-memory region partitioned from the first memory region, used to temporarily store the fourth target data from the fourth actual thread on the first storage node. Similarly, the seventh access information includes the virtual address of the fourth buffer and the corresponding access key.

[0180] Step 1120: The target thread sends the second receiving task to the receiving queue in the first queue pair based on the seventh access information.

[0181] The target thread can package the virtual address and corresponding access key in the seventh access information into a second receiving task, and send the second receiving task to the receiving queue in the first queue pair.

[0182] Step 1130: If the network card in the target computing node receives the fourth target data sent by the network card in the first storage node, consume the second receiving task in the receiving queue of the first queue pair, and cache the fourth target data in the fourth buffer according to the seventh access information, so that the target actual thread can obtain the fourth target data from the fourth buffer.

[0183] In this process, after the network card in the first storage node polls the second sending task in the sending queue of the second queue pair, it reads the fourth target data from the fifth buffer and sends the fourth target data to the network card in the target computing node. The second sending task is generated by the first end thread corresponding to the target actual thread based on the eighth access information of the fifth buffer in the second memory area. The fourth target data is temporarily stored in the fifth buffer by the first end thread corresponding to the target actual thread after it obtains the fourth target data from the fourth actual thread.

[0184] During the data transmission preparation phase, in addition to the target actual thread issuing the second receiving task to the receiving queue in the first queue pair according to the seventh access information, the target actual thread corresponding to the first end thread on the first storage node side temporarily stores the fourth target data from the fourth actual thread into the fifth buffer in the second memory area, and obtains the eighth access information of the fifth buffer. The eighth access information includes the virtual address of the fifth buffer and the corresponding access key.

[0185] The target thread, corresponding to the first peer thread on the first storage node side, can encapsulate a second sending task based on the eighth access information of the fifth buffer and send the second sending task to the sending queue in the second queue pair. After the network card in the first storage node polls the second sending task in the sending queue of the second queue pair, it obtains the virtual address and corresponding access key of the fifth buffer from the eighth access information carried by the second sending task, converts the virtual address of the fifth buffer to the physical address of the fifth buffer, and then obtains the fourth target data from the fifth buffer according to the physical address of the fifth buffer and the corresponding access key. The fourth target data is then encapsulated into a second sending data packet and sent to the target computing node through the physical link.

[0186] After the network interface card (NIC) in the target computing node receives the second data packet, it parses it to obtain the fourth target data and consumes the second receiving task in the receiving queue of the first queue. It then converts the virtual address in the seventh access information carried by the second receiving task into a physical address to obtain the physical address of the fourth buffer. The NIC in the target computing node then caches the fourth target data into the fourth buffer according to the physical address of the fourth buffer and the corresponding access key.

[0187] Afterwards, the target thread retrieves the fourth target data from the fourth buffer and can then perform subsequent processing on the fourth target data, such as using the fourth target data to perform computational tasks.

[0188] Through the above process, based on the establishment of an RDMA connection between the target actual thread on the target computing node and the first paired thread corresponding to the target actual thread on the first storage node, and according to the first queue pair created by the target actual thread and the second queue pair created by the first paired thread corresponding to the target actual thread on the first storage node, the target actual thread on the target computing node can obtain the fourth target data from the fourth actual thread on the first storage node, thus enabling the actual thread on the first storage node to transmit data to the target actual thread on the target computing node.

[0189] As described above, the cluster includes multiple storage nodes, and each storage node may have a proxy thread enabled. In this case, the compute nodes in the cluster can perform data transfer with the storage nodes that have enabled proxy threads, as described in the above embodiment. In some embodiments, there may be storage nodes in the cluster that have not enabled proxy threads. For ease of distinction, these storage nodes are referred to as second storage nodes. The target compute node in the cluster can perform data transfer according to... Figure 12 The process shown involves data transfer with the actual threads on each of the second storage nodes, such as... Figure 12 As shown, the method also includes:

[0190] Step 1210: Among the multiple target actual threads running on the target computing node, determine one target actual thread for the second storage node as the second peer thread corresponding to the second storage node.

[0191] In other words, for each second storage node, one of the multiple target actual threads running on the target computing node is selected as the second peer thread corresponding to each second storage node. In some embodiments, the second peer threads determined for different second storage nodes on the same target computing node can be different; that is, one target actual thread on the target computing node serves as only one second peer thread corresponding to a second storage node.

[0192] In other embodiments, considering that the number of target actual threads running on a target compute node may be less than the number of second storage nodes in the cluster, a target actual thread on a target compute node can also serve as a second peer thread corresponding to at least two second storage nodes. In some embodiments, to avoid large differences in data transmission pressure between different actual threads on the same target compute node, a first number can be set. This first number limits the number of second storage nodes for which a target actual thread on a target compute node serves as a corresponding second peer thread. Assuming the first number is W, a target actual thread on a target compute node can serve as a second peer thread corresponding to at most W second storage nodes.

[0193] In some embodiments, the correspondence between the target actual thread on each computing node and each second storage node can be pre-defined. The correspondence can be one target actual thread corresponding to one second storage node, and different second storage nodes correspond to different target actual threads on the same computing node. The correspondence can also be set according to the first number as described above, ensuring that the number of second storage nodes corresponding to one target actual thread does not exceed the first number in the correspondence.

[0194] In some embodiments, step 1210 includes steps B1 to B3, which are detailed below:

[0195] Step B1: Obtain the node information of the second storage node; the node information includes the node identifier information of the second storage node. The node identifier information refers to the identifier used to uniquely identify the second storage node. Different second storage nodes have different node identifier information.

[0196] Step B2: Perform hash calculation based on the node identifier information of the second storage node to obtain the target hash value.

[0197] In some embodiments, the node identification information of the second storage node can be hashed to obtain the target hash value. It is understood that since the node identification information of different storage nodes in the cluster is different, hashing the node identification information of different second storage nodes will also yield different target hash values.

[0198] In other embodiments, the total number of actual target threads on the current target computing node can also be obtained, assuming it is a second number. Then, the node identification information of the second storage node is used to perform a hash modulo operation on the second number of actual target threads on the current target computing node to obtain the target hash value. This process can be described as follows:

[0199] TY=hash(dn_id)%d_thread_num;

[0200] Wherein, dn_id refers to the node identification information of the second storage node, hash(dn_id) represents the hash calculation result of the node identification information of the second storage node, d_thread_num represents the number of actual target threads on the current target computing node. It can be understood that the number of actual target threads on different computing nodes may be different, and TY refers to the target hash value.

[0201] In the above embodiment, the target hash value is obtained by performing a hash modulo operation on the node identification information of the second storage node and the second number of target actual threads on the current target computing node. Since the node identification information of different second storage nodes is different, but the second number of target actual threads on the same target computing node is fixed, different second storage nodes can be basically evenly distributed to different target actual threads on the same target computing node.

[0202] Step B3: Based on the mapping relationship between hash values ​​and actual threads, the target actual thread corresponding to the target hash value on the target computing node is used as the second peer thread corresponding to the second storage node.

[0203] In some embodiments, a mapping relationship between a hash value and an actual thread on a target computing node can be set for each target computing node. In the mapping relationship between a hash value and an actual thread set for a target computing node, there is a one-to-one correspondence between the hash value and the actual thread. In this way, in step B3, the target actual thread corresponding to the target hash value on the current target computing node can be used as the second peer thread corresponding to the second storage node.

[0204] Step 1220: Establish RDMA connections between the second peer thread corresponding to the second storage node and each actual thread on the second storage node.

[0205] In other words, each actual thread on the second storage node establishes an RDMA connection with the corresponding second peer thread on the second storage node. The process of establishing the RDMA connection between the actual thread on the second storage node and the corresponding second peer thread on the second storage node is similar to the process described above. Figure 3 The process shown is similar and will not be repeated here.

[0206] After establishing an RDMA connection between the second peer thread corresponding to the second storage node and the actual thread on the second storage node, then in the data transmission direction from the target compute node to the second storage node, the second peer thread corresponding to the second storage node serves as the sole data transmission thread from the target compute node to the second storage node. Similarly, in the data transmission direction from the second storage node to the target compute node, the second peer thread corresponding to the second storage node serves as the sole data reception thread from the second storage node to the target compute node.

[0207] Figure 13 This is a schematic diagram illustrating the communication relationship between a compute node and a second storage node in a cluster, according to an embodiment of this application. Figure 13 In the example, storage nodes P3 and P4 are both second storage nodes where the agent thread is not enabled. Figure 13 As shown, following the above process, the actual thread t1-1 on compute node P1 can be identified as the second paired thread corresponding to storage node P4. Then, RDMA connections are established between the actual thread t1-1 on compute node P1 and each actual thread (i.e., actual thread t4-1 to actual thread t4-4) on storage node P4. Correspondingly, the other actual threads on the compute node (i.e., actual threads t1-2 to actual threads t1-4) all use the RDMA connection between actual thread t1-1 and each actual thread on storage node P4 to perform data transmission. This is equivalent to using the second paired thread corresponding to storage node P4 on compute node P1 as an intermediary for data transmission between the other actual threads on compute node P1 and the actual threads on storage node P4.

[0208] exist Figure 13In this process, the actual thread t1-2 on compute node P1 is designated as the second paired thread corresponding to storage node P3, and RDMA connections are established between actual thread t1-2 on compute node P1 and each actual thread (i.e., actual threads t3-1 to t3-4) on storage node P3. Correspondingly, other actual threads on the compute node (i.e., actual threads t1-1, t1-3 to t1-4) all use the RDMA connections between actual thread t1-2 and each actual thread on storage node P3 for data transmission. This is equivalent to using the second paired thread corresponding to storage node P3 on compute node P1 as an intermediary for data transmission between actual threads t1-1, t1-3 to t1-4 on compute node P1 and the actual threads on storage node P3.

[0209] exist Figure 13 In the illustrated embodiment, a management node can be set up in the cluster. This management node can uniformly manage the compute nodes and storage nodes in the cluster. The management node can generate node identification information for each storage node to ensure that the node identification information of different storage nodes is different. The node identification information of each storage node is stored in the node database. Then, the target compute node can obtain the node identification information of each second storage node from the node database.

[0210] Step 1230: Data transfer is performed between the target computing node and each actual thread on the second storage node through the RDMA connection between the second peer thread corresponding to the second storage node and the actual thread on the second storage node.

[0211] Data transfer occurs between the actual threads on the target computing node and the second storage node, including data transfer from the actual threads on the target computing node to the actual threads on the second storage node, and data transfer from the actual threads on the second storage node to the actual threads on the target computing node.

[0212] Taking the example of a first target actual thread on a target computing node needing to transmit fifth target data to a reference actual thread on a second storage node, the first target actual thread can be any target actual thread on the target computing node, and similarly, the reference actual thread can be any actual thread on the second storage node; step 1230 may include: if the first target actual thread is not the second peer thread corresponding to the second storage node, sending the fifth target data to the second peer thread corresponding to the second storage node; the fifth target data is the data transmitted by the first target actual thread to the reference actual thread in the second storage node; and transmitting the fifth target data to the reference actual thread in the second storage node according to the RDMA connection between the second peer thread corresponding to the second storage node and the reference actual thread.

[0213] Specifically, by leveraging the RDMA connection between the second peer thread corresponding to the second storage node and the reference actual thread, the second peer thread corresponding to the second storage node can perform RDMA write operations (i.e., as mentioned above). Figure 9 (corresponding to the process in the embodiment), or the RDMA transmission operation mentioned above (i.e., the process mentioned above). Figure 10 (corresponding to the process shown in the embodiment), the fifth target data is transmitted to the reference actual thread in the second storage node.

[0214] Of course, if the first target actual thread is the second peer thread corresponding to the second storage node, the fifth target data can be directly transmitted to the reference actual thread in the second storage node based on the RDMA connection between the second peer thread corresponding to the second storage node and the reference actual thread, without needing to forward the fifth target data to other actual threads on the target computing node in advance.

[0215] Similarly, taking the first target actual thread on the target computing node receiving the sixth target data from the reference actual thread on the second storage node as an example, step 1230 includes: according to the RDMA connection between the second peer thread corresponding to the second storage node and the reference actual thread, the second peer thread corresponding to the second storage node obtains the sixth target data from the reference actual thread in the second storage node; if the first target actual thread is not the second peer thread corresponding to the second storage node, the second peer thread corresponding to the second storage node sends the sixth target data to the first target actual thread.

[0216] The process by which the second thread corresponding to the second storage node obtains the sixth target data from the reference actual thread in the second storage node can be as described above. Figure 8 The read operation is performed using RDMA as shown in the corresponding embodiment, or it can be performed as described above. Figure 11 The process shown is similar; please refer to the description above for details, which will not be repeated here.

[0217] Still using a cluster with a total of 120 storage nodes, each storage node running 12 actual threads and each compute node running 8 actual threads as an example, and assuming that all storage nodes in the cluster are secondary storage nodes, for a compute node, according to... Figure 12 In a corresponding implementation, in an extreme case, the total number of RDMA connections that need to be established in the cluster is: 12 × 120 = 960.

[0218] As can be seen, the method in this embodiment can also effectively reduce the number of RDMA connections. Correspondingly, even if only some storage nodes in the cluster are secondary storage nodes, this goal can still be achieved. Moreover, this method does not involve thread switching through the proxy thread, so there will be no delay caused by thread switching through the proxy thread.

[0219] In some embodiments, if none of the storage nodes in the cluster have agent threads enabled, each target compute node can perform operations according to... Figure 12 The process shown involves data transfer with each storage node, reducing the overall number of RDMA connections.

[0220] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments described in the above embodiments of this application.

[0221] Figure 14 This is a block diagram of a data transmission apparatus according to an embodiment of the present application. The data transmission apparatus can be applied to a target computing node in a cluster. The target computing node refers to any computing node in the cluster, such as... Figure 14 As shown, the data transmission device includes: a selection module 1410, used to select one of multiple proxy threads enabled on a first storage node for each target actual thread running on a target computing node, as the first peer thread corresponding to the target actual thread; different target actual threads correspond to different proxy threads; the first storage node is any storage node in the cluster that enables proxy threads; a connection establishment module 1420, used to establish a remote direct memory access (RDMA) connection between each target actual thread and the corresponding first peer thread on the first storage node; and a data transmission module 1430, used to perform data transmission between the target actual thread on the target computing node and multiple actual threads running on the first storage node, based on the RDMA connection between each target actual thread and the corresponding first peer thread, using the first peer thread corresponding to each target actual thread as a relay.

[0222] In some embodiments, the connection establishment module 1420 is configured to perform the following processing for each target actual thread: the target actual thread registers a memory region and creates a first queue pair, obtaining the first memory region information of the registered first memory region and the first queue pair information of the first queue pair; based on the first memory region information and the first queue pair information, it exchanges information with the first end thread corresponding to the target actual thread to obtain the second queue pair information of the second queue pair and the second memory region information of the second memory region; the second queue pair is a queue pair created by the first end thread corresponding to the target actual thread, and the second memory region is a memory region registered on the first storage node by the first end thread corresponding to the target actual thread; based on the second queue pair information and the second memory region information, the first queue pair and the second queue pair are associated to establish an RDMA connection between the target actual thread and the corresponding first end thread on the first storage node.

[0223] In some embodiments, the data transmission module 1430 includes: a first partitioning unit, configured to partition a first buffer in a corresponding first memory region by a target actual thread and obtain first access information of the first buffer; a first acquisition unit, configured to acquire a first message sent by a first peer thread corresponding to the target actual thread; the first message includes second access information of first target data in a second memory region; the first target data is data to be transmitted from the first actual thread on the first storage node to the target actual thread; the first target data is stored in the second memory region by the first actual thread through the first peer thread corresponding to the target actual thread; a first sending unit, configured to send a data reading task to the sending queue in the first queue pair by the target actual thread according to the first access information and the second access information; a read operation unit, configured to perform an RDMA read operation by the network card in the target computing node after polling the data reading request in the sending queue, so as to temporarily store the first target data in the first buffer; and a first acquisition unit, configured to acquire the first target data from the first buffer by the target actual thread.

[0224] In some embodiments, the data transmission device further includes a task execution module for performing a computational task using the first target data by a target actual thread.

[0225] In other embodiments, the data transmission module 1430 includes: a first temporary storage unit, used for temporarily storing second target data in a first memory area by the target actual thread, and obtaining third access information of the second target data in the first memory area; a first receiving unit, used for receiving a second message sent by a first peer thread corresponding to the target actual thread, the second message including fourth access information of a second buffer; the second buffer is a buffer partitioned by the first peer thread corresponding to the target actual thread in the second memory area; a second sending unit, used for sending a data write task to the sending queue in the first queue pair by the target actual thread according to the third access information and the fourth access information; and a write operation unit, used for performing an RDMA write operation after the network card in the target computing node polls the data write task in the sending queue, so as to write the second target data into the second buffer, so that the first peer thread corresponding to the target actual thread sends the second target data to the second actual thread.

[0226] In other embodiments, the data transmission module 1430 includes: a second temporary storage unit, used to temporarily store third target data in a first memory area by the target actual thread to obtain fifth access information of the third target data in the first memory area; a third sending unit, used to send a first sending task to the sending queue in the first queue pair according to the fifth access information by the target actual thread; and a sending operation unit, used to execute an RDMA sending operation by the network card in the target computing node after polling the first sending task in the sending queue, so that after the network card in the first storage node receives the third target data, it consumes the first receiving task in the receiving queue in the second queue pair and caches the third target data in a third buffer corresponding to the sixth access information indicated by the receiving task, so that the first peer thread corresponding to the target actual thread sends the third target data to the third actual thread; the third buffer is a buffer partitioned by the first peer thread corresponding to the target actual thread in the second memory area, and the first receiving task is generated by the first peer thread corresponding to the target actual thread according to the sixth access information of the third buffer.

[0227] In other embodiments, the data transmission module 1430 includes: a second partitioning unit, configured to partition a fourth buffer in a first memory region by the target actual thread and obtain the seventh access information of the fourth buffer; a third sending unit, configured to send a second receiving task to the receiving queue in the first queue pair according to the seventh access information by the target actual thread; and a consumption unit, configured to consume the second receiving task in the receiving queue of the first queue pair if the network card in the target computing node receives the fourth target data sent by the network card in the first storage node, and cache the fourth target data in the fourth buffer according to the seventh access information, so that the target actual thread can obtain the fourth target data from the fourth buffer; wherein, after the network card in the first storage node polls the second sending task in the sending queue of the second queue pair, it reads the fourth target data from the fifth buffer and sends the fourth target data to the network card in the target computing node; the second sending task is generated by the first peer thread corresponding to the target actual thread according to the eighth access information of the fifth buffer in the second memory region, and the fourth target data is temporarily stored in the fifth buffer by the first peer thread corresponding to the target actual thread after obtaining the fourth target data from the fourth actual thread.

[0228] In some embodiments, the data transmission apparatus further includes: a proxy thread information acquisition module, configured to acquire proxy thread information allocated by the first storage node to the target computing node, the proxy thread information including proxy thread identifiers of each proxy thread allocated by the first storage node to the target computing node; an establishment module, configured to establish a first mapping relationship based on the proxy thread identifiers of each proxy thread in the proxy thread information and the thread identifiers of each target actual thread on the target computing node; in the first mapping relationship, one target actual thread corresponds to one proxy thread on the first storage node, and different target actual threads correspond to different proxy threads on the first storage node; correspondingly, the selection module is configured to: determine the first peer thread corresponding to the target actual thread based on the first mapping relationship.

[0229] In some embodiments, the cluster has a second storage node where the agent thread is not enabled; the data transmission device further includes: a determination module, configured to determine one target actual thread for the second storage node from among multiple target actual threads running on the target computing node, as the second peer thread corresponding to the second storage node; a first connection establishment module, configured to establish RDMA connections between the second peer thread corresponding to the second storage node and each actual thread on the second storage node; and a first transmission module, configured to perform data transmission between the target computing node and each actual thread on the second storage node through the RDMA connection between the second peer thread corresponding to the second storage node and the actual thread on the second storage node.

[0230] In some embodiments, the determining module includes: a node information acquisition unit, configured to acquire node information of a second storage node; the node information includes node identification information of the second storage node; a hash calculation unit, configured to perform hash calculation based on the node identification information of the second storage node to obtain a target hash value; and a second peer thread determining unit, configured to determine the target actual thread corresponding to the target hash value on the target computing node as the second peer thread corresponding to the second storage node based on the mapping relationship between the hash value and the actual thread.

[0231] In some embodiments, the first transmission module is configured to: perform the following processing on the first target actual thread on the target storage node: if the first target actual thread is not the second peer thread corresponding to the second storage node, send fifth target data to the second peer thread corresponding to the second storage node; the fifth target data is data transmitted by the first target actual thread to the reference actual thread in the second storage node; and transmit the fifth target data to the reference actual thread in the second storage node according to the RDMA connection between the second peer thread corresponding to the second storage node and the reference actual thread.

[0232] Figure 15 A schematic diagram of a computer system suitable for implementing the electronic devices of this application is shown. It should be noted that the electronic device can be a computing node in a cluster. Figure 15 The computer system 1500 of the illustrated electronic device is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this application. This electronic device can be used to perform the data transmission method provided in the application.

[0233] like Figure 15 As shown, the computer system 1500 includes a Central Processing Unit (CPU) 1501, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1502 or programs loaded from storage portion 1508 into Random Access Memory (RAM) 1503. The RAM 1503 also stores various programs and data required for system operation. The CPU 1501, ROM 1502, and RAM 1503 are interconnected via a bus 1504. An Input / Output (I / O) interface 1505 is also connected to the bus 1504.

[0234] The following components are connected to I / O interface 1505: an input section 1506 including a keyboard, mouse, etc.; an output section 1507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1508 including a hard disk, etc.; and a communication section 1509 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1509 performs communication processing via a network such as the Internet. A drive 1510 is also connected to I / O interface 1505 as needed. Removable media 1511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1510 as needed so that computer programs read from them can be installed into storage section 1508 as needed.

[0235] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1509, and / or installed from removable medium 1511. When the computer program is executed by central processing unit (CPU) 1501, it performs various functions defined in the system of this application.

[0236] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0237] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0238] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0239] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.

[0240] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal. It can be implemented wholly or partially using software, hardware (e.g., processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that functions as a whole.

[0241] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods of any of the above embodiments.

[0242] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0243] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0244] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0245] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data transmission method, characterized in that, The method is applied to a target computing node in a cluster, wherein the target computing node refers to any computing node in the cluster, and the method includes: For each target actual thread running on the target computing node, one of the multiple proxy threads enabled on the first storage node is selected as the first peer thread corresponding to the target actual thread; different target actual threads correspond to different proxy threads; the first storage node is any storage node in the cluster that enables proxy threads; Establish a remote direct memory access (RDMA) connection between each of the target actual threads and the corresponding first peer thread on the first storage node; Using the first peer thread corresponding to each of the target actual threads as a relay, data is transmitted between the target actual thread on the target computing node and the multiple actual threads running on the first storage node based on the RDMA connection between each of the target actual threads and the corresponding first peer thread.

2. The method according to claim 1, characterized in that, The establishment of a remote direct memory access (RDMA) connection between each of the target actual threads and the corresponding first peer thread on the first storage node includes: The following processing is performed on each of the target actual threads: The target actual thread registers the memory region and creates a first queue pair, obtaining the first memory region information of the registered first memory region and the first queue pair information of the first queue pair. Based on the first memory region information and the first queue pair information, information is exchanged with the first end thread corresponding to the target actual thread to obtain the second queue pair information and the second memory region information of the second memory region; the second queue pair is a queue pair created by the first end thread corresponding to the target actual thread, and the second memory region is a memory region registered by the first end thread corresponding to the target actual thread on the first storage node; Based on the second queue pair information and the second memory region information, the first queue pair and the second queue pair are associated to establish an RDMA connection between the target actual thread and the corresponding first end thread on the first storage node.

3. The method according to claim 2, characterized in that, The step of using the first peer thread corresponding to each of the target actual threads as a relay, and based on the RDMA connection between each of the target actual threads and the corresponding first peer thread, to perform data transmission between the target actual thread on the target computing node and multiple actual threads running on the first storage node includes: The actual threads of each target are processed according to the following procedure: The target thread allocates a first buffer in the corresponding first memory region and obtains the first access information of the first buffer; Obtain a first message sent by the first peer thread corresponding to the target actual thread; the first message includes second access information of the first target data in the second memory area; the first target data is data to be transmitted from the first actual thread on the first storage node to the target actual thread; the first target data is stored in the second memory area by the first actual thread through the first peer thread corresponding to the target actual thread; The target actual thread issues a data reading task to the sending queue in the first queue pair based on the first access information and the second access information. After the network interface card in the target computing node polls for a data read request in the sending queue, it performs an RDMA read operation to temporarily store the first target data in the first buffer. The target data is obtained from the first buffer by the target actual thread.

4. The method according to claim 3, characterized in that, After the target actual thread obtains the first target data from the first buffer, the method further includes: The target thread performs computational tasks using the first target data.

5. The method according to claim 2, characterized in that, The step of using the first peer thread corresponding to each of the target actual threads as a relay, and based on the RDMA connection between each of the target actual threads and the corresponding first peer thread, to perform data transmission between the target actual thread on the target computing node and multiple actual threads running on the first storage node includes: The actual threads of each target are processed according to the following procedure: The target thread temporarily stores the second target data in the first memory area and obtains the third access information of the second target data in the first memory area; Receive a second message sent by the first peer thread corresponding to the target actual thread, the second message including fourth access information of the second buffer; the second buffer is a buffer partitioned by the first peer thread corresponding to the target actual thread in the second memory area; The target actual thread sends data to the send queue in the first queue pair based on the third and fourth access information. After the network interface card in the target computing node polls the data write task in the sending queue, it performs an RDMA write operation to write the second target data into the second buffer, so that the first peer thread corresponding to the target actual thread sends the second target data to the second actual thread.

6. The method according to claim 2, characterized in that, The step of using the first peer thread corresponding to each of the target actual threads as a relay, and based on the RDMA connection between each of the target actual threads and the corresponding first peer thread, to perform data transmission between the target actual thread on the target computing node and multiple actual threads running on the first storage node includes: The actual threads of each target are processed according to the following procedure: The target actual thread temporarily stores the third target data in the first memory area, thereby obtaining the fifth access information of the third target data in the first memory area; The target actual thread issues a first sending task to the sending queue in the first queue pair based on the fifth access information; After the network interface card (NIC) in the target computing node polls the first transmission task in the transmission queue, it performs an RDMA transmission operation. This allows the NIC in the first storage node to receive the third target data, consume the first reception task in the reception queue of the second queue pair, and cache the third target data in the third buffer corresponding to the sixth access information indicated by the reception task. This enables the first peer thread corresponding to the target actual thread to send the third target data to the third actual thread. The third buffer is a buffer partitioned by the first peer thread corresponding to the target actual thread in the second memory area. The first reception task is generated by the first peer thread corresponding to the target actual thread based on the sixth access information of the third buffer.

7. The method according to claim 2, characterized in that, The step of using the first peer thread corresponding to each of the target actual threads as a relay, and based on the RDMA connection between each of the target actual threads and the corresponding first peer thread, to perform data transmission between the target actual thread on the target computing node and multiple actual threads running on the first storage node includes: The actual threads of each target are processed according to the following procedure: The target actual thread allocates a fourth buffer in the first memory region and obtains the seventh access information of the fourth buffer; The target actual thread issues a second receiving task to the receiving queue in the first queue pair based on the seventh access information; If the network card in the target computing node receives the fourth target data sent by the network card in the first storage node, it consumes the second receiving task in the receiving queue of the first queue pair, and caches the fourth target data in the fourth buffer according to the seventh access information, so that the target actual thread can obtain the fourth target data from the fourth buffer; Wherein, after the network card in the first storage node polls the second sending task in the sending queue of the second queue pair, it reads the fourth target data from the fifth buffer and sends the fourth target data to the network card in the target computing node; the second sending task is generated by the first end thread corresponding to the target actual thread according to the eighth access information of the fifth buffer in the second memory area, and the fourth target data is temporarily stored in the fifth buffer by the first end thread corresponding to the target actual thread after obtaining the fourth target data from the fourth actual thread.

8. The method according to claim 1, characterized in that, Before selecting one of the multiple proxy threads enabled on the first storage node as the first peer thread corresponding to the target actual thread, the method further includes: Obtain the proxy thread information allocated by the first storage node to the target computing node, wherein the proxy thread information includes the proxy thread identifier of each proxy thread allocated by the first storage node to the target computing node; A first mapping relationship is established based on the proxy thread identifier of each proxy thread in the proxy thread information and the thread identifier of each target actual thread on the target computing node; in the first mapping relationship, one target actual thread corresponds to one proxy thread on the first storage node, and different target actual threads correspond to different proxy threads on the first storage node; Selecting one of the multiple proxy threads enabled on the first storage node as the first peer thread corresponding to the target actual thread includes: Based on the first mapping relationship, determine the first peer thread corresponding to the target actual thread.

9. The method according to any one of claims 1 to 8, characterized in that, The cluster has a second storage node where the agent thread is not enabled; the method further includes: For each second storage node, the following processing is performed: Among the multiple target actual threads running on the target computing node, one target actual thread is determined for the second storage node, serving as the second peer thread corresponding to the second storage node; Establish RDMA connections between the second peer thread corresponding to the second storage node and each actual thread on the second storage node; Data transfer is performed between the target computing node and each actual thread on the second storage node via an RDMA connection between the second peer thread corresponding to the second storage node and the actual thread on the second storage node.

10. The method according to claim 9, characterized in that, Among the multiple target actual threads running on the target computing node, one target actual thread is determined for the second storage node as the second peer thread corresponding to the second storage node, including: Obtain the node information of the second storage node; the node information includes the node identifier information of the second storage node; The target hash value is obtained by performing a hash calculation based on the node identifier information of the second storage node; Based on the mapping relationship between hash values ​​and actual threads, the target actual thread corresponding to the target hash value on the target computing node is taken as the second peer thread corresponding to the second storage node.

11. The method according to claim 9, characterized in that, The step of performing data transmission between the target computing node and each actual thread on the second storage node via the RDMA connection between the second peer thread corresponding to the second storage node and the actual thread on the second storage node includes: The following processing is performed on the first target actual thread on the target storage node: If the first target actual thread is not the second peer thread corresponding to the second storage node, the fifth target data is sent to the second peer thread corresponding to the second storage node; the fifth target data is the data transmitted by the first target actual thread to the reference actual thread in the second storage node. The fifth target data is transmitted to the reference actual thread in the second storage node based on the RDMA connection between the second peer thread corresponding to the second storage node and the reference actual thread.

12. A data transmission device, characterized in that, An apparatus applied to a target computing node in a cluster, wherein the target computing node refers to any computing node in the cluster, the apparatus comprising: The selection module is used to select one of multiple proxy threads enabled on the first storage node for each target actual thread running on the target computing node, as the first peer thread corresponding to the target actual thread; different target actual threads correspond to different proxy threads; the first storage node is any storage node in the cluster that enables proxy threads; The connection establishment module is used to establish a remote direct memory access (RDMA) connection between each of the target actual threads and the corresponding first end thread on the first storage node. The data transmission module is used to transmit data between the target actual thread on the target computing node and multiple actual threads running on the first storage node, based on the RDMA connection between each target actual thread and the corresponding first end thread, using the first end thread corresponding to each target actual thread as a relay.

13. An electronic device, characterized in that, include: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1-11.

14. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, When the computer-readable instructions are executed by a processor, the method as described in any one of claims 1-11 is implemented.

15. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method of any one of claims 1-11.

Images