RDMA network data transmission method and device
By realizing data sorting and direct transmission on RDMA devices, the problem of data sorting occupies CPU resources in distributed systems is solved, the sorting efficiency and system performance are improved, and the latency and cost are reduced.
Patent Information
- Application Number
- CN202410992590.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In distributed systems, data sorting operations occupy a large amount of CPU resources, resulting in long computing time and increased latency. A method is urgently needed to achieve efficient inter-node communication and data exchange to speed up the sorting process.
Implement the sorting algorithm on the RDMA device, free the sorting operation from the CPU, sort the data through the RDMA device and directly transmit it to the destination node, reducing the multiple copies and processing of data between the RDMA device and the host.
It improves the utilization rate of sorting resources, reduces the delay of sorting and network transmission, reduces the load on the CPU, and reduces the complexity and cost of the system.
Smart Images

Figure CN119003437A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of communication technology and data processing technology, and in particular relates to a RDMA network data transmission method and device. Background Art
[0002] With the advent of the big data era, the demand for processing massive data is growing. In distributed systems, data is usually distributed on multiple computer nodes and needs to be sorted for further analysis, processing or display. Sorting operations take up a lot of CPU resources for calculation, which takes a long time and occupies a lot of resources. In addition, sorting on the CPU requires switching between the kernel state and user state of the operating system, as well as CPU calculation and scheduling, which will introduce more delays.
[0003] There is an urgent need for a method to achieve efficient inter-node communication and data exchange to accelerate the sorting process.
[0004] RDMA (Remote Direct Memory Access) is a network communication technology that allows a device (usually a network adapter) of a computer system to directly access the memory of another computer system without the intervention of the central processing unit (CPU) or the operating system. This direct access to memory can significantly reduce data transmission delays and CPU overhead, thereby improving the efficiency and performance of network communications. RDMA devices are RDMA network interface controllers that incorporate sorting technology. Using RDMA devices to execute sorting algorithms for hardware can accelerate algorithm execution efficiency while reducing network transmission delays and CPU overhead. Summary of the invention
[0005] In view of the above problems, the present invention proposes a RDMA network data transmission method and device. A sorting algorithm is implemented on the RDMA device, and the sorting operation is added to the RDMA network transmission semantics, so that the sorted data is directly transmitted to the destination node through the network.
[0006] Sorting on RDMA devices can free the sorting operation from the CPU, allowing the CPU to focus on other important computing and management tasks. Especially under high load conditions, placing the sorting operation on the network card can prevent CPU resources from being occupied by the sorting process, improving the overall performance and response speed of the system. At the same time, by using the characteristics of RDMA technology to directly access memory and bypass the operating system kernel, it can reduce the multiple copies and processing of data between RDMA devices and hosts, reducing the complexity and cost of the system.
[0007] The technical solution adopted by this patent application to solve its technical problem is:
[0008] This patent provides a method and device for RDMA network data transmission, which realizes sorting and data sending or sorting and data receiving in RDMA operation, thereby improving sorting resource utilization, reducing sorting and network transmission integrated response delay, and reducing equipment costs. This patent also provides a method for utilizing this capability.
[0009] On the one hand, the present invention provides an RDMA network data transmission method, comprising the following steps:
[0010] Sending side:
[0011] S1. The sender calls the sort and send semantics and obtains the original data to be sent.
[0012] S2. The sender sorts the original data and caches the sorted data.
[0013] S3. The sending end encapsulates the sorted data into an RDMA network data packet, transmits it to the receiving end, and releases the sorting cache space.
[0014] Receiver:
[0015] S4. The receiving end calls the receive and sort semantics, receives and parses the RDMA network data packets, and caches the data packets.
[0016] S5. The receiving end sorts the received data packets.
[0017] S6. The receiving end writes the sorted data into the storage area in the receiving and sorting semantics using the DMA method, releases the receiving buffer space, and completes the network data transmission.
[0018] Furthermore, the step S1 is specifically implemented as follows: the sending end calls the sort and send semantics, converts the sort and send semantics into a sort and send request, and places it in the sending queue of the sending end, and the sort and send request specifies the storage area where the unsorted data to be sent is located; then the sending end obtains the sort and send request, and obtains the original data to be sent.
[0019] Furthermore, the specific process of step S4 is: the receiving end calls the receive and sort semantics, converts the receive and sort semantics into a receive and sort request, and places it in the receiving queue of the receiving end, and the receive and sort request specifies the storage area where the received data should be saved; the receiving end obtains the receive and sort request, receives and parses the RDMA network data packet, and temporarily stores the data packet payload in the receiving cache device.
[0020] On the other hand, the present invention also provides an RDMA network data transmission device, including a sending end and a receiving end.
[0021] The sending end includes a sending end main controller and a sending end RDMA device, which are used to call sorting and sending semantics and send original data.
[0022] The sending end main controller includes a processor and a memory, which is used to convert the sort-and-send semantics into a sort-and-send request and store the original data to be sent.
[0023] Processor, which runs applications and RDMA device drivers and converts sort-and-send semantics into sort-and-send requests.
[0024] Memory, used to store the original data to be sent.
[0025] The sending end RDMA device comprises a request parsing unit, a data acquisition unit, a sorting unit, a sorting buffer storage unit and a data packet encapsulation and sending unit, and is used for parsing RDMA requests, sorting original data and sending them to the receiving end.
[0026] The request parsing unit is used to parse the RDMA request.
[0027] The data acquisition unit is used to acquire the original data to be sent through the DMA engine.
[0028] The sorting unit is used to sort the original data using a preset sorting method to obtain sorted data.
[0029] The sorting buffer storage unit is used to store the data to be sent after being sorted by the sending end.
[0030] The data packet encapsulation and sending unit is used to encapsulate the sorted data into an RDMA network data packet and transmit it to the receiving end.
[0031] The receiving end includes a receiving end main controller and a receiving end RDMA device, which are used to call receiving and sorting semantics, receive and parse data.
[0032] The receiving end main controller includes a processor and a memory, which are used to call the RDMA device driver to convert the receive and sort semantics into a receive and sort request, and store the received data.
[0033] Processor, used to run applications and RDMA device drivers.
[0034] Memory is used to store received raw data or sorted result data.
[0035] The receiving end RDMA device comprises a request parsing unit, a data packet receiving and parsing unit, a receiving buffer storage unit, a sorting unit, and a data writing unit, and is used for receiving and parsing RDMA network data packets and storing the data after sorting.
[0036] The request parsing unit is used to parse the RDMA request.
[0037] The data packet receiving and parsing unit is used to receive RDMA network data packets and parse the packet header and payload information.
[0038] The receiving buffer storage unit is used to store data packets received by the receiving end RDMA device.
[0039] The sorting unit is used to sort the data in the received data packet using a preset sorting method to obtain sorted data.
[0040] The data writing unit is used to write the sorted data into the storage area specified in the RDMA receiving and sorting request.
[0041] Beneficial effects of the present invention:
[0042] The present invention sorts the original data through the RDMA device, and can use the acceleration capability of the RDMA device to quickly sort, shorten the algorithm delay caused by the sorting, and reduce the sorting algorithm burden of the CPU.
[0043] The RDMA extended semantics proposed in the present invention include "sort and send" and "receive and sort". A combination of "sort and send" and "receive" semantics can be used to sort at the sending end of the RDMA network. A combination of "send" and "receive and sort" semantics can also be used to sort at the receiving end of the RDMA network. This has certain flexibility. As long as the RDMA device on one side of the RDMA network has the sorting function, it will suffice.
[0044] The present invention implements sorting and sending or receiving and sorting in RDMA operation, thereby improving sorting resource utilization, reducing sorting and network transmission integration response delay, and reducing equipment costs. The patent also provides a method for utilizing this capability. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 This is an architecture diagram of the RDMA data transmission system in the embodiment;
[0046] Figure 2 It is a flow chart of the method for sorting at the sending end in the embodiment;
[0047] Figure 3 A flow chart of a method for sorting at a receiving end in an embodiment;
[0048] Figure 4 This is a schematic diagram of the structure of the sending end in the embodiment;
[0049] Figure 5 FIG. 4 is a schematic diagram of the structure of the receiving end in the embodiment. DETAILED DESCRIPTION
[0050] like Figure 1 As shown, the present invention provides an RDMA data transmission device, including a sending end and a receiving end, wherein:
[0051] The sending end includes a main controller and an RDMA device, which are used to process sorting and sending semantics; the original data to be sent is obtained through the sending end RDMA device; the original data is sorted by the sending end RDMA device using a preset sorting method to obtain sorted data; the sending end RDMA device encapsulates the sorted data into a data packet, the sorted data is encapsulated in the payload part of the data packet, and the data packet is transmitted to the receiving end.
[0052] The sending end is also used to process common sending semantics, obtain the original data to be sent through the sending end RDMA device; encapsulate the original data into a data packet through the RDMA device, the original data is encapsulated in the payload part of the data packet, and the data packet is transmitted to the receiving end.
[0053] The receiving end includes a main controller and an RDMA device, which is used to process the receiving and sorting semantics; receive the data packet sent by the sending end, parse the data packet, and temporarily store the payload of the data packet in the storage device of the receiving end RDMA device; sort the received data using a preset sorting method to obtain sorted data, and write the sorted data into the storage area specified in the RDMA receiving and sorting semantics; after the sorted data is successfully written into the specified storage area, release the storage space for temporarily storing the sorted data.
[0054] The receiving end is also used to process common receiving semantics; receiving data packets sent by the sending end, parsing the data packets, and writing the payload of the data packets into the storage area specified in the receiving semantics.
[0055] The sender and the receiver are connected via an RDMA network, which includes an Infiniband network, a RoCE network, an iWarp network, and other networks that use RDMA operations. The receiver and the sender can be directly connected via a network cable, or they can be interconnected via a switch or router.
[0056] The present invention extends the RDMA operation, adds "sort and send" semantics and "receive and sort" semantics, and is used in conjunction with the general "send" and "receive" semantics of RDMA.
[0057] In one embodiment, a sending end with a sorting function is used in combination with a receiving end without a sorting function (or without using a sorting function), providing a method for sorting at the sending end, such as Figure 2The sender performs extended "order and send" semantics, and the receiver performs normal "receive" semantics.
[0058] A RDMA network data transmission method comprises the following steps:
[0059] The sender methods include:
[0060] 1) The application invokes "order and send" semantics.
[0061] 2) The RDMA device driver converts the "sort and send" semantics into a "sort and send" request and places it in the sender's send queue. The sort and send request specifies the storage area where the unsorted data to be sent is located.
[0062] 3) The RDMA device gets the "order and send" request.
[0063] 4) The RDMA device obtains the original data to be sent.
[0064] 5) The RDMA device uses a preset sorting method to sort the original data to obtain sorted data, and temporarily stores the sorted data in a sorting cache device of the sending end RDMA device.
[0065] 6) The RDMA device encapsulates the sorted data into an RDMA network data packet.
[0066] 7) The RDMA device transmits the data packet to the receiving end.
[0067] 8) The RDMA device releases the sorting buffer space.
[0068] The receiving end methods include:
[0069] 1) Application calls receive semantics.
[0070] 2) The RDMA device driver converts the "receive" semantics into a "receive" request and places it in the receiving queue of the receiving end. The receive request specifies the storage area where the received data should be saved.
[0071] 3) The RDMA device gets a "receive" request.
[0072] 4) The RDMA device receives and parses the RDMA network data packet and uses DMA to store the data packet payload into the specified storage area.
[0073] In another embodiment, a sending end without a sorting function (or without using a sorting function) is used in conjunction with a receiving end with a sorting function, providing a method for sorting at the receiving end, such as Figure 3 The sender performs general "send" semantics, and the receiver performs "receive and order" semantics.
[0074] The sender methods include:
[0075] 1) The application invokes the "send" semantics.
[0076] 2) The RDMA device driver converts the "send" semantics into a "send" request and places it in the sender's send queue.
[0077] 3) The RDMA device gets a "send" request.
[0078] 4) The RDMA device obtains the original data to be sent.
[0079] 5) The RDMA device encapsulates the original data into an RDMA network data packet.
[0080] 7) The RDMA device transmits the RDMA network data packet to the receiving end.
[0081] The receiving end methods include:
[0082] 1) The application invokes "receive and order" semantics.
[0083] 2) The RDMA device driver converts the "receive and sort semantics" into a "receive and sort" request and places it in the receiving queue of the receiving end. The "receive and sort request" specifies the storage area where the received data should be saved.
[0084] 3) The RDMA device gets a "receive and order" request.
[0085] 4) The RDMA device receives and parses the RDMA network data packet and temporarily stores the data packet payload in the receiving buffer device.
[0086] 5) The RDMA device sorts the received data using a preset sorting method to obtain sorted data.
[0087] 6) The RDMA device uses DMA to write the sorted data to the storage area specified in the "Receive and Sort" request.
[0088] 7) The RDMA device releases the receiving buffer space.
[0089] Typical RDMA data transmission semantics include: Send / Receive and RDMA Read / Write. Based on this, the present invention extends the semantics of "sort and send" and "sort and receive". The implementation method of this semantics is similar to the Send / Receive semantics, which is a bilateral operation semantics. Usually, "sort and send" and "receive" are used in conjunction, and "send" and "sort and receive" are used in conjunction.
[0090] Send / Receive:
[0091] It is a bilateral operation. Host A sends data to host B. Host A performs the sending operation and host B performs the receiving operation. The process is as follows:
[0092] 1. Both host A and host B must create and initialize their own queue pairs (QP) and completion queues (CQ).
[0093] 2. Host A submits a send request to the RDMA device's work queue, and host B submits a receive work request to its RDMA device's work queue.
[0094] 3. Host A's RDMA device polling work queue interprets this as a send request, obtains data from memory and sends the data to host B. After the data stream reaches host B's RDMA device, the receive request in host B's work queue is consumed, and the data is directly stored in the storage location pointed to by the receive request.
[0095] 4. After the communication between hosts A and B is completed, a completion message will be generated in the CQ of host A to indicate that the sending is complete. At the same time, a completion message will also be generated in the CQ of host B to indicate that the receiving is complete.
[0096] RDMA READ and RDMA WRITE are unilateral operations. The local end only needs to specify the source and destination addresses of the information. The remote application does not need to be aware of the communication. The reading or writing of data is completed between the RDMA device and the application buffer through RDMA, and then encapsulated into a message by the remote RDMA device and returned to the local end.
[0097] For unilateral operations, taking RDMA READ storage in a storage network environment as an example, the data flow is as follows:
[0098] 1. First, host A and host B establish a connection, and QP has been created and initialized.
[0099] 2. The data is saved in the buffer address VB of host B. VB should be registered with the RDMA device of host B in advance and obtain the returned key, which is equivalent to the permission of RDMA operation buffer.
[0100] 3. Host B encapsulates the data address VB and the key key into a dedicated message and transmits it to host A. That is, host B hands over the operation rights of the data buffer to host A.
[0101] 4. After host A receives the data VB and R_key sent by host B, the RDMA device will encapsulate them together with its own storage address VA to the RDMA READ request and send this message request to host B. During this process, host A and host B do not need any software to participate, and the data of host B can be stored in the VA virtual address of host A.
[0102] For unilateral operations, taking RDMA WRITE storage in a storage network environment as an example, the data flow is as follows:
[0103] 1. First, host A and host B establish a connection, and QP has been created and initialized.
[0104] 2. The data remote target storage buffer address VB. Note that VB should be registered with the RDMA device of host B in advance and obtain the returned key, which is equivalent to the RDMA operation permission of this buffer.
[0105] 3. Host B encapsulates the data address VB and key key into a dedicated message and transmits it to host A. That is, host B hands over the operation rights of the data buffer to host A.
[0106] 4. After host A receives the data address VB and key key sent by host B, the RDMA device will encapsulate them together with its own sending address VA into the RDMA WRITE request. During this process, host A and host B do not need any software to participate, and can send host A's data to host B's VB virtual address.
[0107] The biggest difference between RDMA and traditional network transmission is the unilateral operation semantics. It only needs to provide direct access to the remote virtual address without the participation of the remote application. This method is suitable for batch data transmission.
[0108] Embodiment: An RDMA network data transmission device specifically includes a sending end and a receiving end. The sending end includes a sending end main controller and a sending end RDMA device, which are used to call sorting and sending semantics and send original data. The receiving end includes a receiving end main controller and a receiving end RDMA device, which are used to call receiving and sorting semantics and receive and parse data.
[0109] The sending end is provided with a main controller and an RDMA device, such as Figure 4 shown.
[0110] The sending end main controller comprises:
[0111] Processor, used to run applications and RDMA device drivers.
[0112] Memory, used to store the original data to be sent.
[0113] The sending end RDMA device includes:
[0114] The request parsing unit is used to parse the RDMA request.
[0115] The data acquisition unit is used to acquire the original data to be sent through the DMA engine.
[0116] The sorting unit is used to sort the original data using a preset sorting method to obtain sorted data.
[0117] The sorting buffer storage unit is used to store the data sorted by the sender and to be sent.
[0118] The data packet encapsulation and sending unit is used to encapsulate the sorted data into an RDMA network data packet and transmit it to the receiving end.
[0119] In some embodiments, the RDMA device at the sending end has a sorting capability, including the above-mentioned request parsing unit, data acquisition unit, sorting unit, sorting buffer storage unit and data packet encapsulation and sending unit.
[0120] In other embodiments, the RDMA device at the sending end does not have the sorting capability, and includes the above-mentioned request parsing unit, data acquisition unit, and data packet encapsulation and sending unit.
[0121] The receiving end is provided with a main controller and an RDMA device, such as Figure 5 shown.
[0122] The receiving end main controller comprises:
[0123] Processor, used to run applications and RDMA device drivers.
[0124] Memory is used to store received raw data or sorted result data.
[0125] The receiving end RDMA device includes:
[0126] The request parsing unit is used to parse the RDMA request.
[0127] The data packet receiving and parsing unit is used to receive RDMA network data packets and parse the packet header and payload information.
[0128] The receiving buffer storage unit is used to store data packets received by the receiving end RDMA device.
[0129] The sorting unit is used to sort the data in the received data packet using a preset sorting method to obtain sorted data.
[0130] The data writing unit is used to write the sorted data into the storage area specified in the RDMA receiving and sorting request.
[0131] In some embodiments, the RDMA device at the receiving end has a sorting capability, including the above-mentioned request parsing unit, data packet receiving and parsing unit, receiving buffer storage unit, sorting unit and data writing unit.
[0132] In some other embodiments, the RDMA device at the receiving end does not have the sorting capability, and includes the above-mentioned request parsing unit, data packet receiving and parsing unit, and data writing unit.
[0133] In a specific embodiment, at least one of the sending end and the receiving end must have a sorting capability in order to utilize the acceleration capability of the RDMA device and achieve the purpose of offloading the sorting algorithm in the RDMA device.
Claims
1. A RDMA network data transmission method, characterized in that: The following steps are involved: Sending side: S1, the sender calls the sort and send semantics and obtains the original data to be sent; S2, the sender sorts the original data and caches the sorted data; S3, the sending end encapsulates the sorted data into an RDMA network data packet, transmits it to the receiving end, and releases the sorting cache space; Receiver: S4, the receiving end calls the receive and sort semantics, receives and parses the RDMA network data packets, and caches the data packets; S5, the receiving end sorts the received data packets; S6. The receiving end writes the sorted data into the storage area in the receiving and sorting semantics using the DMA method, releases the receiving buffer space, and completes the network data transmission.
2. The RDMA network data transmission method according to claim 1, characterized in that: The specific implementation process of step S1 is as follows: the sending end calls the sort and send semantics, converts the sort and send semantics into a sort and send request, and places it in the sending queue of the sending end, and the sort and send request specifies the storage area where the unsorted data to be sent is located; The sender then obtains the sort and sends the request, and obtains the original data to be sent.
3. The RDMA network data transmission method according to claim 2, characterized in that: The specific implementation process of step S4 is as follows: the receiving end calls the receive and sort semantics, converts the receive and sort semantics into a receive and sort request, and places it in the receiving queue of the receiving end, and the receive and sort request specifies the storage area for storing the received data; the receiving end obtains the receive and sort request, receives and parses the RDMA network data packet, and temporarily stores the data packet payload in the receiving cache device.
4. The RDMA network data transmission method according to claim 1, characterized in that: The semantics of sort and send and sort and receive are as follows: Typical RDMA data transmission semantics include: send / receive and RDMA read / write, sort and send and sort and receive semantic implementation methods, which are the same as send / receive semantics, and are bilateral operation semantics, sort and send and receive are used together, and send and sort and receive are used together.
5. An RDMA network data transmission device, used to implement the transmission method according to any one of claims 1 to 4, characterized in that: Includes the sending end and the receiving end; The sending end includes a sending end main controller and a sending end RDMA device, which are used to call sorting and sending semantics, and send the original data; The receiving end includes a receiving end main controller and a receiving end RDMA device, which are used to call receiving and sorting semantics, receive and parse data.
6. The RDMA network data transmission device according to claim 5, characterized in that: The sending end main controller includes a processor and a memory, which is used to convert the sort and send semantics into a sort and send request, and store the original data sent; The processor is used to run the application and the RDMA device driver, and convert the sort and send semantics into a sort and send request; The memory is used to store the original data to be sent; The sending end RDMA device includes a request parsing unit, a data acquisition unit, a sorting unit, a sorting buffer storage unit and a data packet encapsulation and sending unit, which are used to parse the RDMA request and sort the original data to send to the receiving end; The request parsing unit is used to parse the RDMA request; The data acquisition unit is used to acquire the original data to be sent through the DMA engine; The sorting unit is used to sort the original data using a preset sorting method to obtain sorted data; The sorting buffer storage unit is used to store the data to be sent after sorting by the sending end; The data packet encapsulation and sending unit is used to encapsulate the sorted data into an RDMA network data packet and transmit it to the receiving end.
7. The RDMA network data transmission device according to claim 5, characterized in that: The receiving end main controller includes a processor and a memory, which is used to call the RDMA device driver to convert the receive and sort semantics into a receive and sort request, and store the received data; The processor is used to run the application program and the RDMA device driver; The memory is used to store the received original data or sorting result data; The receiving end RDMA device includes a request parsing unit, a data packet receiving and parsing unit, a receiving buffer storage unit, a sorting unit, and a data writing unit, which are used to receive and parse RDMA network data packets and save the data after sorting; The request parsing unit is used to parse the RDMA request; The data packet receiving and parsing unit is used to receive the RDMA network data packet and parse the packet header and payload information; The receiving buffer storage unit is used to store data packets received by the receiving end RDMA device; The sorting unit is used to sort the data in the received data packet using a preset sorting method to obtain sorted data; The data writing unit is used to write the sorted data into the storage area specified in the RDMA receiving and sorting request.
Citation Information
Patent Citations
Mapping rdma semantics to high speed storage
CN102844747A
Data stream transmission method, sending device and receiving device
CN107623646A
Data transmission method and device based on RDMA (Remote Direct Memory Access)
CN115658592A
Reliable communication method for RDMA UD transmission, electronic equipment and readable medium
CN118200253A
