Remote atomic operation hardware offloading method and device based on RDMA network card

Abstract

The application discloses a kind of remote atomic operation hardware unloading method and device based on RDMA network card, the remote atomic operation hardware unloading method based on RDMA network card of the present application includes that full connection or half connection RAO protocol is used between source node and destination node to realize remote atomic operation hardware unloading, and remote atomic operation hardware unloading can be executed according to remote atomic operation type;Remote atomic operation type includes sending post, pull fetch and swap three types of exchange swap.The present application can solve the problem that the delay of traditional RDMA network card bilateral and unilateral communication connection protocol is higher and mainly faces the direct copy of remote memory data of large block data, and remote atomic operation mainly faces frequent small granularity data, directly using traditional connection protocol can cause the problem of high delay and low message rate, without the participation of destination host CPU, software overhead is low, and the delay, message rate and throughput of remote atomic operation can be improved.

Description

Technical Field

[0001] This invention relates to the field of remote atomic operation technology in distributed parallel computing systems, and specifically to a method and apparatus for hardware offloading of remote atomic operations based on an RDMA network card. Background Technology

[0002] Atomic operations (AOs) are the core mechanism for ensuring concurrency safety in computer systems. They are the cornerstone of building efficient and reliable multithreaded programs and can be used to implement high-performance lock-free data structures such as lock-free queues and stacks. They can also be directly used to implement various synchronization primitives such as spinlocks and semaphores. An atomic operation is essentially a series of indivisible instructions (read-modify-write). These operations are executed as a single, uninterruptible unit; they either execute completely or not at all, and the result is immediately visible to other threads without any intermediate states. Traditional single-node processors achieve this by providing dedicated atomic instructions, such as the widely used Compare-And-Swap (CAS) instruction. This instruction first compares the value in memory with an expected value, and only writes the new value if they are the same. The entire process ensures atomicity at the processor level. Furthermore, multi-core processors may also ensure exclusive access to specific memory regions during operations by locking the bus or utilizing cache coherence protocols to prevent memory access interference from other cores.

[0003] Remote Atomic Operations (RAOs) extend traditional atomic operations from within a single node to between nodes, and from operations on local shared memory to operations on distributed memory on high-performance RDMA networks. Many high-performance computing applications, such as climate simulation, fluid mechanics, and cosmology, require thousands of computing nodes to collaborate, resulting in large amounts of globally shared data distributed across different nodes. Nodes need to frequently exchange data and synchronize states. RAOs can achieve extremely low-latency global barrier synchronization, cross-node accumulation (such as reduction operations), and shared counts, accelerating high-performance computing applications. Furthermore, in distributed deep learning training, multiple GPU worker nodes need to frequently update a centralized parameter server or perform All-Reduce operations to synchronize gradients. RAOs can significantly optimize the gradient aggregation process in parameter server architectures, allowing worker nodes to directly and atomically update central parameters, reducing communication rounds and latency, and accelerating training. Therefore, the design and implementation of high-performance remote atomic operations are crucial, as they can significantly reduce the overhead of remote shared data access, provide high-performance cross-node synchronization mechanisms, and efficiently support remote shared data programming interfaces.

[0004] Traditional remote atomic operations are completed using bilateral message sending and receiving communication primitives provided by high-performance RDMA networks. The communication process is as follows: (1) Process A on node 0 initiates a remote atomic operation on the shared variable x on node 1, encapsulates the remote atomic operation information into an immediate message and sends it to node 1; (2) After receiving the message, process B on node 1 copies the message to user space and parses the remote atomic operation information; (3) The CPU on node 1 performs an atomic operation on the shared variable x located in the specified memory, that is, reads the value of variable x, and completes the specified calculation, such as adding 1 and XORing, and then writes the modified result back to memory; (4) After completing the atomic operation, process B on node 1 packages the initial value of the shared variable x and the completion flag into an immediate message and sends it to node 0. Node 0 copies the result from the message buffer and completes a complete remote atomic operation. The disadvantages of the above remote atomic operation based on bilateral message sending and receiving communication primitives are: (1) redundant data copying; (2) large software stack overhead; (3) the need for the CPU of the destination end to participate in the execution. Experimental tests show that in a single remote atomic operation, software overhead and host network card interaction overhead account for more than 70% of the total latency, which greatly limits the cross-node synchronization efficiency of distributed parallel programs. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and apparatus for hardware offloading of remote atomic operations based on an RDMA network card, which addresses the above-mentioned problems of the prior art, requires no CPU participation from the destination host, has low software overhead, and can improve the latency, message rate and throughput of remote atomic operations.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for remote atomic operation hardware offloading based on an RDMA network interface card (NIC) includes remote atomic operation hardware offloading between a source node and a destination node using a fully connected RAO protocol or a half-connected RAO protocol. The method comprises: the source node submitting a RAO descriptor to its local RDMA NIC; the RDMA NIC parsing the descriptor, encapsulating the remote atomic operation information into a remote atomic operation request message RaoRqt, and sending it to the RDMA network via a sending engine; upon receiving the remote atomic operation request message RaoRqt, the destination node's RDMA NIC forwards it to a RAO response processing component based on the message type; this component reads data from the destination node's memory according to the remote atomic operation type and performs corresponding calculations. The updated data is written back to memory. Subsequently, a remote atomic operation response message RaoRsp is encapsulated and sent back to the source node. Upon receiving the RaoRsp message, the source node's RDMA network card notifies the source node of the result as a local event. Then, the source node's RDMA network card sends a remote event request message EvtWtRqt to the destination node. Upon receiving EvtWtRqt, the destination node's RDMA network card notifies the local process of the completion of a remote atomic operation as an event and sends a connection completion response message CnctRspCplt back to the source node. Upon receiving the CnctRspCplt message, the source node marks the completion of a remote atomic operation.

[0007] Optionally, the remote atomic operation hardware offloading using the fully connected RAO protocol includes: ① The source node submits a RAO descriptor to its local RDMA network card. The RDMA network card parses the RAO descriptor, encapsulates the RAO operation information into a remote atomic operation request message RaoRqt, and sends it to the RDMA network through the sending engine; ② After receiving the remote atomic operation request message RaoRqt, the destination node's RDMA network card forwards the message to the RAO response processing unit according to the message type. The RAO response processing unit reads data from the destination node's memory according to the remote atomic operation type of the RaoRqt message and completes the corresponding calculation update for the remote atomic operation. The updated data is then written back to memory. The response data is encapsulated into a remote atomic operation response message RaoRsp and sent back to the source node; ③ After receiving the remote atomic operation response message RaoRsp, the source node's RDMA network card notifies the source node of the result as a local event. 1. The source node's RDMA network card establishes a connection for a remote event to be sent to the destination node, sending a connection message CnctRqt into the RDMA network; 2. After receiving the connection message CnctRqt, the destination node's RDMA network card establishes a connection with the source node, generates a connection response message CnctRspAck, and sends it back to the source node; 3. Upon receiving the connection response message CnctRspAck, the source node marks the establishment of a complete event connection and sends a remote event request message EvtWtRqt to the destination node; 4. Upon receiving the remote event request message EvtWtRqt, the destination node's RDMA network card notifies the local process to complete a remote atomic operation as an event and sends a connection response completion message CnctRspCplt back to the source node; 5. Upon receiving the connection response completion message CnctRspCplt, the source node marks the completion of a remote atomic operation.

[0008] Optionally, the hardware offloading of remote atomic operations using the half-connection RAO protocol includes: ① The source node submits a RAO descriptor to its local RDMA network card. The RDMA network card parses the RAO descriptor, encapsulates the RAO operation information into a remote atomic operation request message RaoRqt, and sends it to the RDMA network through the sending engine; ② After receiving the remote atomic operation request message RaoRqt, the destination node's RDMA network card forwards the message to the RAO response processing unit according to the message type. The RAO response processing unit reads data from the destination node's memory according to the remote atomic operation type of the RaoRqt message and completes the corresponding calculation and update of the remote atomic operation. The updated data is then written back to memory; the response data is encapsulated into a remote atomic operation response. ① The source node's RDMA network card receives the remote atomic operation response message RaoRsp and sends it back to the source node; ② After receiving the remote atomic operation response message RaoRsp, the source node's RDMA network card notifies the source node of the result as a local event; ③ The source node's RDMA network card establishes a connection for the remote event to be sent to the destination node and sends the remote event request message EvtWtRqt to the destination node; ④ After receiving the remote event request message EvtWtRqt, the destination node's RDMA network card notifies the local process to complete a remote atomic operation as an event and sends the connection response completion message CnctRspCplt back to the source node; ⑤ After receiving the connection response completion message CnctRspCplt, the source node marks the completion of a remote atomic operation.

[0009] Optionally, the remote atomic operation types include three types: post, fetch, and swap. Post-based remote atomic operations include four types: add, AND, OR, and XOR. After the operation, the destination node does not return the original data before the operation to the source node. Fetch-based remote atomic operations include four types: add, AND, OR, and XOR. After the operation, the destination node returns the original data before the operation to the source node. Swap-based remote atomic operations include two types: swap and conditional swap (CSWAP). The parameters for SWAP include the remote address ptr and the sent data opd. SWAP swaps the data in the remote address ptr with the sent data opd and returns the original value in the remote address ptr to the source node. The parameters for CSWAP include the remote address ptr and the sent data cmp and opd. CSWAP compares the data in the remote address ptr with the sent data cmp. If they are equal, the data in the remote address ptr and the sent data opd are swapped, and the original value in the remote address ptr is returned to the source node. If they are not equal, no operation is performed, and the original value in the remote address ptr is returned to the source node.

[0010] Optionally, the RAO response processing component reads data from the destination node's memory and performs the corresponding calculation update for the remote atomic operation based on the remote atomic operation type of the remote atomic operation request message RaoRqt. Writing the updated data back to memory includes: checking if the remote atomic operation request message RaoRqt has a CRC error, checking the destination address, performing a virtual-to-real address transformation, reading the original data from the host destination address to the destination node's RDMA network card, performing the corresponding calculation update for the remote atomic operation, and writing the updated data back to the host destination address. If the remote atomic operation type of the remote atomic operation request message RaoRqt is fetch or swap, the original value of the atomic operation is sent to the requesting end; if the remote atomic operation type of the remote atomic operation request message RaoRqt is post, no return is required.

[0011] Optionally, the RAO response processing component includes a RAO request queue, an arbitrator, and multiple arithmetic logic units (ALUs). The RAO request queue is used to cache remote atomic operation request messages RaoRqt. The arbitrator is used to arbitrate and distribute the remote atomic operation request messages RaoRqt cached in the RAO request queue to the ALUs. The multiple ALUs are used to read data from the memory of the destination node in parallel, complete the corresponding calculation updates of the remote atomic operation, and write the updated data back to memory.

[0012] Optionally, the arbitrator is used to arbitrate and distribute the remote atomic operation request messages RaoRqt in each cache of the RAO request queue to the arithmetic logic unit (ALU). This means that, based on the idle status and address conflict status of the ALU, a remote atomic operation request message RaoRqt is taken out and sent to an available ALU that has no address conflict.

[0013] Optionally, the arithmetic logic unit (ALU) includes a RAO message parsing and processing module, a destination address splitting module, a RAO data processing module, and a source address splitting module. The working process of the ALU includes: (1) The message receiving module writes the remote atomic operation request message RaoRqt from other nodes into the RAO request queue; (2) The arbitrator, based on the idle status and address conflict status of the ALU, retrieves a remote atomic operation request message RaoRqt and sends it to an available ALU without address conflicts; (3) RAO message The parsing and processing module first receives the remote atomic operation request message RaoRqt and starts the processing flow. At the same time, it calls the address validity check module in the RDMA network card to perform address validity check, and calls the destination address splitting module to perform destination address splitting operation. The state machine controls the message processing state; (4) The destination address splitting module is responsible for performing the address splitting task. It calls the virtual address translation module in the RDMA network card to translate the virtual address in the message into a virtual address to obtain the physical address; (5) The destination address splitting module returns the obtained physical address to the RAO message parsing unit. (6) The RAO message parsing and processing module sends a DMA read request to the PCIe interface module based on the starting address and length of the physical address; (7) The RAO message parsing and processing module sends the remote atomic operation request message RaoRqt to the RAO data processing module; (8) The RAO data processing module starts the message processing state machine and sends the source address and length information to the source address splitting module for virtual address splitting; (9) After completing the address splitting, the source address splitting module returns the split index number, offset and length to the RAO data processing module; (10) The PCIe interface module will... DMA reads data and writes it into the RAO data FIFO in the RDMA network card. The RAO data processing module reads the original data corresponding to the pending request cached in the RAO data FIFO. (11) If the remote atomic operation type of the remote atomic operation request message RaoRqt is fetch and swap, the RAO data processing module encapsulates the RAO original data into a remote atomic operation response message RaoRsp and sends it to the source node of the RAO request. If the remote atomic operation type of the remote atomic operation request message RaoRqt is post, it is ignored. (12) The RAO data processing module starts the remote atomic operation calculation and submits the calculation result to the message submission module. (13) The message submission module caches the request and submits the result to the PCIe interface module in the manner of DMA write request. After the request is submitted, it sends a response signal ACK to the RAO data processing module to complete a complete RAO operation process.

[0014] Furthermore, the present invention also provides a computer device including a microprocessor and an RDMA network card interconnected thereto, the RDMA network card being programmed or configured to execute the RDMA network card-based remote atomic operation hardware offloading method.

[0015] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: 1. The present invention offloads remote atomic operations to the RDMA network card, allowing the hardware to directly process atomic operations, eliminating redundant data copying, and eliminating the need for the destination CPU to participate, thus reducing software overhead. 2. The present invention employs concurrent pipelined execution of multiple processing units, improving the processing efficiency of remote atomic operations and enhancing the latency, message rate, and throughput of remote atomic operations. Attached Figure Description

[0016] Figure 1 The following are flowcharts of the fully connected RAO protocol and the half-connected RAO protocol in the embodiments of the present invention, wherein (a) is a flowchart of the fully connected RAO protocol and (b) is a flowchart of the half-connected RAO protocol.

[0017] Figure 2 This is a schematic diagram of the RAO response processing component and its working principle in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the arithmetic logic unit (ALU) in an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0020] To address the issues of high latency and low message rate associated with traditional RDMA network interface cards (NICs) bilateral and unilateral communication connection protocols, which primarily handle large blocks of remote memory data copying, and the fact that remote atomic operations (RAOs) mainly deal with frequent, small-granular data, this embodiment provides a hardware offloading method for remote atomic operations based on RDMA NICs. This method utilizes a fully connected or half-connected RAO protocol between the source and destination nodes to achieve hardware offloading of remote atomic operations. The method includes: Source node submits and sends requests: The source node submits the RAO descriptor to the local RDMA network card. The RDMA network card parses the descriptor, encapsulates the remote atomic operation information into a remote atomic operation request message RaoRqt, and sends it to the RDMA network through the sending engine. Destination-side hardware processing and response: After receiving RaoRqt, the RDMA network card of the destination node forwards it to the RAO response processing unit according to the message type. This unit reads data from the destination node's memory according to the remote atomic operation type, completes the corresponding calculation and update, and writes the updated data back to memory. Thereafter, it encapsulates the remote atomic operation response message RaoRsp and sends it back to the source node. If the operation type is POST, the original data before the operation is not returned in RaoRsp. If it is Fetch or Swap, the original data before the operation is returned to the source node through RaoRsp. Source-side notification and event message transmission: After receiving RaoRsp, the source node's RDMA network card notifies the source node of the result as a local event; subsequently, the source node's RDMA network card sends a remote event request message EvtWtRqt to the destination node (for full-connection RAO protocols, a connection message CnctRqt is sent first, and a handshake is completed with the destination node before sending EvtWtRqt; for half-connection RAO protocols, EvtWtRqt is sent directly without sending CnctRqt). Destination-side event notification and completion confirmation: After receiving EvtWtRqt, the destination node's RDMA network card notifies the local process of completing a remote atomic operation in the form of an event, and sends a connection response completion message CnctRspCplt back to the source node; Source node complete marking: After receiving CnctRspCplt, the source node marks the completion of a remote atomic operation.

[0021] like Figure 1As shown in (a), the hardware offloading of remote atomic operations using the fully connected RAO protocol includes: ① The source node submits a RAO descriptor to its local RDMA network card. The RDMA network card parses the RAO descriptor, encapsulates the RAO operation information into a remote atomic operation request message RaoRqt, and sends it to the RDMA network through the sending engine; ② After receiving the remote atomic operation request message RaoRqt, the destination node's RDMA network card forwards the message to the RAO response processing unit according to the message type. The RAO response processing unit reads data from the destination node's memory according to the remote atomic operation type of the RaoRqt message and completes the corresponding calculation update for the remote atomic operation. It then writes the updated data back to memory and encapsulates the response data into a remote atomic operation response message RaoRsp and sends it back to the source node; ③ After receiving the remote atomic operation response message RaoRsp, the source node's RDMA network card notifies the source node of the result as a local event. 1. The source node's RDMA network card establishes a connection for a remote event to be sent to the destination node, sending a connection message CnctRqt into the RDMA network; 2. After receiving the connection message CnctRqt, the destination node's RDMA network card establishes a connection with the source node, generates a connection response message CnctRspAck, and sends it back to the source node; 3. Upon receiving the connection response message CnctRspAck, the source node marks the establishment of a complete event connection and sends a remote event request message EvtWtRqt to the destination node; 4. Upon receiving the remote event request message EvtWtRqt, the destination node's RDMA network card notifies the local process to complete a remote atomic operation as an event and sends a connection response completion message CnctRspCplt back to the source node; 5. Upon receiving the connection response completion message CnctRspCplt, the source node marks the completion of a remote atomic operation.

[0022] like Figure 1As shown in (b), the hardware offloading of remote atomic operations using the half-connection RAO protocol includes: ① The source node submits a RAO descriptor to its local RDMA network card. The RDMA network card parses the RAO descriptor, encapsulates the RAO operation information into a remote atomic operation request message RaoRqt, and sends it to the RDMA network through the sending engine; ② After receiving the remote atomic operation request message RaoRqt, the destination node's RDMA network card forwards the message to the RAO response processing unit according to the message type. The RAO response processing unit reads data from the destination node's memory according to the remote atomic operation type of the RaoRqt message and completes the corresponding calculation update for the remote atomic operation. The updated data is then written back to memory; the response data is encapsulated into a remote atomic operation response. The process involves several steps: 1. The source node's RDMA network card receives the remote atomic operation response message RaoRsp and sends it back to the source node. 2. The source node's RDMA network card establishes a connection for the remote event to be sent to the destination node and sends the remote event request message EvtWtRqt to the destination node. 3. After receiving the remote event request message EvtWtRqt, the destination node's RDMA network card notifies the local process of completing a remote atomic operation as an event and sends the connection completion response message CnctRspCplt back to the source node. 4. After receiving the connection completion response message CnctRspCplt, the source node marks the completion of a remote atomic operation. Compared with the fully connected RAO protocol, the half-connection RAO protocol in this embodiment provides a more streamlined connection protocol. Compared with the fully connected RAO protocol, it omits the handshake for establishing a connection for the remote event message, thus providing lower latency. However, while the fully connected RAO protocol involves three round-trip message communications for a single RAO operation, it offers higher reliability. If a fiber optic link fails and a message is lost, both the source and destination nodes will initiate a timeout retransmission. In contrast, the half-connected RAO protocol cannot complete the RAO operation successfully if the remote event message EvtWtRqt is lost.

[0023] In this embodiment, the remote atomic operation types include three types: post, fetch, and swap. The remote atomic operations under post include four types: summation ADD, AND, OR, and XOR. After the operation is completed, the destination node does not return the original data before the operation to the source node. As an optional implementation, in this embodiment, the names of summation ADD, AND, OR, and XOR under the post type are all prefixed with "ATOMIC_OP_", and the specific definitions are shown in Table 1.

[0024]

[0025] The remote atomic operations under fetch include four types: summation ADD, AND, OR, and XOR. After the operation is completed, the destination node returns the original data before the operation to the source node. As an optional implementation, in this embodiment, the names of summation ADD, AND, OR, and XOR under the fetch type are all prefixed with "ATOMIC_OP_", and the specific definitions are shown in Table 2.

[0026]

[0027] Remote atomic operations under the swap type include two types: SWAP and CSWAP. SWAP parameters include the remote address ptr and the sent data opd. SWAP swaps the data in the remote address ptr with the sent data opd and returns the original value of the remote address ptr to the source node. CSWAP parameters include the remote address ptr and the sent data cmp and opd. CSWAP compares the data in the remote address ptr with the sent data cmp. If they are equal, the data in the remote address ptr and the sent data opd are swapped, and the original value of the remote address ptr is returned to the source node. If they are not equal, no operation is performed, and the original value of the remote address ptr is returned to the source node. As an optional implementation, in this embodiment, the names of both SWAP and CSWAP remote atomic operations under the swap type are prefixed with "ATOMIC_OP_", as shown in Table 3.

[0028]

[0029] Referring to Tables 1, 2, and 3, this embodiment implements 10 remote atomic operations across three main categories: post, fetch, and swap. Data types supported include INT32 and INT64. Post-type remote atomic operations include four operations: ADD, AND, OR, and XOR. After the operation, the destination node does not return the original data to the source node. For example, void ATOMIC_OP_ADD(int *ptr, int opd) adds the data in the remote address ptr to the sent data opd and updates the result in ptr; the corresponding opcode is 6'h0. Fetch-type remote atomic operations also include four operations: ADD, AND, OR, and XOR. The difference from post-type operations is that after the operation, the destination node returns the original data to the source node. For example, void ATOMIC_OP_FAND(int *ptr, int opd) performs a bitwise AND operation on the data in the remote address ptr and the sent data opd, updates the result in ptr, and also returns the original value in that address to the source node; the corresponding opcode is 6'h5. Remote atomic operations of the swap type include two types: SWAP and CSWAP. A SWAP operation involves exchanging the data in the remote address ptr with the sent data opd, and then returning the original value at that address to the source node. A CSWAP operation involves comparing the data in the remote address ptr with the sent cmp; if they are equal, the data in ptr and opd are exchanged, and the original value at that address is returned to the source node; if they are not equal, no operation is performed, and the original value at that address is returned to the source node.

[0030] In this embodiment, the RAO response processing component reads data from the destination node's memory and performs the corresponding calculation update for the remote atomic operation based on the remote atomic operation type of the remote atomic operation request message RaoRqt. Writing the updated data back to memory includes: checking if the remote atomic operation request message RaoRqt has a CRC error, checking the destination address, performing a virtual-to-real address transformation, reading the original data from the host destination address to the destination node's RDMA network card, performing the corresponding calculation update for the remote atomic operation, and writing the updated data back to the host destination address. If the remote atomic operation type of the remote atomic operation request message RaoRqt is fetch or swap, the original value of the atomic operation is sent to the requesting end; if the remote atomic operation type of the remote atomic operation request message RaoRqt is post, no return is required.

[0031] like Figure 2As shown, in this embodiment, the RAO response processing component includes a RAO request queue, an arbitrator, and multiple arithmetic logic units (ALUs). Example 2 shows four ALUs, denoted as ALU0 to ALU3. The RAO request queue is used to buffer remote atomic operation request messages RaoRqt. Figure 2 In the RAO request queue, rqt0 to rqt2 are all remote atomic operation request messages RaoRqt. The arbitrator is used to arbitrate and distribute the remote atomic operation request messages RaoRqt in each cache of the RAO request queue to the arithmetic logic unit (ALU). The multiple arithmetic logic units (ALUs) are used to read data from the memory of the destination node in parallel, complete the corresponding calculation update of the remote atomic operation, and write the updated data back to memory. Figure 2 This demonstrates the specific processing flow of one of the ALUs for two ATOMIC_OP_ADD operations (summing ADD). The remote atomic operation request message RaoRqt for the first ATOMIC_OP_ADD operation is rqt0, and the remote atomic operation request message RaoRqt for the second ATOMIC_OP_ADD operation is rqt1. The arithmetic logic unit (ALU) executes the summation ADD under fetch, which includes: ① The ALU parses the content of the remote atomic operation request message RaoRqt of the summation ADD and reads the value of the operation object, variable x, from the target address in the host memory; ② Increments the value of variable x by 1. Since the RAO type is post, there is no need to return data to the source node; ③ Submits a write-back request for x=1 to the message submission module; ④ The message submission module sends the request to the PCIe interface module after queuing and scheduling, and returns a response signal ACK to the RAO response processing unit after submission. After receiving the response signal ACK, the ALU ends the processing flow of the remote atomic operation request message RaoRqt of the summation ADD. Figure 2 In the diagram, steps ⑤-⑧ correspond to the ALU's processing of the second summation ADD (rqt1). Since the value of variable x in the host memory is 0, the summation ADD operation with the remote atomic operation request message RaoRqt of the first ATOMIC_OP_ADD operation being rqt0 is: x = 0 + 1, and step ④ writes back the value of variable x as 1; the value of variable x in the remote atomic operation request message RaoRqt of the second ATOMIC_OP_ADD operation being rqt1 is the result of the first ATOMIC_OP_ADD operation, which is 1, so the summation ADD operation is: x = 1 + 1, and step ⑧ writes back the value of variable x as 2.

[0032] Because the PCIe interface module has a pre-order function, PCIe read and write operations from different RaoRqt requests will not be out of order and will not cause RAW (Read-After-Write) data hazards. The ACK mechanism of the message submission module ensures both high concurrency of remote atomic operations and the correctness of data results.

[0033] In this embodiment, the arbitrator is used to arbitrate and distribute the remote atomic operation request messages RaoRqt in each cache of the RAO request queue to the arithmetic logic unit (ALU). This means that, based on the idle status and address conflict status of the ALU, a remote atomic operation request message RaoRqt is taken out and sent to the available ALU that has no address conflict.

[0034] like Figure 3As shown, in this embodiment, the arithmetic logic unit (ALU) includes a RAO message parsing and processing module, a destination address splitting module, a RAO data processing module, and a source address splitting module. The working process of the ALU includes: (1) The message receiving module writes the remote atomic operation request message RaoRqt from other nodes into the RAO request queue; (2) The arbitrator, based on the idle status and address conflict status of the ALU, retrieves a remote atomic operation request message RaoRqt and sends it to an available ALU without address conflicts; (3) The RAO message... The message parsing and processing module first receives the remote atomic operation request message RaoRqt and starts the processing flow. At the same time, it calls the address validity check module in the RDMA network card to perform address validity check, and calls the destination address splitting module to perform destination address splitting operation. The state machine controls the message processing state; (4) The destination address splitting module is responsible for performing the address splitting task. It calls the virtual address translation module in the RDMA network card to translate the virtual address of the destination in the message into a virtual address to obtain the physical address; (5) The destination address splitting module returns the obtained physical address to the RAO message parser. Processing module; (6) The RAO message parsing and processing module sends a DMA read request to the PCIe interface module according to the starting address and length of the physical address; (7) The RAO message parsing and processing module sends the remote atomic operation request message RaoRqt to the RAO data processing module; (8) The RAO data processing module starts the message processing state machine and sends the source address and length information to the source address splitting module for virtual address splitting; (9) After completing the address splitting, the source address splitting module returns the split index number, offset and length to the RAO data processing module; (10) PCIe interface module The DMA read data is written into the RAO data FIFO in the RDMA network card. The RAO data processing module reads the original data corresponding to the pending request cached in the RAO data FIFO. (11) If the remote atomic operation type of the remote atomic operation request message RaoRqt is fetch and swap, the RAO data processing module encapsulates the RAO original data into a remote atomic operation response message RaoRsp and sends it to the source node of the RAO request. If the remote atomic operation type of the remote atomic operation request message RaoRqt is post, it is ignored. (12) The RAO data processing module starts the remote atomic operation calculation and submits the calculation result to the message submission module. (13) The message submission module caches the request and submits the result to the PCIe interface module in the manner of DMA write request. After the request is submitted, it sends a response signal ACK to the RAO data processing module to complete a complete RAO operation process.

[0035] Furthermore, this embodiment also provides a computer device including a microprocessor and an RDMA network card interconnected, wherein the RDMA network card is programmed or configured to execute the RDMA network card-based remote atomic operation hardware offloading method.

[0036] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for remote atomic operation hardware offloading based on an RDMA network card, characterized in that, This includes implementing remote atomic operation hardware offloading between the source and destination nodes using a fully connected or half-connected RAO protocol. The process involves: the source node submitting a RAO descriptor to its local RDMA network interface card (NIC); the RDMA NIC parsing the descriptor, encapsulating the remote atomic operation information into a remote atomic operation request message (RaoRqt), and sending it to the RDMA network via a sending engine; the destination node's RDMA NIC receiving the RaoRqt message and forwarding it to the RAO response processing unit based on the message type; and the RAO response processing unit reading data from the destination node's memory based on the remote atomic operation type, performing corresponding calculations and updates, and updating the data. After writing back to memory, each subsequent operation encapsulates a remote atomic operation response message RaoRsp and sends it back to the source node. Upon receiving the RaoRsp message, the source node's RDMA network card notifies the source node of the result as a local event. Subsequently, the source node's RDMA network card sends a remote event request message EvtWtRqt to the destination node. Upon receiving EvtWtRqt, the destination node's RDMA network card notifies the local process of the completion of a remote atomic operation as an event and sends a connection completion response message CnctRspCplt back to the source node. Upon receiving the CnctRspCplt message, the source node marks the completion of a remote atomic operation.

2. The remote atomic operation hardware offloading method based on an RDMA network card according to claim 1, characterized in that, The hardware offloading of remote atomic operations using the fully connected RAO protocol includes: ① The source node submits a RAO descriptor to its local RDMA network card. The RDMA network card parses the RAO descriptor, encapsulates the RAO operation information into a remote atomic operation request message RaoRqt, and sends it to the RDMA network through the sending engine; ② After receiving the remote atomic operation request message RaoRqt, the destination node's RDMA network card forwards the message to the RAO response processing unit according to the message type. The RAO response processing unit reads data from the destination node's memory according to the remote atomic operation type of the RaoRqt message and completes the corresponding calculation and update for the remote atomic operation. The updated data is then written back to memory. The response data is encapsulated into a remote atomic operation response message RaoRsp and sent back to the source node; ③ After receiving the remote atomic operation response message RaoRsp, the source node's RDMA network card notifies the source node of the result as a local event.

1. The source node's RDMA network card establishes a connection for a remote event to be sent to the destination node, sending a connection message CnctRqt into the RDMA network; 2. After receiving the connection message CnctRqt, the destination node's RDMA network card establishes a connection with the source node, generates a connection response message CnctRspAck, and sends it back to the source node; 3. Upon receiving the connection response message CnctRspAck, the source node marks the establishment of a complete event connection and sends a remote event request message EvtWtRqt to the destination node; 4. Upon receiving the remote event request message EvtWtRqt, the destination node's RDMA network card notifies the local process to complete a remote atomic operation as an event and sends a connection response completion message CnctRspCplt back to the source node; 5. Upon receiving the connection response completion message CnctRspCplt, the source node marks the completion of a remote atomic operation.

3. The remote atomic operation hardware offloading method based on an RDMA network card according to claim 1, characterized in that, The hardware offloading of remote atomic operations using the half-connection RAO protocol includes: ① The source node submits a RAO descriptor to its local RDMA network card. The RDMA network card parses the RAO descriptor, encapsulates the RAO operation information into a remote atomic operation request message RaoRqt, and sends it to the RDMA network through the sending engine; ② After receiving the remote atomic operation request message RaoRqt, the destination node's RDMA network card forwards the message to the RAO response processing unit according to the message type. The RAO response processing unit reads data from the destination node's memory according to the remote atomic operation type in the RaoRqt message, completes the corresponding calculation and update for the remote atomic operation, writes the updated data back to memory, and encapsulates the response data into a remote atomic operation response. ① The source node's RDMA network card receives the remote atomic operation response message RaoRsp and sends it back to the source node; ② After receiving the remote atomic operation response message RaoRsp, the source node's RDMA network card notifies the source node of the result as a local event; ③ The source node's RDMA network card establishes a connection for the remote event to be sent to the destination node and sends the remote event request message EvtWtRqt to the destination node; ④ After receiving the remote event request message EvtWtRqt, the destination node's RDMA network card notifies the local process to complete a remote atomic operation as an event and sends the connection response completion message CnctRspCplt back to the source node; ⑤ After receiving the connection response completion message CnctRspCplt, the source node marks the completion of a remote atomic operation.

4. The remote atomic operation hardware offloading method based on an RDMA network card according to claim 2 or 3, characterized in that, The remote atomic operation types include three types: post, fetch, and swap. Post operations include four types: summation (ADD), AND, OR, and XOR. After the operation, the destination node does not return the original data to the source node. Fetch operations also include four types: summation (ADD), AND, OR, and XOR. After the operation, the destination node returns the original data to the source node. Swap operations include two types: swap and conditional swap (CSWAP). Swap parameters include the remote address ptr and the sent data opd. Swap swaps the data in the remote address ptr and the sent data opd, and returns the original value in the remote address ptr to the source node. The parameters of Conditional Swap (CSWAP) include the remote address ptr and the sent data cmp and opd. CSWAP is used to compare the data in the remote address ptr with the sent data cmp. If they are equal, the data in the remote address ptr and the sent data opd are swapped, and the original value in the remote address ptr is returned to the source node; if they are not equal, no operation is performed, and the original value in the remote address ptr is returned to the source node.

5. The remote atomic operation hardware offloading method based on an RDMA network card according to claim 4, characterized in that, The RAO response processing component reads data from the destination node's memory and performs the corresponding calculation and update for the remote atomic operation based on the remote atomic operation type of the remote atomic operation request message RaoRqt. Writing the updated data back to memory includes: checking for CRC errors in the remote atomic operation request message RaoRqt, checking the destination address, performing a virtual-to-real address transformation, reading the original data from the host destination address to the destination node's RDMA network card, performing the corresponding calculation and update for the remote atomic operation, and writing the updated data back to the host destination address. If the remote atomic operation type of the remote atomic operation request message RaoRqt is fetch or swap, the original value of the atomic operation is sent to the requesting end; if the remote atomic operation type of the remote atomic operation request message RaoRqt is post, no return is required.

6. The remote atomic operation hardware offloading method based on an RDMA network card according to claim 5, characterized in that, The RAO response processing component includes a RAO request queue, an arbitrator, and multiple arithmetic logic units (ALUs). The RAO request queue is used to cache remote atomic operation request messages RaoRqt. The arbitrator is used to arbitrate and distribute the remote atomic operation request messages RaoRqt cached in the RAO request queue to the ALUs. The multiple ALUs are used to read data from the memory of the destination node in parallel, complete the corresponding calculation updates of the remote atomic operation, and write the updated data back to memory.

7. The remote atomic operation hardware offloading method based on an RDMA network card according to claim 6, characterized in that, The arbitrator is used to arbitrate and distribute the remote atomic operation request messages RaoRqt from each cache of the RAO request queue to the arithmetic logic unit (ALU). This means that, based on the idle status and address conflict status of the ALU, a remote atomic operation request message RaoRqt is retrieved and sent to an available ALU that is not in conflict with its address.

8. The remote atomic operation hardware offloading method based on an RDMA network card according to claim 6, characterized in that, The arithmetic logic unit (ALU) includes a RAO message parsing and processing module, a destination address splitting module, a RAO data processing module, and a source address splitting module. The working process of the ALU includes: (1) The message receiving module writes the remote atomic operation request message RaoRqt from other nodes into the RAO request queue; (2) The arbitrator, based on the idle status and address conflict status of the ALU, retrieves a remote atomic operation request message RaoRqt and sends it to an available ALU without address conflicts; (3) The RAO message parsing and processing module... The processing module first receives the remote atomic operation request message RaoRqt and starts the processing flow. At the same time, it calls the address validity check module in the RDMA network card to perform address validity check, and calls the destination address splitting module to perform destination address splitting operation. The state machine controls the message processing state; (4) The destination address splitting module is responsible for performing the address splitting task. It calls the virtual address translation module in the RDMA network card to translate the virtual address of the destination in the message into a virtual address to obtain the physical address; (5) The destination address splitting module returns the obtained physical address to the RAO message parsing processing module. (6) The RAO message parsing and processing module sends a DMA read request to the PCIe interface module based on the starting address and length of the physical address; (7) The RAO message parsing and processing module sends the remote atomic operation request message RaoRqt to the RAO data processing module; (8) The RAO data processing module starts the message processing state machine and sends the source address and length information to the source address splitting module for virtual address splitting; (9) After completing the address splitting, the source address splitting module returns the split index number, offset and length to the RAO data processing module; (10) The PCIe interface module sends the D MA reads data and writes it into the RAO data FIFO in the RDMA network card. The RAO data processing module reads the original data corresponding to the pending request cached in the RAO data FIFO. (11) If the remote atomic operation type of the remote atomic operation request message RaoRqt is fetch and swap, the RAO data processing module encapsulates the RAO original data into a remote atomic operation response message RaoRsp and sends it to the source node of the RAO request. If the remote atomic operation type of the remote atomic operation request message RaoRqt is post, it is ignored. (12) The RAO data processing module starts the remote atomic operation calculation and submits the calculation result to the message submission module. (13) The message submission module caches the request and submits the result to the PCIe interface module in the manner of DMA write request. After the request is submitted, it sends a response signal ACK to the RAO data processing module to complete a complete RAO operation process.

9. A computer device comprising a microprocessor and an RDMA network card interconnected, characterized in that, The RDMA network card is programmed or configured to perform the remote atomic operation hardware offloading method based on any one of claims 1 to 8.

Images