Methods, apparatus, computer devices, readable storage media, and program products for out-of-order reception of remote direct memory access.

Abstract

This application relates to a method, apparatus, computer device, computer-readable storage medium, and computer program product for out-of-order reception of remote direct memory access, applied to a responding end. It includes: receiving a request message from a requesting end; the request message is obtained by the requesting end by adding total message length, offset information, and message reception sequence number to each data message; determining whether the message reception sequence number already exists in a preset out-of-order reception table entry; if not, determining the out-of-order offset, pre-allocating reception queues, and creating an out-of-order reception table entry corresponding to each reception queue; if the offset information is valid, obtaining the index information corresponding to the offset information and calculating the write address of the request message; writing the data carried by the request message into memory; and if the received message length corresponding to the message reception sequence number is equal to the total message length, sending an acknowledgment message to the requesting end. This method can accurately identify the message to which a message belongs when messages are out of order or lost.

Description

Technical Field

[0001] This application relates to the field of network communication technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for remote direct memory access out-of-order reception. Background Technology

[0002] In modern data transmission scenarios such as intelligent computing networks, HPC (High Performance Computing) data centers, and VPC (Virtual Private Cloud) networks, RDMA (Remote Direct Memory Access) technology has become the core choice for high-frequency, high-volume data transmission because it can bypass the CPU (Central Processing Unit) to directly realize data transmission between memory and reduce latency. Among them, RDMASend (Remote Direct Memory Access Send Operation) is a key data transmission mechanism that widely supports core businesses such as distributed computing and storage access.

[0003] Existing RDMA Send methods struggle to identify the message to which a packet belongs when packets are out of order or lost, and they are not well-suited for handling non-contiguous memory. In RDMA Read processing, when the requesting end receives an out-of-order Response, it cannot directly extract the correct Read destination address. Summary of the Invention

[0004] Therefore, it is necessary to provide a remote direct memory access method, apparatus, computer device, computer-readable storage medium, and computer program product that can accurately identify the message to which a message belongs when messages are out of order or lost, in order to address the aforementioned technical problems.

[0005] Firstly, this application provides a method for out-of-order reception of remote direct memory access, applied to the response end, including:

[0006] Receive request messages from the requesting end; the request message is obtained by the requesting end by adding the total message length, offset information and message reception sequence number to each data message; the data message is obtained by the requesting end by splitting the data to be sent based on the current network's maximum transmission unit; the total message length and the current message reception sequence number are obtained by the requesting end based on the data to be sent.

[0007] Based on the preset maximum message out-of-order level, the received message sequence number, and the message receiving sequence number, the validity of the request message is determined; if the determination result is that the request message is valid, then it is determined whether the message receiving sequence number already exists in the preset out-of-order reception entry.

[0008] If it does not exist, then based on the message receiving sequence number and the received message receiving sequence number, determine the out-of-order offset; based on the out-of-order offset and the message receiving sequence number, pre-allocate receiving queues and add corresponding out-of-order receiving entries to each receiving queue;

[0009] Based on the offset information of the request message and the out-of-order receive table entry corresponding to the receive queue where the request message is located, determine whether the offset information is valid; if the determination result is that the offset information is valid, obtain the index information corresponding to the offset information; based on the index information, calculate the write address of the request message; based on the write address, write the data carried by the request message into memory.

[0010] Determine if the length of the message received corresponding to the message receiving sequence number is equal to the total message length. If it is equal, send an acknowledgment message to the requesting end.

[0011] In one embodiment, a receive queue is established based on the out-of-order offset and the message reception sequence number, including:

[0012] Increment the out-of-order offset by 1 to obtain the first quantity; create the first quantity of receive queues; establish a mapping relationship between each received message sequence number and the first quantity of receive queues; the receive queues are used to receive the request messages corresponding to the corresponding message sequence numbers.

[0013] In one embodiment, the validity of a request message is determined based on a preset maximum message out-of-order level, the received message sequence number, and the message reception sequence number, including:

[0014] If the message receiving sequence number is less than the received message receiving sequence number, the request message is a duplicate message and is deemed invalid. If the result of subtracting the received message receiving sequence number from the received message receiving sequence number is not less than the preset maximum message out-of-order level, the request message is an out-of-order message and is deemed invalid. If the message receiving sequence number is not less than the received message receiving sequence number and is not greater than the sum of the received message receiving sequence number and the preset maximum message out-of-order level, the request message is deemed valid.

[0015] In one embodiment, the method further includes:

[0016] If it exists, then the validity of the offset information is determined based on the offset information of the request message and the out-of-order receive entry corresponding to the receive queue where the request message is located.

[0017] In one embodiment, the method further includes:

[0018] If the determination result is that the offset information is invalid, a response message is sent to the requesting end, indicating that the request message is invalid.

[0019] In one embodiment, after sending the confirmation information to the requesting end, the method further includes:

[0020] The message receiving sequence number is added to the completion queue corresponding to the receiving queue, and the entry associated with the message receiving sequence number in the out-of-order receiving table entry corresponding to the receiving queue is deleted; the completion queue is used to record message identifiers that have been fully received and whose data verification is error-free.

[0021] In one embodiment, the method further includes:

[0022] The system receives a read message from the requesting end and verifies its validity. The read message is obtained by the requesting end adding a hash extension header to the read message to be transmitted. The hash extension header is generated based on hash list information, which is obtained by the requesting end from its local memory. The hash list information includes the number of hash elements and the hash elements themselves, which include a virtual address, access key, and length. If valid, the system determines the target data to be read. Based on the virtual address, access key, length, and the maximum transmission unit of the current network within the hash elements in the read message, the system splits the target data to obtain a reply data message. A read response extension header is added to the reply data message to obtain a reply message. Each reply message corresponds to only one hash element. The read response extension header includes the virtual address and access key of the matching hash element. The system sends the reply message to the requesting end, instructing it to write the data in the reply message into its local memory based on the access key and virtual address of the matching hash element.

[0023] In one embodiment, reading the message includes: a remote access key; verifying the validity of the read message includes:

[0024] Obtain the remote access key, target virtual address, and request data length from the read message; locate the memory region corresponding to the remote access key in the protection domain associated with the current queue pair; the queue pair refers to a corresponding send queue and receive queue; verify whether the remote access key is a registered valid key, whether the associated memory region has read permissions, whether the address range formed by the target virtual address and the request data length is within the registered memory range, and whether the corresponding protection domain is consistent with the queue pair bound to the current queue pair; if all four verifications pass, the verification result is valid; if any verification fails, the verification result is invalid.

[0025] Secondly, this application also provides a method for out-of-order reception of remote direct memory access, applied to the requesting end, including:

[0026] Obtain the total message length and current message reception sequence number of the data to be sent; split the data to be sent based on the current network's maximum transmission unit to obtain multiple data packets;

[0027] For each data packet, add the total message length, offset information, and message reception sequence number to obtain multiple request packets. Send the request packets to the response end to instruct it to determine the validity of the request packets based on a preset maximum message out-of-order level, the received message reception sequence number, and the message reception sequence number. If the determination result is that the request packet is valid, check if the message reception sequence number already exists in a preset out-of-order reception table entry. If not, determine the out-of-order offset based on the message reception sequence number and the received message reception sequence number. Based on the out-of-order offset and the message reception sequence number, pre-allocate reception queues and add corresponding out-of-order reception table entries to each reception queue. Based on the offset information of the request packet and the out-of-order reception table entry corresponding to the reception queue where the request packet is located, determine if the offset information is valid. If the determination result is that the offset information is valid, obtain the index information corresponding to the offset information. Based on the index information, calculate the write address of the request packet. Based on the write address, write the data carried by the request packet into memory. Determine if the received message length corresponding to the message reception sequence number is equal to the total message length. If equal, send an acknowledgment message to the request end.

[0028] In one embodiment, the method further includes:

[0029] Obtain the hash list information from the local memory; generate a hash extended transmission header based on the hash list information; the hash list information includes the number of hash elements and the hash elements themselves, each including a virtual address, access key, and length; add the hash extended transmission header to the read message to be transmitted, obtaining the read message, and send the read message to the response end to instruct the response end to verify the validity of the read message; if valid, determine the target data to be read; based on the virtual address, access key, length, and the current network's maximum transmission unit within the hash elements in the read message, split the target data to obtain a reply data message; match the reply data message with the hash elements; add a read response extended header to the reply data message to obtain a reply message; a single reply message corresponds to only one hash element; the read response extended header includes the virtual address and access key of the matching hash element; receive the reply message from the response end, and write the data in the reply message into the local memory based on the access key and virtual address of the matching hash element in the reply message.

[0030] Thirdly, this application also provides a remote direct memory access out-of-order receiving device for use at a response end, the device implementing the steps of the method as described in the first aspect.

[0031] Fourthly, this application also provides an out-of-order receiving device for remote direct memory access, applied to the requesting end, the device implementing the steps of the method as described in the second aspect.

[0032] Fifthly, this application also provides a computer device applied to a response end, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method as described in the first aspect.

[0033] In a sixth aspect, this application also provides a computer device applied to a requesting end, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method as described in the second aspect.

[0034] In a seventh aspect, this application also provides a computer-readable storage medium applied to a response end, on which a computer program is stored, which, when executed by a processor, implements the steps of the method as described in the first aspect.

[0035] Eighthly, this application also provides a computer-readable storage medium applied to a requesting end, having stored thereon a computer program that, when executed by a processor, implements the steps of the method as described in the second aspect.

[0036] Ninthly, this application also provides a computer program product applied to a response end, including a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect.

[0037] In a tenth aspect, this application also provides a computer program product applied to a requesting end, comprising a computer program that, when executed by a processor, implements the steps of the method as described in the second aspect.

[0038] The aforementioned remote direct memory access out-of-order reception method, apparatus, computer device, computer-readable storage medium, and computer program product receive a request message from a requesting end. The request message is obtained by the requesting end adding total message length, offset information, and message reception sequence number to each data message. The data message is obtained by the requesting end by splitting the data to be sent based on the current network's maximum transmission unit (MTBF). The total message length and the current message reception sequence number are obtained by the requesting end based on the data to be sent. The validity of the request message is determined based on a preset maximum message out-of-order level, already received message reception sequence numbers, and the message reception sequence number. If the determination result is that the request message is valid, it is then determined whether the message reception sequence number already exists in a preset out-of-order reception table. If the message does not exist, the out-of-order offset is determined based on the message receiving sequence number and the received message receiving sequence number. Based on the out-of-order offset and the message receiving sequence number, a receiving queue is pre-allocated, and a corresponding out-of-order receiving entry is added to each receiving queue. Based on the offset information of the request message and the out-of-order receiving entry corresponding to the receiving queue where the request message is located, the validity of the offset information is determined. If the determination result is that the offset information is valid, the index information corresponding to the offset information is obtained. Based on the index information, the write address of the request message is calculated. Based on the write address, the data carried by the request message is written to memory. It is determined whether the length of the message already received corresponding to the message receiving sequence number is equal to the total message length. If it is equal, an acknowledgment message is sent to the requesting end. By configuring a message receiving sequence number for each RDMA data packet and embedding the request message, it is possible to quickly associate each data packet corresponding to the message receiving sequence number, thereby accurately identifying the message to which the message belongs when the message is out of order / packet. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a diagram illustrating the application environment of an out-of-order reception method for remote direct memory access in one embodiment.

[0041] Figure 2 This is a flowchart illustrating the application of a remote direct memory access out-of-order reception method to the response end in one embodiment.

[0042] Figure 3 This is a schematic diagram of the structure of a request message in one embodiment;

[0043] Figure 4 This is a schematic diagram of the structure for reading a message in one embodiment;

[0044] Figure 5 This is a schematic diagram of the structure of a reply data packet in one embodiment;

[0045] Figure 6 This is a flowchart illustrating the application of a remote direct memory access out-of-order reception method to the requesting end in one embodiment.

[0046] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0049] The out-of-order reception method for remote direct memory access provided in this application can be applied to, for example... Figure 1In the application environment shown, the requesting end 102 communicates with the responding end 104 via a network. A data storage system can store the data that the requesting end 102 and the responding end 104 need to process. The data storage system can be integrated on the requesting end 102 and the responding end 104, or it can be placed on the cloud or other network servers. The requesting end 102 obtains the total message length and the current message reception sequence number of the data to be sent; it splits the data to be sent based on the maximum transmission unit of the current network to obtain multiple data packets; it adds the total message length, offset information, and message reception sequence number to each data packet to obtain multiple request packets; it sends the request packets to the responding end; the responding end 104 receives the request packets from the requesting end; based on the preset maximum message out-of-order level, the received message reception sequence number, and the message reception sequence number, it determines the validity of the request packets; if the determination result is that the request packets are valid, it determines whether the message reception sequence number already exists in the preset out-of-order reception entry; if not, it determines the validity of the request packets based on the message reception sequence number. The system receives the sequence number and the received message sequence number to determine the out-of-order offset. Based on the out-of-order offset and the message sequence number, it pre-allocates receiving queues and adds corresponding out-of-order receiving entries to each receiving queue. Based on the offset information of the request message and the out-of-order receiving entry corresponding to the receiving queue where the request message is located, it determines whether the offset information is valid. If the determination result is that the offset information is valid, it obtains the index information corresponding to the offset information. Based on the index information, it calculates the write address of the request message. Based on the write address, it writes the data carried by the request message into memory. It determines whether the message receiving sequence number is equal to the total message length. If it is equal, it sends an acknowledgment message to the requesting end 102. The requesting end 102 and the responding end 104 can be, but are not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0050] In one exemplary embodiment, such as Figure 2 As shown, a method for out-of-order reception of remote direct memory access is provided, which is applied to... Figure 1 Taking response terminal 104 as an example, the explanation includes the following steps 202 to 210. Wherein:

[0051] Step 202: Receive a request message from the requesting end; the request message is obtained by the requesting end by adding the total message length, offset information and message receiving sequence number to each data message; the data message is obtained by the requesting end by splitting the data to be sent based on the current network's maximum transmission unit; the total message length and the current message receiving sequence number are obtained by the requesting end based on the data to be sent.

[0052] Among them, such as Figure 3 As shown, a standard RDMA Send message includes an Ethernet header, an IP header, a UDP header, an InfiniBand Base Transport Header (IB BTH), an InfiniBand Payload, an Invariant Cyclic Redundancy Check (ICRC), and a Frame Check Sequence (FCS). In addition to the above, the request message in this application adds a METH Message Extended Transport Header between the IB BTH header and the IB Payload. The METH extension header includes: MsgLen (Message Length, total number of bytes in the message), offset information (Offset, the byte offset position of the current message data in the complete message), and Receive Sequence Number (RxSN, Receive Sequence Number, a unique sequence number used to identify the message to which the message belongs).

[0053] Step 204: Based on the preset maximum message out-of-order level, the received message sequence number, and the message receiving sequence number, determine the validity of the request message; if the determination result is that the request message is valid, then determine whether the message receiving sequence number already exists in the preset out-of-order reception entry.

[0054] The preset maximum message out-of-order degree (Maximum Message Out-of-Order Degree, the upper limit on the number of out-of-order messages that the responder can process) is a key parameter configured by the responder when switching the QP (Queue Pair) to the read-to-receive (RTR) state. It is used to prevent the requesting end from sending excessive out-of-order attack packets and exhausting the responder's resources. For example, the preset value can be set to 5, indicating that the responder can support processing a maximum of 5 out-of-order messages that are ahead of the current expected sequence. The received message sequence number (RxSN) is the set of message sequence numbers that the responder has successfully received and recorded. It is usually taken as the maximum received message sequence number (MaximumReceived RxSN). The default Out-of-Order Receive Table is a metadata table maintained by the responder for each Receive Queue (RQ) or Shared Receive Queue (SRQ). The table entries include the message reception sequence number (RxSN), the corresponding Receive Queue element index (RQE Index, an identifier associated with the receive memory), the Completion Queue element index (CQE Index, an identifier associated with the completion notification), the reception status (RxStatus, indicating whether the message has been fully received), and a memory range sub-table (recording the offset mapping of non-contiguous memory SGE). It is used to temporarily store the metadata of out-of-order received messages and support subsequent address calculation and data writing.

[0055] Step 206: If it does not exist, determine the out-of-order offset based on the message receiving sequence number and the received message receiving sequence number; pre-allocate receiving queues based on the out-of-order offset and the message receiving sequence number, and add corresponding out-of-order receiving entries to each receiving queue.

[0056] In this step, the key reference for the received message sequence number is the expected received sequence number (Expected RxSN, the sequence number of the next message to be received in order, initially set to 0) maintained by the responding end. This is because the expected received sequence number represents the baseline sequence to be received in the current non-out-of-order condition and is the core basis for calculating the out-of-order offset, rather than referring to all historically received sequence numbers. The calculation logic for the out-of-order offset is as follows: subtract the expected received sequence number (Expected RxSN) from the current received sequence number (Current RxSN) carried in the request message; the difference is the out-of-order offset. For example, if the expected received sequence number is 3, the current request message's received sequence number is 5, and the preset maximum message out-of-order level is ≥2, then the out-of-order offset = 5 - 3 = 2. This value represents the number of messages in the current message that are ahead of the expected sequence, and also provides a quantitative basis for subsequent prefetching of received resources. The pre-allocated receive queue (RQ, used for receiving and temporarily storing request message data) pre-allocates resources from the existing receive queue (or shared receive queue, SRQ) on the responding end based on the out-of-order offset. Specifically, based on the out-of-order offset d (i.e., Current RxSN - Expected RxSN), d+1 receive queue elements (RQE, containing information such as receive memory address and access key) are pre-fetched from the receive queue at once. A one-to-one mapping relationship is established between the "expected receive sequence number ~ current message receive sequence number" (e.g., Expected RxSN=3 to Current RxSN=5, corresponding to sequence numbers 3, 4, and 5) and the pre-fetched RQEs, ensuring that each out-of-order message has dedicated receive memory resources. When creating the out-of-order receive table (the table used to store out-of-order message metadata) for each receive queue, an independent table entry is created for each mapped message sequence number (e.g., 3, 4, and 5) based on the pre-fetched RQE resources.The table entries should include at least the following: message reception sequence number (RxSN, used to identify the message identity), the corresponding RQE index (RQE Index, associated with the prefetched receive memory), completion queue element index (CQE Index, pre-allocated completion notification resource identifier), reception status (RxStatus, initialized to "incomplete", used to track whether the message has been received), and a memory range sub-table (if the receive memory is a non-contiguous SGL (Scatter / Gather List) structure, the sub-table will record the offset of each SGE, such as SGE[0] corresponding to offset 0~1023, SGE[1] corresponding to offset 1024~2047, supporting subsequent address calculation), thereby providing metadata support for subsequent message validity verification and data writing address location.

[0057] Step 208: Based on the offset information of the request message and the out-of-order receive table entry corresponding to the receive queue where the request message is located, determine whether the offset information is valid; if the determination result is that the offset information is valid, obtain the index information corresponding to the offset information; based on the index information, calculate the write address of the request message; based on the write address, write the data carried by the request message into memory.

[0058] Among them, the offset information of the request message (the byte offset position of the current data message in the complete message) is the core basis for determining whether the data can be correctly written to the target memory. The index information corresponding to the offset information (i.e., the index of SGE in SGL) is used to locate non-contiguous memory segments. When the receiving memory is an SGL (Scatter / Gather List) structure, the "memory range sub-table" of the out-of-order receiving entries will pre-store the offset range of each SGE (Scatter / Gather Element) in the complete message (for example, SGE[0] corresponds to offset 0~1023, SGE[1] corresponds to offset 1024~2047). The responding end can determine the SGE index that the current message data should be written to by matching the offset with the offset range in the memory range sub-table. For example, when the offset is 1500, the offset range of SGE[1] will be matched, and the index information is 1. The write address of the request message needs to be calculated based on the metadata (including VA, LKey, Len) of the corresponding SGE in the out-of-order reception table based on the index information: if the receiving memory is contiguous, the write address = SGE.VA (virtual address) + offset; if it is non-contiguous memory, after locating the target SGE by the index, the write address = target SGE.VA + (offset - the starting offset of the SGE in the message). For example, if the starting offset of SGE[1] is 1024 and the offset is 1500, the write address = SGE[1].VA + (1500-1024) = SGE[1].VA + 476, to ensure that the data is accurately written to the memory segment of the corresponding SGE. Writing the data carried in the request message to memory is achieved through DMA (Direct Memory Access) operations: The responding hardware, based on the calculated write address and the SGE's LKey (LocalKey), verifies memory access permissions and directly writes the payload data in the message to the target area of ​​the receiving application's memory without CPU intervention, reducing latency. At the same time, the responding end updates the "received data length" of the message in the out-of-order reception table in real time (received data length = original length + current message data length). When the received data length equals the total message length, the message is marked as "received complete", and CQEs (Completion Queue Elements) are generated in the order of message reception sequence numbers to notify the application to process the received complete data.

[0059] Step 210: Determine whether the length of the message received corresponding to the message receiving sequence number is equal to the total length of the message. If it is equal, send an acknowledgment message to the requesting end.

[0060] The Received Data Length of the Message Corresponding to RxSN is the total number of bytes of data successfully written to memory by the responding end for a specific message (identified by the unique Receive Sequence Number RxSN). After completing the DMA write of each data packet, the responding end reads the payload length of the current packet from the METH Extended Transport Header of the packet and adds it to the "Received Data Length" of the message corresponding to that RxSN (e.g., if the total length of the message with RxSN=2024 is 4000 bytes, and two packets of 1500 bytes and 1000 bytes respectively have been received, then the current received data length = 1500 + 1000 = 2500 bytes). The total message length (MsgLen) is a fixed value determined by the requesting end when splitting the data to be sent based on the total number of bytes in the complete message, and is carried to the responding end through the METH extension header of each data packet. When the responding end receives the packet corresponding to the RxSN for the first time, it stores MsgLen in the dedicated entry of that RxSN in the Out-of-Order Receive Table as a benchmark for determining whether the message has been "completely received" (for example, if the total length of the data to be sent is 4000 bytes, then all packets carrying RxSN=2024 will have MsgLen of 4000 bytes in their METH header). The core logic for determining whether the received length equals the total message length is as follows: After each update of the "received length", the responding end retrieves the "received length" corresponding to the RxSN from the out-of-order reception entry and compares it with the "total message length". If the received length < MsgLen, it means that some packets of the message have not been received and it is necessary to continue waiting for subsequent out-of-order or undelivered packets; if the received length = MsgLen, it means that all the split packets of the message have been received and written to memory, satisfying the condition of "complete message reception".The acknowledgment message sent to the requesting end is specifically an ACK (Acknowledgment, transport layer positive acknowledgment message). Its functions include: first, to inform the requesting end that the message corresponding to the RxSN has been completely received and no timeout retransmission is required; second, to trigger the requesting end to release the sending buffer resources (such as WQE, Work Queue Element related buffers) corresponding to the message; and third, to provide a pre-trigger signal for the responding end to subsequently generate CQE (Completion Queue Element) in the order of RxSNs and notify the application to process complete data. After sending the ACK, the responding end will update the reception status (RxStatus) of the RxSN in the out-of-order reception table to "Completed" and subsequently return CQEs to the application in order (starting from the smallest completed RxSN), satisfying the order preservation requirements of the IB protocol for RC / XRC / DCT type QPs.

[0061] The aforementioned out-of-order reception method for remote direct memory access is applied at the responding end to receive request messages from the requesting end. The request message is obtained by the requesting end by adding total message length, offset information, and message reception sequence number to each data message. The data message is obtained by the requesting end by splitting the data to be sent based on the current network's maximum transmission unit (MTBF). The total message length and the current message reception sequence number are obtained by the requesting end based on the data to be sent. The validity of the request message is determined based on a preset maximum message out-of-order level, the already received message reception sequence number, and the message reception sequence number. If the determination result is that the request message is valid, it is then determined whether the message reception sequence number already exists in a preset out-of-order reception table entry. If it does not exist, the validity of the request message is determined based on the message reception sequence number. The system uses the message reception sequence number and the received message reception sequence number to determine the out-of-order offset. Based on the out-of-order offset and the message reception sequence number, it pre-allocates reception queues and adds corresponding out-of-order reception entries to each reception queue. Based on the offset information of the request message and the out-of-order reception entry corresponding to the reception queue where the request message is located, it determines whether the offset information is valid. If the determination result is that the offset information is valid, it obtains the index information corresponding to the offset information. Based on the index information, it calculates the write address of the request message. Based on the write address, it writes the data carried by the request message into memory. It determines whether the length of the message already received corresponding to the message reception sequence number is equal to the total message length. If it is equal, it sends an acknowledgment message to the requesting end. By configuring a message reception sequence number for each RDMA data packet and embedding the request message, it is possible to quickly associate each data packet corresponding to the message reception sequence number, thereby accurately identifying the message to which the message belongs when the message is out of order.

[0062] In an exemplary embodiment, a receive queue is established based on the out-of-order offset and the message reception sequence number, including:

[0063] Increment the out-of-order offset by 1 to obtain the first quantity; create the first quantity of receive queues; establish a mapping relationship between each received message sequence number and the first quantity of receive queues; the receive queues are used to receive the request messages corresponding to the corresponding message sequence numbers.

[0064] For example, assuming the core reference indicator for the received message sequence number maintained by the responding end is the expected received sequence number (Expect RxSN), and the current Expect RxSN = 2 (meaning the responding end should currently receive messages with RxSN = 2 in order), the message received sequence number (Current RxSN) carried in the request message sent by the requesting end is 5, and the preset maximum message out-of-order severity is ≥ 3 (ensuring message validity and meeting the out-of-order handling range), first calculate the out-of-order offset: Out-of-order offset = CurrentRxSN - Expect RxSN = 5 - 2 = 3. Add 1 to the out-of-order offset to get the first quantity = 3 + 1 = 4, meaning 4 receive queues (RQ, Receive Queue) need to be created. Since each receive queue corresponds to one message, a completion queue element (CQE, Completion Queue Element) is needed to record the status after each message is received. Therefore, it is also necessary to create these 4 receive queues from the current QP (Queue) of the responding end. Four CQE positions are pre-allocated in the dedicated completion queue bound to the pair (each QP is usually associated with a CQ, or shares CQ resources).Next, create four receive queues and label them RQ-2, RQ-3, RQ-4, and RQ-5 respectively. Determine the range of "each sequence number between the received message sequence number and the current message sequence number" as Expect RxSN(2) to Current. RxSN(5), corresponding to sequence numbers 2, 3, 4, 5 (a total of 4, consistent with the first number), establish the mapping relationship between the receiving queue and the sequence number: RQ-2 maps to sequence number 2 (used to receive request messages for RxSN=2), RQ-3 maps to sequence number 3 (used to receive request messages for RxSN=3), RQ-4 maps to sequence number 4 (used to receive request messages for RxSN=4), RQ-5 maps to sequence number 5 (used to receive the current request message), and at the same time, the 4 pre-allocated CQE positions are denoted as CQE-2, CQE-3, CQE-4, and CQE-5, and a three-way mapping relationship of "receiving queue-sequence number-CQE" is established: RQ-2 (corresponding to RxSN=2) is associated with CQE-2, RQ-3 (corresponding to RxSN=3) is associated with CQE-3, RQ-4 ( RQ-4 is associated with RxSN=4, and RQ-5 is associated with RxSN=5. Subsequently, when the current request message of RxSN=5 receives all the split messages (the length of received data = the total length of the message), the responding end will first update the "reception status" of the out-of-order reception table entry corresponding to RQ-5 to "completed", and then trigger hardware filling of CQE-5. After the messages of RxSN=2, 3, and 4 are also received and the corresponding CQE-2, CQE-3, and CQE-4 are filled, the responding end will submit the CQEs to the completion queue in ascending order of sequence number (2→3→4→5). The application layer can know the reception completion status of all messages in order by reading the CQEs in the CQ, which meets the order preservation requirements of the IB protocol for RC / XRC / DCT type QPs.

[0065] In this embodiment, by determining the first quantity based on the out-of-order offset, creating a receiving queue, and allocating dedicated resources, message packet confusion can be avoided, ensuring accurate packet matching. By synchronously pre-allocating CQEs and establishing a "receiving queue-sequence number-CQE" mapping, the foundation for subsequent in-order notification applications is laid. By submitting CQEs in sequence according to the sequence number, message order preservation can be achieved at the response end, eliminating the inefficient caching of the traditional Fence mechanism and improving transmission efficiency. At the same time, there is no need for the receiving end to cache packets, reducing hardware usage and adapting to multi-path out-of-order and lossy network scenarios.

[0066] In an exemplary embodiment, the validity of a request message is determined based on a preset maximum message out-of-order level, the received message sequence number, and the message reception sequence number, including:

[0067] If the message receiving sequence number is less than the received message receiving sequence number, the request message is a duplicate message and is deemed invalid. If the result of subtracting the received message receiving sequence number from the received message receiving sequence number is not less than the preset maximum message out-of-order level, the request message is an out-of-order message and is deemed invalid. If the message receiving sequence number is not less than the received message receiving sequence number and is not greater than the sum of the received message receiving sequence number and the preset maximum message out-of-order level, the request message is deemed valid.

[0068] For example, suppose the preset maximum message out-of-order level is 4 (meaning the responder can handle a maximum of 4 out-of-order messages ahead of the Max RxSN), and the current Max RxSN is 8. If the request message sent by the requesting end carries a message reception sequence number (Current RxSN) of 6, since 6 < 8 (less than the maximum received message reception sequence number), the message is determined to be a duplicate message, and the request message is invalid. If the Current RxSN of the request message is 13, calculate 13 - 8 = 5, and 5 ≥ 4 (not less than the preset maximum message out-of-order level), the message is determined to be an out-of-order message, and the request message is invalid. If the Current RxSN of the request message is 10, 10 ≥ 8 and 10 ≤ 8 + 4 = 12 (not greater than the sum of the maximum received message reception sequence number and the maximum out-of-order level), the request message is determined to be valid and can proceed to the subsequent out-of-order reception processing flow.

[0069] In this embodiment, by using the maximum received message sequence number (Max RxSN) as a benchmark and combining it with a preset maximum message out-of-order level to construct an validity judgment range (Max RxSN≤Current RxSN≤Max RxSN+Maximum out-of-order level), it can accurately filter duplicate messages smaller than Max RxSN (avoiding redundant processing and wasting resources) and also intercept out-of-order messages that exceed the out-of-order limit (preventing the requesting end from exhausting the response end's resources with excessive out-of-order messages). At the same time, it ensures that messages within a reasonable out-of-order range are received and processed normally, ultimately achieving efficient screening and control of out-of-order request messages. This provides effective input for subsequent out-of-order reception entry queries, data writing, and other processes, adapting to the complex transmission scenarios of multi-path out-of-order in intelligent computing networks and lossy VPC networks.

[0070] In one exemplary embodiment, the method further includes:

[0071] If it exists, then the validity of the offset information is determined based on the offset information of the request message and the out-of-order receive entry corresponding to the receive queue where the request message is located.

[0072] For example, suppose the responding end has previously received a request message fragment with message reception sequence number (RxSN) = 7, and created a receiving queue and corresponding out-of-order receiving entry for it (the table records that the total length of the message MsgLen = 3000 bytes, the length of the received data = 1200 bytes, and the receiving memory is an SGL structure containing two SGEs: SGE[0] corresponds to offset 0~1999 bytes, and SGE[1] corresponds to offset 2000~2999 bytes). At this time, another request message carrying RxSN = 7 is received. First, the out-of-order receiving entry is queried. Since RxSN = 7 already exists in the table (i.e., the scenario of "existence"), the validity is judged based on the offset information in the METH extension header of the message (assuming it is 1500 bytes) and the out-of-order receiving entry: First, it is verified whether the offset of 1500 bytes is within the range of "0~2999 bytes" (MsgLen = 3000 bytes), and then it is confirmed whether 1500 bytes is greater than the length of the received data of 1200 bytes. (To avoid duplicate reception), and at the same time match the offset range of SGE in the out-of-order reception table (1500 bytes belong to the 0~1999 byte range of SGE[0], without exceeding the boundary), and comprehensively determine that the offset information is valid; if the received message offset is 3200 bytes (exceeding MsgLen=3000 bytes), or the offset is 1000 bytes (less than the length of the received data of 1200 bytes), then the offset information is determined to be invalid, and the responding end will return NACK to the requesting end, requesting the resend of the correct message.

[0073] In this embodiment, when the message reception sequence number (RxSN) of the request message already exists in the out-of-order reception table, the validity is determined by combining the message offset information with key data (total message length, length of received data, and SGE offset range) in the out-of-order reception table. This can accurately intercept out-of-bounds messages with offsets exceeding the total message length and duplicate messages with offsets less than the received length, avoiding invalid data being written to memory or wasting resources through repeated processing. Furthermore, SGE offset range matching ensures that the offset corresponds to a legitimate receive memory segment, providing a prerequisite for subsequent accurate calculation of the DMA write address and correct writing of data to non-contiguous memory. At the same time, this judgment logic does not rely on complex CPU intervention and can be executed quickly by hardware, taking into account both the accuracy and low latency requirements of out-of-order reception, and further improving the closed loop of out-of-order message processing.

[0074] In one exemplary embodiment, the method further includes:

[0075] If the determination result is that the offset information is invalid, a response message is sent to the requesting end, indicating that the request message is invalid.

[0076] For example, suppose that in the out-of-order reception table maintained by the responder, the total length (MsgLen) of the message with the received sequence number (RxSN) = 5 is 2000 bytes, the length of the received data is 800 bytes, and the SGE[0] of the receiving memory SGL corresponds to offset 0~999 bytes, and the SGE[1] corresponds to offset 1000~1999 bytes. At this point, the responding end receives a request message carrying RxSN=5, with an offset of 2100 bytes in its METH extension header. It first checks the out-of-order reception table to confirm that RxSN=5 exists, then checks the validity of the offset: since 2100 bytes is greater than the total message length of 2000 bytes, the offset information is deemed invalid. The responding end then sends a response message to the requesting end, specifically a NACK (Negative Acknowledgment), which carries an invalidity reason flag indicating "offset out of bounds" (or feedback on the invalidity type through a field specified in the protocol). This clearly informs the requesting end that the currently sent request message is invalid because the offset exceeds the total message length. After receiving this NACK, the requesting end will re-verify the offset configuration of the RxSN=5 message, correct it, and resend a compliant request message. If a subsequent RxSN=5 request message has an offset of 600 bytes (less than the length of the received data of 800 bytes, which is considered a duplicate reception), the offset will also be determined to be invalid, and the responding end will send NACK again to prompt the requesting end to avoid resending the received message fragment.

[0077] In this embodiment, when the offset information of the request message is determined to be invalid (e.g., the offset exceeds the total message length or is less than the length of received data), the responding end sends a NACK (Negative Acknowledgment) response message to the requesting end, which explicitly indicates that the "request message is invalid." This allows the requesting end to promptly report the reason for the invalidity (which can be indicated by flags such as "offset out of bounds" or "repeated reception") to the requesting end. This prevents the requesting end from waiting for a long time due to the lack of acknowledgment or triggering unnecessary timeout retransmissions, thus reducing the waste of network resources. At the same time, the requesting end can quickly locate the problem based on the reason for invalidity in the NACK (e.g., correct the offset configuration or stop repeatedly sending received segments), accelerating the retransmission and transmission of subsequent valid messages. This ensures the efficient operation of the closed loop of "invalid message processing - problem feedback - message correction and retransmission" in the out-of-order reception process, further guaranteeing the reliability and transmission efficiency of RDMA out-of-order reception.

[0078] In an exemplary embodiment, after sending the confirmation information to the requesting end, the method further includes:

[0079] The message receiving sequence number is added to the completion queue corresponding to the receiving queue, and the entry associated with the message receiving sequence number in the out-of-order receiving table entry corresponding to the receiving queue is deleted; the completion queue is used to record message identifiers that have been fully received and whose data verification is error-free.

[0080] For example, suppose the responding end receives three out-of-order messages sent by the requesting end through the receive queue (RQ) of a queue pair (QP). The corresponding message reception sequence numbers (RxSN) are 0x001, 0x002, and 0x003, and are all cached in the out-of-order reception table (each RxSN is associated with a corresponding message data segment and reception status). When the responding end completes the reception and data verification of the last message (RxSN=0x002), and confirms that all three messages are complete and error-free, it sends an acknowledgment message (ACK) to the requesting end. Subsequently, the responding end adds RxSN=0x001, 0x002, and 0x003 to the completion queue (CQ) corresponding to the receive queue in sequence, and deletes the entries associated with these three RxSNs from the out-of-order reception table, releasing the storage resources occupied by the entries. The RxSN recorded in the completion queue serves as a message identifier that has been legally received, which is read by the application layer to trigger subsequent data reassembly and business processing.

[0081] In this embodiment, after receiving the read message carrying the scattered extended transmission header from the requesting end through the receive queue (RQ) of the queue pair (QP), the responding end sends an acknowledgment message (ACK) to the requesting end after completing message validity verification (including four Rkey checks) and complete reception and verification of the target data. Subsequently, the message reception sequence number (RxSN) corresponding to the read message is added to the completion queue (CQ) associated with the receive queue, and the entry corresponding to the RxSN in the out-of-order reception table is deleted to release storage resources. The completion queue serves as a message identifier record carrier for messages that have been completely received and whose data verification is error-free. The application layer can obtain the legally received message information by reading this queue, thereby triggering subsequent processes such as splitting the target data according to scattered elements, adding the read response extended header, and sending the reply message. This ensures the traceability of message reception status and achieves efficient recycling of storage resources.

[0082] In one exemplary embodiment, the method further includes:

[0083] The system receives a read message from the requesting end and verifies its validity. The read message is obtained by the requesting end adding a hash extension header to the read message to be transmitted. The hash extension header is generated based on hash list information, which is obtained by the requesting end from its local memory. The hash list information includes the number of hash elements and the hash elements themselves, which include a virtual address, access key, and length. If valid, the system determines the target data to be read. Based on the virtual address, access key, length, and the maximum transmission unit of the current network within the hash elements in the read message, the system splits the target data to obtain a reply data message. A read response extension header is added to the reply data message to obtain a reply message. Each reply message corresponds to only one hash element. The read response extension header includes the virtual address and access key of the matching hash element. The system sends the reply message to the requesting end, instructing it to write the data in the reply message into its local memory based on the access key and virtual address of the matching hash element.

[0084] Among them, such as Figure 4 and Figure 5As shown, a standard RDMA Read message includes an Ethernet header, an IP header, a UDP header, an InfiniBand Base Transport Header (IB BTH), a Read / Write Extended Transport Header (RETH), an ICRC (InfiniBand Cyclic Redundancy Check), and a FrameCheck Sequence (FCS). In addition to the above-mentioned parts, the read message of this application also adds SGETH (Scatter-Gather Extended Transport Header), which is located between RETH and ICRC. It includes: number of scatter elements (SGECnt, used to identify the total number of elements in the scatter list of the requesting end), scatter elements (SGE(0)-SGE(m-1), each scatter element contains a virtual address subfield (VA, Virtual Address), an access key subfield (LKey, Local Key), and a length subfield (Len, Length), which respectively record the memory virtual address of the local scatter element of the requesting end, the access authorization key, and the length of the memory segment).In addition to the regular Ethernet header, IP header, InfiniBand Base Transport Header (IB BTH), AETH (ACK Extended Transport Header), data payload segment (InfiniBand Payload, carrying the split target data fragments, corresponding one-to-one with the requesting end's hash elements), ICRC (InfiniBand Cyclic Redundancy Check), and FCS (Frame Check Sequence), the reply data packet contains an RRETH (Read Response Extended Transport Header) between the AETH and the data payload segment. The read response extended header includes a Virtual Address (VA) subfield matching the hash elements and an Access Key (LKey) subfield matching the hash elements, used by the requesting end to locate the local memory write position.

[0085] For example, suppose the requesting end needs to read "Device Status Log" data (total length 4096 bytes) stored in the responding end's memory. This data is distributed across two SGEs (Small Set Elements) in the requesting end's local memory: SGE1 (virtual address 0x2000, access key 0xDEF456, length 2048 bytes) and SGE2 (virtual address 0x5000, access key 0xDEF456, length 2048 bytes). The requesting end first generates a SGE list based on these two SGEs (number of SGE elements = 2, containing complete information from the two SGEs), then creates a SGE extended transport header based on this information, adds it to the read message to be transmitted, and then sends the read message to the responding end. After receiving the read message, the responding end first verifies its validity (e.g., checking whether the access key matches the authorization key pre-stored by the responding end, and whether the virtual address is within the allowed memory range). After confirming its validity, it determines that the target data is the 4096 bytes of "Device Status Log" data. Next, the responding end checks the current network's Maximum Transmission Unit (MTU), which is 2048 bytes. Combining this with the lengths of the two SGEs in the read packet (both 2048 bytes), it splits the target data into two reply data packets (each corresponding to one SGE, each with a data length of 2048 bytes). Then, it adds a read response header to each reply data packet: the first reply packet's read response header carries "virtual address 0x2000 matching SGE1, access key 0xDEF456 matching SGE1," and the second reply packet's read response header carries "virtual address 0x5000 matching SGE2, access key 0xDEF456 matching SGE2." Finally, it obtains two reply packets and sends them to the requesting end. After receiving the reply packets, the requesting end uses the virtual address and access key in the read response header to accurately locate the corresponding SGE memory region locally, and writes the data from the reply packets into memory segments starting at 0x2000 and 0x5000 respectively, completing the data read and write operations.

[0086] In this embodiment, in addition to solving the problem that existing RDMA Send cannot identify the message to which the message belongs when the message is out of order or lost, this application also solves the problem that existing RDMA Read cannot accurately match the target memory in scenarios with multiple hash elements, and that the split reply message is prone to the requesting end being unable to locate the write position due to the lack of association identifier: by carrying a hash extension transmission header generated based on the hash list in the read message, and adding a read response extension header containing "the virtual address of the matching hash element + access key" in the reply message, a strong association is established between the read request and the reply data. Even if the reply message is out of order or partially lost, the requesting end can quickly match the corresponding local hash element through the identifier in the read response extension header, ensuring that the data is accurately written to the target memory and avoiding the problem of memory write misalignment or data not belonging.

[0087] In an exemplary embodiment, reading the message includes: a remote access key; verifying the validity of the read message includes:

[0088] Obtain the remote access key, target virtual address, and request data length from the read message; locate the memory region corresponding to the remote access key in the protection domain associated with the current queue pair; the queue pair refers to a corresponding send queue and receive queue; verify whether the remote access key is a registered valid key, whether the associated memory region has read permissions, whether the address range formed by the target virtual address and the request data length is within the registered memory range, and whether the corresponding protection domain is consistent with the queue pair bound to the current queue pair; if all four verifications pass, the verification result is valid; if any verification fails, the verification result is invalid.

[0089] For example, suppose the requesting end needs to access the target memory data of the responding end through an RDMA Read operation. The constructed read message carries the remote access key (Rkey=0x12345678), the target virtual address (0x7F000000), and the length of the requested data (4096 bytes), and is sent to the responding end through a queue pair (QP, consisting of a send queue SQ and a receive queue RQ). Upon receiving the read message, the responding end first extracts the aforementioned Rkey, target virtual address, and requested data length from the message. Then, within the protection field (PD) bound to the QP currently processing the message, it searches for the memory region (MR) corresponding to Rkey=0x12345678, confirming that the Rkey is a valid key registered by the responding end and that the associated MR has been granted READ permission. It further calculates the address range (starting address 0x7F000000, ending address 0x7F000000 + 4096 = 0x7F001000), verifying that this range falls entirely within the MR's registered memory range (0x7F000000-0x7F00FFFF). Finally, it confirms that the PD to which the MR belongs matches the PD bound to the current QP. Since all four verifications pass, the read message is deemed valid, and the responding end executes the subsequent target data read and split reply process. If, during the above verification, Rkey=0x12345678 is not found in the current PD (existence check failed), or the associated MR is only configured with WRITE permission (permission check failed), or the address range exceeds the registered memory range (range check failed), or the PD to which the MR belongs is inconsistent with the PD bound to the current QP (protection domain matching check failed), then the read message is directly determined to be invalid, the responding end refuses to execute the data read operation, and generates an Error Completion Queue Element (CQE) to record the specific error type.

[0090] In this embodiment, by performing four compliance checks on the remote access key (Rkey) carried in the read message—namely, key existence, memory permissions, address range, and protection domain matching—risks such as unregistered key access, unauthorized access, address out-of-bounds access, and illegal access across protection domains can be strictly blocked. This ensures that only legitimate RDMA Read requests can trigger data read operations on the responding end, which not only complies with the secure access mechanism of the RDMA standard but also provides a secure prerequisite for subsequent target data splitting and accurate response based on scattered elements, effectively protecting the privacy and integrity of the local memory data on the responding end.

[0091] In one exemplary embodiment, such as Figure 6 As shown, a method for out-of-order reception of remote direct memory access is provided, which is applied to... Figure 1 Taking requester 102 as an example, the explanation includes steps 602 to 604. Wherein:

[0092] Step 602: Obtain the total message length of the data to be sent and the current message receiving sequence number; split the data to be sent based on the current network's maximum transmission unit to obtain multiple data packets.

[0093] The data to be sent refers to the complete data extracted from local memory by the responding end according to the target data range specified in the message read by the requesting end. When splitting the data to be sent based on the current network's Maximum Transmission Unit (MTU), the maximum payload length of each data packet needs to be calculated in conjunction with the MTU value.

[0094] Step 604: Add message total length, offset information, and message reception sequence number to each data packet to obtain multiple request packets; send the request packets to the response end to instruct the response end to determine the validity of the request packets based on the preset maximum message out-of-order level, the received message reception sequence number, and the message reception sequence number; if the determination result is that the request packet is valid, then determine whether the message reception sequence number already exists in the preset out-of-order reception entry; if it does not exist, then determine the out-of-order offset based on the message reception sequence number and the received message reception sequence number; based on the out-of-order offset and The message receiving sequence number is used to pre-allocate receiving queues and add corresponding out-of-order receiving entries to each receiving queue. Based on the offset information of the request message and the out-of-order receiving entry corresponding to the receiving queue where the request message is located, it is determined whether the offset information is valid. If the determination result is that the offset information is valid, the index information corresponding to the offset information is obtained. Based on the index information, the write address of the request message is calculated. Based on the write address, the data carried by the request message is written to memory. It is determined whether the length of the message already received corresponding to the message receiving sequence number is equal to the total length of the message. If it is equal, an acknowledgment message is sent to the requesting end.

[0095] The total message length is the byte length of the complete message to be sent. It needs to be added to the METH extension header of each request message so that the responding end can obtain the overall message size when receiving the first message, providing a basis for creating out-of-order reception entries and allocating reception resources. The offset information refers to the starting byte position of the current data packet in the complete message. It is used by the responding end to locate the position where the data is written to memory, avoiding data overlap or misalignment between different messages. The message reception sequence number (RxSN) is a unique identifier assigned by the requesting end to each message (e.g., RxSN=9). It needs to be uniformly added to the header of all split request messages to ensure that the responding end can identify the message to which the message belongs through this sequence number. Even if the message arrives out of order, it can be accurately associated with the corresponding out-of-order reception entry. After the requesting end completes the field addition according to the above rules, it will send multiple request messages to the responding end sequentially or in parallel through the RDMA network. During the sending process, multi-path transmission (such as the ECMP path of RoCEv2) can be selected according to the network conditions. The RxSN and offset information in each request message can ensure that the responding end can still correctly identify, receive and reassemble data in out-of-order / packet loss scenarios, laying the foundation for the subsequent out-of-order processing.

[0096] The aforementioned out-of-order reception method for remote direct memory access is applied to the requesting end. It obtains the total message length and current message reception sequence number of the data to be sent; it splits the data to be sent based on the current network's maximum transmission unit (MTBF) to obtain multiple data packets; it adds the total message length, offset information, and message reception sequence number to each data packet to obtain multiple request packets; it sends the request packets to the responding end to instruct the responding end to determine the validity of the request packets based on a preset maximum message out-of-order level, the received message reception sequence number, and the message reception sequence number; if the determination result is that the request packet is valid, it then determines whether the message reception sequence number already exists in a preset out-of-order reception table; if it does not exist, it then determines the validity of the request packets based on the message reception sequence number. The process involves: receiving the sequence number and the received message sequence number to determine the out-of-order offset; pre-allocating receive queues and adding corresponding out-of-order receive entries to each receive queue based on the out-of-order offset and the message sequence number; determining the validity of the offset information based on the offset information of the request message and the out-of-order receive entry corresponding to the receive queue containing the request message; obtaining the index information corresponding to the offset information based on the index information; calculating the write address of the request message based on the write address; writing the data carried by the request message to memory based on the write address; and determining whether the received message length corresponding to the message sequence number is equal to the total message length. If equal, sending an acknowledgment message to the requesting end. By configuring a message receive sequence number for each RDMA data packet and embedding the request message, it is possible to quickly associate each data packet corresponding to the message receive sequence number, thereby accurately identifying the message to which the message belongs when the message is out of order.

[0097] In one exemplary embodiment, the method further includes:

[0098] Obtain the hash list information from the local memory; generate a hash extended transmission header based on the hash list information; the hash list information includes the number of hash elements and the hash elements themselves, each including a virtual address, access key, and length; add the hash extended transmission header to the read message to be transmitted, obtaining the read message, and send the read message to the response end to instruct the response end to verify the validity of the read message; if valid, determine the target data to be read; based on the virtual address, access key, length, and the current network's maximum transmission unit within the hash elements in the read message, split the target data to obtain a reply data message; match the reply data message with the hash elements; add a read response extended header to the reply data message to obtain a reply message; a single reply message corresponds to only one hash element; the read response extended header includes the virtual address and access key of the matching hash element; receive the reply message from the response end, and write the data in the reply message into the local memory based on the access key and virtual address of the matching hash element in the reply message.

[0099] For example, suppose the requesting end needs to read "business log data" from the responding end. This data needs to be written to a non-contiguous memory region locally on the requesting end: SGE1 (virtual address 0x3000, access key 0xABC123, length 2048 bytes) and SGE2 (virtual address 0x7000, access key 0xABC123, length 1536 bytes). The requesting end first obtains the hash list information (number of hash elements = 2, containing the complete parameters of the above SGEs) composed of these two SGEs from its local memory management module, and generates a hash extended transmission header based on this information (recording the number of SGEs, the virtual address of each SGE, the access key, and the length), adds it to the read message to be transmitted, forms a complete read message, and sends it to the responding end. After receiving the read message, the responding end first verifies whether the access key (0xABC123) is consistent with the local authorization key. After confirming its validity, it determines that the target data is the "business log data" (total length 3584 bytes) specified by the requesting end. Next, the responding end combines the current network MTU (note: the MTU of RDMA is the maximum payload length, excluding the packet header) with the SGE information read from the packet to perform data splitting: because the network MTU limits the maximum payload length of a single packet to 2048 bytes, and the length of SGE1 (2048 bytes) does not exceed this limit, the payload of the first reply data packet directly matches the length of SGE1 (2048 bytes); the length of SGE2 (1536 bytes) also does not exceed the MTU limit, and is used as the payload of the second reply data packet. Finally, two reply data packets are obtained: the first packet corresponds to SGE1 (payload length 2048 bytes), and the second packet corresponds to SGE2 (payload length 1536 bytes). The responding end then adds a read response header to each reply data packet: the first reply packet's header carries "virtual address 0x3000 matching SGE1, access key 0xABC123", and the second carries "virtual address 0x7000 matching SGE2, access key 0xABC123", generating a complete reply packet and sending it to the requesting end. Upon receiving the reply packets, the requesting end uses the virtual address and access key in the read response header to precisely match the corresponding SGE memory region locally. It writes the payload data of the first reply packet to the memory segment starting at 0x3000, and the payload data of the second reply packet to the memory segment starting at 0x7000, completing the precise data write to non-contiguous memory.

[0100] In this embodiment, by carrying a hash-extended transmission header generated based on the local hash list in the read message, the responding end can accurately obtain the memory distribution requirements of the requesting end, and also provide a basis for the splitting and matching of subsequent reply messages. The responding end splits the target data according to SGE and adds a read response extended header containing matching information, ensuring a one-to-one correspondence between the reply message and the hash elements of the requesting end. The requesting end writes data based on the virtual address and access key in the extended header, which can complete the accurate filling of non-contiguous memory without additional memory copying, solving the problem of data writing misalignment in non-contiguous memory scenarios of RDMA Read. At the same time, key verification ensures access security, improves the adaptability and reliability of Read operation, and adapts to the complex requirements of non-contiguous memory access in intelligent computing networks.

[0101] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0102] Based on the same inventive concept, this application also provides a remote direct memory access out-of-order receiving device for implementing the aforementioned remote direct memory access out-of-order receiving method, applied at a response end. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more remote direct memory access out-of-order receiving device embodiments provided below can be found in the limitations of the remote direct memory access out-of-order receiving method described above, and will not be repeated here.

[0103] Based on the same inventive concept, this application also provides a remote direct memory access out-of-order receiving apparatus for implementing the aforementioned remote direct memory access out-of-order receiving method, applied to the requesting end. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the remote direct memory access out-of-order receiving apparatus provided below can be found in the limitations of the remote direct memory access out-of-order receiving method described above, and will not be repeated here.

[0104] Each module in the aforementioned out-of-order receiving device for remote direct memory access can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can invoke and execute the operations corresponding to each module.

[0105] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data required by the responding and requesting ends. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a remote direct memory access out-of-order reception method.

[0106] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0107] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0108] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0109] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0111] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0112] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for out-of-order reception of remote direct memory access, characterized in that, Applied to the response end, the method includes: Receive a request message from the requesting end; the request message is obtained by the requesting end by adding the total message length, offset information and message reception sequence number to each data message; the data message is obtained by the requesting end by splitting the data to be sent based on the current network's maximum transmission unit; the total message length and the current message reception sequence number are obtained by the requesting end based on the data to be sent; Based on the preset maximum message out-of-order level, the received message sequence number, and the received message sequence number, the validity of the request message is determined; if the determination result is that the request message is valid, then it is determined whether the received message sequence number already exists in the preset out-of-order reception table. If it does not exist, then based on the message receiving sequence number and the received message receiving sequence number, determine the out-of-order offset; based on the out-of-order offset and the message receiving sequence number, pre-allocate receiving queues and add corresponding out-of-order receiving entries to each receiving queue; Based on the offset information of the request message and the out-of-order receive entry corresponding to the receive queue where the request message is located, determine whether the offset information is valid; if the determination result is that the offset information is valid, obtain the index information corresponding to the offset information; based on the index information, calculate the write address of the request message; based on the write address, write the data carried by the request message into memory; Determine whether the length of the message received corresponding to the message receiving sequence number is equal to the total length of the message. If it is equal, send an acknowledgment message to the requesting end.

2. The method according to claim 1, characterized in that, The pre-allocation of the receiving queue based on the out-of-order offset and the message receiving sequence number includes: Increment the disordered offset by 1 to obtain the first quantity; Create the first number of receive queues; Establish a mapping relationship between each received message receiving sequence number and the first number of receiving queue elements; the receiving queue is used to receive the request message corresponding to the corresponding message receiving sequence number.

3. The method according to claim 1, characterized in that, The determination of the validity of the request message based on the preset maximum message out-of-order level, the received message sequence number, and the received message sequence number includes: If the message receiving sequence number is less than the already received message receiving sequence number, then the request message is a duplicate message and is determined to be invalid. If the result of subtracting the received message sequence number from the received message sequence number is not less than the preset maximum message out-of-order level, then the request message is an out-of-order limit message, and the request message is determined to be invalid. If the message receiving sequence number is not less than the received message receiving sequence number and is not greater than the sum of the received message receiving sequence number and the preset maximum message out-of-order degree, then the request message is determined to be valid.

4. The method according to claim 1, characterized in that, The method further includes: If it exists, then based on the offset information of the request message and the out-of-order receive entry corresponding to the receive queue where the request message is located, it is determined whether the offset information is valid.

5. The method according to claim 1, characterized in that, The method further includes: If the determination result indicates that the offset information is invalid, a response message is sent to the requesting end, and the response message indicates that the request message is invalid.

6. The method according to claim 1, characterized in that, After sending confirmation information to the requesting end, the process also includes: The message receiving sequence number is added to the completion queue corresponding to the receiving queue, and the entry associated with the message receiving sequence number in the out-of-order receiving table entry corresponding to the receiving queue is deleted; the completion queue is used to record message identifiers that have been fully received and whose data verification is error-free.

7. The method according to claim 1, characterized in that, The method further includes: The system receives a read message from the requesting end and verifies whether the read message is valid. The read message is obtained by the requesting end by adding a hash set extension header to the read message to be transmitted. The hash set extension header is generated based on hash set list information. The hash set list information is obtained by the requesting end from its local memory. The hash set list information includes the number of hash set elements and the hash set elements, and the hash set elements include: virtual address, access key, and length. If valid, determine the target data to be read from the read message; based on the virtual address, access key, length, and maximum transmission unit of the current network in the hash element of the read message, split the target data to obtain a reply data message; A read response extension header is added to the reply data packet to obtain a reply packet; a single reply packet corresponds to only one hash element; the read response extension header includes: the virtual address of the matching hash element and the access key of the matching hash element; The reply message is sent to the requesting end to instruct the requesting end to write the data in the reply message into its local memory based on the matching hash element access key and the matching hash element virtual address in the reply message.

8. The method according to claim 7, characterized in that, The read message includes: a remote access key; the verification of the validity of the read message includes: Obtain the remote access key, target virtual address, and request data length from the read message; In the protection domain associated with the current queue pair, search for the memory region corresponding to the remote access key; the queue pair refers to a corresponding send queue and a receive queue. Verify whether the remote access key is a registered valid key, whether the associated memory region has read permissions, whether the address range formed by the target virtual address and the length of the requested data is within the registered memory range, and whether the corresponding protection domain is consistent with the queue pair bound to the current queue pair; If all four verifications pass, the verification result is valid; If any validation fails, the validation result is invalid.

9. A method for out-of-order reception of remote direct memory access, characterized in that, Applied to the requesting end, the method includes: Obtain the total message length and current message reception sequence number of the data to be sent; split the data to be sent based on the current network's maximum transmission unit to obtain multiple data packets; For each data packet, add total message length, offset information, and message reception sequence number to obtain multiple request packets. Send these request packets to the response end to instruct it to determine the validity of the request packets based on a preset maximum message out-of-order level, the received message reception sequence number, and the received message reception sequence number. If the determination result is that the request packet is valid, then check whether the message reception sequence number already exists in a preset out-of-order reception table. If not, determine the out-of-order offset based on the message reception sequence number and the received message reception sequence number. Finally, based on the out-of-order offset and the message reception sequence number... The system pre-allocates receiving queues and adds corresponding out-of-order receiving entries to each receiving queue. Based on the offset information of the request message and the out-of-order receiving entry corresponding to the receiving queue where the request message is located, it determines whether the offset information is valid. If the determination result is that the offset information is valid, it obtains the index information corresponding to the offset information. Based on the index information, it calculates the write address of the request message. Based on the write address, it writes the data carried by the request message into memory. It determines whether the length of the message already received corresponding to the message receiving sequence number is equal to the total length of the message. If it is equal, it sends an acknowledgment message to the requesting end.

10. The method according to claim 9, characterized in that, The method further includes: Obtain the hash list information of the local memory; generate a hash extended transmission header based on the hash list information; the hash list information includes the number of hash elements and the hash elements, and the hash elements include: virtual address, access key and length; The scattered set extended transport header is added to the read message to be transmitted to obtain the read message, and the read message is sent to the response end to instruct the response end to verify whether the read message is valid; if valid, the target data to be read by the read message is determined; based on the virtual address, access key, length and the maximum transmission unit of the current network in the scattered set element of the read message, the target data is split to obtain a reply data message; the reply data message and the scattered set element are matched; a read response extended header is added to the reply data message to obtain a reply message; a single reply message corresponds to only one scattered set element; the read response extended header includes: the virtual address of the matching scattered set element and the access key of the matching scattered set element; Receive a reply message from the responding end, and write the data in the reply message into the local memory based on the access key of the matching hash element and the virtual address of the matching hash element in the reply message.

11. A remote direct memory access out-of-order receiving device, characterized in that, Applied to the response end, the device includes: The receiving module is used to receive request messages from the requesting end; the request message is obtained by the requesting end by adding the total message length, offset information and message receiving sequence number to each data message; the data message is obtained by the requesting end by splitting the data to be sent based on the current network's maximum transmission unit; the total message length and the current message receiving sequence number are obtained by the requesting end based on the data to be sent. The judgment module is used to determine the validity of the request message based on the preset maximum message out-of-order degree, the received message sequence number, and the message received sequence number; if the judgment result is that the request message is valid, then it determines whether the message received sequence number already exists in the preset out-of-order reception table. The processing module is configured to, if the message receiving sequence number does not exist, determine the out-of-order offset based on the message receiving sequence number and the received message receiving sequence number; and pre-allocate receiving queues and add corresponding out-of-order receiving entries to each receiving queue based on the out-of-order offset and the message receiving sequence number. The writing module is used to determine whether the offset information is valid based on the offset information of the request message and the out-of-order receive table entry corresponding to the receive queue where the request message is located; if the determination result is that the offset information is valid, then obtain the index information corresponding to the offset information; calculate the write address of the request message based on the index information; and write the data carried by the request message into memory based on the write address. The first sending module is used to determine whether the length of the message received corresponding to the message receiving sequence number is equal to the total length of the message. If it is equal, an acknowledgment message is sent to the requesting end.

12. A remote direct memory access out-of-order receiving device, characterized in that, Applied to the requesting end, the device includes: The splitting module is used to obtain the total message length of the data to be sent and the current message reception sequence number; it splits the data to be sent based on the current network's maximum transmission unit to obtain multiple data packets. The second sending module is used to add message total length, offset information, and message reception sequence number to each data packet to obtain multiple request packets; send the request packets to the response end to instruct the response end to determine the validity of the request packets based on a preset maximum message out-of-order level, the received message reception sequence number, and the message reception sequence number; if the determination result is that the request packet is valid, then it is determined whether the message reception sequence number already exists in a preset out-of-order reception entry; if it does not exist, then based on the message reception sequence number and the received message reception sequence number, the out-of-order offset is determined; based on the out-of-order offset and the message reception sequence number, the out-of-order offset is determined; and based on the out-of-order offset and the message reception sequence number, the out-of-order offset is determined. Receive sequence number, pre-allocate receiving queues and add corresponding out-of-order receiving entries to each receiving queue; based on the offset information of the request message and the out-of-order receiving entry corresponding to the receiving queue where the request message is located, determine whether the offset information is valid; if the determination result is that the offset information is valid, obtain the index information corresponding to the offset information; based on the index information, calculate the write address of the request message; based on the write address, write the data carried by the request message into memory; determine whether the message length received corresponding to the message receiving sequence number is equal to the total message length, if equal, send an acknowledgment message to the requesting end.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images