RDMA read message processing method and RDMA read message processing unit

Abstract

The invention relates to an RDMA read message processing method, an RDMA read message processing unit, a network interface card, an RDMA communication system, a computer readable storage medium and a computer program product. Determining the number n of request messages which can be sent at present according to the number of read response messages to be received at the local RDMA end and the maximum read response message window threshold allowed by the single QP; and according to the receiving sequence of the RDMA read messages and the initial position of the current RDMA read message to be processed, sequentially selecting one or more RDMA read messages from the RDMA read messages to be processed, and according to the selected one or more RDMA read messages, generating n request messages and sending the n request messages to a far-end RDMA end. And the network effective bandwidth is improved.

Description

Technical Field

[0001] The present application relates to the field of chip technology, and in particular to an RDMA read message processing method, an RDMA read message processing unit, a network interface card, an RDMA communication system, a computer-readable storage medium, and a computer program product. Background Art

[0002] In Remote Direct Memory Access (RDMA) communications, an RDMA read operation refers to the process of actively reading remote memory from the local RDMA end. During the execution of the RDMA read operation, after the remote RDMA end receives the RDMA read request message, it reads data from the memory and packages a read response message and returns it to the local RDMA end. During the entire execution process, the remote CPU does not need to participate and is unaware of the process of data being read from the memory. In the existing technology, the local RDMA end transmits the entire RDMA read message to the remote RDMA end at one time via a request message. After receiving the RDMA read request message, the remote RDMA end restores the entire RDMA read message, records it locally, and then, based on the working status of the remote RDMA end, sequentially returns all the data required by the RDMA read message.

[0003] According to the existing RDMA protocol, a single RDMA read message can read up to 2GB of memory data at a time. When the remote RDMA peer processes an RDMA read request message, it must generate up to 8M RDMA read response messages to the local RDMA peer. This means that for every RDMA read request message the local RDMA peer sends to the remote RDMA peer, the remote RDMA peer must return 8M RDMA read response messages to complete processing. This can lead to the following problems: 1. A severe imbalance in network traffic between the local and remote peers, and between the remote and local peers. By simply sending a few consecutive RDMA read request messages to the remote RDMA peer, the local RDMA peer can easily consume all of the remote RDMA peer's bandwidth, crowding out its resources and impacting its own sending tasks, leading to a network traffic attack. 2. When the remote RDMA peer continuously returns a large number of RDMA read response messages to the local RDMA peer, the message window is no longer controlled by the remote RDMA peer, easily leading to network traffic congestion. Once packet loss occurs, all subsequent RDMA read response packets become invalid. The remote RDMA end cannot detect the packet loss in time and cannot stop sending invalid response packets, causing the network to be flooded with invalid packets, severely reducing the effective network bandwidth. Summary of the Invention

[0004] Based on this, it is necessary to provide an RDMA read message processing method, an RDMA read message processing unit, a network interface card, an RDMA communication system, a computer-readable storage medium and a computer program product that can improve the effective network bandwidth to address the above technical problems.

[0005] In a first aspect, the present application provides an RDMA read message processing method, comprising:

[0006] Determine the number n of request messages that can be sent based on the number of read response messages currently to be received by the local RDMA end and the maximum read response message window threshold allowed by a single QP (Queue Pairs);

[0007] Select one or more RDMA read messages from the pending RDMA read messages according to the order in which the RDMA read messages are received and the starting position of the currently pending RDMA read message. Generate n request messages based on the selected one or more RDMA read messages and send them to the remote RDMA end.

[0008] Receive the read response message returned by the remote RDMA end, process the read response message, and update the number of read response messages currently to be received.

[0009] In a second aspect, the present application further provides an RDMA read message processing unit, comprising:

[0010] A message splitting module is configured to determine the number n of request messages that can be sent based on the number of read response messages currently to be received by the local RDMA end and the maximum read response message window threshold allowed by a single QP; and to select one or more RDMA read messages from the RDMA read messages to be processed in sequence based on the order in which the RDMA read messages are received and the starting position of the RDMA read message currently to be processed;

[0011] A first message sending processing module is used to generate n request messages according to one or more selected RDMA read messages and send them to the remote RDMA end;

[0012] The first message receiving and processing module is configured to receive the read response message returned by the remote RDMA end, process the read response message, and update the number of read response messages currently to be received.

[0013] In a third aspect, the present application further provides a network interface card, comprising the above-mentioned RDMA read message processing unit.

[0014] In a fourth aspect, the present application also provides an RDMA communication system, including the above-mentioned RDMA read message processing unit.

[0015] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned RDMA read message processing method is implemented.

[0016] In a sixth aspect, the present application further provides a computer program product, comprising a computer program, which implements the above-mentioned RDMA read message processing method when executed by a processor.

[0017] In the above-mentioned RDMA read message processing method, RDMA read message processing unit, network interface card, RDMA communication system, computer-readable storage medium, and computer program product, the local RDMA end determines the number of request messages n that can be sent based on the number of read response messages currently to be received by the local RDMA end and the maximum read response message window threshold allowed by a single QP; selects one or more RDMA read messages from the pending RDMA read messages in sequence based on the order in which the RDMA read messages are received and the starting position of the currently pending RDMA read message; generates n request messages based on the selected one or more RDMA read messages and sends them to the remote RDMA end; receives the read response message returned by the remote RDMA end, processes the read response message, and updates the number of currently pending read response messages. This method of controlling the sending of request messages based on the maximum read response message window threshold allowed by a single QP can reduce the imbalance of network communication traffic between the local end and the remote end and the remote end and local end, reduce network traffic congestion, and greatly improve the processing speed of RDMA read messages, thereby increasing the effective bandwidth of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a schematic diagram of a communication system in one embodiment;

[0020] Figure 2 1 is a flow chart of an RDMA read message processing method according to an embodiment;

[0021] Figure 3A structural block diagram of an RDMA read message processing unit in one embodiment;

[0022] Figure 4 FIG. 4 is an example diagram of a method for processing an RDMA read message in one embodiment. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0024] It should be noted that the terms "first", "second", etc. used in this application may be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "including" and "having" used in this application and any variations thereof are intended to cover non-exclusive inclusions. The term "plurality" used in this application refers to two or more. The term "and / or" used in this application refers to one of the solutions or any combination of multiple solutions.

[0025] See also Figure 1 As shown, the RDMA read message processing method provided in the embodiment of the present application can be applied to Figure 1 In the communication system shown, Figure 1 The communication system shown includes a local RDMA end and a remote RDMA end, and the local RDMA end and the remote RDMA end can communicate with each other.

[0026] In some embodiments, in the RDMA read message processing method provided in the embodiments of the present application, see Figure 2 As shown, the steps performed by the local RDMA end include:

[0027] Step 202: Determine the number n of request messages that can be sent currently based on the number of read response messages to be received by the local RDMA end and the maximum read response message window threshold allowed by a single QP.

[0028] Among them, the request message mentioned in the embodiment of the present application is an RDMA read request message, and the read response message mentioned in the embodiment of the present application is an RDMA read response message.

[0029] Optionally, a ratio parameter N is configured on the local RDMA end. This ratio parameter N is referred to as a message splitting factor in this embodiment of the present application and is used to indicate the maximum number of RDMA read response messages that can be generated for each RDMA read request message. The value of N can be flexibly set based on actual conditions. For example, N=2 indicates that for each RDMA read request message, a maximum of two RDMA read response messages can be generated.

[0030] Optionally, the local RDMA end may monitor the number of read response messages currently to be received.

[0031] Optionally, the maximum read response message window threshold allowed by a single QP can be configured as the bandwidth delay product (BDP). This ensures the effective RDMA bandwidth while reducing the overshoot of the RDMA read response message returned by the remote RDMA peer.

[0032] When executing step 202, the number of read response messages that the local RDMA end can still receive can be determined based on the current number of read response messages to be received by the local RDMA end and the maximum read response message window threshold allowed by a single QP. Then, the number of request messages n that can currently be sent can be determined based on the number of read response messages that can still be received and the message splitting factor.

[0033] Step 204: Select one or more RDMA read messages from the RDMA read messages to be processed according to the order in which the RDMA read messages are received and the starting position of the RDMA read message to be processed. Generate n request messages based on the selected one or more RDMA read messages and send them to the remote RDMA end.

[0034] Optionally, the local RDMA end processes each RDMA read message in sequence according to the order in which the RDMA read messages are received.

[0035] Among them, for each RDMA read message received, when certain conditions are met, the local RDMA end will split it to obtain multiple sub-messages, generate request messages corresponding to the multiple sub-messages, and send all the multiple request messages to the remote RDMA end. When other conditions are met, a certain number of sub-messages will be obtained from the multiple sub-messages, and request messages corresponding to these sub-messages will be generated and sent to the remote RDMA end. In this case, there will be remaining sub-messages. In other words, the RDMA read message currently to be processed by the local RDMA end may be a complete RDMA read message, that is, an unsplit RDMA read message, or it may be the remaining unprocessed sub-message of a split RDMA read message. Optionally, whether it is an unsplit RDMA read message or the remaining unprocessed sub-message of a split RDMA read message can be determined based on the starting position of the RDMA read message currently to be processed.

[0036] Optionally, the local RDMA end may sequentially select one or more RDMA read messages from the RDMA read messages to be processed, generate n request messages according to the selected one or more RDMA read messages, and send the n request messages to the remote RDMA end.

[0037] Step 206: Receive the read response message returned by the remote RDMA end, process the read response message, and update the number of read response messages currently to be received.

[0038] The remote RDMA end processes the request message generated after the splitting process as a normal RDMA read request message. The remote RDMA end does not need to know whether multiple adjacent request messages belong to the same RDMA read message. When processing each request message, the remote RDMA end only needs to read N * PMTU length data from the starting address VA_n of the request message, generate N read response messages, and return them to the local RDMA end.

[0039] Optionally, after receiving the request message, the remote RDMA end generates a response message descriptor for the request message and sends the response message descriptor to the read response queue to wait for scheduling. The remote RDMA end can schedule the response message descriptors in the read response queue based on the traffic management and congestion management mechanisms. For the currently scheduled response message descriptor, data of N*PMTU length can be read starting from the starting address in the response message descriptor, N read response messages can be generated, and the N read response messages can be returned to the local RDMA end. After receiving the read response message, the local RDMA end processes the read response message, updates the number of read response messages currently to be received, generates a CQE based on the response message, and writes the CQE to the host side of the local end.

[0040] In the above embodiment, the local RDMA end determines the number of request messages n that can be sent based on the number of read response messages currently to be received by the local RDMA end and the maximum read response message window threshold allowed by a single QP; selects one or more RDMA read messages from the pending RDMA read messages in sequence based on the order in which the RDMAread messages are received and the starting position of the currently pending RDMA read message; generates n request messages based on the selected one or more RDMA read messages and sends them to the remote RDMA end; receives the read response message returned by the remote RDMA end, processes the read response message, and updates the number of currently pending read response messages. This method of controlling the sending of request messages based on the maximum read response message window threshold allowed by a single QP can reduce the imbalance of network communication traffic between the local end and the remote end and the remote end and local end, reduce network traffic congestion, and greatly improve the processing speed of RDMA read messages, thereby increasing the effective bandwidth of the network.

[0041] In some embodiments, determining the number n of request messages that can be sent currently based on the number of read response messages currently to be received by the local RDMA end and the maximum read response message window threshold allowed by a single QP specifically includes:

[0042] Use the following formula to calculate the number n of request messages that can be sent currently:

[0043] n=(wnd-k) / N

[0044] Wherein, wnd is the maximum read response message window threshold allowed by a single QP, and k is the number of read response messages currently to be received.

[0045] Specifically, as described above, the maximum read response message window threshold wnd allowed for a single QP can be configured as the bandwidth delay product (BDP). Subtracting the current number of pending read response messages k from wnd yields the number of read response messages that the local RDMA end can still receive. N is the message splitting factor, which specifies the maximum number of RDMA read response messages that each RDMA read request message can generate. Dividing wnd - k by N yields the number of request messages n that can currently be sent.

[0046] Among them, since the number of read response messages corresponding to each RDMA read request message is fixed (N), the local RDMA end can calculate the number of RDMA read response messages currently waiting to be returned based on the number of RDMA read request messages that have been sent but have not yet returned RDMA read response messages, that is, the number of read response messages currently to be received.

[0047] Specifically, the number of read response messages currently to be received can be calculated using the following formula:

[0048]

[0049] Wherein, f is the number of request messages for which no corresponding read response messages have been received among all request messages sent by the local RDMA end, and N is a preset message splitting factor.

[0050] In the above embodiment, a specific method for calculating the number of read response messages to be received and a specific method for calculating the number n of request messages that can be sent are provided. The sending of request messages is subsequently controlled based on n, which greatly improves the processing speed of RDMA read messages and increases the effective bandwidth of the network.

[0051] In some embodiments, selecting one or more RDMA read messages from the RDMA read messages to be processed in sequence according to the order in which the RDMA read messages are received and the starting position of the currently processed RDMA read message, and generating n request messages according to the selected one or more RDMA read messages and sending them to the remote RDMA end specifically includes:

[0052] For any currently pending RDMA read message, if it is determined based on the starting position that the currently pending RDMA read message is an unsplit RDMA read message, then calculate the number of request messages X1 into which the currently pending RDMA read message can be split according to the set rule;

[0053] If X1 is greater than the number of request messages that can currently be sent, n, the currently pending RDMA read message is split into multiple sub-messages according to the set rules, n sub-messages are obtained from the multiple sub-messages, n request messages are generated based on the n sub-messages and sent to the remote RDMA end, and the number of request messages that can currently be sent is updated to 0;

[0054] If X1 is less than or equal to the number n of request messages that can be sent currently, then X1 request messages are generated according to the currently pending RDMA read message and sent to the remote RDMA end, the number of request messages that can be sent currently is updated to Y1, Y1=n-X1, and the number of request messages X2 that the next pending RDMA read message of the currently pending RDMA read message can be split into according to the set rules is further calculated. If X2 is less than or equal to Y1, then X2 request messages are generated according to the next pending RDMA read message and sent to the remote RDMA end, and the number of request messages that can be sent currently is updated to Y2, Y2=Y1-X2, and the number of request messages X3 that the next pending RDMA read message can be split into is further calculated. The relationship between X3 and Y2 is judged, and corresponding request messages are generated until n request messages are generated, and the number of request messages that can be sent currently is updated to 0; if X2 is greater than Y1, the next pending RDMA read message is split according to the set rules. Read message, select Y1 sub-messages from the sub-messages after the next RDMA read message to be processed, generate Y1 request messages according to the Y1 sub-messages and send them to the remote RDMA end, and update the number of request messages that can be sent to 0.

[0055] As described above, the RDMA read message currently pending on the local RDMA end may be a complete RDMA read message (that is, an unsplit RDMA read message), or it may be the remaining unprocessed sub-message of a split RDMA read message. The starting position of the currently pending RDMA read message can be used to determine whether it is an unsplit RDMA read message or the remaining unprocessed sub-message of a split RDMA read message.

[0056] Optionally, for any currently pending RDMA read message, if the starting position of the message is consistent with the starting address of the corresponding complete RDMA read message, then the currently pending RDMA read message is determined to be an unsplit RDMA read message; if they are inconsistent, then the currently pending RDMA read message is determined to be the remaining unprocessed sub-message portion of a split RDMA read message.

[0057] Optionally, if it is determined according to the starting position that the RDMA read message currently to be processed is an unsplit RDMA read message, the number X1 of request messages into which the RDMA read message currently to be processed can be split according to a set rule is calculated.

[0058] Specifically, the length of the currently pending RDMA read message can be obtained, and based on the length, a preset message splitting factor, and a preset PMTU size of each read response message, the number of request messages that can be split into X1 can be calculated.

[0059] After calculating the number of request messages X1 that the currently pending RDMA read message can be split into according to the set rules, compare X1 with the number of request messages that can currently be sent, n. If X1 is greater than n, split the currently pending RDMA read message into X1 sub-messages. The starting address of the i-th sub-message can be determined as follows:

[0060] ;

[0061] Where VA0 is the starting address of the RDMA read message that is split into the X1 sub-messages, N is the preset message splitting factor, PMTU is the preset size of each read response message, and VA_i is the starting address of the i-th sub-message. Then, n sub-messages are sequentially retrieved from the X1 sub-messages, and a request message corresponding to each sub-message is generated. These n request messages are sent to the remote RDMA peer, and the number of request messages that can currently be sent is updated to 0. In this case, there may be remaining sub-messages, which the local RDMA peer will subsequently process in sequence.

[0062] If X1 is less than or equal to n, the currently pending RDMA read message is split into X1 sub-messages, with the starting address of the i-th sub-message also determined by the above formula. A request message corresponding to each sub-message is then generated and sent to the remote RDMA end. The number of request messages that can currently be sent is updated to Y1, where Y1 = n - X1. Furthermore, the number of request messages X2, into which the next pending RDMA read message of the currently pending RDMA read message can be split according to the set rules, is calculated. X2 is calculated similarly to X1. X2 is compared with Y1. If X2 is less than or equal to Y1, the next pending RDMA read message is split into X2 sub-messages. A request message corresponding to each sub-message is then generated and sent to the remote RDMA end. The number of request messages that can currently be sent is updated to Y2, where Y2 = Y1 - X2. The next pending RDMA read message is further calculated as the number of request messages, X3, that can be split into. Using the same logic as above, X3 is compared with Y2. If X3 is less than or equal to Y2, the next pending RDMA read message is split into X3 sub-messages. A request message corresponding to each sub-message is generated and sent to the remote RDMA end. This process is repeated until the number of request messages sent reaches n, at which point the number of request messages that can currently be sent is updated to 0. During the above process, if a comparison result is greater than, for example, if X2 is greater than Y1, the next pending RDMA read message is split into X2 sub-messages according to the above message splitting method. Y1 sub-messages are then sequentially selected from the X2 sub-messages, a request message corresponding to each sub-message is generated, and Y1 request messages are sent to the remote RDMA end. The number of request messages that can currently be sent is updated to 0.

[0063] The above embodiment provides a specific implementation method for sending a request message when the currently pending RDMA read message is an unsplit RDMA read message. This method of controlling the sending of request messages based on the maximum read response message window threshold allowed by a single QP can reduce network traffic congestion, greatly improve the processing speed of RDMA read messages, and increase the effective network bandwidth.

[0064] In some embodiments, the RDMA read message processing method provided by the embodiments of the present application further includes: for any currently-to-be-processed RDMA read message, if it is determined according to the starting position that the currently-to-be-processed RDMA read message belongs to the remaining unprocessed sub-message portion of the split RDMA read message;

[0065] Calculate the number U of request messages that can be generated by the remaining unprocessed sub-message part;

[0066] If U is greater than the number n of request messages that can currently be sent, n sub-messages are selected from the remaining unprocessed sub-messages, n request messages are generated based on the n sub-messages and sent to the remote RDMA end, and the number of request messages that can currently be sent is updated to 0;

[0067] If U is less than or equal to the number n of request messages that can currently be sent, then U request messages are generated based on the remaining unprocessed sub-messages and sent to the remote RDMA end, the number of request messages that can currently be sent is updated to V1, V1=nU, and the number of request messages U1 that the next pending RDMA read message can be split into according to the set rules is further calculated. If U1 is greater than V1, the next pending RDMA read message is split according to the set rules, V1 sub-messages are selected from the sub-messages after the next pending RDMA read message is split, V1 request messages are generated based on the V1 sub-messages, and sent to the remote RDMA end, and the number of request messages that can currently be sent is updated to 0.

[0068] As described above, it is possible to determine whether the RDMA read message to be processed is an unsplit RDMA read message or a remaining unprocessed sub-message of a split RDMA read message according to the starting position of the RDMA read message to be processed.

[0069] If it is determined based on the starting position that the currently-to-be-processed RDMA read message belongs to the remaining unprocessed sub-message portion of the split RDMA read message, the number of sub-messages contained in the remaining unprocessed sub-message portion can be obtained. This number is the number U of request messages that can be generated by the remaining unprocessed sub-message portion. Compare U with the number n of request messages that can be sent currently. If U is greater than n, select n sub-messages from the remaining unprocessed sub-messages in sequence, then generate request messages corresponding to each sub-message, send n request messages to the remote RDMA end, and update the number of request messages that can currently be sent to 0; if U is less than or equal to n, generate U request messages based on the remaining unprocessed sub-messages and send them to the remote RDMA end, update the number of request messages that can currently be sent to V1, V1=nU, and further calculate the number of request messages U1 that can be split into according to the set rules for the next complete RDMA read message to be processed. The following process is similar to the processing process after calculating X1 above. Specifically, compare U1 with V1. If U1 is greater than V1, the RDMA to be processed The read message is split into U1 sub-messages, and V1 sub-messages are selected from these U1 sub-messages in sequence. A request message corresponding to each sub-message is generated and sent to the remote RDMA end. The number of request messages that can currently be sent is updated to 0. If U1 is less than or equal to V1, the pending RDMA read message is split into U1 sub-messages, and a request message corresponding to each sub-message is generated and sent to the remote RDMA end. The number of request messages that can currently be sent is updated to V2, where V2 = V1 - U. This process continues until the number of request messages sent reaches n, at which point the number of request messages that can currently be sent is updated to 0.

[0070] The above embodiment provides a specific implementation method for sending a request message when the currently pending RDMA read message belongs to the remaining unprocessed sub-message portion of a split RDMA read message. This method of controlling the sending of request messages based on the maximum read response message window threshold allowed by a single QP can reduce network traffic congestion, significantly improve the processing speed of RDMA read messages, and increase the effective network bandwidth.

[0071] In some embodiments, calculating the number X1 of request messages into which the currently-to-be-processed RDMA read message can be split according to a set rule specifically includes:

[0072] Calculate X1 using the following formula:

[0073] ;

[0074] Wherein, msg_len is the length of the RDMA read message currently to be processed, N is a preset message splitting factor, and PMTU is a preset size of each read response message.

[0075] It should be noted that the above formula is not only applicable to calculating the number of request messages X1 into which the currently pending RDMA read message can be split according to the set rules, but also applicable to calculating the number of request messages X1 corresponding to other RDMA read messages. For example, the number of request messages X2 into which the next pending RDMA read message after the currently pending RDMA read message can be split according to the set rules, and the number of request messages X3 into which the next pending RDMA read message can be split can both be calculated using the above formula by substituting the length of the corresponding RDMA read message into msg_len in the above formula.

[0076] In the above embodiment, a specific method for calculating the number of request messages that an RDMA read message can be split into according to a set rule is provided. The calculated number of request messages can be used to control the generation of request messages, thereby reducing network traffic congestion and increasing the effective bandwidth of the network.

[0077] In some embodiments, obtaining n sub-messages from a plurality of sub-messages and generating n request messages according to the n sub-messages specifically includes:

[0078] Select n sub-messages in sequence from multiple sub-messages to generate n request messages, where the starting address of the i-th request message is determined as follows:

[0079] ;

[0080] VA0 is the starting address of the RDMA read message obtained by splitting the multiple sub-messages, N is the preset message splitting factor, PMTU is the preset size of each read response message, and VA_i is the starting address of the i-th message.

[0081] Optionally, when the currently pending RDMA read message is split into multiple sub-messages, the starting address of the i-th sub-message can be calculated according to the above formula. For the i-th sub-message, a corresponding request message can be generated according to a preset rule. The starting address of the request message is consistent with the starting address of the corresponding sub-message. Each sub-message corresponds to a request message.

[0082] It should be noted that, for any RDMA read message, when it is split to obtain a request message, the above formula can be used to determine the starting address of the request message.

[0083] Optionally, in the process of splitting the RDMA read message, all sub-messages can be split at one time, and then a corresponding number of sub-messages can be obtained in sequence from all sub-messages as needed; or it can be split on demand. For example, if the number of request messages that can be sent currently is n, then n sub-messages are first split, and n request messages are generated and sent to the remote RDMA end. This embodiment of the present application does not limit this.

[0084] The above embodiment provides a specific method for determining the starting address of a request message. This method of splitting an RDMA read message to generate multiple request messages can reduce the imbalance of network communication traffic between the local end and the remote end and between the remote end and the local end.

[0085] It should be understood that, although the various steps in the flowcharts involved in the above embodiments are displayed in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless clearly stated herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps. It is understandable that the various steps in different embodiments can be freely combined as needed, and the various non-contradictory schemes formed by the combination all fall within the scope of protection of this application.

[0086] In some embodiments, see Figure 3 As shown, an RDMA read message processing unit is provided, comprising:

[0087] A message splitting module is configured to determine the number n of request messages that can be sent based on the number of read response messages currently to be received by the local RDMA end and the maximum read response message window threshold allowed by a single QP; and to select one or more RDMA read messages from the RDMA read messages to be processed in sequence based on the order in which the RDMA read messages are received and the starting position of the RDMA read message currently to be processed;

[0088] A first message sending processing module is used to generate n request messages according to one or more selected RDMA read messages and send them to the remote RDMA end;

[0089] The first message receiving and processing module is configured to receive the read response message returned by the remote RDMA end, process the read response message, and update the number of read response messages currently to be received.

[0090] In some embodiments, the message splitting module is specifically configured to:

[0091] Use the following formula to calculate the number n of request messages that can be sent currently:

[0092] n=(wnd-k) / N

[0093] Wherein, wnd is the maximum read response message window threshold allowed by a single QP, and k is the number of read response messages currently to be received;

[0094] The number of read response messages currently to be received is calculated using the following formula:

[0095]

[0096] Wherein, f is the number of request messages for which no corresponding read response messages have been received among all request messages sent by the local RDMA end, and N is a preset message splitting factor.

[0097] In some embodiments, the first message sending processing module is specifically configured to:

[0098] For any currently pending RDMA read message, if it is determined based on the starting position that the currently pending RDMA read message is an unsplit RDMA read message, then calculate the number of request messages X1 into which the currently pending RDMA read message can be split according to the set rule;

[0099] If X1 is greater than the number of request messages that can currently be sent, n, the currently pending RDMA read message is split into multiple sub-messages according to the set rules, n sub-messages are obtained from the multiple sub-messages, n request messages are generated based on the n sub-messages and sent to the remote RDMA end, and the number of request messages that can currently be sent is updated to 0;

[0100] If X1 is less than or equal to the number n of request messages that can be sent currently, then X1 request messages are generated according to the currently pending RDMA read message and sent to the remote RDMA end, the number of request messages that can be sent currently is updated to Y1, Y1=n-X1, and the number of request messages X2 that the next pending RDMA read message of the currently pending RDMA read message can be split into according to the set rules is further calculated. If X2 is less than or equal to Y1, then X2 request messages are generated according to the next pending RDMA read message and sent to the remote RDMA end, and the number of request messages that can be sent currently is updated to Y2, Y2=Y1-X2, and the number of request messages X3 that the next pending RDMA read message can be split into is further calculated. The relationship between X3 and Y2 is judged, and corresponding request messages are generated until n request messages are generated, and the number of request messages that can be sent currently is updated to 0; if X2 is greater than Y1, the next pending RDMA read message is split according to the set rules. Read message, select Y1 sub-messages from the sub-messages after the next RDMA read message to be processed, generate Y1 request messages according to the Y1 sub-messages and send them to the remote RDMA end, and update the number of request messages that can be sent to 0.

[0101] In some embodiments, the first message sending processing module is specifically configured to:

[0102] For any currently pending RDMA read message, if it is determined according to the starting position that the currently pending RDMA read message belongs to the remaining unprocessed sub-message part of the split RDMA read message;

[0103] Calculate the number U of request messages that can be generated by the remaining unprocessed sub-message part;

[0104] If U is greater than the number n of request messages that can currently be sent, n sub-messages are selected from the remaining unprocessed sub-messages, n request messages are generated based on the n sub-messages and sent to the remote RDMA end, and the number of request messages that can currently be sent is updated to 0;

[0105] If U is less than or equal to the number n of request messages that can currently be sent, then U request messages are generated based on the remaining unprocessed sub-messages and sent to the remote RDMA end, the number of request messages that can currently be sent is updated to V1, V1=nU, and the number of request messages U1 that the next pending RDMA read message can be split into according to the set rules is further calculated. If U1 is greater than V1, the next pending RDMA read message is split according to the set rules, V1 sub-messages are selected from the sub-messages after the next pending RDMA read message is split, V1 request messages are generated based on the V1 sub-messages, and sent to the remote RDMA end, and the number of request messages that can currently be sent is updated to 0.

[0106] In some embodiments, the first message sending processing module is specifically configured to:

[0107] Calculate X1 using the following formula:

[0108] ;

[0109] Wherein, msg_len is the length of the RDMA read message currently to be processed, N is a preset message splitting factor, and PMTU is a preset size of each read response message.

[0110] In some embodiments, the first message sending processing module is specifically configured to:

[0111] Select n sub-messages in sequence from multiple sub-messages to generate n request messages, where the starting address of the i-th request message is determined as follows:

[0112] ;

[0113] VA0 is the starting address of the RDMA read message obtained by splitting the multiple sub-messages, N is the preset message splitting factor, PMTU is the preset size of each read response message, and VA_i is the starting address of the i-th message.

[0114] Each module in the RDMA read message processing unit can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0115] Here are some examples:

[0116] See also Figure 4As shown, a local RDMA end is provided, comprising: a message splitting module, a first message sending processing module, and a first message receiving processing module, wherein the first message sending processing module is represented by RDMA TX1, and the first message receiving processing module is represented by RDMA RX1. A remote RDMA end is provided, comprising: a second message receiving processing module and a second message sending processing module, wherein the second message receiving processing module is represented by RDMA RX2, and the second message sending processing module is represented by RDMATX2.

[0117] 1) The message splitting module is used to split an RDMA read message into multiple sub-messages. This example assumes that the preset message splitting factor N = 2 and the maximum read response message window threshold max_rd_rsp_pkt_wnd allowed for a single QP = 4. The message splitting module receives a new RDMA read message. Assuming that the length of the RDMA read message, msg_len, is 8 PMTUs, the formula is:

[0118]

[0119] It can be determined that the RDMA read message needs to be split into four sub-messages, and a corresponding request message must be generated for each of the four sub-messages and sent to the remote RDMA peer. Assuming that the number of read response messages to be received is rd_rsp_inflight_pkt_num = 0 (that is, there are currently no read response messages waiting to be returned), the message splitting module decides to split the message into two sub-messages first according to max_rd_rsp_pkt_wnd / N = 2.

[0120] 2) The message splitting module transmits the two sub-messages to RDMA TX1, and RDMA TX1 generates the request messages corresponding to the two sub-messages ( Figure 4 RDMA TX1 sends two request packets to the remote RDMA peer. At this point, rd_rsp_inflight_pkt_num is updated to 4. At this point, rd_rsp_inflight_pkt_num is equal to max_rd_rsp_pkt_wnd.

[0121] 3) The remote RDMA end receives two request messages and sends them to RDMA RX2.

[0122] 4) RDMA RX2 processes the two request messages separately, generates two independent response message descriptors, and sends them to the read response queue for scheduling.

[0123] 5) The remote RDMA end schedules a certain read response descriptor according to the traffic management and congestion management mechanisms, and sends the read response descriptor into the RDMA TX2. The RDMA TX2 generates a corresponding read response packet based on the read response descriptor.

[0124] 6) According to two independent read response descriptors, the RDMA TX2 returns a total of 4 read response packets (read response packets 1 / 2 / 3 / 4).

[0125] 7) After the RDMA RX1 in the local RDMA end receives the read response packet, it starts to process. After processing, the RDMA RX1 writes the data carried by the read response packet into the local memory.

[0126] 8) Each time the RDMA RX1 in the local RDMA end receives a read response packet, it decrements the rd_rsp_inflight_pkt_num by 1. When rd_rsp_inflight_pkt_num < max_rd_rsp_pkt_wnd, the RDMA RX1 notifies the message splitting module to restart splitting and sends the remaining two request packets (request packets 3 / 4) to the remote RDMA end.

[0127] In an exemplary embodiment, a network interface card is provided, which includes the RDMA read message processing unit in the foregoing embodiment.

[0128] In an exemplary embodiment, an RDMA communication system is provided, which includes the RDMA read message processing unit in the foregoing embodiment.

[0129] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the RDMA read message processing method provided in the foregoing embodiment.

[0130] In an embodiment, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, it implements the RDMA read message processing method provided in the foregoing embodiment.

[0131] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0132] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0133] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A RDMA read message processing method, characterized in that: Applied to the local RDMA end, the method includes: Determine the number n of request messages that can be sent currently based on the number of read response messages currently to be received by the local RDMA end and the maximum read response message window threshold allowed by a single QP; Select one or more RDMA read messages from the pending RDMA read messages according to the order in which the RDMA read messages are received and the starting position of the currently pending RDMA read message. Generate n request messages based on the selected one or more RDMA read messages and send them to the remote RDMA end. Receive the read response message returned by the remote RDMA end, process the read response message, and update the number of read response messages currently to be received.

2. The method according to claim 1, characterized in that The determining, based on the number of read response messages currently to be received by the local RDMA end and the maximum read response message window threshold allowed by a single QP, the number n of request messages that can currently be sent specifically includes: Use the following formula to calculate the number n of request messages that can be sent currently: n=(wnd-k) / N Wherein, wnd is the maximum read response message window threshold allowed by a single QP, and k is the number of read response messages currently to be received; The number of read response messages currently to be received is calculated using the following formula: Wherein, f is the number of request messages for which no corresponding read response messages have been received among all request messages sent by the local RDMA end, and N is a preset message splitting factor, which is used to indicate the maximum number of RDMAread response messages that can be generated by an RDMA request message.

3. The method according to claim 1, characterized in that The method of selecting one or more RDMA read messages from the RDMA read messages to be processed according to the order in which the RDMA read messages are received and the starting position of the currently processed RDMA read message, and generating n request messages according to the selected one or more RDMA read messages and sending the n request messages to the remote RDMA end specifically includes: For any currently pending RDMA read message, if it is determined based on the starting position that the currently pending RDMA read message is an unsplit RDMA read message, then calculate the number of request messages X1 into which the currently pending RDMA read message can be split according to the set rule; If X1 is greater than the number of request messages that can currently be sent, n, the currently pending RDMA read message is split into multiple sub-messages according to the set rules, n sub-messages are obtained from the multiple sub-messages, n request messages are generated based on the n sub-messages and sent to the remote RDMA end, and the number of request messages that can currently be sent is updated to 0; If X1 is less than or equal to the number n of request messages that can be sent currently, then X1 request messages are generated according to the currently pending RDMA read message and sent to the remote RDMA end, the number of request messages that can be sent currently is updated to Y1, Y1=n-X1, and the number of request messages X2 that the next pending RDMA read message of the currently pending RDMA read message can be split into according to the set rules is further calculated. If X2 is less than or equal to Y1, then X2 request messages are generated according to the next pending RDMA read message and sent to the remote RDMA end, and the number of request messages that can be sent currently is updated to Y2, Y2=Y1-X2, and the number of request messages X3 that the next pending RDMA read message can be split into is further calculated. The relationship between X3 and Y2 is judged, and corresponding request messages are generated until n request messages are generated, and the number of request messages that can be sent currently is updated to 0; if X2 is greater than Y1, the next pending RDMA read message is split according to the set rules. read message, select Y1 sub-messages from the sub-messages after the next RDMAread message to be processed is split, generate Y1 request messages according to the Y1 sub-messages and send them to the remote RDMA end, and update the number of request messages that can currently be sent to 0.

4. The method according to claim 1, wherein The method further comprises: For any currently pending RDMA read message, if it is determined according to the starting position that the currently pending RDMA read message belongs to the remaining unprocessed sub-message part of the split RDMA read message; Calculate the number U of request messages that can be generated by the remaining unprocessed sub-message part; If U is greater than the number n of request messages that can currently be sent, n sub-messages are selected from the remaining unprocessed sub-messages, n request messages are generated based on the n sub-messages and sent to the remote RDMA end, and the number of request messages that can currently be sent is updated to 0; If U is less than or equal to the number n of request messages that can currently be sent, then U request messages are generated based on the remaining unprocessed sub-messages and sent to the remote RDMA end, the number of request messages that can currently be sent is updated to V1, V1=nU, and the number of request messages U1 that the next pending RDMA read message can be split into according to the set rules is further calculated. If U1 is greater than V1, the next pending RDMA read message is split according to the set rules, V1 sub-messages are selected from the sub-messages after the next pending RDMA read message is split, V1 request messages are generated based on the V1 sub-messages, and sent to the remote RDMA end, and the number of request messages that can currently be sent is updated to 0.

5. The method according to claim 3 or 4, characterized in that The calculating of the number X1 of request messages into which the currently-to-be-processed RDMA read message can be split according to a set rule specifically includes: Calculate X1 using the following formula: ； Wherein, msg_len is the length of the RDMA read message currently to be processed, N is a preset message splitting factor, and PMTU is a preset size of each read response message.

6. The method according to claim 3, characterized in that The obtaining of n sub-messages from the plurality of sub-messages and generating n request messages according to the n sub-messages specifically includes: Select n sub-messages in sequence from multiple sub-messages to generate n request messages, where the starting address of the i-th request message is determined as follows: ； VA0 is the starting address of the RDMA read message obtained by splitting the multiple sub-messages, N is the preset message splitting factor, PMTU is the preset size of each read response message, and VA_i is the starting address of the i-th message.

7. An RDMA read message processing unit, characterized in that: include: The message splitting module is used to determine the number of request messages that can be sent based on the number of read response messages currently to be received by the local RDMA end and the maximum read response message window threshold allowed by a single QP; Select one or more RDMA read messages from the RDMA read messages to be processed in sequence according to the order in which the RDMA read messages are received and the starting position of the RDMA read message to be processed. A first message sending processing module is used to generate n request messages according to one or more selected RDMA read messages and send them to the remote RDMA end; The first message receiving and processing module is configured to receive the read response message returned by the remote RDMA end, process the read response message, and update the number of read response messages currently to be received.

8. The RDMA read message processing unit according to claim 7, characterized in that: The message splitting module is specifically used to: Use the following formula to calculate the number n of request messages that can be sent currently: n=(wnd-k) / N Wherein, wnd is the maximum read response message window threshold allowed by a single QP, and k is the number of read response messages currently to be received; The number of read response messages currently to be received is calculated using the following formula: Wherein, f is the number of request messages for which no corresponding read response messages have been received among all request messages sent by the local RDMA end, and N is a preset message splitting factor.

9. The RDMA read message processing unit according to claim 7, characterized in that: The first message sending processing module is specifically configured to: For any currently pending RDMA read message, if it is determined based on the starting position that the currently pending RDMA read message is an unsplit RDMA read message, then calculate the number of request messages X1 into which the currently pending RDMA read message can be split according to the set rule; If X1 is greater than the number of request messages that can currently be sent, n, the currently pending RDMA read message is split into multiple sub-messages according to the set rules, n sub-messages are obtained from the multiple sub-messages, n request messages are generated based on the n sub-messages and sent to the remote RDMA end, and the number of request messages that can currently be sent is updated to 0; If X1 is less than or equal to the number n of request messages that can be sent currently, then X1 request messages are generated according to the currently pending RDMA read message and sent to the remote RDMA end, and the number of request messages that can be sent currently is updated to Y1, where Y1=n-X1. The number X2 of request messages that the next pending RDMA read message of the currently pending RDMA read message can be split into according to the set rule is further calculated. If X2 is less than or equal to Y1, then X2 request messages are generated according to the next pending RDMA read message and sent to the remote RDMA end, and the number of request messages that can be sent currently is updated to Y2, where Y2=Y1-X2. The number X3 of request messages that the next pending RDMA read message can be split into is further calculated. The relationship between X3 and Y2 is determined, and corresponding request messages are generated until n request messages are generated, and the number of request messages that can be sent currently is updated to 0. If X2 is greater than Y1, the next pending RDMA read message is split according to the set rules, Y1 sub-messages are selected from the sub-messages after the next pending RDMA read message is split, Y1 request messages are generated based on the Y1 sub-messages and sent to the remote RDMA end, and the number of request messages that can currently be sent is updated to 0.

10. The RDMA read message processing unit according to claim 7, characterized in that: The first message sending processing module is specifically configured to: For any currently pending RDMA read message, if it is determined according to the starting position that the currently pending RDMA read message belongs to the remaining unprocessed sub-message part of the split RDMA read message; Calculate the number U of request messages that can be generated by the remaining unprocessed sub-message part; If U is greater than the number n of request messages that can currently be sent, n sub-messages are selected from the remaining unprocessed sub-messages, n request messages are generated based on the n sub-messages and sent to the remote RDMA end, and the number of request messages that can currently be sent is updated to 0; If U is less than or equal to the number n of request messages that can currently be sent, then U request messages are generated based on the remaining unprocessed sub-messages and sent to the remote RDMA end, the number of request messages that can currently be sent is updated to V1, V1=nU, and the number of request messages U1 that the next pending RDMA read message can be split into according to the set rules is further calculated. If U1 is greater than V1, the next pending RDMA read message is split according to the set rules, V1 sub-messages are selected from the sub-messages after the next pending RDMA read message is split, V1 request messages are generated based on the V1 sub-messages, and sent to the remote RDMA end, and the number of request messages that can currently be sent is updated to 0.

11. The RDMA read message processing unit according to claim 9 or 10, characterized in that: The first message sending processing module is specifically configured to: Calculate X1 using the following formula: ； Wherein, msg_len is the length of the RDMA read message currently to be processed, N is a preset message splitting factor, and PMTU is a preset size of each read response message.

12. The RDMA read message processing unit according to claim 9, characterized in that: The first message sending processing module is specifically configured to: Select n sub-messages in sequence from multiple sub-messages to generate n request messages, where the starting address of the i-th request message is determined as follows: ； VA0 is the starting address of the RDMA read message obtained by splitting the multiple sub-messages, N is the preset message splitting factor, PMTU is the preset size of each read response message, and VA_i is the starting address of the i-th message.

13. A network interface card, characterized in that: The method comprises the RDMA read message processing unit according to any one of claims 7 to 12.

14. An RDMA communication system, characterized in that: The method comprises the RDMA read message processing unit according to any one of claims 7 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images