A remote direct memory access device and method

Abstract

The application provides a remote direct memory access device and method. The remote direct memory access device comprises a sending module and a receiving module. The sending module acquires a current work queue element, and determines that the current work queue element is an abnormal work queue element after the current work queue element appears abnormal. The sending module determines whether all work queue elements before the abnormal work queue element have received corresponding response responses. If all work queue elements before the abnormal work queue element have received corresponding response responses, and the receiving module no longer receives response responses, the sending module generates an abnormal completion queue element for the abnormal work queue element, and writes the abnormal completion queue element into a completion queue. The application can still write corresponding CQEs into the completion queue for the WQE that has not been sent out after the work queue appears abnormal, and can also ensure that the order of the CQEs is consistent with the order of the corresponding WQE.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of network communication, and particularly relates to a remote direct memory access device and method. BACKGROUND

[0002] Under normal circumstances, after a sender of an HCA (Host Channel Adapter) obtains a WQE (Work Queue Element) from a work queue (SQ) and composes a request data packet, the sender sends the request data packet out, and when the sender receives an ACK from an opposite end, a corresponding CQE (Completion Queue Entry) is created and sent to a corresponding completion queue (CQ), as shown in FIG. 1. Figure 1 The WQEs in the same SQ are sequentially sent, and thus the CQEs are sequentially recycled. After an application layer obtains a corresponding CQE from the CQ, the application layer can know the WQE that has been completed, and thus can release the memory resource corresponding to the completed WQE. SUMMARY

[0003] The present application aims to provide a remote direct memory access device and method, and aims to solve the problem that after a work queue appears an exception, the remaining WQEs in the work queue cannot be sent out, and thus the CQEs corresponding to the remaining WQEs cannot be generated according to a normal process, and the CQEs between the normally sent WQEs and the not sent WQEs cannot be guaranteed to be consistent with the corresponding WQEs in sequence.

[0004] According to a first aspect of the present application, a remote direct memory access device is provided, comprising a sending module and a receiving module, wherein

[0005] The sending module obtains a current work queue element, and after determining that the current work queue element appears an exception, takes the current work queue element as an abnormal work queue element, and determines whether the work queue elements before the abnormal work queue element have all received corresponding response.

[0006] If the work queue elements before the abnormal work queue element have all received corresponding response, and the receiving module no longer receives a response, the sending module generates an abnormal completion queue element for the abnormal work queue element, and writes the abnormal completion queue element into a completion queue.

[0007] In an optional implementation, if the sending module determines that the work queue elements before the abnormal work queue element have not all received corresponding response, a data retransmission request corresponding to the abnormal work queue element is initiated.

[0008] In an optional implementation, if the sending module determines that all work queue elements before the abnormal work queue element have all received corresponding acknowledgement responses, and the receiving module is still able to receive an acknowledgement response, the receiving module is initiated with a notification of no longer continuing to receive an acknowledgement response of the abnormal work queue element;

[0009] After the receiving module receives the notification, it no longer receives an acknowledgement response, and initiates a data retransmission request of the abnormal work queue element.

[0010] In an optional implementation, the remote direct memory access device further comprises: an on-chip work queue cache module; wherein,

[0011] After the sending module writes the abnormal completion queue element into the completion queue, it initiates a request to the on-chip work queue cache module to delete a cache chain corresponding to an abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

[0012] In an optional implementation, the remote direct memory access device further comprises: a doorbell module and a scheduling module; wherein, the data retransmission request is a retransmission doorbell request;

[0013] After the doorbell module receives the retransmission doorbell request, it sets an abnormal work queue where an abnormal work queue element corresponding to the retransmission doorbell request is located as a current to-be-scheduled work queue.

[0014] When the scheduling module schedules the current to-be-scheduled work queue, it judges whether the abnormal work queue element is a work queue element for which the receiving module has not received an acknowledgement response, if so, it pushes the abnormal work queue element to the sending module after taking it out from the current to-be-scheduled work queue.

[0015] After the sending module obtains the abnormal work queue element, it further judges whether the current work queue element appears abnormal.

[0016] In an optional implementation, after the receiving module receives an acknowledgement response, if it judges that a work queue element corresponding to the acknowledgement response appears abnormal, it takes the abnormal work queue element as an abnormal work queue element, writes abnormal information into a work queue context where the abnormal work queue element is located, and initiates a data retransmission request of the abnormal work queue element.

[0017] The sending module stops sending the request data packet corresponding to the work queue element after reading the exception information from the work queue context, and initiates a data retransmission request corresponding to the work queue element after the exception work queue element and not receiving the response if it is determined that the corresponding response of the request data packet that has been sent out is not all received.

[0018] In an optional embodiment, if the sending module reads the exception information from the work queue context and has received the data retransmission request initiated by the receiving module, and the exception work queue element corresponding to the data retransmission request is the same as the work queue element that has not received the response, and the receiving module no longer continues to receive the response, an exception completion queue element is generated for the exception work queue element, and the exception completion queue element is written into the completion queue.

[0019] In an optional embodiment, after the sending module writes the exception completion queue element into the completion queue, a request is initiated to the on-chip work queue cache module to delete the cache chain corresponding to the exception work queue where the exception work queue element is located, so as to release the cache occupied by the exception work queue.

[0020] In an optional embodiment, the sending module also writes a first exception completed report identifier into the context of the exception work queue where the exception work queue element is located, and generates a non-first exception completion queue element corresponding to the work queue element after the exception work queue element and not receiving the response based on the data retransmission request initiated by the sending module, and writes the non-first exception completion queue element into the completion queue.

[0021] According to a second aspect of the present application, a remote direct memory access method is provided, comprising:

[0022] The sending module of the remote direct memory access device obtains the current work queue element, and determines whether the work queue element before the exception work queue element has all received the corresponding response after the current work queue element is determined as an exception work queue element after the exception work queue element occurs.

[0023] If the work queue element before the exception work queue element has all received the corresponding response, and the receiving module of the remote direct memory access device no longer receives the response, the sending module generates an exception completion queue element for the exception work queue element, and writes the exception completion queue element into the completion queue.

[0024] In an optional embodiment, the method further comprises:

[0025] If the sending module determines that the corresponding acknowledgement responses of the work queue elements before the abnormal work queue element have all been received, and the receiving module is still able to receive the acknowledgement responses, the sending module initiates a notification to the receiving module that the receiving of the acknowledgement responses of the abnormal work queue element will no longer be continued.

[0026] In an optional embodiment, the method further comprises:

[0027] If the sending module determines that the corresponding acknowledgement responses of the work queue elements before the abnormal work queue element have all been received, and the receiving module is still able to receive the acknowledgement responses, the sending module initiates a notification to the receiving module that the receiving of the acknowledgement responses of the abnormal work queue element will no longer be continued.

[0028] After the receiving module receives the notification, the receiving of the acknowledgement responses is no longer continued, and the sending module initiates a data retransmission request corresponding to the abnormal work queue element.

[0029] In an optional embodiment, the method further comprises:

[0030] After the sending module writes the abnormal completion queue element into the completion queue, the sending module initiates a request to the on-chip work queue cache module of the remote direct memory device to delete the cache chain corresponding to the abnormal work queue in which the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

[0031] In an optional embodiment, the data retransmission request is a retransmission doorbell request; the method further comprises:

[0032] After the doorbell module of the remote direct memory access device receives the retransmission doorbell request, the abnormal work queue in which the abnormal work queue element corresponding to the retransmission doorbell request is located is set as the current work queue to be scheduled.

[0033] When the scheduling module of the remote direct memory access device schedules the current work queue to be scheduled, it is determined whether the abnormal work queue element is a work queue element for which the receiving module has not received the acknowledgement response; if so, the abnormal work queue element is taken out from the current work queue to be scheduled and pushed to the sending module.

[0034] After the sending module obtains the abnormal work queue element, the sending module takes the abnormal work queue element as the current work queue element, and jumps to the step of determining whether the work queue elements before the abnormal work queue element have all received the corresponding acknowledgement responses.

[0035] In an optional embodiment, the method further comprises:

[0036] After the receiving module receives the response, if it is determined that the work queue element corresponding to the response is abnormal, the work queue element is taken as an abnormal work queue element, abnormal information is written into a work queue context in which the abnormal work queue element is located, and a data retransmission request of the abnormal work queue element is initiated.

[0037] After the sending module reads the abnormal information from the work queue context, the sending of the request data packet corresponding to the work queue element is stopped, and if it is determined that the corresponding response is not received for all the work queue elements for which the data packet has been sent, a data retransmission request of the work queue element after the abnormal work queue element for which the response is not received is initiated.

[0038] In an optional implementation, the method further comprises:

[0039] If the sending module reads the abnormal information from the work queue context, and the data retransmission request initiated by the receiving module is received, the abnormal work queue element corresponding to the data retransmission request is the same as the work queue element after the last work queue element for which the response is received, and the receiving module no longer continues to receive the response, an abnormal completion queue element is generated for the abnormal work queue element, and the abnormal completion queue element is written into a completion queue.

[0040] In an optional implementation, the method further comprises:

[0041] After the sending module writes the first abnormal completion report identifier into the abnormal work queue in which the abnormal work queue element is located and writes the abnormal completion queue element into the completion queue, the sending module initiates a request to the on-chip work queue cache module to delete the cache chain corresponding to the abnormal work queue in which the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

[0042] In an optional implementation, the method further comprises:

[0043] Based on the data retransmission request initiated by the sending module, the sending module generates a non-first abnormal completion queue element corresponding to the work queue element after the abnormal work queue element for which the response is not received, and writes the non-first abnormal completion queue element into the completion queue.

[0044] Compared with the related art, the technical solution of the present application has at least the following advantages:

[0045] The present application can still write a corresponding CQE into a completion queue for a WQE that has not been sent out after the work queue is abnormal, and can also ensure that the order of the CQE is consistent with the order of the corresponding WQE.

[0046] Other features and advantages of the present application will be set forth in the following description, and in part will be apparent from the description, or can be learned by practice of the application. The objects and other advantages of the present application will be realized and attained by the structure and processes particularly pointed out in the description and appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings described below are only some embodiments of the present application, and the ordinary skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0048] Figure 1 is a working queue consumption flow effect diagram according to the related art.

[0049] Figure 2 is a structural block diagram of a remote direct memory access device according to an exemplary embodiment of the present application.

[0050] Figure 3 is a normal processing flow effect diagram of a remote direct memory access device according to an exemplary embodiment of the present application.

[0051] Figure 4 is a processing flow effect diagram of a remote direct memory access device after the sending module discovers an exception according to an exemplary embodiment of the present application.

[0052] Figure 5 is a processing flow effect diagram of a remote direct memory access device after the receiving module discovers an exception according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purposes, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the ordinary skilled in the art without any creative effort are within the protection scope of the present application.

[0054] In the case of an exception of the working queue, the application layer also needs to know which memory resources of the WQE can be released, so the underlying hardware needs to generate corresponding exception completion queue elements according to the original order of all the unfinished WQE, and write them into the completion queue to inform the application layer to release the memory resources of the corresponding WQE.

[0055] But after the work queue exception, the following two problems need to be considered when writing CQE to the completion queue: first, the remaining WQE in the work queue will not be sent out, so it will not receive the corresponding ACK response, and thus it cannot write the CQE corresponding to the remaining WQE to the completion queue according to the normal process; second, the CQE between the normally sent WQE and the unsent WQE needs to meet the order preserving requirement, that is, the order of the CQE and the order of the corresponding WQE should be consistent. At present, no related technology has been found to solve the above two problems.

[0056] Based on this, the present application proposes a remote direct memory access device and method to solve the above two problems, that is, the present application can still write the corresponding CQE to the completion queue for the unsent WQE after the work queue exception, and can also ensure that the order of the CQE and the order of the corresponding WQE are consistent.

[0057] Exemplarily, as shown in Figure 2 The remote direct memory access device proposed by the present application comprises a sending module and a receiving module; wherein,

[0058] The sending module obtains the current work queue element, and determines that the current work queue element is an abnormal work queue element after the current work queue element appears abnormal, and determines whether the work queue elements before the abnormal work queue element have all received the corresponding ACK response.

[0059] If the work queue elements before the abnormal work queue element have all received the corresponding ACK response, and the receiving module no longer receives the ACK response, the sending module generates an abnormal completion queue element for the abnormal work queue element, and writes the abnormal completion queue element into the completion queue.

[0060] Exemplarily, in the case where the work queue does not appear abnormal, as shown in Figure 3 The sending module of the HCA _ A obtains the WQE, generates the corresponding request data packet, and sends it to the HCA_B, and writes the write CQE information of the current WQE to the on-chip work queue cache module. The HCA_B receives the request data packet of the HCA_A, and returns an ACK response to the HCA _ A after verification, the receiving module of the HCA _ A receives the ACK and reads the write CQE information of the ACK response from the on-chip work queue cache module after verification, and generates the corresponding CQE based on the write CQE information, and writes the generated CQE into the CQ, and the hardware WQE recycling process is completed.

[0061] In the process, after the sending module obtains the current work queue element, if it is found in the checking process that the work queue in which the current work queue element is located is abnormal, the starting WQE of the abnormality is the current work queue element. In order to ensure that the order of the CQE corresponding to the WQE sent is consistent with the sending order of the WQE, in the present application, the current work queue element is taken as an abnormal work queue element, and the sending module determines whether the work queue elements before the abnormal work queue element have all received the corresponding response based on the state identifier set by the receiving module. If the corresponding response has all been received, and the receiving module is in a state of no longer receiving the response of the destination device, at this time the sending module generates an error completion queue element (ERR CQE) for the abnormal work queue element and writes it into the completion queue. The subsequent application layer can read the ERR CQE from the completion queue, thereby performing corresponding processing.

[0062] In some optional implementations, if the sending module determines that the work queue elements before the abnormal work queue element have not all received the corresponding response, the sending module initiates a data retransmission request corresponding to the abnormal work queue element.

[0063] That is, if the sending module determines that the work queue elements before the abnormal work queue element have not all received the corresponding response, the sending module can initiate a data retransmission request, thereby re-executing the sending process of the request data packet of the abnormal work queue element. When re-executing the sending process of the request data packet, if all the work queue elements before the abnormal work queue element have received the corresponding response, and the receiving module no longer continues to receive the response, the sending module can generate an error completion queue element (ERR CQE) for the abnormal work queue element and write it into the completion queue according to the above description.

[0064] In some optional implementations, if the sending module determines that the work queue elements before the abnormal work queue element have all received the corresponding response, and the receiving module can still receive the response, the sending module initiates a notification to the receiving module that it no longer continues to receive the response of the abnormal work queue element.

[0065] After the receiving module receives the notification, it no longer receives the response, and initiates a data retransmission request corresponding to the abnormal work queue element.

[0066] If the sending module determines that the response to the work queue element before the abnormal work queue element has been received, but the receiving module is still in the state of continuing to receive the response, the sending module initiates a notification to the receiving module that the response to the abnormal work queue element is no longer received, so that the receiving module sets its state to the state of no longer receiving the response; and the receiving module initiates a data retransmission request corresponding to the abnormal work queue element.

[0067] The data retransmission request causes the sending process of the abnormal work queue element to be executed again. After the sending process is transferred to the sending module, the sending module can continue to determine whether the responses to the work queue elements before the abnormal work queue element have been received and whether the receiving module is in the state of receiving the response. Since the data retransmission request is initiated by the receiving module, the responses to the work queue elements before the abnormal work queue element have been received, and the state of the receiving module is set to the state of no longer receiving the response. Therefore, as described above, the sending module can generate the abnormal completion queue element corresponding to the abnormal work queue element and write the abnormal completion queue element into the completion queue.

[0068] In some optional implementations, the remote direct memory access device further includes: the remote direct memory access device further includes: an on-chip work queue cache module; wherein,

[0069] After the sending module writes the abnormal completion queue element into the completion queue, the sending module initiates a request to the on-chip work queue cache module to delete the cache chain corresponding to the abnormal work queue in which the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

[0070] All work queues can be cached in the on-chip work queue cache module, each work queue can include a plurality of work queue elements, and the sending module can also generate the corresponding write CQE information after generating the corresponding request data packet for the current work queue element, so that the receiving module can generate the completion queue element after reading the corresponding write CQE information from the on-chip work queue cache module after receiving the corresponding response, and then write the completion queue element into the completion queue.

[0071] It can be understood that after the abnormal completion queue element is generated and written into the completion queue, the other work queue elements in the abnormal work queue in which the abnormal work queue element is located are no longer sent, and the corresponding information is no longer recorded in the on-chip work queue cache module, so that the cache occupied by the abnormal work queue in the on-chip work queue cache module can be released, that is, the data chain of the abnormal work queue can be deleted from the on-chip work queue cache module.

[0072] In some optional implementations, the remote direct memory access device further comprises a doorbell module and a scheduling module; wherein the data retransmission request is a retransmission doorbell request;

[0073] After the doorbell module receives the retransmission doorbell request, the doorbell module sets an exception work queue in which an exception work queue element corresponding to the retransmission doorbell request is located as a current work queue to be scheduled;

[0074] When the scheduling module schedules the current work queue to be scheduled, the scheduling module determines whether the exception work queue element is a work queue element for which the receiving module has not received an acknowledgement response, and if so, the scheduling module takes the exception work queue element from the current work queue to be scheduled and pushes the exception work queue element to the sending module;

[0075] After the sending module obtains the exception work queue element, the sending module takes the exception work queue element as a current work queue element and further determines whether the current work queue element is abnormal.

[0076] Exemplarily, the doorbell module is configured to receive a doorbell request, the doorbell request including a data retransmission request initiated by the sending module and the receiving module and a doorbell request initiated by an application layer. The doorbell module, in response to the doorbell request, sets a work queue in which a work queue element corresponding to the doorbell request is located as a current work queue to be scheduled, so as to cause the scheduling module to schedule the work queue element corresponding to the doorbell request from the current work queue to be scheduled and push the scheduled work queue element to the sending module, so that the sending module generates a corresponding request data packet and then sends the request data packet to a destination device.

[0077] When the doorbell request is a data retransmission request corresponding to an exception work queue element, the scheduling module schedules the exception work queue element. When the scheduling module schedules the exception work queue element, the scheduling module first determines whether the exception work queue element and a work queue element for which the receiving module has not received an acknowledgement response are the same. If they are the same, the scheduling module pushes the exception work queue element to the sending module. If they are not the same, it indicates that an acknowledgement response has not been received for a work queue element before the exception work queue element, and the scheduling module needs to wait until an acknowledgement response has been received for all work queue elements before the exception work queue element. To this end, the scheduling module reinitiates a doorbell request to reschedule the exception work queue element until the exception work queue element and the work queue element for which the receiving module has not received an acknowledgement response are the same.

[0078] In the sending process of an abnormal work queue element triggered by a data retransmission request, after the sending module obtains the abnormal work queue element, it will still perform verification according to the normal process. During verification, it will find that the current work queue element is an abnormal work queue element, and then it will re-determine whether all the work queue elements included in the previously sent request data have been responded to, according to the method described above: a) If not all have been responded to, the sending module sends a doorbell request and a data retransmission request to the doorbell module; b) If all have been responded to, but the receiving module can still receive response responses, the sending module sends a doorbell request to the receiving module to notify the receiving module to set its own state to no longer receive response responses; after receiving the doorbell request, the receiving module will initiate a data retransmission request; c) If all have been responded to and the receiving module no longer receives response responses, the sending module starts generating an ERR CQE and writes it to the completion queue.

[0079] The following is through Figure 4 The exemplary embodiment shown describes the processing flow when the sending module detects an anomaly in the work queue.

[0080] like Figure 4 As shown, when sending a request data packet, the sending direction performs checks on the corresponding WQE, such as protocol verification (whether it is illegal), QP status detection, and interface ERR signal checks. If a verification error occurs, the QP ERROR will be set, indicating that an error has occurred in the work queue containing the current work queue element.

[0081] 1. Assume that the starting work queue element of the sending direction Tx_error is WQE3, and SSN=3 (indicating that the last request data packet sent by the sending module was the one corresponding to WQE3). At this time, WQE3 is an abnormal work queue element. If the receiving module does not receive the response of WQE2, it means that the request data packet sent normally before the abnormality occurred did not complete the response. The sending module needs to press the retransmission doorbell and record the starting position of retransmission WQE3 in the QP context (QPC) to initiate the data retransmission request of WQE3.

[0082] 2. When the doorbell module receives a retransmission of the doorbell from the sending module, it sets the request of the work queue where WQE3 is located to be valid, and the work queue is dequeued, that is, the work queue is determined as the current work queue to be scheduled.

[0083] 3. After receiving a request to schedule the currently scheduled work queue, the scheduling module determines whether the abnormal work queue element recorded by the sending module is a WQE that the receiving module has not yet received a response to. If so, the WQE3 in the currently scheduled work queue can be scheduled. Otherwise, the request to schedule the currently scheduled work queue should be returned to the doorbell module to wait for the next scheduling.

[0084] 4、QP exception, the sending module receives the data retransmission request, that is, the data retransmission request sent by the sending module in step 1 passes through the doorbell module and the scheduling module, and is finally pushed to the sending module again. If the receiving module does not currently receive the response WQE equal to the currently requested retransmission WQE, that is, WQE3, and it is judged that the receiving module is still in a state capable of receiving a response, the receiving module is notified to no longer continue to collect the response of the work queue in which WQE3 is located. The receiving module rings the doorbell request, that is, the receiving module initiates a data retransmission request, so as to return to the retransmission process of WQE3 again. This is to ensure that after the receiving module does not operate the on-chip work queue cache module, the sending module deletes the data link of the work queue in which the abnormal work queue element is located on the on-chip work queue cache module, and releases the cache resources occupied by the abnormal work queue element.

[0085] 5、The sending module receives the data retransmission request initiated by the receiving module again. The WQE currently not received by the receiving module is equal to the currently requested WQE, that is, WQE3. The receiving module is in a state of no longer receiving a response. An ERR CQE is generated for the current WQE, that is, WQE3, and is written into the CQ (completion queue). The on-chip work queue cache module initiates a chain deletion request for the current QP. The flush flag (the first abnormal completion reporting flag) is recorded in the QPC. After receiving the remaining WQE of the QP, the flush flag is read, and the flush CQE (non-first abnormal completion queue element) is generated for the remaining WQE and written into the completion queue, without the need to send the remaining WQE again and record the related information of the WQE to the on-chip work queue cache module.

[0086] In some optional implementations, after the receiving module receives the response, if it is judged that the work queue element corresponding to the response is abnormal, the receiving module regards the abnormal work queue element as an abnormal work queue element, writes the abnormal information into the work queue context in which the abnormal work queue element is located, and initiates a data retransmission request for the abnormal work queue element.

[0087] After the sending module reads the abnormal information from the work queue context, the sending module stops sending the request data packet corresponding to the work queue element, and if it is determined that not all corresponding response responses have been received for the work queue element for which the request data packet has been sent, the sending module initiates a data retransmission request for the work queue element for which no response response has been received after the abnormal work queue element.

[0088] Exemplarily, after receiving the response, the receiving module performs a relevant check. If it is found that the current response corresponds to an abnormal work queue element, the work queue element is taken as an abnormal work queue element, and the receiving module initiates a data retransmission request.

[0089] Before the receiving module initiates the data retransmission request, the abnormal information of the abnormal work queue element, such as an abnormal reason, is written in the context of the abnormal work queue where the abnormal work queue element is located. After the receiving module initiates the data retransmission request, the retransmission process of the abnormal work queue element is to the sending module, and the sending module still generates a corresponding abnormal completion queue element and writes it into the queue. It can be understood that the premise of generating the corresponding abnormal completion queue element is that all the work queue elements that have sent the request data packet have received the corresponding response, and the receiving module is in a state of no longer receiving the response.

[0090] After the sending module reads the abnormal information written by the receiving module in the work queue context, it stops sending the request data packet corresponding to the subsequent work queue element in the work queue, and if all the work queue elements that have sent the request data packet have not received the corresponding response, it initiates a data retransmission request. The data retransmission request is a data retransmission request of the work queue element that has sent the request data packet after the abnormal work queue element. In this way, the sending module can generate an abnormal completion queue element (ERR CQE) for the abnormal work queue element and a non-first abnormal completion queue element (flush CQE) for the subsequent work queue element of the abnormal work queue element under the premise of generating the corresponding abnormal completion queue element.

[0091] In some optional implementations, if the sending module reads the abnormal information from the work queue context and has received the data retransmission request initiated by the receiving module, the abnormal work queue element corresponding to the data retransmission request is the same as the work queue element that has not received the response, and the receiving module has no longer continued to receive the response, an abnormal completion queue element is generated for the abnormal work queue element, and the abnormal completion queue element is written into the completion queue.

[0092] Exemplarily, if the data retransmission request initiated by the receiving module returns to the sending module, and the sending module reads the abnormal information written by the receiving module in the work queue context, if the work queue element that has not completed the response by the receiving module is the same as the abnormal work queue element corresponding to the data retransmission request, and the state of the receiving module is no longer to continue to receive the response at this time, the sending module generates an abnormal completion queue element for the abnormal work queue element and writes it into the completion queue.

[0093] In some optional implementation manners, after the sending module writes the exception completion queue element into the completion queue, the sending module initiates a request to the on-chip work queue cache module to delete the cache chain corresponding to the exception work queue in which the exception work queue element is located, so as to release the cache occupied by the exception work queue.

[0094] For example, after the sending module writes the exception completion queue element generated for the exception work queue element into the completion queue, the sending module requests the on-chip work queue cache module to release the cache resource occupied by the exception work queue.

[0095] In some optional implementation manners, the sending module further writes a first exception completion report identifier into the context of the exception work queue in which the exception work queue element is located, and generates a non-first exception completion queue element corresponding to a work queue element that has not received a response after the sending module initiates a data retransmission request, and writes the non-first exception completion queue element into the completion queue.

[0096] For example, after the receiving module discovers the exception of the work queue, the receiving module takes the starting work queue element in which the exception occurs in the work queue as the exception work queue element, and generates the corresponding exception completion queue element for the exception work queue element after the above-mentioned conditions are met. Then, in order to generate the corresponding non-first exception completion queue element for the subsequent work queue element that has completed the request data packet sending after the exception work queue element, the sending module writes a flush identifier (first exception completion report identifier) into the exception queue context, so that after receiving the data retransmission request for the candidate work queue element, the flush CQE (non-first exception completion queue element) corresponding to the candidate work queue element is directly generated.

[0097] The processing flow when the receiving module discovers the exception of the work queue is described below with reference to the exemplary embodiment shown in Figure 5

[0098] As shown in Figure 5 After the receiving module receives the response and performs protocol verification on the received response data packet, the receiving module discovers the exception, and the QP ERROR is set. The set QP ERROR indicates that the work queue in which the work queue element currently receiving the response is located has an exception:

[0099] ​1. If the first exception work queue element in the exception work queue is WQE3, the receiving module records the error reason of the current WQE, i.e., WQE3, in the QPC; and presses the retransmission doorbell, i.e. initiates a data retransmission request so that the sending module can generate an ERR CQE (Exception Completion Queue Element); the WQE4 to WQE6 included in the sent request data are considered as exception work queue elements after the first exception work queue element, and can be retransmitted, and finally the sending module generates the corresponding flushCQE.

[0100] 2. When the sending module detects an anomaly, it stops sending subsequent WQEs. For example, if the sending module detects an anomaly while sending WQE7, then WQE7 and subsequent WQEs will not be sent. If WQE7 is greater than WQE3, which has not yet received a response, and if it is determined that not all previously sent request data packets have been acknowledged, then a retransmission doorbell is pressed, i.e., a retransmission doorbell is sent to initiate a data retransmission request.

[0101] 3. When the receiving module detects that the QP ERROR is set and receives a retransmission request for WQE3 from the sending module, if it determines that the WQE for which the receiving module has not yet received a completion acknowledgment response is equal to the requested WQE in the data retransmission request, and the receiving module is in a state where it can no longer receive acknowledgment responses, then the sending module can generate an ERRCQE for the current WQE, i.e., WQE3, and write it into the CQ. Simultaneously, it initiates a chain deletion request for the current QP to the on-chip QP buffer module to delete the buffer chain of the work queue containing WQE3. The sending module also records the flush flag in the QPC. Subsequent reception of the remaining WQEs of this QP (such as...) Figure 5 After WQE4 to WQE6 are processed, the flush flag is detected, and a flush CQE can be directly generated for the remaining WQEs and written to the CQE. In other words, WQE3 is generated as an ERR CQE by the sending module, and the WQEs following WQE3 are generated as flush CQEs by the sending module.

[0102] 4. Timeouts and data retransmission requests initiated by the receiving module have higher priority than data retransmission requests initiated by the sending module.

[0103] The above is the remote direct memory access device proposed in this application, which has the following advantages compared with related technologies:

[0104] This application can still write the corresponding CQE to the completion queue for WQEs that have not been sent out after an anomaly occurs in the work queue, and can also ensure that the order of CQEs is consistent with the order of the corresponding WQEs.

[0105] Accordingly, this application also provides an exemplary remote direct memory access method, comprising:

[0106] The sending module of the remote direct memory access device acquires a current work queue element, and determines that the current work queue element is an abnormal work queue element after the current work queue element is abnormal, and determines whether work queue elements before the abnormal work queue element have all received corresponding response messages;

[0107] If the work queue elements before the abnormal work queue element have all received the corresponding response messages, and the receiving module of the remote direct memory access device no longer receives the response messages, the sending module generates an abnormal completion queue element for the abnormal work queue element, and writes the abnormal completion queue element into a completion queue.

[0108] In some optional implementations, the method further includes:

[0109] If the sending module determines that the work queue elements before the abnormal work queue element have not all received the corresponding response messages, the sending module initiates a data retransmission request corresponding to the abnormal work queue element.

[0110] In some optional implementations, the method further includes:

[0111] If the sending module determines that the work queue elements before the abnormal work queue element have all received the corresponding response messages, and the receiving module can still receive the response messages, the sending module initiates a notification to the receiving module that the receiving module no longer continues to receive the response messages of the abnormal work queue element.

[0112] After the receiving module receives the notification, the receiving module no longer receives the response messages, and initiates the data retransmission request corresponding to the abnormal work queue element.

[0113] In some optional implementations, the method further includes:

[0114] After the sending module writes the abnormal completion queue element into the completion queue, the sending module initiates a request to a cache module on a chip of the remote direct memory device to delete a cache chain corresponding to an abnormal work queue in which the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

[0115] In some optional implementations, the data retransmission request is a retransmission doorbell request; the method further includes:

[0116] After the doorbell module of the remote direct memory access device receives the retransmission doorbell request, the doorbell module sets an abnormal work queue in which an abnormal work queue element corresponding to the retransmission doorbell request is located as a current work queue to be scheduled.

[0117] When the scheduling module of the remote direct memory access device schedules the current work queue to be scheduled, it is determined whether the abnormal work queue element is a work queue element for which the receiving module currently does not receive a response. If so, the abnormal work queue element is taken out from the current work queue to be scheduled and is pushed to the sending module.

[0118] After the sending module obtains the abnormal work queue element, the abnormal work queue element is taken as a current work queue element, and the step of determining whether work queue elements before the abnormal work queue element have all received corresponding response is executed.

[0119] In some optional implementation manners, the method further includes:

[0120] After the receiving module receives the response, if it is determined that the work queue element corresponding to the response is abnormal, the work queue element is taken as an abnormal work queue element, abnormal information is written into a work queue context in which the abnormal work queue element is located, and a data retransmission request of the abnormal work queue element is initiated.

[0121] After the sending module reads the abnormal information from the work queue context, the sending module stops sending a request data packet corresponding to the work queue element, and if it is determined that corresponding responses have not been received for all work queue elements for which data packets have been sent, the sending module initiates a data retransmission request of a work queue element after the abnormal work queue element for which no response has been received.

[0122] In some optional implementation manners, the method further includes:

[0123] If the sending module reads the abnormal information from the work queue context, has received the data retransmission request initiated by the receiving module, the abnormal work queue element corresponding to the data retransmission request is the same as a work queue element after the last work queue element for which a response has been received, and the receiving module no longer continues to receive a response, an abnormal completion queue element of the abnormal work queue element is generated, and the abnormal completion queue element is written into a completion queue.

[0124] In some optional implementation manners, the method further includes:

[0125] After the sending module writes a first abnormal completion report identifier into an abnormal work queue in which the abnormal work queue element is located and writes an abnormal completion queue element into a completion queue, the sending module initiates a request to delete a cache chain corresponding to the abnormal work queue in which the abnormal work queue element is located to the on-chip work queue cache module, so as to release a cache occupied by the abnormal work queue.

[0126] In some optional implementation manners, the method further includes:

[0127] The sending module generates a non-first abnormal completion queue element corresponding to a work queue element for which no response is received after the sending module generates the abnormal work queue element based on a data retransmission request initiated by the sending module, and writes the non-first abnormal completion queue element into the completion queue.

[0128] The above method can be implemented by the remote direct memory access device provided in the above embodiments, and specific details can be referred to the description in the embodiments of the remote direct memory access device, which will not be described herein again.

[0129] It can be understood that the circuit structure, name and parameter described in the above embodiments are only examples. Those skilled in the art can also easily think of combinations and adjustments of the structural features of the above multiple embodiments according to the use needs, and the concept of the present application should not be limited to the specific details of the above examples.

[0130] Although the present application is described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A remote direct memory access device, characterized by, The application comprises: a sending module and a receiving module; wherein, the sending module acquires a current work queue element, and determines that the current work queue element is an abnormal work queue element after the current work queue element appears abnormal, and determines whether all work queue elements before the abnormal work queue element have received corresponding response; if all work queue elements before the abnormal work queue element have received corresponding response, and the receiving module no longer receives response, the sending module generates an abnormal completion queue element for the abnormal work queue element, and writes the abnormal completion queue element into a completion queue; after the receiving module receives response, if it is determined that the work queue element corresponding to the response appears abnormal, the work queue element appearing abnormal is taken as an abnormal work queue element, and abnormal information is written into a work queue context where the abnormal work queue element is located, and a data retransmission request of the abnormal work queue element is initiated; after the sending module reads the abnormal information from the work queue context, it stops sending a request data packet corresponding to the work queue element, and if it is determined that corresponding response has not been received for all work queue elements for which request data packets have been sent, it initiates a data retransmission request of work queue elements after the abnormal work queue element for which no response has been received.

2. The remote direct memory access device of claim 1, wherein, if the sending module determines that corresponding response has not been received for all work queue elements before the abnormal work queue element, it initiates a data retransmission request of the abnormal work queue element.

3. The remote direct memory access device of claim 1, wherein, if the sending module determines that corresponding response has been received for all work queue elements before the abnormal work queue element, and the receiving module can still receive response, it initiates a notification to the receiving module that it will no longer continue to receive response of the abnormal work queue element; after the receiving module receives the notification, it no longer receives response, and initiates a data retransmission request of the abnormal work queue element.

4. The remote direct memory access device of claim 1, wherein, The remote direct memory access device further comprises an on-chip work queue cache module; wherein, after the sending module writes the abnormal completion queue element into the completion queue, it initiates a request to the on-chip work queue cache module to delete a cache chain corresponding to an abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

5. The remote direct memory access device of claim 2 or 3, wherein, The remote direct memory access device further comprises a doorbell module and a scheduling module; wherein, the data retransmission request is a retransmission doorbell request; after the doorbell module receives the retransmission doorbell request, it sets an abnormal work queue where the abnormal work queue element corresponding to the retransmission doorbell request is located as a current to-be-scheduled work queue; when the scheduling module schedules the current to-be-scheduled work queue, it determines whether the abnormal work queue element is a work queue element for which the receiving module has not received response, if yes, it takes out the abnormal work queue element from the current to-be-scheduled work queue and pushes it to the sending module. The sending module obtains the abnormal work queue element, takes the abnormal work queue element as a current work queue element, and further determines whether the current work queue element is abnormal.

6. The remote direct memory access device of claim 1, wherein, If the sending module reads the abnormal information from the work queue context and has received a data retransmission request initiated by the receiving module, the data retransmission request corresponds to the same abnormal work queue element as a work queue element for which no response has been received, and the receiving module no longer continues to receive the response, an abnormal completion queue element is generated for the abnormal work queue element, and the abnormal completion queue element is written into a completion queue.

7. The remote direct memory access device of claim 6, wherein, After the sending module writes the abnormal completion queue element into the completion queue, a request is initiated to the on-chip work queue cache module to delete a cache chain corresponding to an abnormal work queue in which the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

8. The remote direct memory access device of claim 1, wherein, The sending module further writes a first abnormal completion report identifier into a context of an abnormal work queue in which the abnormal work queue element is located, generates a non-first abnormal completion queue element corresponding to a work queue element for which no response has been received after the abnormal work queue element based on a data retransmission request initiated by the sending module, and writes the non-first abnormal completion queue element into the completion queue.

9. A method of remote direct memory access, characterized by, Comprise: A sending module of a remote direct memory access device obtains a current work queue element, and determines that the current work queue element is abnormal, takes the current work queue element as an abnormal work queue element, and determines whether work queue elements before the abnormal work queue element have all received corresponding response responses; If the work queue elements before the abnormal work queue element have all received the corresponding response responses, and a receiving module of the remote direct memory access device no longer receives the response, the sending module generates an abnormal completion queue element for the abnormal work queue element, and writes the abnormal completion queue element into a completion queue; The method further comprises: After the receiving module receives the response, if it is determined that a work queue element corresponding to the response is abnormal, the work queue element that is abnormal is taken as an abnormal work queue element, abnormal information is written into a work queue context in which the abnormal work queue element is located, and a data retransmission request of the abnormal work queue element is initiated; After the sending module reads the abnormal information from the work queue context, the sending of a request data packet corresponding to the work queue element is stopped, and if it is determined that corresponding response responses have not been received for work queue elements for which data packets have been sent, a data retransmission request corresponding to work queue elements for which no response has been received after the abnormal work queue element is initiated.

10. The remote direct memory access method of claim 9, wherein, The method further comprises: If the sending module determines that the work queue elements before the abnormal work queue element have not all received the corresponding response responses, a data retransmission request corresponding to the abnormal work queue element is initiated.

11. The remote direct memory access method of claim 9, wherein, The method further comprises: If the sending module determines that all work queue elements before the abnormal work queue element have received corresponding response, and the receiving module can still receive the response, the sending module initiates a notification to the receiving module that the receiving module no longer continues to receive the response of the abnormal work queue element; After receiving the notification, the receiving module no longer receives the response, and initiates a data retransmission request of the abnormal work queue element.

12. The remote direct memory access method of claim 9, wherein, The method further comprises: After the sending module writes the abnormal completion queue element into the completion queue, the sending module initiates a request to the on-chip work queue cache module of the remote direct memory device to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

13. The remote direct memory access method of claim 10 or 11, wherein, The data retransmission request is a retransmission doorbell request; the method further comprises: After the doorbell module of the remote direct memory access device receives the retransmission doorbell request, the abnormal work queue where the abnormal work queue element corresponding to the retransmission doorbell request is located is set as the current work queue to be scheduled; When the scheduling module of the remote direct memory access device schedules the current work queue to be scheduled, it is determined whether the abnormal work queue element is a work queue element for which the receiving module has not received the response; if so, the abnormal work queue element is taken out from the current work queue to be scheduled and pushed to the sending module; After the sending module obtains the abnormal work queue element, the sending module takes the abnormal work queue element as the current work queue element and jumps to the step of determining whether all work queue elements before the abnormal work queue element have received corresponding responses.

14. The remote direct memory access method of claim 9, wherein, The method further comprises: If the sending module reads the abnormal information from the work queue context, and has received the data retransmission request initiated by the receiving module, and the abnormal work queue element corresponding to the data retransmission request is the same as the next work queue element for which the response is last received, and the receiving module no longer continues to receive the response, an abnormal completion queue element is generated for the abnormal work queue element, and the abnormal completion queue element is written into the completion queue.

15. The remote direct memory access method of claim 14, wherein, The method further comprises: After the sending module writes the first abnormal completed reporting identifier into the abnormal work queue where the abnormal work queue element is located, and writes the abnormal completion queue element into the completion queue, the sending module initiates a request to the on-chip work queue cache module to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

16. The remote direct memory access method according to claim 14 or 15, characterized in that, The method further comprises: Based on the data retransmission request initiated by the sending module, the sending module generates a non-first abnormal completion queue element corresponding to a work queue element for which no response has been received after the abnormal work queue element, and writes the non-first abnormal completion queue element into the completion queue.

Images