Remote direct memory access device and method

Abstract

The invention provides remote direct memory access equipment and a remote direct memory access method, and the remote direct memory access equipment comprises a sending module and a receiving module, the sending module obtains a current work queue element, takes the current work queue element as an abnormal work queue element after determining that the current work queue element is abnormal, and determines whether all work queue elements before the abnormal work queue element receive corresponding response responses or not; if all the work queue elements before the abnormal work queue element receive the corresponding response responses and the receiving module does not receive the response responses any more, the sending module generates an abnormal completion queue element for the abnormal work queue element and then writes the abnormal completion queue element into a completion queue. According to the method and the device, after the working queue is abnormal, the corresponding CQE is still written into the completion queue for the unsent WQE, and the sequence of the CQE can be ensured to be consistent with the sequence of the corresponding WQE.

Description

Technical Field

[0001] The present application belongs to the field of network communication technology, and in particular relates to a remote direct memory access device and method. Background Art

[0002] Under normal circumstances, the initiator of the HCA (Host Channel Adapter) obtains the WQE (Work Queue Element) from the work queue (SQ) and composes a request data packet and sends it out. When the initiator receives the ACK from the other end, it creates a corresponding CQE (Completion Queue Entry) and sends it to the corresponding completion queue (CQ). Figure 1 As shown; and WQEs in the same SQ are sent sequentially, so CQEs are also recycled sequentially; after the application layer obtains the corresponding CQE from the CQ, it can know the currently completed WQE, so that the memory resources corresponding to the completed WQE can be released. Summary of the invention

[0003] The purpose of the present application is to provide a remote direct memory access device and method, which aims to solve the problem that after an abnormality occurs in a work queue, the remaining WQEs in the work queue will not be sent out, so that the CQEs corresponding to these remaining WQEs cannot be generated according to the normal process, and the CQEs between the normally sent WQEs and the unsent WQEs cannot be guaranteed to be consistent with the corresponding WQE order.

[0004] According to a first aspect of the present application, a remote direct memory access device is provided, comprising: a sending module and a receiving module; wherein:

[0005] The sending module obtains the current work queue element, and after determining that the current work queue element is abnormal, takes the current work queue element as an abnormal work queue element, and determines whether all the work queue elements before the abnormal work queue element have received corresponding responses;

[0006] If all the work queue elements before the abnormal work queue element have received corresponding responses and the receiving module no longer receives responses, the sending module generates an abnormal completion queue element for the abnormal work queue element and writes the abnormal completion queue element into the completion queue.

[0007] In an optional implementation manner, if the sending module determines that all the work queue elements before the abnormal work queue element have not received corresponding acknowledgements, a data retransmission request corresponding to the abnormal work queue element is initiated.

[0008] In an optional implementation, if the sending module determines that the work queue elements before the abnormal work queue element have all received corresponding response responses, and the receiving module is still able to receive response responses, a notification is initiated to the receiving module that the response responses of the abnormal work queue elements will no longer be received;

[0009] After receiving the notification, the receiving module no longer receives a response and initiates a data retransmission request corresponding to the abnormal work queue element.

[0010] In an optional implementation, the remote direct memory access device further comprises: an on-chip work queue cache module; wherein,

[0011] After the sending module writes the abnormal completion queue element into the completion queue, the sending module initiates a request to the on-chip work queue cache module to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

[0012] In an optional implementation, the remote direct memory access device further comprises: a doorbell module and a scheduling module; wherein the data retransmission request is a retransmission doorbell request;

[0013] After receiving the retransmitted doorbell request, the doorbell module sets the abnormal work queue where the abnormal work queue element corresponding to the retransmitted doorbell request is located as the current work queue to be scheduled;

[0014] When the scheduling module schedules the current work queue to be scheduled, it determines whether the abnormal work queue element is a work queue element to which the receiving module has not currently received a response. If so, the abnormal work queue element is taken out from the current work queue to be scheduled and pushed to the sending module;

[0015] After the sending module obtains the abnormal work queue element, it uses the abnormal work queue element as the current work queue element, and further determines whether the current work queue element is abnormal.

[0016] In an optional implementation, after the receiving module receives the response, if it is determined that an abnormality occurs in the work queue element corresponding to the response, the work queue element with the abnormality is treated as an abnormal work queue element, and the abnormal information is written into the work queue context where the abnormal work queue element is located, and a data retransmission request for the abnormal work queue element is initiated;

[0017] After the sending module reads the exception information from the work queue context, it stops sending the request data packet corresponding to the work queue element, and if it is determined that the work queue elements that have sent out the request data packets have not all received corresponding response responses, it initiates a data retransmission request corresponding to the work queue elements that have not received any response responses after the abnormal work queue element.

[0018] In an optional implementation, if the sending module reads the exception information from the work queue context and has received the data retransmission request initiated by the receiving module, and the abnormal work queue element corresponding to the data retransmission request is the same as the work queue element that has not currently received a response, and the receiving module no longer continues to receive a response, then an abnormal completion queue element is generated for the abnormal work queue element, and the abnormal completion queue element is written into the completion queue.

[0019] In an optional implementation, after the sending module writes the abnormal completion queue element into the completion queue, it initiates a request to the on-chip work queue cache module to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

[0020] In an optional embodiment, the sending module also writes the first exception completed reporting identifier into the context of the abnormal work queue where the abnormal work queue element is located, and based on the data retransmission request initiated by the sending module, generates a non-first abnormal completion queue element corresponding to the work queue element that has not received a response after the abnormal work queue element, and writes the non-first abnormal completion queue element into the completion queue.

[0021] According to a second aspect of the present application, a remote direct memory access method is provided, comprising:

[0022] The sending module of the remote direct memory access device obtains the current work queue element, and after determining that the current work queue element is abnormal, takes the current work queue element as an abnormal work queue element, and determines whether all the work queue elements before the abnormal work queue element have received corresponding responses;

[0023] If all the work queue elements before the abnormal work queue element have received corresponding response responses, and the receiving module of the remote direct memory access device no longer receives response responses, the sending module generates an abnormal completion queue element for the abnormal work queue element and writes the abnormal completion queue element into the completion queue.

[0024] In an optional embodiment, the method further comprises:

[0025] If the sending module determines that all the work queue elements before the abnormal work queue element have not received corresponding responses, a data retransmission request corresponding to the abnormal work queue element is initiated.

[0026] In an optional embodiment, the method further comprises:

[0027] If the sending module determines that all the work queue elements before the abnormal work queue element have received corresponding response, and the receiving module can still receive response, then the receiving module is notified of not continuing to receive the response of the abnormal work queue element;

[0028] After receiving the notification, the receiving module no longer receives a response and initiates a data retransmission request corresponding to the abnormal work queue element.

[0029] In an optional embodiment, the method further comprises:

[0030] After the sending module writes the abnormal completion queue element into the completion queue, it initiates a request to the on-chip work queue cache module of the remote direct storage device to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

[0031] In an optional implementation manner, the data retransmission request is a retransmission doorbell request; and the method further includes:

[0032] After receiving the retransmitted doorbell request, the doorbell module of the remote direct memory access device sets the abnormal work queue where the abnormal work queue element corresponding to the retransmitted doorbell request is located as the current work queue to be scheduled;

[0033] When the scheduling module of the remote direct memory access device schedules the current work queue to be scheduled, it is determined whether the abnormal work queue element is a work queue element to which the receiving module has not currently received a response, and if so, the abnormal work queue element is taken out from the current work queue to be scheduled and pushed to the sending module;

[0034] After the sending module obtains the abnormal work queue element, it uses the abnormal work queue element as the current work queue element and jumps to the step of determining whether all the work queue elements before the abnormal work queue element have received corresponding response responses.

[0035] In an optional embodiment, the method further comprises:

[0036] After the receiving module receives the response, if it is determined that the work queue element corresponding to the response is abnormal, the work queue element with the abnormality is used as an abnormal work queue element, and the abnormal information is written into the work queue context where the abnormal work queue element is located, and a data retransmission request for the abnormal work queue element is initiated;

[0037] After the sending module reads the exception information from the work queue context, it stops sending the request data packet corresponding to the work queue element, and if it is determined that the work queue elements for which the data packets have been sent have not all received corresponding response responses, it initiates a data retransmission request corresponding to the work queue elements that have not received any response responses after the abnormal work queue element.

[0038] In an optional embodiment, the method further comprises:

[0039] If the sending module reads the exception information from the work queue context and has received the data retransmission request initiated by the receiving module, and the abnormal work queue element corresponding to the data retransmission request is the same as the next work queue element to which the response was last received, and the receiving module no longer continues to receive the response, then an abnormal completion queue element is generated for the abnormal work queue element, and the abnormal completion queue element is written into the completion queue.

[0040] In an optional embodiment, the method further comprises:

[0041] The sending module also writes the first exception completed reporting flag to the abnormal work queue where the abnormal work queue element is located, and after writing the abnormal completion queue element into the completion queue, initiates a request to the on-chip work queue cache module to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

[0042] In an optional embodiment, the method further comprises:

[0043] The sending module generates a non-first abnormal completion queue element corresponding to a work queue element that has not received a response after the abnormal work queue element based on the data retransmission request initiated by the sending module, and writes the non-first abnormal completion queue element into the completion queue.

[0044] Compared with the related art, the technical solution of the present application has at least the following advantages:

[0045] This application can write the corresponding CQE to the completion queue for the WQE that has not been sent out after an exception occurs in the work queue, and can also ensure that the order of the CQE is consistent with the order of the corresponding WQE.

[0046] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures and processes indicated in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. It is obvious that the drawings described below are certain embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 It is a schematic diagram of the work queue consumption process effect based on related technologies.

[0049] Figure 2 is a structural block diagram of a remote direct memory access device according to an exemplary embodiment of the present application.

[0050] Figure 3 It is a schematic diagram of the normal processing flow effect of the remote direct memory access device according to the exemplary embodiment of the present application.

[0051] Figure 4 It is a schematic diagram of the processing flow effect after the sending module of the remote direct memory access device finds an abnormality according to the exemplary embodiment of the present application.

[0052] Figure 5 It is a schematic diagram of the processing flow effect after the receiving module of the remote direct memory access device finds an abnormality according to the exemplary embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] When an exception occurs in the work queue, the application layer also needs to know which WQE memory resources can be released. Therefore, the underlying hardware needs to generate corresponding exception completion queue elements for all corresponding unfinished WQEs in the original order and write them into the completion queue to inform the application layer to release the corresponding WQE memory resources.

[0055] However, after the work queue is abnormal, the following two issues need to be considered when writing CQE to the completion queue: first, the remaining WQEs in the work queue will not be sent out, so the corresponding ACK will not be received, and the CQEs corresponding to the remaining WQEs cannot be written to the completion queue according to the normal process; second, the CQEs between the normally sent WQEs and the unsent WQEs need to meet the order preservation requirements, that is, the order of the CQEs must be consistent with the order of the corresponding WQEs. At present, no related technology has been found to solve the above two problems.

[0056] Based on this, the present application proposes a remote direct memory access device and method for solving the above two problems. That is, after an exception occurs in the work queue, the present application can still write the corresponding CQE to the completion queue for the WQE that has not been sent out, and can also ensure that the order of the CQE is consistent with the order of the corresponding WQE.

[0057] For example, Figure 2 As shown, a remote direct memory access device proposed in the present application includes: a sending module and a receiving module; wherein,

[0058] The sending module obtains the current work queue element, and after determining that the current work queue element is abnormal, takes the current work queue element as an abnormal work queue element, and determines whether all the work queue elements before the abnormal work queue element have received corresponding responses;

[0059] If all the work queue elements before the abnormal work queue element have received corresponding responses and the receiving module no longer receives responses, the sending module generates an abnormal completion queue element for the abnormal work queue element and writes the abnormal completion queue element into the completion queue.

[0060] For example, if there is no abnormality in the work queue, Figure 3 As shown, HCA _ After A's sending module obtains WQE, it generates the corresponding request data packet and sends it to HCA_B, and writes the write CQE information of the current WQE to the on-chip work queue cache module. HCA_B receives HCA_A's request data packet, verifies it, and sends it to HCA_B. _ A returns an ACK response, HCA _ A's receiving module receives the ACK, and after verification, reads the write CQE information of the response from the on-chip work queue cache module, generates a corresponding CQE based on the write CQE information, and writes the generated CQE into the CQ. At this point, the hardware WQE recovery process is completed.

[0061] In this process, after the sending module obtains the current work queue element, if the work queue where the current work queue element is located is found to be abnormal during the verification process, the abnormal starting WQE is the current work queue element. In order to ensure that the order of CQEs corresponding to the WQEs that have been sent is consistent with the sending order of WQEs, in this application, the current work queue element is used as an abnormal work queue element, and the sending module determines whether the work queue elements before the abnormal work queue element have all received the corresponding response based on the status identifier set by the receiving module. If all the corresponding response responses have been received, and the receiving module is in a state where the response response of the destination device is no longer received, the sending module generates an abnormal completion queue element (ERR CQE) for the abnormal work queue element and writes it to the completion queue. The subsequent application layer can read the ERR CQE from the completion queue to perform corresponding processing.

[0062] In some optional implementations, if the sending module determines that all the work queue elements before the abnormal work queue element have not received corresponding acknowledgements, a data retransmission request corresponding to the abnormal work queue element is initiated.

[0063] That is to say, if the sending module determines that the work queue elements before the abnormal work queue element have not all received the corresponding response, the sending module can initiate a data retransmission request, thereby re-executing the sending process of the request data packet of the abnormal work queue element. When re-executing the sending process of the request data packet, if all the work queue elements before the abnormal work queue element have received the corresponding response, and the receiving module no longer continues to receive the response, the sending module can generate an abnormal completion queue element (ERR CQE) for the abnormal work queue element as described above, and write it into the completion queue.

[0064] In some optional implementations, if the sending module determines that the work queue elements before the abnormal work queue element have all received corresponding response responses, and the receiving module is still able to receive response responses, a notification is initiated to the receiving module that the response responses of the abnormal work queue elements will no longer be received;

[0065] After receiving the notification, the receiving module no longer receives a response and initiates a data retransmission request corresponding to the abnormal work queue element.

[0066] As described above, if the sending module determines that the corresponding response responses of the working queue elements before the abnormal working queue element have been received, but the receiving module is still in a state of waiting to receive the response response, the sending module initiates a notification to the receiving module that it will no longer continue to receive the response response of the abnormal working queue, causing the receiving module to set its own state to a state where it no longer continues to receive the response response; and the receiving module initiates a data retransmission request corresponding to the abnormal working queue element.

[0067] The data retransmission request will cause the sending process of the abnormal work queue element to be re-executed. After the sending process flows to the sending module, the sending module can continue to determine whether the work queue elements before the abnormal work queue element have all received the corresponding response, and whether the receiving module is in the state of receiving the response. Since the data retransmission request is initiated by the receiving module, the work queue elements before the abnormal work queue element have all received the corresponding response, and the receiving module has set its own state to a state where it no longer receives the response. Therefore, as described above, the sending module can generate an abnormal completion queue element for the abnormal work queue element and write it into the completion queue.

[0068] In some optional implementations, the remote direct memory access device further includes: the remote direct memory access device further includes: an on-chip work queue cache module; wherein,

[0069] After the sending module writes the abnormal completion queue element into the completion queue, the sending module initiates a request to the on-chip work queue cache module to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

[0070] All work queues can be cached in the on-chip work queue cache module, and each work queue can include multiple work queue elements. After the sending module generates a corresponding request data packet for the current work queue element, the corresponding write CQE information can also be written into the on-chip work queue cache module, so that after the receiving module receives the corresponding response, it can read the corresponding write CQE information from the on-chip work queue cache module and generate a completion queue element, which is then written into the completion queue.

[0071] It can be understood that after an abnormal completion queue element is generated for an abnormal work queue element and written into the completion queue, other work queue elements in the abnormal work queue where the abnormal work queue element is located will no longer continue to be sent, and the corresponding information will no longer be recorded to the on-chip work queue cache module, so that the cache occupied by the abnormal work queue in the on-chip work queue cache module can be released, that is, the data link of the abnormal work queue can be deleted from the on-chip work queue cache module.

[0072] In some optional implementations, the remote direct memory access device further includes: a doorbell module and a scheduling module; wherein the data retransmission request is a retransmission doorbell request;

[0073] After receiving the retransmitted doorbell request, the doorbell module sets the abnormal work queue where the abnormal work queue element corresponding to the retransmitted doorbell request is located as the current work queue to be scheduled;

[0074] When the scheduling module schedules the current work queue to be scheduled, it determines whether the abnormal work queue element is a work queue element to which the receiving module has not currently received a response. If so, the abnormal work queue element is taken out from the current work queue to be scheduled and pushed to the sending module;

[0075] After the sending module obtains the abnormal work queue element, it uses the abnormal work queue element as the current work queue element, and further determines whether the current work queue element is abnormal.

[0076] Exemplarily, the doorbell module is used to receive a doorbell request, which includes a data retransmission request initiated by the sending module and the receiving module, and a doorbell request initiated by the application layer. In response to the doorbell request, the doorbell module sets the work queue where the work queue element corresponding to the doorbell request is located as the current work queue to be scheduled, thereby causing the scheduling module to schedule the work queue element corresponding to the doorbell request from the current work queue to be scheduled, and push the scheduled work queue element to the sending module, so that the sending module generates a corresponding request data packet and then sends it to the destination device.

[0077] When the doorbell request is a data retransmission request corresponding to an abnormal work queue element, the scheduling module schedules the abnormal work queue element. When scheduling the abnormal work queue element, the scheduling module will first determine whether the abnormal queue element is the same as the work queue element to which the receiving module has not currently received a response. If they are the same, the abnormal work queue element will be pushed to the sending module. If they are not the same, it means that the work queue element before the abnormal work queue element has not yet received a response, and it is necessary to wait until the work queue elements before the abnormal work queue element have all received a response before scheduling. To this end, the scheduling module will re-initiate the doorbell request to reschedule the abnormal work queue element until the abnormal queue element is the same as the work queue element to which the receiving module has not currently received a response.

[0078] In the sending process of the abnormal work queue element triggered by the data retransmission request, after the sending module obtains the abnormal work queue element, it will still be verified according to the normal process. During the verification, it will be found that the current work queue element is an abnormal work queue element, and then it will be judged again in the manner described above whether the work queue elements included in the previously issued request data have all been answered: a. If not all are answered, the sending module sends a doorbell request to the doorbell module, and also a data retransmission request; b. If all are answered, but the receiving module can continue to receive acknowledgment responses, the sending module sends a doorbell request to the receiving module to notify the receiving module to set its own state to a state where it no longer receives acknowledgment responses; after receiving the doorbell request, the receiving module will initiate a data retransmission request; c. If all are answered and the receiving module no longer receives acknowledgment responses, the sending module starts to generate ERR CQE and writes it to the completion queue.

[0079] Below through Figure 4 The exemplary embodiment shown describes the processing flow when the sending module finds that the work queue is abnormal.

[0080] like Figure 4 As shown, when sending a request data packet, the sending direction will check the corresponding WQE, such as protocol check (whether it is illegal), QP status check, interface ERR signal check, etc. If the check fails, QP ERROR will be set. The setting of QP ERROR indicates that the work queue where the current work queue element is located has an exception:

[0081] 1. Assume that the starting work queue element of the sending direction Tx_error is WQE3, SSN=3 (indicating that the last request data packet sent by the sending module is the request data packet corresponding to WQE3), and WQE3 is an abnormal work queue element; if the receiving module does not receive the response of WQE2, it means that the request data packet sent normally before the exception occurs has not completed the response. The sending module needs to press the retransmission doorbell and record the starting position of the retransmission WQE3 in the QP context (QPC) to initiate a data retransmission request for WQE3.

[0082] 2. When the doorbell module receives the retransmitted doorbell from the sending module, it sets the request of the work queue where WQE3 is located to be valid, and the work queue is scheduled out of the queue, that is, the work queue is determined as the current work queue to be scheduled.

[0083] 3. After receiving the request to schedule the current work queue to be scheduled, the scheduling module determines whether the abnormal work queue element recorded by the sending module is the WQE to which the receiving module currently receives no response. If so, the WQE3 in the current work queue to be scheduled can be scheduled. Otherwise, the request to schedule the current work queue to be scheduled is returned to the doorbell module to wait for the next scheduling.

[0084] 4. In the case of QP abnormality, the sending module receives a data retransmission request, that is, the data retransmission request sent by the sending module in the above step 1 passes through the doorbell module and the scheduling module, and is finally pushed to the sending module. If the WQE for which the receiving module has not currently received a response is equal to the WQE for the current request for retransmission, that is, WEQ3, and it is determined that the receiving module is still in a state where it can receive a response; then the receiving module is notified to no longer continue to receive the response from the work queue where WQE3 is located, and the receiving module presses the doorbell request, that is, the receiving module initiates a data retransmission request, thereby returning to the retransmission process of WQE3. This is to ensure that the receiving module does not operate the on-chip work queue cache module, and the sending module then deletes the data link of the work queue where the abnormal work queue element is located on the on-chip work queue cache module to release the cache resources occupied by it.

[0085] 5. After the sending module receives the data retransmission request initiated by the receiving module again, the WQE for which the receiving module has not received a response is equal to the current request WQE, that is, WQE3; and the receiving module is no longer in a state of receiving a response; then an ERR CQE is generated for the current WQE, that is, WQE3, and written into CQ (completion queue), and a chain deletion request for the current QP is initiated to the on-chip work queue cache module; and the flush flag (the first exception has been completed reporting flag) is recorded in QPC. After the remaining WQEs of the QP are received subsequently, after the flush flag is read, a flush CQE (non-first exception completion queue element) can be directly generated for these remaining WQEs and written into the completion queue, without sending these remaining WQEs again, and no longer recording the relevant information of these WQEs to the on-chip work queue cache module.

[0086] In some optional implementations, after the receiving module receives the response, if it is determined that an abnormality occurs in the work queue element corresponding to the response, the work queue element with the abnormality is used as an abnormal work queue element, and the abnormal information is written into the work queue context where the abnormal work queue element is located, and a data retransmission request for the abnormal work queue element is initiated;

[0087] After the sending module reads the exception information from the work queue context, it stops sending the request data packet corresponding to the work queue element, and if it is determined that the work queue elements that have sent out the request data packets have not all received corresponding response responses, it initiates a data retransmission request corresponding to the work queue elements that have not received any response responses after the abnormal work queue element.

[0088] Exemplarily, after receiving the acknowledgement response, the receiving module performs relevant verification. If it is found during the verification that the work queue element corresponding to the current acknowledgement response is abnormal, the work queue element is treated as an abnormal work queue element, and the receiving module initiates a data retransmission request.

[0089] Before the receiving module initiates a data retransmission request, the abnormal information of the abnormal working queue element, such as the abnormal cause, is written in the context of the abnormal working queue where the abnormal working queue element is located. After the receiving module initiates a data retransmission request, after the retransmission process of the abnormal working queue element reaches the sending module, the sending module is still used to generate the corresponding abnormal completion queue element and write it into the queue. It can be understood that the premise for generating the corresponding abnormal completion queue element is that the working queue elements that have sent out the request data packet have all received the corresponding response, and the receiving module is in a state of no longer receiving the response.

[0090] After the sending module reads the abnormal information written by the receiving module in the work queue context, it stops sending the request data packets corresponding to the subsequent work queue elements in the work queue, and if the work queue elements that have sent out the request data packets have not received all the corresponding response responses, a data retransmission request will be initiated, and the data retransmission request is a data retransmission request for the work queue elements that have sent the request data packets after the abnormal work queue elements. In this way, the sending module can generate an abnormal completion queue element (ERR CQE) for the abnormal work queue element and generate a non-first abnormal completion queue element (flush CQE) for the subsequent work queue elements of the abnormal work queue element under the premise of meeting the requirements for generating the corresponding abnormal completion queue element.

[0091] In some optional implementations, if the sending module reads the exception information from the work queue context and has received the data retransmission request initiated by the receiving module, and the abnormal work queue element corresponding to the data retransmission request is the same as the work queue element that has not currently received a response, and the receiving module no longer continues to receive a response, then an abnormal completion queue element is generated for the abnormal work queue element, and the abnormal completion queue element is written into the completion queue.

[0092] Exemplarily, if the data retransmission request initiated by the receiving module is returned to the sending module, and the sending module reads the exception information written to the work queue context of the receiving module, then if the work queue element of the receiving module that has not yet completed the response is the same as the abnormal work queue element corresponding to the data retransmission request, and if the state of the receiving module has been set to no longer continue to receive response, the sending module generates an abnormal completion queue element for the abnormal work queue element and writes it into the completion queue.

[0093] In some optional implementations, after the sending module writes the abnormal completion queue element into the completion queue, it initiates a request to the on-chip work queue cache module to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

[0094] Exemplarily, after the sending module completes writing the abnormal completion queue element generated for the abnormal work queue element, it requests the on-chip work queue cache module to release the cache resources occupied by the abnormal work queue.

[0095] In some optional implementations, the sending module also writes the first exception completed reporting identifier into the context of the abnormal work queue where the abnormal work queue element is located, and based on the data retransmission request initiated by the sending module, generates a non-first abnormal completion queue element corresponding to the work queue element that has not received a response after the abnormal work queue element, and writes the non-first abnormal completion queue element into the completion queue.

[0096] Exemplarily, after the receiving module finds that the work queue is abnormal, the starting work queue element in the work queue where the abnormality occurs is used as the abnormal work queue element, and after the premise described above is met, the sending module generates a corresponding abnormal completion queue element for the abnormal work queue element. Afterwards, in order to generate a corresponding non-first abnormal completion queue element for the subsequent work queue elements that have completed the sending of the request data packet after the abnormal work queue element, the sending module will write a flush flag (the first abnormal completed reporting flag) in the abnormal queue context, so that after receiving the data retransmission request for the candidate work queue element, the flush CQE (non-first abnormal completion queue element) corresponding to the candidate work queue element is directly generated.

[0097] Below through Figure 5 The exemplary embodiment shown describes the processing flow when the receiving module finds that the work queue is abnormal.

[0098] like Figure 5 As shown, after the receiving direction receives the response, it performs protocol verification on the received response data packet. If an abnormality is found, QP ERROR will be set. The setting of QP ERROR indicates that an abnormality has occurred in the work queue where the work queue element that currently receives the response is located:

[0099] 1. If the first abnormal work queue element in the abnormal work queue is WQE3, the receiving module records the error cause of the current WQE, that is, WQE3, in QPC; and presses the retransmission doorbell, that is, initiates a data retransmission request, so that the sending module generates ERR CQE (abnormal completion queue element); WQE4 to WQE6 included in the sent request data are abnormal work queue elements after the first abnormal work queue element, and data can be retransmitted, and finally the sending module generates the corresponding flushCQE.

[0100] 2. When the sending module finds an abnormality, it stops sending subsequent WQEs. Assuming that the sending module finds an abnormality when sending WQE7, WQE7 and subsequent WQEs will no longer be sent. WQE7 is greater than WQE3, which has not received a response. If it is determined that the WQE that has sent the request data packet has not been fully responded to, the retransmission doorbell is pressed, that is, the retransmission doorbell is sent to initiate a data retransmission request.

[0101] 3. The receiving module finds that QP ERROR is set, and receives a request to retransmit WQE3 initiated by the sending module. If it is determined that the WQE for which the receiving module has not yet received a completion response is equal to the request WQE in the data retransmission request, and the receiving module is in a state where it can no longer receive a response, the sending module can generate an ERRCQE for the current WQE, that is, WQE3, and write it into CQ. At the same time, it initiates a chain deletion request for the current QP to the on-chip QP cache module to delete the cache chain of the work queue where WQE3 is located; the sending module also records the flush flag in QPC, and subsequently receives the remaining WQEs of the QP (such as Figure 5 After the WQE4 to WQE6 in the above example are detected, the flush flag can be directly generated for the remaining WQEs and written into the CQ. That is, WQE3 is generated by the sending module as an ERR CQE, and the WQEs after WQE3 are generated as flush CQEs by the sending module.

[0102] 4. The priority of the data retransmission request initiated by the timeout and receiving module is higher than the data retransmission request initiated by the sending module.

[0103] The above is the remote direct memory access device proposed in this application, which has the following advantages compared with the related technologies:

[0104] This application can write the corresponding CQE to the completion queue for the WQE that has not been sent out after an exception occurs in the work queue, and can also ensure that the order of the CQE is consistent with the order of the corresponding WQE.

[0105] Accordingly, the present application also exemplarily provides a remote direct memory access method, comprising:

[0106] The sending module of the remote direct memory access device obtains the current work queue element, and after determining that the current work queue element is abnormal, takes the current work queue element as an abnormal work queue element, and determines whether all the work queue elements before the abnormal work queue element have received corresponding responses;

[0107] If all the work queue elements before the abnormal work queue element have received corresponding response responses, and the receiving module of the remote direct memory access device no longer receives response responses, the sending module generates an abnormal completion queue element for the abnormal work queue element and writes the abnormal completion queue element into the completion queue.

[0108] In some optional implementations, the method further includes:

[0109] If the sending module determines that all the work queue elements before the abnormal work queue element have not received corresponding responses, a data retransmission request corresponding to the abnormal work queue element is initiated.

[0110] In some optional implementations, the method further includes:

[0111] If the sending module determines that all the work queue elements before the abnormal work queue element have received corresponding response, and the receiving module can still receive response, then the receiving module is notified of not continuing to receive the response of the abnormal work queue element;

[0112] After receiving the notification, the receiving module no longer receives a response and initiates a data retransmission request corresponding to the abnormal work queue element.

[0113] In some optional implementations, the method further includes:

[0114] After the sending module writes the abnormal completion queue element into the completion queue, it initiates a request to the on-chip work queue cache module of the remote direct storage device to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

[0115] In some optional implementations, the data retransmission request is a retransmission doorbell request; and the method further includes:

[0116] After receiving the retransmitted doorbell request, the doorbell module of the remote direct memory access device sets the abnormal work queue where the abnormal work queue element corresponding to the retransmitted doorbell request is located as the current work queue to be scheduled;

[0117] When the scheduling module of the remote direct memory access device schedules the current work queue to be scheduled, it is determined whether the abnormal work queue element is a work queue element to which the receiving module has not currently received a response, and if so, the abnormal work queue element is taken out from the current work queue to be scheduled and pushed to the sending module;

[0118] After the sending module obtains the abnormal work queue element, it uses the abnormal work queue element as the current work queue element and jumps to the step of determining whether all the work queue elements before the abnormal work queue element have received corresponding response responses.

[0119] In some optional implementations, the method further includes:

[0120] After the receiving module receives the response, if it is determined that the work queue element corresponding to the response is abnormal, the work queue element with the abnormality is used as an abnormal work queue element, and the abnormal information is written into the work queue context where the abnormal work queue element is located, and a data retransmission request for the abnormal work queue element is initiated;

[0121] After the sending module reads the exception information from the work queue context, it stops sending the request data packet corresponding to the work queue element, and if it is determined that the work queue elements for which the data packets have been sent have not all received corresponding response responses, it initiates a data retransmission request corresponding to the work queue elements that have not received any response responses after the abnormal work queue element.

[0122] In some optional implementations, the method further includes:

[0123] If the sending module reads the exception information from the work queue context and has received the data retransmission request initiated by the receiving module, and the abnormal work queue element corresponding to the data retransmission request is the same as the next work queue element to which the response was last received, and the receiving module no longer continues to receive the response, then an abnormal completion queue element is generated for the abnormal work queue element, and the abnormal completion queue element is written into the completion queue.

[0124] In some optional implementations, the method further includes:

[0125] The sending module also writes the first exception completed reporting flag to the abnormal work queue where the abnormal work queue element is located, and after writing the abnormal completion queue element into the completion queue, initiates a request to the on-chip work queue cache module to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

[0126] In some optional implementations, the method further includes:

[0127] The sending module generates a non-first abnormal completion queue element corresponding to a work queue element that has not received a response after the abnormal work queue element based on the data retransmission request initiated by the sending module, and writes the non-first abnormal completion queue element into the completion queue.

[0128] The above method can be implemented by the remote direct memory access device provided in the above embodiment. For specific details, please refer to the description in the embodiment of the above remote direct memory access device, which will not be repeated here.

[0129] It is understood that the circuit structures, names and parameters described in the above embodiments are only examples. Those skilled in the art can also easily combine and adjust the structural features of the above embodiments according to the use requirements, and should not limit the concept of the present application to the specific details of the above examples.

[0130] Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that he or she may still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A remote direct memory access device, characterized in that: include: A sending module and a receiving module; wherein, The sending module obtains the current work queue element, and after determining that the current work queue element is abnormal, takes the current work queue element as an abnormal work queue element, and determines whether all the work queue elements before the abnormal work queue element have received corresponding responses; If all the work queue elements before the abnormal work queue element have received corresponding responses and the receiving module no longer receives responses, the sending module generates an abnormal completion queue element for the abnormal work queue element and writes the abnormal completion queue element into the completion queue.

2. The remote direct memory access device according to claim 1, characterized in that: If the sending module determines that all the work queue elements before the abnormal work queue element have not received corresponding acknowledgements, a data retransmission request corresponding to the abnormal work queue element is initiated.

3. The remote direct memory access device according to claim 1, characterized in that: If the sending module determines that all the work queue elements before the abnormal work queue element have received corresponding response, and the receiving module can still receive response, then the receiving module is notified of not continuing to receive the response of the abnormal work queue element; After receiving the notification, the receiving module no longer receives a response and initiates a data retransmission request corresponding to the abnormal work queue element.

4. The remote direct memory access device according to claim 1, characterized in that: The remote direct memory access device further comprises: an on-chip work queue cache module; wherein, After the sending module writes the abnormal completion queue element into the completion queue, the sending module initiates a request to the on-chip work queue cache module to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

5. The remote direct memory access device according to claim 2 or 3, characterized in that: The remote direct memory access device further comprises: a doorbell module and a scheduling module; wherein the data retransmission request is a retransmission doorbell request; After receiving the retransmitted doorbell request, the doorbell module sets the abnormal work queue where the abnormal work queue element corresponding to the retransmitted doorbell request is located as the current work queue to be scheduled; When the scheduling module schedules the current work queue to be scheduled, it determines whether the abnormal work queue element is a work queue element to which the receiving module has not currently received a response. If so, the abnormal work queue element is taken out from the current work queue to be scheduled and pushed to the sending module; After the sending module obtains the abnormal work queue element, it uses the abnormal work queue element as the current work queue element, and further determines whether the current work queue element is abnormal.

6. The remote direct memory access device according to any one of claims 1 to 4, characterized in that: After the receiving module receives the response, if it is determined that the work queue element corresponding to the response is abnormal, the work queue element with the abnormality is used as an abnormal work queue element, and the abnormal information is written into the work queue context where the abnormal work queue element is located, and a data retransmission request for the abnormal work queue element is initiated; After the sending module reads the exception information from the work queue context, it stops sending the request data packet corresponding to the work queue element, and if it is determined that the work queue elements that have sent out the request data packets have not all received corresponding response responses, it initiates a data retransmission request corresponding to the work queue elements that have not received any response responses after the abnormal work queue element.

7. The remote direct memory access device according to claim 6, characterized in that: If the sending module reads the exception information from the work queue context and has received the data retransmission request initiated by the receiving module, and the abnormal work queue element corresponding to the data retransmission request is the same as the work queue element that has not currently received a response, and the receiving module no longer continues to receive a response, then an abnormal completion queue element is generated for the abnormal work queue element, and the abnormal completion queue element is written into the completion queue.

8. The remote direct memory access device according to claim 7, characterized in that: After the sending module writes the abnormal completion queue element into the completion queue, it initiates a request to the on-chip work queue cache module to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

9. The remote direct memory access device according to claim 6, characterized in that: The sending module also writes the first exception completed reporting identifier into the context of the abnormal work queue where the abnormal work queue element is located, and based on the data retransmission request initiated by the sending module, generates a non-first abnormal completion queue element corresponding to the work queue element that has not received a response after the abnormal work queue element, and writes the non-first abnormal completion queue element into the completion queue.

10. A remote direct memory access method, characterized in that: include: The sending module of the remote direct memory access device obtains the current work queue element, and after determining that the current work queue element is abnormal, takes the current work queue element as an abnormal work queue element, and determines whether all the work queue elements before the abnormal work queue element have received corresponding responses; If all the work queue elements before the abnormal work queue element have received corresponding response responses, and the receiving module of the remote direct memory access device no longer receives response responses, the sending module generates an abnormal completion queue element for the abnormal work queue element and writes the abnormal completion queue element into the completion queue.

11. The remote direct memory access method according to claim 10, characterized in that: The method further comprises: If the sending module determines that all the work queue elements before the abnormal work queue element have not received corresponding acknowledgements, a data retransmission request corresponding to the abnormal work queue element is initiated.

12. The remote direct memory access method according to claim 10, characterized in that: The method further comprises: If the sending module determines that all the work queue elements before the abnormal work queue element have received corresponding response, and the receiving module can still receive response, then the receiving module is notified of not continuing to receive the response of the abnormal work queue element; After receiving the notification, the receiving module no longer receives a response and initiates a data retransmission request corresponding to the abnormal work queue element.

13. The remote direct memory access method according to claim 10, characterized in that: The method further comprises: After the sending module writes the abnormal completion queue element into the completion queue, it initiates a request to the on-chip work queue cache module of the remote direct storage device to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

14. The remote direct memory access method according to claim 11 or 12, characterized in that: The data retransmission request is a retransmission doorbell request; the method further includes: After receiving the retransmitted doorbell request, the doorbell module of the remote direct memory access device sets the abnormal work queue where the abnormal work queue element corresponding to the retransmitted doorbell request is located as the current work queue to be scheduled; When the scheduling module of the remote direct memory access device schedules the current work queue to be scheduled, it is determined whether the abnormal work queue element is a work queue element to which the receiving module has not currently received a response, and if so, the abnormal work queue element is taken out from the current work queue to be scheduled and pushed to the sending module; After the sending module obtains the abnormal work queue element, it uses the abnormal work queue element as the current work queue element and jumps to the step of determining whether all the work queue elements before the abnormal work queue element have received corresponding responses.

15. The remote direct memory access method according to any one of claims 10 to 13, characterized in that: The method further comprises: After the receiving module receives the response, if it is determined that the work queue element corresponding to the response is abnormal, the work queue element with the abnormality is used as an abnormal work queue element, and the abnormal information is written into the work queue context where the abnormal work queue element is located, and a data retransmission request for the abnormal work queue element is initiated; After the sending module reads the exception information from the work queue context, it stops sending the request data packet corresponding to the work queue element, and if it is determined that the work queue elements for which the data packets have been sent have not all received corresponding response responses, it initiates a data retransmission request corresponding to the work queue elements that have not received any response responses after the abnormal work queue element.

16. The remote direct memory access method according to claim 15, characterized in that: The method further comprises: If the sending module reads the exception information from the work queue context and has received the data retransmission request initiated by the receiving module, and the abnormal work queue element corresponding to the data retransmission request is the same as the next work queue element to which the response was last received, and the receiving module no longer continues to receive the response, then an abnormal completion queue element is generated for the abnormal work queue element, and the abnormal completion queue element is written into the completion queue.

17. The remote direct memory access method according to claim 16, characterized in that: The method further comprises: The sending module also writes the first exception completed reporting flag to the abnormal work queue where the abnormal work queue element is located, and after writing the abnormal completion queue element into the completion queue, initiates a request to the on-chip work queue cache module to delete the cache chain corresponding to the abnormal work queue where the abnormal work queue element is located, so as to release the cache occupied by the abnormal work queue.

18. The remote direct memory access method according to claim 16 or 17, characterized in that: The method further comprises: The sending module generates a non-first abnormal completion queue element corresponding to a work queue element that has not received a response after the abnormal work queue element based on the data retransmission request initiated by the sending module, and writes the non-first abnormal completion queue element into the completion queue.

Images