A message processing system

Abstract

In the message processing system provided by the present invention, the request message verification module verifies the received request message and maintains QP status information, and filters out the request messages that pass the verification; the RQE processing verification module verifies the RQE and maintains QP status information, and determines whether the RQE space is sufficient to allocate the request message; the request message payload upload module writes the request message payload to the host according to the remote address of the request message or the RQE address and maintains QP status information; the completion verification module determines and notifies the request message processing method according to the above QP status information to maintain the synchronization of QP status information, and the response construction module constructs the response message. Through the QP status information maintenance and synchronization mechanism, the processing system can further decouple each processing operation, avoid similar head-of-line blocking problems in each module, and can realize the full pipelining processing of the RDMA network card for the request message, improving the processing performance of the response end of the RDMA network card.

Description

Technical Field

[0001] The present invention relates to the technical field of remote direct memory access, and particularly to a message processing system. Background Art

[0002] In recent years, with the rapid development of technologies such as artificial intelligence and cloud computing, RoCEv2 (RDMA over Converged Ethernet) technology has been widely applied in data centers, and numerous domestic and foreign manufacturers have thus launched their respective RDMA-related IPs and products. However, on the one hand, although there is already a series of open-source supports for current RDMA-related software, such as OFED (OpenFabrics Enterprise Distribution), the hardware implementation architectures of major manufacturers are still not publicly available; on the other hand, for the RDMA network cards deployed in data centers, the link bandwidth they support is mostly in the order of hundreds of Gbps. For the FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) for implementing the RDMA network card, precise pipelining design is required to meet such data processing requirements of this order of magnitude. Taking the processing flow of the Send request message of the RC (Reliable Connection) service type by the response end of the RDMA network card as an example, based on the requirements of the IB (InfiniBand) protocol, its processing includes at least a series of information validations such as Opcode and PSN (Packet Sequence Number), obtaining RQE (Receive Queue Element) from the host, cumulative detection of the message length, sending the message data to the host, generating CQE (Completion Queue Element) and sending it to the host, etc. Moreover, the message processing of the same QP may have problems such as head-of-line blocking, which greatly increases the coupling degree between each step. For example, only after the previous message of the QP (Queue Pair) completes updating the ePSN (Expected Packet Sequence Number), the next message can use this ePSN to decide its processing method, and the maximum processing delay of each message often requires hundreds of clock cycles. Obviously, a simple message processing mechanism is difficult to meet the bandwidth requirements of the order of hundreds of Gbps.

[0003] Therefore, it is an urgent problem for those skilled in the art to provide a message processing system to solve the above problems. Summary of the Invention

[0004] The object of the present invention is to provide a message processing system, which has a simple structure, is safe, effective, reliable and easy to operate, and realizes hardware pipelining processing to improve the processing performance of the response end of the RDMA network card on the premise of meeting the requirements of the IB protocol.

[0005] Based on the above object, the technical solution provided by the present invention is as follows:

[0006] A message processing system, comprising:

[0007] A request message verification module, configured to verify the received request message, maintain the first QP status information, and screen out the request messages that pass the verification;

[0008] An RQE processing verification module, configured to verify the RQE obtained from the host, maintain the second QP status information, and verify whether the space of the RQE is sufficient to be allocated for use by the request message;

[0009] A request message payload upload module, configured to write the request message payload into the host according to the remote address in the request message or the address in the RQE, and maintain the third QP status information;

[0010] A completion verification module, configured to determine the request message processing method according to the first QP status information, the second QP status information, and the third QP status information, and notify the request message processing method to maintain the synchronization of the first QP status information, the second QP status information, and the third QP status information;

[0011] A response message construction module, configured to construct a response message according to the request message processing method.

[0012] Preferably, the request message verification module includes: a sending verification module, a rationality verification module, a first screening module, and a first QP status module;

[0013] The sending verification module is configured to verify whether the received request message is the request message of the message processing system;

[0014] The rationality verification module is configured to verify whether the request message is reasonable;

[0015] The first screening module is configured to screen out the request messages for which the judgment results of the sending verification module and the rationality verification module are both yes;

[0016] The first QP status module is configured to mark the request message as abnormal and update the first QP status information when any one of the judgment results of the sending verification module and the rationality verification module is no.

[0017] Preferably, when the QP type in the first QP status information is the RC type, the first QP status module is further configured to process the request message according to the RC type QP hardware status;

[0018] The first QP status module includes: a first status switching sub-module;

[0019] Specifically, the first status switching sub-module is configured to switch to the normal state when the host creates or restarts the QP;

[0020] If the judgment result of the rationality verification module is negative, it switches to the wait_send_nack state;

[0021] If the completion verification module notifies that the request message is a type B error defined by the IB protocol, it switches to the wait_epsn state;

[0022] If the request message triggers any one of a type A or type C error defined by the IB protocol, a host configuration QP error, or the completion verification module notifies a type A or type C error defined by the IB protocol, it switches to the error state.

[0023] Preferably, the first QP status module is further configured to

[0024] When in the error state, call the network card to directly and silently discard all the request messages;

[0025] When in the normal state, call the network card to discard the request messages for error retransmission, and mark and process the remaining request messages according to the judgment result of the rationality verification module according to the IB protocol;

[0026] When in the wait_send_nack state, call the network card to process the request messages for correct retransmission and discard the remaining request messages;

[0027] When in the wait_epsn state, call the network card to process the request messages for which the judgment result of the rationality verification module is positive or for correct retransmission, and discard the remaining request messages. Among them, the processing method for the request messages for which the judgment result of the rationality verification module is positive is the same as the processing method when in the normal state.

[0028] Preferably, the RQE processing verification module includes: a cache module, an RQE status verification module, an RQE space verification module, a second screening module, and a second QP status module;

[0029] The cache module is configured to obtain the RQEs of the active QPs from the host and cache them into the network card by QP;

[0030] The RQE status verification module is used to verify whether the RQE is correct. If not, it is marked as a deformed RQE;

[0031] The RQE space verification module is used to check whether there is an RQE caching the corresponding QP in the network card according to the request message. If so, the RQE caching the corresponding QP is allocated to the request message. If not, an RNR error is triggered and the request message is marked;

[0032] The second screening module is used to screen the request messages for which the judgment results of both the RQE status verification module and the RQE space verification module are yes;

[0033] The second QP status module is used to mark the request message as abnormal and update the second QP status information when any one of the judgment results of the RQE status verification module and the RQE space verification module is no.

[0034] Preferably, the second QP status module is further used to process the request message according to the QP hardware status of the RC type when the QP type in the second QP status information is the RC type;

[0035] The second QP status module includes: a second status switching sub-module;

[0036] The second status switching sub-module is specifically used to switch to the normal state when the host creates or restarts the QP;

[0037] If the judgment result of the RQE space verification module is no or the request message has been marked as a type B error defined by the IB protocol, it switches to the wait_send_nack state;

[0038] If the verification module notifies that the request message is a type B error defined by the IB protocol, it switches to the wait_epsn state;

[0039] If the request message has been marked as a type A or type C error defined by the IB protocol or the host configures the QP incorrectly or the verification module notifies any one of the type A or type C errors defined by the IB protocol, it switches to the error state.

[0040] Preferably, the second QP status module is further used for,

[0041] When in the error state, it calls the network card to directly and silently discard all the request messages;

[0042] When in the wait_send_nack state, call the network card to process the correct retransmission of the said request message and discard the remaining said request messages;

[0043] When in the normal state or the wait_epsn state, if the said request message triggers an RNR error and the said request message has been marked as a Class B error with the judgment result of the rationality verification module being no, then mark the said request message as an RNR error again.

[0044] Preferably, the RQE space verification module further includes: a double verification sub-module;

[0045] The double verification sub-module is used to verify whether the RQE is deformed and whether the remaining host memory space corresponding to the RQE supports the writing of the load data of the current said request message after the RQE corresponding to the QP already exists in the network card;

[0046] If the RQE is marked as deformed, then mark the said request message as a deformed RQE error;

[0047] If the remaining host memory space corresponding to the RQE does not support the writing of the load data of the current said request message, then mark the said request message as an invalid request error.

[0048] Preferably, the second QP state module further includes: a third state switching sub-module;

[0049] The third state switching sub-module is specifically used to switch to the normal state when the host creates or restarts the QP;

[0050] If the said request message has been marked as an error of Class A or Class C defined by the IB protocol or a host configuration QP error or the completion verification module notifies an error of Class A or Class C defined by the IB protocol or the said request message is marked as any one of the deformed RQE error or the invalid request error, then switch to the error state.

[0051] Preferably, the request message load upload module includes: a load upload module, a load data writing module, and a third QP state module;

[0052] The load upload module is used to judge whether the said request message needs to perform load upload;

[0053] The load data writing module is used to, when the judgment result of the load upload module is yes, obtain the corresponding physical page address from the host according to the remote address in the said request message or the address in the RQE, write the request message load into the host memory according to the physical page address, and construct a request message descriptor;

[0054] The third QP status module is configured to process the request message according to the QP hardware status of the RC type when the QP type in the third QP status information is of the RC type.

[0055] Preferably, the third QP status module includes: a fourth status switching sub-module;

[0056] The fourth status switching sub-module is specifically configured to switch to the normal state when the host creates or restarts the QP;

[0057] If the request message has been marked as an error of type A or C defined by the IB protocol, or the host configures the QP incorrectly, or the completion verification module notifies an error of type A or C defined by the IB protocol, then it switches to the error state.

[0058] Preferably, the completion verification module includes: a request message processing method determination module and a notification module;

[0059] The request message processing method determination module is configured to determine the request message processing method according to the first QP status information, the second QP status information, and the third QP status information included in the request message descriptor;

[0060] The notification module is configured to send the message information in the request message descriptor to the request message verification module, the RQE processing verification module, and the request message payload upload module respectively to update the corresponding QP status information.

[0061] Preferably, the response message construction module is specifically configured to generate a response message descriptor according to the request message descriptor and construct a response message according to the response message descriptor;

[0062] If the response message is a read response message, the response message construction module is further specifically configured to construct the response message according to the response message descriptor, the remote address in the request message, and the physical page address.

[0063] The message processing system provided by the present invention verifies the received request message through the request message verification module and maintains the QP status information, and filters out the request messages that pass the verification; verifies the RQE through the RQE processing verification module and maintains the QP status information to determine whether the RQE space is sufficient to allocate the request message; through the request message payload upload module, according to the remote address of the request message or the RQE address, writes the request message payload to the host and maintains the QP status information; determines and announces the request message processing method according to the QP status information of the above modules through the completion verification module to maintain the synchronization of the QP status information, and constructs the response message through the response construction module. Through the QP status information maintenance and synchronization mechanism, each processing operation can be further decoupled, avoiding similar head blocking problems caused by each module for implementing exception handling, and realizing the full pipelining processing of the RDMA network card for the request message, improving the processing performance of the RDMA network card response end. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0065] Figure 1 It is a schematic structural diagram of a message processing system provided by an embodiment of the present invention;

[0066] Figure 2 It is a schematic structural diagram of the request message verification module provided by an embodiment of the present invention;

[0067] Figure 3 It is a schematic structural diagram of the RQE processing verification module provided by an embodiment of the present invention;

[0068] Figure 4 It is a schematic structural diagram of the request message payload upload module provided by an embodiment of the present invention;

[0069] Figure 5 It is a schematic structural diagram of the completion verification module provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0070] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0071] The embodiments of the present invention are written in a progressive manner.

[0072] The embodiments of the present invention provide a message processing system, mainly solving the technical problem in the prior art that the processing performance of the RDMA network card response end is reduced due to similar head blocking problems caused by each module implementing exception handling.

[0073] As Figure 1 shown, a message processing system includes:

[0074] A request message verification module, used to verify the received request message, maintain the first QP status information, and filter out the request messages that pass the verification;

[0075] An RQE processing verification module, used to verify the RQE obtained from the host, maintain the second QP status information, and verify whether the space of the RQE is sufficient to be allocated for the use of the request message;

[0076] A request message payload upload module, used to write the request message payload to the host according to the remote address in the request message or the address in the RQE, and maintain the third QP status information;

[0077] A completion verification module, used to determine the request message processing method according to the first QP status information, the second QP status information, and the third QP status information, and announce the request message processing method to maintain the synchronization of the first QP status information, the second QP status information, and the third QP status information;

[0078] A response message construction module, used to construct a response message according to the request message processing method.

[0079] During the actual operation process, the request message verification module receives the request message transmitted by the network card, verifies the message, maintains the first QP status information, and filters out the request messages that pass the verification; sends the request messages that pass the verification to the RQE processing verification module. The RQE processing verification module pre-obtains the RQE of the active QP from the host, verifies the RQE, maintains the second QP status information, and verifies whether the space of the RQE is sufficient to be allocated for the request messages that pass the verification; if so, the request message payload upload module writes the request message payload to the host according to the remote address in the request message or the address in the RQE, and maintains the third QP status information; the completion verification module receives the QP status information maintained by the above modules, determines the request message processing method, sends it to the response message construction module, and announces it to the above modules, so as to synchronize the QP status information of each module; the response construction module constructs a response message according to the request message processing method.

[0080] Preferably, the request message verification module includes: a sending verification module, a rationality verification module, a first screening module, and a first QP status module;

[0081] The sending verification module is used to verify whether the received request message is a request message of the message processing system;

[0082] The rationality verification module is used to verify whether the request message is reasonable;

[0083] The first screening module is used to screen the request messages for which the judgment results of both the sending verification module and the rationality verification module are yes;

[0084] The first QP status module is used to mark the request message as abnormal and update the first QP status information when any one of the judgment results of the sending verification module and the rationality verification module is no.

[0085] In the actual application process, when screening RoCEv2 messages, first judge whether it is a RoCEv2 message sent to the local end based on the Ethernet layer 2, 3, and 4 header information of the message, and then judge whether it is a reasonable RoCEv2 request message according to the BTH (Base Transport Header) header information of the IB protocol and the corresponding QPC (Queue Pair Context) information. Only reasonable RoCEv2 request messages will be processed by the subsequent modules above, otherwise they will be directly discarded silently or sent to other processing logics of the network card according to the protocol requirements;

[0086] The preliminary verification for RoCEv2 request messages refers to the verification that can be directly judged based on the BTH and ETH (Extended Transport Header) information, QPC, MRC (Memory Region Context) of the message, and the QP-related status information maintained by the module itself. If the preliminary verification passes, the subsequent module processing will continue, otherwise, relevant exception handling operations will be executed according to the error reason. The content of the preliminary verification includes: whether the Opcode type is supported, the correctness of the Opcode order, the PSN correctness, the matching of the R_Key (Remote Key) and the access requirement, the Q_Key (Queue Key) correctness, the matching of the message length and the PMTU (Path Maximum Transmission Unit), the correctness of the Pad Count, etc. Among them:

[0087] The Opcode type verification is used to check whether the corresponding QP of the message supports the request type corresponding to the Opcode field in the BTH header information of the message;

[0088] Opcode sequence check is used to check whether the Opcode of a message is sequential with the Opcode of the previous correct message of the corresponding QP; therefore, the request message verification module needs to record the Opcode of the previous correct message of each QP.

[0089] PSN check is used to check whether the PSN of a message is equal to the ePSN of the corresponding QP; therefore, the request message verification module needs to maintain the ePSN value of each QP.

[0090] R_Key check is used to check whether the operation type and spatial size of the MR required to process an RDMA Write or RDMA Read request message are within the access permissions of its R_Key, and the access permissions of the R_Key can be obtained from the MRC.

[0091] Q_Key check is used to check whether the Q_Key of a request message matches the corresponding destination QP.

[0092] Message length check includes checking whether the message payload length matches the MTU and Opcode, and whether the total length of all message payloads of an RDMA Write request matches the DMA Length field in its RETH (RDMA Extended Transport Header); therefore, the request message verification module needs to record the RETH in the RDMA Write request message based on the QP, as well as the cumulative value of the lengths of all message payloads of this request.

[0093] Pad Count check is used to check whether the Pad Count field in the BTH matches the Opcode.

[0094] When performing a preliminary verification of a message, relevant QP status information will be maintained. If an error is found during the verification process, the message will be marked as abnormal, and at the same time, the relevant QP status information will be updated to facilitate decision-making on the processing method of subsequent messages of the corresponding QP.

[0095] As Figure 2 shown, preferably, the first QP status module is further configured to, when the QP type in the first QP status information is of the RC type, process the request message according to the hardware status of the RC type QP.

[0096] The first QP status module includes: a first status switching sub-module;

[0097] The first status switching sub-module is specifically configured to switch to the normal state when the host creates or restarts a QP.

[0098] If the judgment result of the rationality verification module is negative, switch to the wait_send_nack state;

[0099] If the completion verification module notifies that the request message is a Class B error defined by the IB protocol, switch to the wait_epsn state;

[0100] If the request message triggers any of the Class A or Class C errors defined by the IB protocol, or the host configures the QP error, or the completion verification module notifies any of the Class A or Class C errors defined by the IB protocol, switch to the error state.

[0101] In the actual application process, when the QP type in the first QP status information is of the RC type, four state transitions are performed, and the request message is processed correspondingly in each state to further determine the processing method of the message. The QP hardware status includes four states: normal, wait_send_nack, wait_epsn, and error. When the host creates or restarts the QP, it enters the normal state, which is used for normal pipelining processing of the request message; in this state, if a message with an invalid PSN is detected, it enters the wait_send_nack state, if the completion verification module notifies a Class B error defined by the IB protocol, it enters the wait_epsn state, and if it detects any of the three situations where the request message being processed by this module triggers a Class A or Class C error defined by the IB protocol, the host configures the QP to enter an error-related state, and the completion verification module notifies a Class A or Class C error defined by the IB protocol, it enters the error state;

[0102] The wait_send_nack state is used to wait for the network card to send the corresponding Unacknowledge message after detecting a message with an invalid PSN; in this state, if the completion verification module notifies a Class B error defined by the IB protocol, it enters the wait_epsn state, and if it detects any of the two situations where the host configures the QP to enter an error-related state and the completion verification module notifies a Class A or Class C error defined by the IB protocol, it enters the error state;

[0103] The wait_epsn state is used to wait for the peer to resend the correct request message after the network card sends the Unacknowledge message caused by a Class B error; in this state, if a message with all verifications including PSN verification being correct is detected, it enters the normal state, and if it detects any of the two situations where the PSN of the request message being processed by this module is correct but triggers a Class A or Class C error defined by the IB protocol, or the host configures the QP to enter an error-related state, it enters the error state;

[0104] The error state is used for the network card to discard the packets corresponding to the QP; in this state, if the host creates or restarts the QP, it can enter the normal state.

[0105] Preferably, the first QP state module is also used for

[0106] When in the error state, call the network card to directly and silently discard all request packets;

[0107] When in the normal state, call the network card to discard the request packets for error retransmission, and mark and process the remaining request packets according to the judgment result of the rationality verification module according to the IB protocol;

[0108] When in the wait_send_nack state, call the network card to process the request packets for correct retransmission and discard the remaining request packets;

[0109] When in the wait_epsn state, call the network card to process the request packets for which the judgment result of the rationality verification module is yes or for correct retransmission, and discard the remaining request packets. Among them, the processing method of the request packets for which the judgment result of the rationality verification module is yes is the same as the processing method in the normal state.

[0110] In the actual application process, the first QP state module operates as follows by calling the network card's packet processing method based on the QP state: if it is in the error state, directly and silently discard all packets; if it is in the normal state, silently discard the incorrect retransmission packets, and other packets are marked and processed according to the packet verification result according to the requirements of the IB protocol, such as being marked as class A / B / C errors defined by the IB protocol such as invalid request error, remote access error, PSN sequence error, etc.; if it is in the wait_send_nack state, only process the correct retransmission packets and discard all other packets; if it is in the wait_epsn state, only process the packets with correct PSN or correct retransmission packets. Among them, the processing method of the packets with correct PSN is the same as that in the normal state, while all other packets are discarded.

[0111] Preferably, the RQE processing and verification module includes: a cache module, an RQE status verification module, an RQE space verification module, a second screening module, and a second QP state module;

[0112] The cache module is used to obtain the RQEs of the active QP from the host and cache them into the network card by QP;

[0113] The RQE status verification module is used to verify whether the RQE is correct. If not, mark it as a deformed RQE;

[0114] The RQE space verification module is used to check whether there is an RQE corresponding to the QP cached in the network card according to the request message. If there is, the RQE corresponding to the cached QP is allocated to the request message. If not, an RNR error is triggered and the request message is marked.

[0115] The second screening module is used to screen the request messages for which the judgment results of both the RQE status verification module and the RQE space verification module are yes.

[0116] The second QP status module is used to mark the request message as abnormal and update the second QP status information when either the judgment result of the RQE status verification module or the RQE space verification module is no.

[0117] In the actual application process, when performing RQE processing, the RQE of the active QP is pre-obtained from the host and cached in the network card, and at the same time, the correctness of the RQE is checked based on information such as MRC. When a request message that needs to consume the RQE is received, if there is an RQE for the corresponding QP, it can be directly taken out from the cache for use. If there is no RQE, the corresponding exception handling operation is performed. In addition, when the corresponding QP is in an error state, etc., the RQE processing and verification module will also actively read the RQE of the corresponding QP to facilitate the completion of the RQE refresh error operation. When judging whether the space of the RQE is sufficient to be allocated to the message for use, that is, judging whether the remaining host memory space corresponding to the RQE is sufficient to support the writing of the current message load data, the correctness of the RQE is also checked. If the RQE is correct and sufficient, the processing of the subsequent module is continued. Otherwise, the corresponding exception handling operation is performed. When performing the above RQE processing and judging whether the space of the RQE is sufficient to be allocated to the message, the corresponding QP status information is maintained. When any exception is found during message processing, the message is marked as abnormal, and at the same time, the QP status information is updated to facilitate the decision-making of the subsequent message or RQE processing method for the corresponding QP.

[0118] In this embodiment, specifically, the RQE of the active QP is obtained from the host in advance and cached in the network card by QP. At the same time, based on the information such as SGE (Scatter / Gather Elements), L_Key, and MRC in the RQE, it is checked whether the RQE is correct. If the RQE is incorrect, it is marked as a deformed RQE for caching. When the RQE space verification module receives a request message that needs to consume the RQE, it checks whether there is an RQE corresponding to the QP in the network card cache. If there is an RQE, it is taken out and allocated to this request message. If not, an RNR (Receiver Not Ready) error may be triggered, and it is necessary to jointly decide the processing method of the message in combination with the marked state of the message and the QP status, etc.

[0119] Such as Figure 3As shown, preferably, the second QP status module is further configured to process the request message according to the QP hardware status corresponding to the RC type when the QP type in the second QP status information is the RC type;

[0120] The second QP status module includes: a second status switching sub-module;

[0121] The second status switching sub-module is specifically configured to switch to the normal state when the host creates or restarts the QP;

[0122] If the judgment result of the RQE space check module is negative or the request message has been marked as a Class B error defined by the IB protocol, it switches to the wait_send_nack state;

[0123] If the completion check module notifies that the request message is a Class B error defined by the IB protocol, it switches to the wait_epsn state;

[0124] If the request message has been marked as any one of a Class A or Class C error defined by the IB protocol, a host configuration QP error, or the completion check module notifies a Class A or Class C error defined by the IB protocol, it switches to the error state.

[0125] In the actual application process, when the QP type in the second QP status information is the RC type, four state transitions are performed, and the request message is processed correspondingly in each state, which is used to further determine the processing method of the message. When the host creates or restarts the QP, it enters the normal state, and this state is used for normal pipelining to process the request message; in this state, if a message marked as normal but triggering an RNR error or a message marked as a Class B error defined by the IB protocol is detected, it enters the wait_send_nack state. If it is detected that the completion check module notifies a Class B error defined by the IB protocol, it enters the wait_epsn state. If it is detected that any one of the three situations where the request message being processed has been marked as a Class A or Class C error defined by the IB protocol, the host configuration QP enters an error-related state, and the completion check module notifies a Class A or Class C error defined by the IB protocol, it enters the error state;

[0126] The wait_send_nack state is used to wait for the network card to send the corresponding Unacknowledge message after detecting an RNR error; in this state, if it is detected that the completion check module notifies a Class B error defined by the IB protocol, it enters the wait_epsn state. If it is detected that any one of the two situations where the host configuration QP enters an error-related state and the completion check module notifies a Class A or Class C error defined by the IB protocol, it enters the error state;

[0127] The wait_epsn state is used to wait for the peer to resend the correct request message after the network card sends an Unacknowledge message due to a Class B error; in this state, if a message marked as normal and with correct RNR verification is detected, it enters the normal state. If a request message is detected to be marked with a Class A or Class C error defined by the IB protocol or the host configures the QP to enter an error-related state, it enters the error state. If a message marked as normal but triggering an RNR error or a message already marked with a Class B error defined by the IB protocol is detected, it enters the wait_send_nack state;

[0128] The error state is used for the network card to discard the messages corresponding to the QP and complete the RQE refresh error; in this state, if the host creates or restarts the QP, it can enter the normal state.

[0129] Preferably, the second QP state module is also used for,

[0130] When in the error state, call the network card to directly and silently discard all request messages;

[0131] When in the wait_send_nack state, call the network card to process the correctly retransmitted request messages and discard the remaining request messages;

[0132] When in the normal state or the wait_epsn state, if the request message triggers an RNR error and the request message has been marked as a Class B error with the judgment result of the rationality verification module being no, the request message is re-marked as an RNR error.

[0133] In the actual application process, the second QP state module operates as follows by calling the network card's message processing method based on the QP state: When the QP is in the normal state or the wait_epsn state, if the message triggers an RNR error and the message has been marked as a Class B error with a PSN sequence error by the request message verification module, the message is re-marked as an RNR error; if the message does not trigger an RNR error or has been marked as other Class A or Class C errors by the request message verification module, there is no need to modify the processing method decided by the request message verification module;

[0134] When the QP is in the wait_send_nack state, only the retransmitted messages marked as normal can perform subsequent processing, and other messages will be discarded;

[0135] When the QP is in the error state, all messages are discarded.

[0136] Preferably, the RQE space verification module further includes: a dual parity check sub-module;

[0137] The double syndrome sub-module is used to check whether the RQE is deformed and whether the remaining host memory space corresponding to the RQE supports the writing of the load data of the current request message after the RQE corresponding to the QP already exists in the network card;

[0138] If the RQE is marked as deformed, the request message is marked as a deformed RQE error;

[0139] If the remaining host memory space corresponding to the RQE does not support the writing of the load data of the current request message, the request message is marked as an invalid request error.

[0140] In the actual application process, when processing the message based on the read RQE, it is also necessary to check whether the RQE is deformed and whether the remaining host memory space corresponding to the RQE is sufficient to support the writing of the load data of the current message: if it is found that the RQE is marked as deformed, the message is marked as a deformed RQE error; if the remaining host memory space corresponding to the RQE is not sufficient to write the load data of the current message, the message is marked as an invalid request error. The above two errors are both Class A or Class C errors defined by the IB protocol.

[0141] Preferably, the second QP status module further includes: a third status switching sub-module;

[0142] The third status switching sub-module is specifically used to switch to the normal state when the host creates or restarts the QP;

[0143] If the request message has been marked as any one of the Class A or Class C errors defined by the IB protocol or the host configures the QP error or the completion check module notifies the Class A or Class C errors defined by the IB protocol or the request message is marked as a deformed RQE error or an invalid request error, it switches to the error state.

[0144] During actual operation, when the QP type in the second QP status information is the RC type, two state conversions are performed, and request packets are processed correspondingly in each state for further decision on the packet processing method: When processing packets based on the read RQE, in addition to recording the remaining host memory space size corresponding to the above RQE based on the QP, it is also necessary to maintain the QP hardware status for the RC type QP, which includes two states: normal and error. When the host creates or restarts the QP, it enters the normal state, which is used for normal pipelining to process request packets; in this state, if it is detected that a request packet being processed has been marked as an error of type A or C defined by the IB protocol, a request packet being processed is marked as one of the above two errors in this module, the host configures the QP to enter an error-related state, or the completion verification module notifies an error of type A or C defined by the IB protocol, then it enters the error state; the error state is used for the network card to discard packets corresponding to the QP; in this state, the host can enter the normal state only after creating or restarting the QP.

[0145] As Figure 4 shown, preferably, the request packet payload upload module includes: a payload upload module, a payload data writing module, and a third QP status module;

[0146] The payload upload module is used to determine whether the request packet needs to upload the payload;

[0147] The payload data writing module is used to, when the judgment result of the payload upload module is yes, obtain the corresponding physical page address from the host according to the remote address in the request packet or the address in the RQE, write the request packet payload into the host memory according to the physical page address, and construct a request packet descriptor;

[0148] The third QP status module is used to, when the QP type in the third QP status information is the RC type, process the request packet according to the RC type QP hardware status.

[0149] During actual operation, when the request packet payload upload module writes the packet payload into the host memory, it first determines whether the packet needs to upload the payload. If not, it directly proceeds to the processing of the subsequent module. If so, it obtains the corresponding physical page address from the host based on the remote address in the request packet or the address in the RQE, then writes the packet payload data into the host memory according to the physical page address, and finally constructs a request packet descriptor according to the processing situation of the packet and sends it to the completion verification module.

[0150] In this embodiment, when the first message of the request for load upload is received, the remote access address and length in the message RETH or the SGE address and length in the allocated RQE are recorded, and then the physical page addresses required for the corresponding request are read in batches and cached in the network card, so that the subsequent messages of the request can directly obtain the corresponding physical page addresses from the cache, thereby reducing the PCIE bandwidth occupancy rate and the subsequent message processing delay.

[0151] Preferably, the third QP status module includes: a fourth status switching sub-module;

[0152] The fourth status switching sub-module is specifically configured to switch to the normal state when the host creates or restarts the QP;

[0153] If the request message has been marked as an error of type A or C defined by the IB protocol, or a host configuration QP error, or the completion verification module notifies an error of type A or C defined by the IB protocol, then it switches to the error state.

[0154] In the actual operation process, it is necessary to maintain the access address and remaining length of the next message of the current request based on the QP, so as to allocate the corresponding memory cache space for the load data of the subsequent messages of the request. In addition, the request message load upload module does not verify the message, but in order to ensure the consistency of the pipelined message processing, it is still necessary to maintain the QP hardware state for the RC type QP, which includes two states: normal and error, where: the normal state is used for normal pipelined processing of request messages and enters the normal state when the host creates or restarts the QP; in this state, if it is detected that a request message being processed has been marked as an error of type A or C defined by the IB protocol, the host configures the QP to enter an error-related state, or the completion verification module notifies an error of type A or C defined by the IB protocol, then it enters the error state; the error state is used for the network card to discard the messages corresponding to the QP; in this state, it can enter the normal state only after the host creates or restarts the QP.

[0155] As Figure 5 shown, preferably, the completion verification module includes: a request message processing method determination module and a notification module;

[0156] The request message processing method determination module is used to determine the request message processing method according to the first QP status information, the second QP status information, and the third QP status information included in the request message descriptor;

[0157] The notification module is used to send the message information in the request message descriptor to the request message verification module, the RQE processing verification module, and the request message load upload module respectively to update the corresponding QP status information.

[0158] In the actual operation process, the completion verification module determines the processing conditions of each previous module for the message according to the information recorded in the request message descriptor, which are specifically reflected as the first QP status information, the second QP status information, and the third QP status information. Then, based on the IB protocol, it decides how to construct the response message descriptor and generate the CQE. Among them, the response message includes Acknowledge message, Unacknowledge message, and Read Response message. When constructing the descriptor of the Unacknowledge message, information such as QPN (Queue Pair Number), error type, ePSN, etc. in the corresponding request message descriptor will be notified to the request message verification module, the RQE processing verification module, and the request message payload upload module, so that these three modules can update the status information of the corresponding QP.

[0159] Preferably, the response message construction module is specifically used to generate a response message descriptor according to the request message descriptor and construct a response message according to the response message descriptor;

[0160] If the response message is a read response message, the response message construction module is also specifically used to construct a response message according to the response message descriptor, the remote address in the request message, and the physical page address.

[0161] In the actual operation process, when constructing the Acknowledge message and the Unacknowledge message, it can be directly completed based on descriptor information, QPC, etc. As for the construction of the Read Response message, data also needs to be obtained from the host to fill the message payload. The processing flow of constructing the Read Response message is divided into three steps: physical page address acquisition, MR data reading, and message payload data filling and encapsulation. Among them: the physical page address acquisition step is used to obtain the physical page address of the access space required for the RDMA read request from the host memory according to information such as the remote address of the Read request, R_Key, and MRC cached by the network card; the MR data acquisition step is used to read the data in the corresponding space of the MR from the host memory again according to the physical page address; the message payload data filling and encapsulation step is used to fill the read MR data into the message payload field, and then construct a message header for message encapsulation and forwarding.

[0162] In the embodiments provided in this application, it should be understood that the disclosed system can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, and can be electrical, mechanical, or other forms.

[0163] In addition, in each embodiment of the present invention, each functional module can be fully integrated in a processor, or each module can be separately used as a device, or two or more modules can be integrated in a device; each functional module in each embodiment of the present invention can be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0164] It should be understood that in this application, if terms such as "system", "device", "unit" and / or "module" are used, they are only a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the term can be replaced by other expressions.

[0165] As shown in this application and the claims, unless the context clearly indicates an exception, words such as "a", "an", "one" and / or "the" are not specifically singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. A method or device may also include other steps or elements. An element defined by the statement "comprising one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

[0166] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0167] The above has introduced in detail a message processing system provided by the present invention. The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A message processing system, characterized in that, Including: A request message verification module, which is used to verify the received request message, maintain the first QP status information, and filter out the request messages that pass the verification; An RQE processing verification module, which is used to verify the RQE obtained from the host, maintain the second QP status information, and verify whether the space of the RQE is sufficient to be allocated for the use of the request message; A request message payload upload module, which is used to write the request message payload into the host according to the remote address in the request message or the address in the RQE, and maintain the third QP status information; A completion verification module, which is used to determine the request message processing method according to the first QP status information, the second QP status information, and the third QP status information, and announce the request message processing method to maintain the synchronization of the first QP status information, the second QP status information, and the third QP status information; A response message construction module, which is used to construct a response message according to the request message processing method; The request message verification module includes: a sending verification module, a rationality verification module, a first screening module, and a first QP status module; the first QP status module includes: a first status switching sub-module, which is used to switch to the normal state when the host creates or restarts the QP; if the judgment result of the rationality verification module is no, it switches to the wait_send_nack state; if the completion verification module announces that the request message is a type B error defined by the IB protocol, it switches to the wait_epsn state; if the request message triggers a type A or type C error defined by the IB protocol, or the host configures the QP incorrectly, or the completion verification module announces any of the type A or type C errors defined by the IB protocol, it switches to the error state; When in the error state, the network card is called to directly and silently discard all the request messages; when in the normal state, the network card is called to discard the request messages that are retransmitted due to errors, and the remaining request messages are marked according to the judgment result of the rationality verification module according to the IB protocol; when in the wait_send_nack state, the network card is called to process the request messages that are correctly retransmitted, and the remaining request messages are discarded; when in the wait_epsn state, the network card is called to process the request messages for which the judgment result of the rationality verification module is yes or the request messages that are correctly retransmitted, and the remaining request messages are discarded, wherein the processing method of the request messages for which the judgment result of the rationality verification module is yes is the same as the processing method when in the normal state.

2. The message processing system according to claim 1, wherein The sending verification module is used to verify whether the received request message is the request message of the message processing system; The rationality verification module is used to verify whether the request message is reasonable; The first screening module is used to screen the request messages for which the judgment results of the sending verification module and the rationality verification module are both yes; The first QP status module is used to mark the request message as abnormal and update the first QP status information when either the judgment result of the sending verification module or the judgment result of the rationality verification module is negative. The first QP status module is further used to process the request message according to the QP hardware status of the RC type when the QP type in the first QP status information is of the RC type.

3. The message processing system according to claim 2, wherein The RQE processing verification module includes: a cache module, an RQE status verification module, an RQE space verification module, a second screening module, and a second QP status module. The cache module is used to obtain the RQEs of the active QP from the host and cache them into the network card by QP. The RQE status verification module is used to verify whether the RQE is correct. If not, it is marked as a deformed RQE. The RQE space verification module is used to check whether there is a cached RQE of the corresponding QP in the network card according to the request message. If so, the cached RQE of the corresponding QP is assigned to the request message. If not, an RNR error is triggered and the request message is marked. The second screening module is used to screen the request messages for which both the judgment result of the RQE status verification module and the judgment result of the RQE space verification module are positive. The second QP status module is used to mark the request message as abnormal and update the second QP status information when either the judgment result of the RQE status verification module or the judgment result of the RQE space verification module is negative.

4. The message processing system according to claim 3, wherein the second QP status module is further used to process the request message according to the QP hardware status of the RC type when the QP type in the second QP status information is of the RC type; the second QP status module includes: a second status switching sub-module; the second status switching sub-module is specifically used to switch to the normal state when the host creates or restarts the QP; if the judgment result of the RQE space verification module is negative or the request message has been marked as a type B error defined by the IB protocol, then it switches to the wait_send_nack state; if the completion verification module notifies that the request message is a type B error defined by the IB protocol, then it switches to the wait_epsn state; if the request message has been marked as a type A or type C error defined by the IB protocol, or the host configures the QP incorrectly, or the completion verification module notifies any of the type A or type C errors defined by the IB protocol, then it switches to the error state.

5. The message processing system according to claim 4, wherein the second QP status module is further used for when in the error state, calling the network card to directly and silently discard all the request messages; when in the wait_send_nack state, calling the network card to process the correctly retransmitted request messages and discard the remaining request messages. When in the normal state or the wait_epsn state, if the request message triggers an RNR error and the request message has been marked as a Class B error with the judgment result of the rationality verification module being no, then the request message is re-marked as an RNR error.

6. The message processing system according to claim 3, wherein The RQE space verification module further includes: a double verification sub-module; The double verification sub-module is used to verify whether the RQE is deformed and whether the remaining host memory space corresponding to the RQE supports the writing of the load data of the current request message after the RQE corresponding to the QP already exists in the network card; If the RQE is marked as deformed, then the request message is marked as a deformed RQE error; If the remaining host memory space corresponding to the RQE does not support the writing of the load data of the current request message, then the request message is marked as an invalid request error.

7. The message processing system according to claim 6, characterized in that, The second QP status module further includes: a third status switching sub-module; The third status switching sub-module is specifically used to switch to the normal state when the host creates or restarts the QP; If the request message has been marked as a Class A or Class C error defined by the IB protocol or the host configures the QP incorrectly or the completion verification module notifies a Class A or Class C error defined by the IB protocol or the request message is marked as any one of the deformed RQE error or the invalid request error, then it switches to the error state.

8. The message processing system according to claim 1, wherein The request message load upload module includes: a load upload module, a load data writing module, and a third QP status module; The load upload module is used to determine whether the request message needs to perform load upload; The load data writing module is used to, when the judgment result of the load upload module is yes, obtain the corresponding physical page address from the host according to the remote address in the request message or the address in the RQE, write the request message load into the host memory according to the physical page address, and construct a request message descriptor; The third QP status module is used to process the request message according to the QP hardware status corresponding to the RC type when the QP type in the third QP status information is the RC type.

9. The message processing system according to claim 8, wherein The third QP status module includes: a fourth status switching sub-module; The fourth status switching sub-module is specifically used to switch to the normal state when the host creates or restarts the QP; If the request message has been marked as any one of a Class A or Class C error defined by the IB protocol or the host configures the QP incorrectly or the completion verification module notifies a Class A or Class C error defined by the IB protocol, then it switches to the error state.

10. The message processing system according to claim 8, wherein The completion verification module includes: a request message processing method determination module and a notification module; The request message processing method determination module is used to determine the request message processing method according to the first QP status information, the second QP status information, and the third QP status information included in the request message descriptor; The notification module is used to send the message information in the request message descriptor to the request message verification module, the RQE processing verification module, and the request message payload upload module respectively, so as to update the corresponding QP status information.

11. The message processing system according to claim 8, wherein The response message construction module is specifically used to generate a response message descriptor according to the request message descriptor, and construct a response message according to the response message descriptor; If the response message is a read response message, the response message construction module is further specifically used to construct the response message according to the response message descriptor, the remote address in the request message, and the physical page address.

Images