A virtual channel scheduling system for RDMA transmission

Abstract

The present application discloses a virtual channel scheduling system for RDMA transmission. The system includes: a relationship establishment module, a first virtual channel scheduling module, a second virtual channel scheduling module, a third virtual channel scheduling module, and a message construction module; the first virtual channel scheduling module is respectively connected to the relationship establishment module and the second virtual channel scheduling module; the third virtual channel scheduling module is respectively connected to the second virtual channel scheduling module and the message construction module; the relationship establishment module, the first virtual channel scheduling module, the second virtual channel scheduling module, the third virtual channel scheduling module, and the message construction module are all connected to a host; through the collaborative work of each module, the entire processing flow of SQE corresponding to each virtual channel's QP can be completed in a pipelined manner without mutual interference and blocking, thereby solving problems such as a sharp increase in delay, performance degradation, and head-of-line blocking caused by simple scheduling methods.

Description

Technical Field

[0001] The present application relates to the technical field of remote direct memory access, and particularly to a virtual channel scheduling system for RDMA transmission. Background Art

[0002] In recent years, with the rapid development of technologies such as artificial intelligence and cloud computing, RoCEv2 (RDMA over Converged Ethernet) technology has been widely applied in data centers. Its characteristics of high bandwidth, low latency, multi-link, and zero-copy enable it to have better performance in multi-node distributed computing. In the application scenarios of data centers, the number of links it needs to support is often more than 10K, and the link bandwidth is mostly in the order of hundreds of Gbps.

[0003] However, on the one hand, the total amount of on-chip RAM (Random Access Memory) storage resources of an RDMA network card implemented based on FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) is generally in the order of ten megabytes. However, based on the requirements of RDMA protocol technology, the RDMA network card needs to cache information such as QPC (Queue Pair Context), CQC (Completion Queue Context), MRC (Memory Region Context), SQE (Send Queue Element), RQE (Receive Queue Element), and OR (Outstanding Request). Taking the number of QPs (Queue Pairs) as 10K and each QPC as 128B, the space required for only QPC is 1.28MB. Obviously, it is unreasonable to use on-chip storage resources to cache the relevant information of all links. On the other hand, such a large number of links require a corresponding number of QPs to support RDMA transmission. For an RDMA network card, it is difficult to implement the scheduling between such a large number of QPs using simple polling. The simple polling operation of 10K-level QPs will bring huge delays, and in practical applications, only a small number of QPs need to be processed in parallel by the network card. Such polling will also cause a serious decline in the performance of the network card when processing these QPs in parallel. Finally, affected by the RoCEv2 network congestion control algorithm and PFC (Priority-based Flow Control), it is necessary to implement fine-grained scheduling based on QPs, and more importantly, to solve the head-of-line blocking problem where all QPs cannot send data when a certain QP is restricted by congestion control or PFC and cannot continue to send data.

[0004] In view of this, how to implement the scheduling and processing of a huge number of QPs, while solving the problems of soaring delays, performance degradation, and head-of-line blocking caused by simple scheduling methods is a technical problem that those skilled in the art urgently need to solve. Summary of the Invention

[0005] To solve the above technical problems, the objective of the present invention is to provide a virtual channel scheduling system for RDMA transmission. Through the collaborative work of each module, the entire processing flow of SQE corresponding to each virtual channel's QP can be completed in a pipelined manner without mutual interference and blocking, thereby solving problems such as a sharp increase in latency, performance degradation, and head-of-line blocking caused by simple scheduling methods.

[0006] The first objective of the present invention is to provide a virtual channel scheduling system for RDMA transmission;

[0007] The technical solution provided by the present invention is as follows:

[0008] A virtual channel scheduling system for RDMA transmission, comprising: a relationship establishment module, a first virtual channel scheduling module, a second virtual channel scheduling module, a third virtual channel scheduling module, and a packet construction module; the first virtual channel scheduling module is respectively connected to the relationship establishment module and the second virtual channel scheduling module; the third virtual channel scheduling module is respectively connected to the second virtual channel scheduling module and the packet construction module; the relationship establishment module, the first virtual channel scheduling module, the second virtual channel scheduling module, the third virtual channel scheduling module, and the packet construction module are all connected to the host;

[0009] The relationship establishment module is used to obtain active QPs according to the Doorbell of each QP and establish a mapping relationship between the active QPs and the network card virtual channels;

[0010] The first virtual channel scheduling module is used to poll and schedule the active QPs obtained by the relationship establishment module to obtain SQEs from the host and write the SQEs into the SQE cache space;

[0011] The second virtual channel scheduling module is used to poll and read the SQEs in the first virtual channel scheduling module to obtain physical page addresses from the host and write the physical page addresses into the page address cache space;

[0012] The third virtual channel scheduling module is used to poll and read the physical page addresses and the SQEs in the second virtual channel scheduling module to obtain message data cached in the corresponding MR (Memory Region) from the host;

[0013] The packet construction module is used to construct a RoCEv2 packet according to the QPC of the active QP, the SQE, and the message data and forward it.

[0014] Preferably, the relationship establishment module is specifically used for:

[0015] Determine whether the corresponding QP is the active QP according to the WQE (Work Queue Element) type, PI (Producer Index), and CI (Consumer Index) maintained by the hardware in the Doorbell; if so, then:

[0016] Query whether the active QP already has a virtual channel mapping; if the active QP already has a mapped virtual channel, there is no need to establish a new mapping relationship, and only the PI value in the hardware cache needs to be updated; if the active QP does not yet have a mapped virtual channel, then:

[0017] Query whether there is an idle virtual channel. If there is an idle virtual channel, establish a mapping relationship between the active QP and the idle virtual channel. If there is no idle virtual channel, do nothing.

[0018] Preferably, the relationship establishment module is further configured to: when it is detected that the QP corresponding to the virtual channel is destroyed or when the network card has processed the existing SQE of the virtual channel and there is no new SQE to be processed within a preset time, delete the mapping relationship between the virtual channel and the active QP.

[0019] Preferably, the first virtual channel scheduling module is specifically configured to:

[0020] Obtain the remaining SQE cache space of the virtual channel and the remaining number of SQEs of the QP corresponding to the virtual channel in the host memory by polling and scheduling the virtual channel to calculate the number of SQEs;

[0021] Read a specified number of SQEs from the host according to the number of SQEs and write the SQEs into the SQE cache space.

[0022] Preferably, the second virtual channel scheduling module is specifically configured to:

[0023] Read the SQE cache space of the virtual channel by polling and scheduling the virtual channel to obtain the remaining page address cache space of the virtual channel and the remaining length of the SQE message;

[0024] Calculate the number of page addresses according to the remaining page address cache space and the remaining length of the SQE message;

[0025] Obtain the physical page address of the MR corresponding to the message data in the SQE from the host according to the number of page addresses and write the physical page address into the page address cache space of the corresponding virtual channel.

[0026] Preferably, the third virtual channel scheduling module is specifically configured to:

[0027] Read the physical page address and the SQE of the virtual channel by polling the virtual channel, and at the same time obtain the PMTU (Path Maximum Transmission Unit) of the QP corresponding to the virtual channel, the remaining token number, the hardware status, the available space of the ORT (Outstanding Request Table), the remaining length of the SQE message, the PFC backpressure status, and the number of page addresses;

[0028] Calculate the number of packets according to the PMTU, the remaining token number, the hardware status, the available space of the ORT, the remaining length of the SQE message, the PFC backpressure status, and the number of page addresses of the QP corresponding to the virtual channel;

[0029] Obtain the corresponding MR message data from the host memory according to the number of packets.

[0030] Preferably, the packet construction module is specifically configured to:

[0031] Cache the MR message data of the corresponding MR cache into the shared packet payload cache, and at the same time cache the fields required for constructing the packet in the corresponding SQE into the shared packet descriptor cache in sequence; wherein, the fields include: the operation type of the corresponding SQE, the immediate number, the remote access address, the remote access key, and the remote access memory size;

[0032] Construct a RoCEv2 packet according to the MR message data in the shared packet payload cache, the fields in the shared packet descriptor cache, and the QPC in the corresponding QP, and send the RoCEv2 packet to the Ethernet interface for forwarding.

[0033] A virtual channel scheduling system for RDMA transmission provided by the present invention includes: a relationship establishment module, a first virtual channel scheduling module, a second virtual channel scheduling module, a third virtual channel scheduling module, and a packet construction module; the first virtual channel scheduling module is respectively connected to the relationship establishment module and the second virtual channel scheduling module; the third virtual channel scheduling module is respectively connected to the second virtual channel scheduling module and the packet construction module; the relationship establishment module, the first virtual channel scheduling module, the second virtual channel scheduling module, the third virtual channel scheduling module, and the packet construction module are all connected to the host; through the collaborative work of each module, the entire processing flow of the SQE of the QP corresponding to each virtual channel can be completed in a pipelined manner without mutual interference and blocking, thereby solving problems such as a sharp increase in delay, performance degradation, and head-of-line blocking caused by simple scheduling methods. Description of the Drawings

[0034] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments recorded in the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0035] Figure 1 It is a schematic structural diagram of a virtual channel scheduling system for RDMA transmission in an embodiment of the present invention. Detailed implementation manners

[0036] In order to enable those skilled in the art to better understand the technical solutions in the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope protected by the present application.

[0037] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly disposed on the other element; when an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0038] It should be understood that the orientation or positional relationship indicated by terms such as "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the accompanying drawings. It is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be understood as a limitation to the present application.

[0039] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present application, the meanings of "a plurality" and "several" are two or more, unless otherwise specifically defined.

[0040] It should be noted that the structures, proportions, sizes, etc. shown in the attached drawings of this specification are only used to cooperate with the content disclosed in the specification for those familiar with this technology to understand and read, and are not used to limit the implementation conditions of this application. Therefore, they do not have substantial technical significance. Any modification of the structure, change of the proportional relationship, or adjustment of the size, without affecting the effects that this application can produce and the purposes that can be achieved, should still fall within the scope covered by the technical content disclosed in this application.

[0041] As Figure 1 shown, an embodiment of the present invention provides a virtual channel scheduling system for RDMA transmission, including: a relationship establishment module 101, a first virtual channel scheduling module 102, a second virtual channel scheduling module 103, a third virtual channel scheduling module 104, and a message construction module 105; the first virtual channel scheduling module 102 is respectively connected to the relationship establishment module 101 and the second virtual channel scheduling module 103; the third virtual channel scheduling module 104 is respectively connected to the second virtual channel scheduling module 103 and the message construction module 105; the relationship establishment module 101, the first virtual channel scheduling module 102, the second virtual channel scheduling module 103, the third virtual channel scheduling module 104, and the message construction module 105 are all connected to the host 11;

[0042] The relationship establishment module 101 is used to obtain active QPs according to the doorbells of each QP, and establish a mapping relationship between the active QPs and the network card virtual channels;

[0043] The first virtual channel scheduling module 102 is used to poll and schedule the active QPs obtained by the relationship establishment module 101, so as to obtain SQEs from the host and write the SQEs into the SQE cache space;

[0044] The second virtual channel scheduling module 103 is used to poll and read the SQEs in the first virtual channel scheduling module 102, so as to obtain physical page addresses from the host and write the physical page addresses into the page address cache space;

[0045] The third virtual channel scheduling module 104 is used to poll and read the physical page addresses and the SQEs in the second virtual channel scheduling module 103, so as to obtain MR message data corresponding to the MR cache from the host;

[0046] The message construction module 105 is used to construct a RoCEv2 message according to the QPC of the active QP, the SQE, and the MR message data and forward it.

[0047] In the actual application process, a virtual channel scheduling system for RDMA transmission is provided, which can realize the pipelining and parallelization of the SQE processing process on the RDMA network card 10. The system includes: a relationship establishment module 101, which is used to screen active QPs according to the Doorbells of each QP and establish a mapping relationship between the active QPs and the virtual channels of the network card; a first virtual channel scheduling module 102, which is used to poll and schedule the corresponding QPs of each virtual channel, obtain SQEs from the host and write them into the SQE cache space of the corresponding virtual channel; a second virtual channel scheduling module 103, which is used to poll and read the SQEs of each virtual channel, obtain the physical page address of the MR corresponding to the message data in the SQE from the host, and write it into the page address cache space of the corresponding virtual channel; a third virtual channel scheduling module 104, which is used to poll and read the page addresses and SQEs of each virtual channel, calculate the number of packets that can be sent, and read the message data in the MR according to the page address; a packet construction module 105, which is used to construct and forward RoCEv2 packets based on the read MR message data, SQEs and relevant information of QPC; the first virtual channel scheduling module 102 is respectively connected to the relationship establishment module 101 and the second virtual channel scheduling module 103; the third virtual channel scheduling module 104 is respectively connected to the second virtual channel scheduling module 103 and the packet construction module 105, so that each module can work together to complete all the processing processes of the SQEs of the corresponding QPs of each virtual channel in a pipelined manner without mutual interference and blockage, thereby solving problems such as a sharp increase in delay, performance decay and head blockage caused by simple scheduling methods; moreover, each virtual channel only needs to independently cache a small amount of information such as SQEs and physical page addresses, and the data load is shared by all QPs, and the on-chip storage resources of the network card can be used to achieve this.

[0048] Preferably, the relationship establishment module 101 is specifically used for:

[0049] Judge whether the corresponding QP is the active QP according to the WQE type, PI producer index in the Doorbell and the CI consumer index maintained by the hardware; if so, then:

[0050] Query whether the active QP already has a virtual channel mapping; if the active QP already has a mapped virtual channel, there is no need to establish a new mapping relationship, and only the PI value cached by the hardware needs to be updated; if the active QP does not have a mapped virtual channel yet, then:

[0051] Query whether there is an idle virtual channel. If there is an idle virtual channel, establish a mapping relationship between the active QP and the idle virtual channel. If there is no idle virtual channel, do nothing.

[0052] In the actual application process, in the relationship establishment module 101, the virtual channel is a logical channel formed by the network card based on the above three scheduling modules. Each virtual channel can process SQEs in parallel and will not block or interfere with each other. When the virtual channel establishes a mapping relationship with the QP, it will only process the SQEs of the corresponding QP. The virtual channel mapping relationship is to establish a mapping table based on the binding relationship between the virtual channel and the QP. The mapping table also records status information such as whether each virtual channel is idle. The relationship establishment module 101 uses a mapping table to record the corresponding relationship between the QP and the virtual channel. The mapping table specifically includes the number of each virtual channel, the QPN (Queue Pair Number) of the corresponding QP, and status information such as whether it is idle. As for the number of virtual channels, it can be based on the specific implementation of the network card, and it is necessary to balance the resources of the network card and the design requirements. At the same time, the relationship establishment module is also used to release the virtual channel, that is, when it detects that the QP corresponding to the virtual channel is destroyed or when the network card has processed the existing SQEs of the virtual channel and there are no new SQEs to be processed within the preset time, it deletes the mapping relationship between the virtual channel and the active QP.

[0053] Preferably, the first virtual channel scheduling module 102 is specifically configured to:

[0054] Obtain the remaining SQE cache space of the virtual channel and the remaining number of SQEs of the QP corresponding to the virtual channel in the host memory by polling and scheduling the virtual channel, so as to calculate the number of SQEs;

[0055] Read SQEs from the host according to the number of SQEs and write the SQEs into the SQE cache space.

[0056] In the actual application process, the first virtual channel scheduling module 102 polls each virtual channel to obtain information such as the remaining SQE cache space of the corresponding virtual channel and the remaining number of SQEs of the QP corresponding to the virtual channel in the host memory. Based on this information, it calculates and decides the number of SQEs that can be read, and then reads the specified number of SQEs from the host QP from the host memory and writes them into the SQE cache space of the corresponding virtual channel. The specified number in this embodiment needs to be less than or equal to the number of SQEs that can be read, and can be set according to actual situations such as the size of the SQE and the memory access method.

[0057] Preferably, the second virtual channel scheduling module 103 is specifically configured to:

[0058] Read the SQE cache space of the virtual channel by polling and scheduling the virtual channel to obtain the remaining page address cache space of the virtual channel and the remaining length of the SQE message;

[0059] Calculate the number of page addresses according to the remaining page address cache space and the remaining length of the SQE message;

[0060] Obtain the physical page address of the MR corresponding to the message data in the SQE from the host according to the number of page addresses, and write the physical page address into the page address cache space of the corresponding virtual channel.

[0061] In the actual application process, the second virtual channel scheduling module 103 polls and schedules each virtual channel, reads its SQE cache space, calculates and determines the number of page addresses that can be read according to information such as the remaining page address cache space of the corresponding virtual channel and the remaining length of the SQE message, and then reads the host MTR (Memory Translate Region) to obtain the physical page address of the MR corresponding to the message data in the specified number of SQEs from the host, and writes the physical page address into the page address cache space of the corresponding virtual channel; the specified number in this embodiment needs to be less than or equal to the number of page addresses that can be read, and can be set according to actual situations such as the memory access mode.

[0062] Preferably, the third virtual channel scheduling module 104 is specifically configured to:

[0063] Poll and schedule the virtual channel to read the physical page address and the SQE of the virtual channel, and at the same time obtain the PMTU, remaining token number, hardware status, ORT available space, remaining length of the SQE message, PFC backpressure status, and the number of page addresses of the QP corresponding to the virtual channel;

[0064] Calculate the number of packets according to the PMTU, the remaining token number, the hardware status, the ORT available space, the remaining length of the SQE message, the PFC backpressure status, and the number of page addresses of the QP corresponding to the virtual channel;

[0065] Obtain the corresponding MR message data from the host memory according to the number of packets.

[0066] In the actual application process, the third virtual channel scheduling module 104 polls and schedules each virtual channel, reads its page address and SQE, obtains information such as the PMTU of the QP corresponding to the virtual channel, the remaining token number in the token bucket, the hardware status, the ORT available space, the remaining length of the SQE message, the PFC backpressure status, and the number of page addresses, calculates and determines the number of packets that can be sent based on this information, and then reads the host MR to obtain the corresponding MR message data of the specified number from the host memory; the specified number in this embodiment is the amount of MR message data corresponding to the packets that can be sent.

[0067] Among them, the remaining number of tokens in the token bucket is used to control the packet sending rate of the QP corresponding to each virtual channel, thereby supporting the QP-based congestion control algorithm required by the RoCEv2 network; the token bucket is activated when the mapping relationship between the virtual channel and the QP is established, and the speed limit value of the token bucket is calculated by the congestion control algorithm implemented by the network card; the hardware status is used to maintain the status of the QP corresponding to each virtual channel on the network card, and is used to decide whether the corresponding QP can continue to process SQE normally; the available space of the ORT is used to determine whether an OR cache can be constructed into the ORT when processing the SQE of the QP corresponding to the virtual channel; the remaining length of the SQE message is the size of the message data that has not been read from the MR for constructing the packet, and is used to calculate the amount of data that the SQE still needs to read, the number of corresponding page addresses, and other information; the PFC backpressure status is used to determine whether the QP corresponding to the virtual channel can continue to send packets. When the PFC queue mapped by the corresponding QP is in a paused state, the packet cannot be sent continuously, and the corresponding virtual channel of the third virtual channel scheduling module 104 also pauses reading the MR message data; the number of page addresses is the number of page addresses in the page address cache space of the corresponding virtual channel in the second virtual channel scheduling module 103.

[0068] It should be noted that to support the QP-based congestion control required by the RoCEv2 network, such as DCQCN (DataCenter Quantized Congestion Notification), the third virtual channel scheduling module 104 maintains an independent token bucket for each virtual channel. The token bucket is activated when the mapping relationship between the virtual channel and the QP is established, and starts accumulating tokens. The rate of accumulating tokens is calculated by the congestion control algorithm implemented by the network card.

[0069] Preferably, the packet construction module 105 is specifically used for:

[0070] Caching the MR message data in the corresponding MR cache into the shared packet payload cache, and at the same time caching the fields required for constructing the packet in the corresponding SQE into the shared packet descriptor cache in sequence; among them, the fields include: the operation type, immediate number, remote access address, remote access key word, and remote access memory size in the corresponding SQE;

[0071] Constructing a RoCEv2 packet according to the MR message data in the shared packet payload cache, the fields in the shared packet descriptor cache, and the QPC in the corresponding QP, and sending the RoCEv2 packet to the Ethernet interface for forwarding.

[0072] In the actual operation process, the message construction module 105 caches the read MR message data of all virtual channels into the shared message payload cache, and at the same time sequentially caches the message construction related fields such as the operation type, immediate number, remote access address, remote access key, and remote access memory size in the corresponding SQE into the shared message descriptor cache; then the message construction module 105 constructs the header of the RoCEv2 message according to the message construction related fields in the above SQE and the corresponding QPC, including the Ethernet layer 2, layer 3, layer 4 headers and the header of the IB (InfiniBand) protocol, etc., and then fills the MR message data into the IB payload interval of the RoCEv2 message; finally, the constructed RoCEv2 message is forwarded.

[0073] It should be understood that in this application, if the terms "method", "device", "unit" and / or "module" are used, they are only a way to distinguish different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, then the term can be replaced by other expressions.

[0074] As shown in this application and the claims, unless the context clearly indicates an exception, the words "a", "an", "one" and / or "the" etc. do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. A method or device may also include other steps or elements. An element defined by the statement "comprising one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

[0075] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0076] If a flowchart is used in this application, the flowchart is used to illustrate the operations performed by the system according to the embodiments of this application. It should be understood that the previous or subsequent operations do not necessarily need to be executed precisely in sequence. On the contrary, they can be executed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or one or several operations can be removed from these processes.

[0077] The foregoing description of the disclosed embodiments enables those skilled in the art to practice or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A virtual channel scheduling system for RDMA transmission, characterized in that, Including: A relationship establishment module, a first virtual channel scheduling module, a second virtual channel scheduling module, a third virtual channel scheduling module, and a packet construction module; The first virtual channel scheduling module is respectively connected to the relationship establishment module and the second virtual channel scheduling module; the third virtual channel scheduling module is respectively connected to the second virtual channel scheduling module and the packet construction module; the relationship establishment module, the first virtual channel scheduling module, the second virtual channel scheduling module, the third virtual channel scheduling module, and the packet construction module are all connected to the host; The relationship establishment module is used to obtain active QPs according to the Doorbells of each QP and establish a mapping relationship between the active QPs and the network card virtual channels; The first virtual channel scheduling module is used to poll and schedule the active QPs obtained by the relationship establishment module to obtain SQEs from the host and write the SQEs into the SQE cache space; The second virtual channel scheduling module is used to poll and read the SQEs in the first virtual channel scheduling module to obtain physical page addresses from the host and write the physical page addresses into the page address cache space; The third virtual channel scheduling module is used to poll and read the physical page addresses and the SQEs in the second virtual channel scheduling module to obtain MR message data corresponding to the MR cache from the host; The packet construction module is used to construct a RoCEv2 packet according to the QPC of the active QP, the SQE, and the MR message data and forward it; The relationship establishment module is specifically used for: Judging whether the corresponding QP is the active QP according to the WQE type, PI producer index in the Doorbell, and the CI consumer index maintained by the hardware; if so, then: Querying whether the active QP already has a virtual channel mapping; if the active QP already has a mapped virtual channel, there is no need to establish a new mapping relationship, and only the PI value in the hardware cache needs to be updated; if the active QP has no virtual channel mapping, then: Querying whether there is an idle virtual channel; if there is an idle virtual channel, establishing a mapping relationship between the active QP and the idle virtual channel; if there is no idle virtual channel, no operation is performed.

2. The virtual channel scheduling system for RDMA transmission according to claim 1, wherein The relationship establishment module is further used for: when it is detected that the QP corresponding to the virtual channel is destroyed or when the network card has processed the existing SQEs of the virtual channel and there are no new SQEs to be processed within a preset time, deleting the mapping relationship between the virtual channel and the active QP.

3. The virtual channel scheduling system for RDMA transmission according to claim 1, wherein The first virtual channel scheduling module is specifically used for: Calculating the number of SQEs by polling and scheduling virtual channels to obtain the remaining SQE cache space of the virtual channels and the remaining number of SQEs of the QPs corresponding to the virtual channels in the host memory; Reading a specified number of SQEs from the host according to the number of SQEs and writing the SQEs into the SQE cache space.

4. The virtual channel scheduling system for RDMA transmission according to claim 1, wherein The second virtual channel scheduling module is specifically used for: Reading the SQE cache space of the virtual channel by polling and scheduling the virtual channel to obtain the remaining page address cache space of the virtual channel and the remaining length of the SQE message; Calculating the number of page addresses according to the remaining page address cache space and the remaining length of the SQE message; Obtain the physical page address of the MR corresponding to the message data in the SQE from the host according to the number of page addresses, and write the physical page address into the page address cache space of the corresponding virtual channel.

5. The virtual channel scheduling system for RDMA transmission according to claim 4, characterized in that, The third virtual channel scheduling module is specifically configured to: Read the physical page address and the SQE of the virtual channel by polling and scheduling the virtual channel, and simultaneously obtain the PMTU, remaining token count, hardware status, ORT available space, remaining length of the SQE message, PFC backpressure status, and the number of page addresses corresponding to the virtual channel; Calculate the number of packets according to the PMTU, the remaining token count, the hardware status, the ORT available space, the remaining length of the SQE message, the PFC backpressure status, and the number of page addresses corresponding to the virtual channel of the QP corresponding to the virtual channel; Obtain the corresponding MR message data from the host memory according to the number of packets.

6. The virtual channel scheduling system for RDMA transmission according to claim 5, wherein The packet construction module is specifically configured to: Cache the MR message data cached in the corresponding MR into the shared packet payload cache, and simultaneously cache the fields required for constructing packets in the corresponding SQE into the shared packet descriptor cache in sequence; wherein, the fields include: operation type, immediate number, remote access address, remote access key, and remote access memory size in the corresponding SQE; Construct a RoCEv2 packet according to the MR message data in the shared packet payload cache, the fields in the shared packet descriptor cache, and the QPC of the corresponding QP, and send the RoCEv2 packet to the Ethernet interface for forwarding.

Images