Circuit and method for managing registration memory information of RDMA network cards

Abstract

This application relates to the field of RDMA network technology, specifically providing a registration memory information management circuit and method for RDMA network cards, including an arbitration module for responding to MR registration information requests from a target request source; a matching module for receiving and parsing the physical address of the MR registration information request and performing physical address to logical address conversion; a counting module for recording the number of currently occupied MR registration information; a checking module for determining whether the logical address group corresponding to the current queue number is valid; and a storage module for storing MR registration information. This application caches MR registration information from external memory in the storage module. After the arbitration module sends the MR registration information request to the matching module, the matching module, in conjunction with the counting and checking modules, performs physical address to logical address conversion for retrieval, enabling the registration memory information management circuit to promptly prepare MR registration information for receiving data packets.

Description

Technical Field

[0001] This application relates to the field of RDMA network technology, and in particular to a registration memory information management circuit and method for RDMA network cards. Background Technology

[0002] In RDMA (Remote Direct Memory Access) communication, MR (Memory Region) refers to a region of memory allocated by the RDMA software layer for storing transmitted and received data. In the IB protocol, after requesting a memory region for data storage, the user needs to register the MR by calling the API provided by the IB framework before the RDMA network card can access this memory region. The MR is essentially a block of memory requested and used during RDMA communication.

[0003] However, MR registration information is typically stored in external memory (such as DDR SDRAM, Double Data Rate Synchronous Dynamic Random Access Memory). When an RDMA network card sends or receives data packets, it needs to access the external memory to obtain the corresponding MR registration information. However, accessing external memory requires a long waiting time, especially when multiple RDMA operations are running simultaneously. If the MR registration information for receiving data packets cannot be prepared in time, data packets will be dropped, thereby reducing data transmission bandwidth and increasing latency. Therefore, the lack of MR registration information caching capabilities and a reasonable scheduling mechanism can lead to performance bottlenecks in the system due to data packet loss or delay. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a registration memory information management circuit and method for RDMA network cards. By caching the registration information of the MR in the external memory to a high-speed memory, after the arbitration module sends the MR registration information request to the matching module, the matching module, in conjunction with the counting module and the checking module, performs a physical address to logical address conversion for reading, enabling the registration memory information management circuit to promptly prepare the MR registration information for receiving data packets.

[0005] To achieve the objectives of this application, the following technical solution is provided:

[0006] In a first aspect, this application provides a registration memory information management circuit for an RDMA network card, comprising: an arbitration module, a matching module, a counting module, a checking module, a storage module, and an identification module;

[0007] The arbitration module is used to respond to the MR registration information request corresponding to the target request source based on a preset arbitration strategy, and send the MR registration information request to the matching module;

[0008] The matching module is used to parse the physical address in the MR registration information request and determine whether a valid logical address exists in the address mapping table based on the physical address. If the valid logical address exists in the address mapping table, the matching module reads the first logical address corresponding to the physical address from the address mapping table and sends a count increment request to the counting module. If the valid logical address does not exist in the address mapping table, the matching module requests a second logical address from the counting module and requests external MR registration information from the external memory based on the physical address. The second logical address is used to store the external MR registration information in the storage module.

[0009] The counting module is used to record the number of references to the MR registration information cached in the storage module. After the matching module reads the logical address corresponding to the physical address from the address mapping table, it receives a count increment request from the matching module. After the controller reads the logical address corresponding to the physical address from the address mapping table, it receives a count decrement request from the matching module. After performing the count increment or decrement, it updates the counting table. The counting table stores the mapping relationship between the logical address and the physical address of the MR registration information, as well as the reference count.

[0010] The inspection module is used to receive the logical address group corresponding to the current queue number, and when it is determined that all logical addresses in the logical address group are valid, the identification module is used to mark the current queue number as ready, so that the controller can obtain the logical address group through the matching module and read the MR registration information corresponding to the logical address group from the storage module.

[0011] A further improvement of the present invention is that, in the case where the valid logical address does not exist in the address mapping table, the matching module requests a second logical address from the counting module and requests external MR registration information from the external memory based on the physical address, comprising: in the case where the valid logical address does not exist in the address mapping table, the matching module sends a logical address request to the counting module; in response to the logical address request, the counting module allocates a second logical address to the matching module for storing MR registration information in the storage module, and the matching module updates the address mapping table based on the second logical address; the matching module requests the external MR registration information from the external memory based on the physical address, and writes the external MR registration information into the MR registration information memory of the storage module using the second logical address as an index.

[0012] A further improvement of the present invention is that the counting module stores the mapping relationship between the logical address and physical address of the MR registration information and the reference count through a counting table. At the same time, a set of 512-bit flag bits records whether the logical address is occupied. When the reference count is zero, the flag bit is set to zero, and when the reference count is not zero, the flag bit is set to one. When the counting module receives a count increment request or count decrement request from the matching module, it reads the reference count in the counting table according to the logical address determined by the matching module, increments or decrements it by one, and writes it back.

[0013] A further improvement of the present invention is that the depth of the address mapping table is the maximum number of MR registration information stored in the external memory, and the width of the address mapping table is the bit width of the first logical address or the second logical address.

[0014] A further improvement of the present invention is that the inspection module stores the queue number of the request originating from the sending queue and the logical address of its corresponding MR registration information in the storage module through a sending queue mapping table; it stores the queue number of the request originating from the receiving queue and the logical address of its corresponding MR registration information in the storage module through a receiving queue mapping table; and it stores the queue number of the request originating from the unpacking module and the logical address of its corresponding MR registration information in the storage module through an unpacking module mapping table; the sending queue mapping table corresponds to one set of 256-bit flag bits, the receiving queue mapping table corresponds to another set of 256-bit flag bits, and the unpacking module mapping table corresponds to yet another set of 256-bit flag bits, wherein each flag bit corresponds to a queue number; when the inspection module determines whether all logical addresses in the logical address group corresponding to the current queue number are valid, it determines whether the logical address group needs to be checked for validity based on the flag bits; wherein, the current queue number corresponds to... A flag position of 1 indicates that the current queue number needs to be checked for validity, while a flag position of 0 indicates that the current queue number does not need to be checked for validity. When the flag position of the current queue number is 1, all logical addresses of the logical address group corresponding to the current queue number are read through fair polling, and the validity of all logical addresses is checked according to the validity of the logical addresses. The checking module records the validity of each logical address through a set of 512-bit logical address validity bits. Each bit of the logical address validity bit corresponds to a logical address; a valid bit of 1 indicates that the logical address is valid, and a valid bit of 0 indicates that the logical address is invalid. When the checking module receives a logical address validity signal from the storage module, it sets the valid bit of the logical address to 1; when the checking module receives a logical address release signal from the counting module, it sets the valid bit of the logical address to 0.

[0015] A further improvement of the present invention is that the arbitration module is further configured to, upon receiving the arbitration enable signal from the matching module, respond to the MR registration information request from the target request source based on a preset arbitration strategy, and send the MR registration information request to the matching module; the arbitration strategy is to prioritize receiving the MR registration information request from the unpacking module, and to conduct fair round-robin polling of the sending queue and the receiving queue.

[0016] A further improvement of the present invention is that the counting module is also configured to: when the occupancy count of the counting module reaches a saturation threshold, send a storage saturation signal to the arbitration module through the counting module, so that the arbitration module stops receiving any MR registration information request in response to the storage saturation signal; and execute the arbitration strategy when the occupancy count indicates that the occupancy status of the MR registration information is not saturated.

[0017] A further improvement of the present invention is that the MR registration information storage of the storage module uses four BRAMs to store the MR registration information; the inspection module uses a first storage block to store the mapping relationship between the queue number of the sending queue and the corresponding logical address, a second storage block to store the mapping relationship between the queue number of the receiving queue and the corresponding logical address, and a third storage block to store the mapping relationship between the queue number of the unpacking module and the corresponding logical address; wherein, the third storage block is Ultra-Scale RAM; the counting module uses a fourth storage block to store the mapping relationship between the logical address and physical address of the MR registration information and the reference count.

[0018] A further improvement of the present invention is that, after the user ends the current RDMA communication, the physical addresses of all MR registration information used by the user are sent to the matching module. Based on the physical addresses of all MR registration information corresponding to the current RDMA communication, the matching module sets all logical addresses mapped in the address mapping table to 0, that is, releases all logical addresses mapped in the address mapping table.

[0019] A further improvement of the present invention is that the identification module uses three sets of 256-bit identification bits to indicate whether the MR registration information of all queue numbers of the sending queue, receiving queue, and unpacking module is ready.

[0020] Secondly, this application provides a registration memory information management method for an RDMA network card, applied to the aforementioned registration memory information management circuit for an RDMA network card, the method comprising:

[0021] The arbitration module responds to the MR registration information request corresponding to the target request source based on a preset arbitration strategy, and sends the MR registration information request to the matching module;

[0022] The matching module parses the physical address in the MR registration information request and determines whether a valid logical address exists in the address mapping table based on the physical address. If the valid logical address exists in the address mapping table, the matching module reads the first logical address corresponding to the physical address from the address mapping table, and sends a count increment request to the counting module. If the valid logical address does not exist in the address mapping table, the matching module requests a second logical address from the counting module and requests external MR registration information from the external memory based on the physical address. The second logical address is used to store the external MR registration information in the storage module.

[0023] After the physical addresses of all MR registration information for the current queue number are obtained through the matching module to obtain the corresponding logical addresses, the matching module sends the logical address group corresponding to the current queue number to the inspection module.

[0024] When the inspection module determines that all logical addresses in the logical address group are valid, the identification module marks the current queue number as ready, so that the controller can obtain the logical address group through the matching module and read the MR registration information corresponding to the logical address group from the storage module.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The registration memory information management circuit and method for RDMA network cards provided in this application cache the registration information of MR in the external memory to the high-speed memory. After the arbitration module sends the MR registration information request to the matching module, the matching module, together with the counting module and the checking module, performs the conversion of physical address to logical address for reading. This enables the registration memory information management circuit to prepare the MR registration information for receiving data packets in a timely manner, avoiding performance bottlenecks that may be caused by data packet loss or delay. Attached Figure Description

[0027] The accompanying drawings are provided to further understand this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof.

[0028] Figure 1 A schematic block diagram of the overall circuit of the memory management system for an RDMA network card provided in an embodiment of this application;

[0029] Figure 2 A schematic block diagram of the arbitration module circuit for a memory management system of an RDMA network card provided in an embodiment of this application;

[0030] Figure 3A schematic block diagram of the matching module circuit for a memory management system of an RDMA network card provided in an embodiment of this application;

[0031] Figure 4 A schematic block diagram of a checking module circuit for a memory management system of an RDMA network card provided in an embodiment of this application;

[0032] Figure 5 A schematic block diagram of a counting module circuit for a memory management system for an RDMA network card, provided in an embodiment of this application;

[0033] Figure 6 A schematic block diagram of the identification module circuit for a memory management system of an RDMA network card provided in an embodiment of this application;

[0034] Figure 7 A schematic block diagram of the storage module circuit of the memory management system for an RDMA network card provided in an embodiment of this application;

[0035] Figure 8 A schematic flowchart of an optional memory management method for an RDMA network card provided in an embodiment of this application;

[0036] Figure 9 The topology for RDMA network card testing provided in the embodiments of this application;

[0037] Figure 10 A comparison chart of Send operation bandwidth tests for the memory management method for RDMA network cards provided in the embodiments of this application;

[0038] Figure 11 A comparison chart of RDMA Write operation bandwidth tests for the memory management method for RDMA network cards provided in the embodiments of this application;

[0039] Figure 12 A comparison chart of RDMA Read operation bandwidth tests for the memory management method for RDMA network cards provided in the embodiments of this application;

[0040] Figure 13 Comparison chart of Send operation latency test for the memory management method for RDMA network card provided in the embodiments of this application;

[0041] Figure 14 Comparison chart of RDMA Write operation latency test for the memory management method for RDMA network cards provided in the embodiments of this application;

[0042] Figure 15 Comparison chart of RDMA Read operation latency test for the memory management method for RDMA network cards provided in the embodiments of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature; in the description of this application, unless otherwise stated, "multiple" means two or more.

[0045] The following is combined with Figures 1 to 15 The illustrated embodiments describe the technical solution of the present invention:

[0046] Example 1

[0047] Reference Figure 1 As shown in the figure, a registration memory information management circuit for an RDMA network card according to an embodiment of this application includes: an arbitration module, a matching module, a counting module, a checking module, an identification module, and a storage module.

[0048] Reference Figure 2 As shown, the arbitration module responds to the MR registration information request corresponding to the target request source based on a preset arbitration strategy, and sends the MR registration information request to the matching module, which then performs the conversion from the physical address to the logical address of the MR registration information. In this embodiment, the preset arbitration strategy is to prioritize receiving the MR registration information request from the unpacking module, and to conduct fair round-robin polling of the sending and receiving queues. That is, after the arbitration module receives the arbitration enable signal from the matching module, when the sending queue, receiving queue, and unpacking module all need to request MR registration information, they use the arbitration strategy to arbitrate the different MR registration information requests received from these three sources, arbitrate the target request source that receives the request first (prioritizing receiving the MR registration information request from the unpacking module, and conducting fair round-robin polling of the sending and receiving queues), and send the MR registration information request from this target request source to the matching module.

[0049] In this embodiment of the application, the MR registration information request is used to request the reading of the MR registration information corresponding to the target request source.

[0050] Reference Figure 3As shown, the matching module, upon receiving a MapReduce (MR) registration information request from a target request source, parses the physical address in the MR registration information request and determines whether a valid logical address exists in the address mapping table within the matching module based on the physical address. This address mapping table stores the mapping relationship between the physical address of the MR registration information in external memory and its logical address in the storage module. In this embodiment, if the matching module determines that a valid logical address exists in the address mapping table, it reads the first logical address corresponding to the physical address of the MR registration information request from the address mapping table and notifies the counting module to increment the number of work queues using the MR registration information. If the matching module determines that no valid logical address exists in the address mapping table, it requests a second logical address from the counting module and fills the allocated logical address into the address mapping table with the physical address as the address in the address mapping table. It then notifies the counting module to increment the number of work queues using the MR registration information. The matching module requests external MR registration information from external memory based on the physical address in the MR registration information request and stores the external MR registration information in the storage module using the second logical address. Thus, the matching module completes the physical-to-logical conversion for reading, and reading triggers a decrement of the reference count. Among them, a valid logical address refers to a logical address that has been legally bound to the physical address of the MR registration information in the address mapping table of the matching module, and whose corresponding MR registration information can be accessed normally by the current request.

[0051] In this embodiment, the depth of the address mapping table is the maximum number of MR registration information stored in the external memory, and the width of the address mapping table is the bit width of the logical address. All logical addresses in the address mapping table are initially 0, meaning a logical address of 0 indicates that the mapping relationship does not exist. As a feasible implementation, the external memory in this embodiment can store up to 16384 MR registration information entries, and the storage module can store up to 512 MR registration information entries. The corresponding logical address is 9 bits, therefore the size of the address mapping table is 16384 * 9 bits. Four BRAMs (Block RAM, high-speed memory, a type of dedicated memory inside an FPGA chip) can be used to store the mapping relationship between physical addresses and logical addresses.

[0052] In this embodiment, when all MR registration information requested by queue number A in the matching module is allocated to logical addresses, the characterization storage module can send the logical addresses corresponding to the registration information of all MRs in queue number A, along with the source information of the request, to the inspection module by either caching the MR registration information or reserving space for the MR registration information. As a feasible implementation, a single request in this embodiment can request registration information for up to six MRs.

[0053] Reference Figure 4As shown, the inspection module is used to receive the logical address group corresponding to the current queue number, and when it is determined that all logical addresses in the logical address group are valid, the identification module marks the current queue number as ready, so that the controller can obtain the logical address group through the matching module and read the MR registration information corresponding to the logical address group from the storage module.

[0054] In this embodiment, the inspection module includes a sending queue mapping table, a receiving queue mapping table, and an unpacking module mapping table. The sending queue mapping table stores the request queue number from the sending queue source and the logical address of the corresponding MR registration information in the storage module. The receiving queue mapping table stores the request queue number from the receiving queue source and the logical address of the corresponding MR registration information in the storage module. The unpacking module mapping table stores the request queue number from the unpacking module source and the logical address of the corresponding MR registration information in the storage module. Each of these three mapping tables has a set of 256-bit flag bits, with each bit corresponding to a queue number. A flag bit set to 1 indicates that the queue number needs to be checked for logical address validity; otherwise, it does not. In this embodiment, when the checking module receives a set of logical addresses and their source information from the matching module, it stores these logical addresses in the corresponding mapping table according to the source information. After storage, it sets the corresponding flag bit to 1 (a flag bit of 1 indicates that the queue number needs to check whether the logical address is valid). Therefore, when it is necessary to check whether the logical address is valid, the logical address of the queue number with the flag bit set to 1 is read through fair polling, and the validity of these logical addresses is checked according to the validity of the read logical addresses. Specifically, the checking module records whether the logical address is valid through a set of 512 logical address valid bits, with each bit corresponding to a logical address. When the checking module receives a logical address valid signal from the storage module, it sets the valid bit of the logical address to 1. When the checking module receives a logical address release signal from the counting module, it sets the valid bit of the logical address to 0. Therefore, when it is determined through the sending queue mapping table, the receiving queue mapping table, or the unpacking module mapping table that the queue number B needs to check whether the logical address is valid, it judges the logical address b corresponding to the logical address valid bit of the queue number B. If the valid bit is set to 1, it means that the logical address b is valid; otherwise, the logical address b is invalid.

[0055] In this embodiment, when all logical addresses of a queue number in the send queue mapping table or receive queue mapping table are valid, the checking module sends the queue number and source information to the identification module, and sets the flag of the queue number to zero. When all logical addresses of a queue number in the unpacking module mapping table are valid, the checking module sends the queue number and source information to the identification module, and when the read and write pointers of a queue number in the unpacking module mapping table are the same, the flag of the queue number is set to zero. Here, this embodiment sets a read pointer and a write pointer (both the initial values ​​of the read pointer and the write pointer are zero) for each queue number in the unpacking module mapping table. When a logical address is written to a queue number, the queue number is used as the high-order bit and the write pointer as the low-order bit as the write address of the logical address, and then the write pointer is incremented by one; when the controller reads the registration information of the MR through the logical address of the queue number, the read pointer is incremented by one.

[0056] As a feasible implementation, this embodiment uses a first storage block (BRAM) to store the mapping relationship between the queue number of the sending queue and its corresponding logical address (sending queue mapping table), and a second storage block (BRAM) to store the mapping relationship between the queue number of the receiving queue and its corresponding logical address (receiving queue mapping table). Since the unpacking module may send multiple MR registration information requests for a certain queue number within a short period, the unpacking module mapping table requires a URAM (Ultra-Scale RAM, a dedicated storage within UltraScale Plus series FPGA chips, i.e., a third storage block) to store the mapping relationship between the queue number and the logical address. Simultaneously, since the unpacking module requests only one MR registration information per request, the width of the unpacking module mapping table is the bit width of one logical address, and the depth is 32768. Therefore, the size of the unpacking module mapping table is 32768 * 9 bits. The unpacking module mapping table is allocated to one queue number every 128 rows from low to high, and one queue number can store a maximum of 128 mapping relationships with logical addresses simultaneously. Here, since MR registration information requests for the next MR will not be sent if the MR registration information for the same queue number in both the send and receive queues is not read, the depth of both the send queue mapping table and the receive queue mapping table is the number of queue numbers, and the width of these two mapping tables is the width of six logical addresses. Therefore, the size of these two mapping tables is 256*54 bits. Here, URAM constructs a 32768×9-bit table, divides it into blocks by queue (maximum 128 items per queue), and sets independent read / write pointers.

[0057] Reference Figure 5As shown, the counting module is used to receive a count increment request from the matching module after the matching module reads the logical address corresponding to the physical address from the address mapping table, and to receive a count decrement request from the matching module after the controller reads the logical address corresponding to the physical address from the address mapping table, and to update the counting table after performing the count increment or decrement; wherein, the counting table stores the mapping relationship between the logical address and the physical address of the MR registration information, and the reference count.

[0058] In this embodiment, the counting module stores the mapping relationship between the logical address and physical address of the MR registration information and the reference count through a counting table. It also uses a 512-bit flag to record whether a logical address is occupied (the flag is set to zero when the reference count is zero, and set to one when the reference count is not zero). Specifically, the depth of the counting table is the maximum amount of MR registration information that the storage module can store; that is, the counting table records the physical address and reference count for each logical address, and its width is 64 bits. Bit 64 represents the valid physical address, bits 63 to 50 represent the physical address, and bits 49 to 1 represent the reference count. In this embodiment, the size of the counting table is 512 * 64 bits, and a single BRAM (fourth storage block) can be used to store the mapping relationship between the logical address and physical address of the MR registration information and the reference count.

[0059] As one possible implementation, the counting module records the number of occupied logical addresses using a 9-bit occupancy counter.

[0060] In this embodiment, the counting module first finds and prepares logical addresses with a flag bit of zero using a fair polling method. When the counting module receives a logical address request from the matching module, it allocates the prepared logical address to the matching module, sets the flag bit of that logical address to one, and simultaneously finds the next logical address with a flag bit of zero using a fair polling method, prepares it in advance, sets the physical address validity bit and reference count to one, and writes the physical address requesting the logical address, along with the physical address, into the counting table. Using the allocated logical address as the address, the counting module reads the physical address validity bit and the physical address from the counting table. If the physical address validity bit is one, it means that the logical address was previously allocated to that physical address, and the logical address of that physical address needs to be released from the matching module. Simultaneously, the counting module sets the logical address validity bit of the checking module to zero. If the physical address validity bit is zero, it means that the logical address was not previously allocated to any physical address, and the logical address does not need to be released. Therefore, when the counting module receives a count increment or decrement request from the matching module, it reads the reference count from the counting table based on the logical address obtained from the matching module, increments or decrements it, and writes it back. It should be noted that if a count increment request and a count decrement request are received in the same clock cycle, or in two consecutive clock cycles, and both are counting requests for the same logical address, no counting operation is required to avoid write-write conflicts and read-write conflicts. Therefore, through logical address pre-allocation and conflict self-release mechanism, the table entries are read back during allocation, and if a historical address is detected as occupied, the original mapping is automatically released and the valid bit is cleared.

[0061] As a feasible implementation, the occupancy count saturation threshold of the counting module in this embodiment is 506. When the occupancy count of the counting module reaches the saturation threshold, the counting module sends a storage saturation signal to the arbitration module, indicating that there are not enough logical addresses available for allocation and that it is necessary to wait for the MR registration information controller to read a small amount of MR registration information until the occupancy count has not reached the saturation threshold, at which point the sending of the storage saturation signal stops.

[0062] Reference Figure 6 As shown, the identification module uses three sets of 256-bit flags to indicate whether the MR registration information for all queue numbers in the sending queue, receiving queue, and unpacking module is ready. When the identification module receives the queue number and source information from the inspection module, it means that all MR registration information corresponding to the current queue number is valid, and the corresponding flag is set to 1; when the identification module receives the queue number and source information from the MR registration information controller, the corresponding flag is set to zero.

[0063] Reference Figure 7As shown, the storage module is used to store the registration information of MRs through the MR registration information memory. In this embodiment, the MR registration information memory uses four BRAMs to store the MR registration information. The storage module can store a maximum of 512 MR registration information entries, each with a bit width of 256 bits. Therefore, the size of the MR registration information memory is 512 * 256 bits. When the storage module receives MR registration information requested from the external memory by the matching module, it stores it in the MR registration information memory with a logical address. When the storage module receives the logical address obtained by the controller from the matching module, it sends the MR registration information to the controller through the MR registration information memory.

[0064] The registration memory information management circuit for RDMA network cards in this embodiment caches the registration information of MR in the external memory to the high-speed memory. After the arbitration module sends the MR registration information request to the matching module, the matching module, together with the counting module and the checking module, performs the conversion of physical address to logical address for reading. This enables the registration memory information management circuit to prepare the MR registration information for receiving data packets in a timely manner, avoiding performance bottlenecks that may be caused by data packet loss or delay.

[0065] Example 2

[0066] Based on the foregoing embodiments, referring to Figure 8 As shown, this application provides a registration memory information management method for an RDMA network card, applied to the registration memory information management circuit of the above embodiment 1, including the following:

[0067] Step S201: The matching module provides an arbitration enable signal to the arbitration module;

[0068] Step S202: The arbitration module interacts with the sending queue, receiving queue and unpacking module using a three-way handshake mechanism. When the three need to request MR registration information, they need to send a request to the arbitration module. After receiving the arbitration enable signal from the matching module, the arbitration module decides which request to respond to according to the arbitration strategy.

[0069] The MR registration information request includes the physical address of the MR registration information in external memory and the source information of the request. The request sources are divided into three types: the sending queue, the receiving queue, and the unpacking module, with each source having 256 queue numbers. The arbitration strategy used in this embodiment prioritizes receiving MR registration information requests from the unpacking module, while the sending and receiving queues are round-robin fairly. Each of the three queues sends the required MR registration information's physical address in external memory and its corresponding queue number to the arbitration module after receiving a response. When the arbitration module receives a storage saturation signal from the counting module, it means that all MR registration information stored in the storage module is occupied, and there is no space to store new MR registration information. Regardless of whether arbitration is enabled, it will no longer receive MR registration information requests.

[0070] In this way, this embodiment uses a priority scheduling mechanism to prioritize the execution of the MR registration information request of the unpacking module, ensuring that the unpacking module will not be discarded before the MR registration information is ready. This avoids the loss of data packets due to the failure to prepare the MR registration information in time, thereby improving the stability of data transmission and reducing the latency of data transmission.

[0071] Step S203: The matching module determines whether the MR registration information has been cached in the storage module based on the physical address; if it has not been cached, the counting module allocates a logical address for storing the MR registration information in the storage module and requests the MR registration information from the external memory based on the physical address; if it has been cached, the matching module sends a count increment request for the logical address to the counting module; when the physical addresses of all the MR registration information required by a certain queue number are obtained through the matching module, these logical addresses along with the queue number are sent to the checking module.

[0072] Thus, this embodiment supports caching of MR registration information, caching the MR registration information of external memory (such as DDR SDRAM) to high-speed memory (such as BRAM), thereby avoiding the need for frequent access to external memory, effectively reducing storage access latency, speeding up data access, and improving the bandwidth of RDMA operation.

[0073] The matching module uses an address mapping table to store the mapping relationship between the physical address of the MR registration information in external memory and its logical address in the storage module. The depth of this address mapping table is the maximum number of MR registration information entries that can be stored in external memory. In other words, the address mapping table provides a space for storing the logical address for each physical address of MR registration information. In this embodiment, the external memory can store a maximum of 16384 MR registration information entries, and the width of the address mapping table is the bit width of the logical address. In this embodiment, the storage module can store a maximum of 512 MR registration information entries, corresponding to a 9-bit logical address. Therefore, the size of the address mapping table is 16384 * 9 bits. Four BRAMs (Block RAM, a high-speed memory, a type of dedicated memory inside an FPGA chip) can be used to store the mapping relationship between physical and logical addresses. All logical addresses in the address mapping table are initially set to 0, meaning a logical address of 0 indicates that the mapping relationship does not exist.

[0074] When the matching module receives a MR registration information request, it checks the address mapping table to see if a logical address matches it. If a mapping exists, it notifies the counting module to increment the number of work queues using the MR registration information. If no mapping exists, it requests a logical address from the counting module, fills the address mapping table with the allocated logical address as its physical address, notifies the counting module to increment the number of work queues using the MR registration information, and requests MR registration information from external memory using the physical address. The requested MR registration information is then stored in the storage module using the logical address. When the storage module receives MR registration information from the matching module, it stores it in the MR registration information storage using the logical address. When all MR registration information requested by a queue number is allocated a logical address, it means the storage module has either cached or reserved space for MR registration information. It then sends these logical addresses, along with the source information of the requests, to the checking module. When the storage module receives a logical address from the controller through the matching module, it sends the MR registration information to the controller via the MR registration information storage.

[0075] When the controller needs to read the MR registration information, it converts the physical address to a logical address through the address mapping table in the matching module and reads the MR registration information from the storage module. When reading the MR registration information, it notifies the counting module to decrement the number of work queues using the MR registration information by one.

[0076] When the matching module receives a logical address release signal from the counting module, it sets the logical address mapped to the corresponding physical address in the address mapping table to 0. When the user ends RDMA communication, it sends the physical addresses of all MR registration information used by the user to the matching module. The matching module sets all logical addresses mapped in the address mapping table to 0 based on these physical addresses to avoid misreading of MR information across sessions.

[0077] Step S204: Check the module record and check whether the MR registration information is valid. When the MR registration information is stored in the storage module according to the logical address, check the valid position of the corresponding logical address in the module.

[0078] If all the MR registration information required for a certain queue number is valid, the identification module marks the queue number as ready, thereby notifying the controller. The controller then uses the matching module to convert the physical address to a logical address and read the MR registration information from the storage module. When reading the MR registration information, a decrement request for the logical address is sent to the counting module. In this embodiment, the counting module stores the mapping relationship between the logical address and physical address of the MR registration information, as well as the reference count, through a counting table. The depth of the counting table is the maximum number of MR registration information entries that the storage module can store. The counting module uses a set of 512 flag bits to record whether a logical address is occupied. When the reference count is zero, the flag is set to zero; when the reference count is not zero, the flag is set to one. Simultaneously, the counting module also uses a 9-bit occupancy count bit to record the number of occupied logical addresses. In this embodiment, the counting module first finds logical addresses with a flag of zero in advance through fair polling. When the counting module receives a logical address request from the matching module, it allocates the pre-prepared logical address to the matching module and sets the flag of that logical address to one. Furthermore, a fair polling method is used to find the next logical address with a zero flag in advance, and the physical address validity bit and reference count are set to one. This, along with the physical address requesting the logical address, is written to the counting table. Using the allocated logical address as the address, the counting table is read to retrieve the physical address validity bit and the physical address itself. If the physical address validity bit is one, it means the logical address was previously allocated to that physical address, and the logical address in the matching module needs to be released. Simultaneously, the logical address validity bit in the checking module is set to zero. If the physical address validity bit is zero, it means the logical address was not previously allocated to any physical address, and the logical address does not need to be released. When the counting module receives a count increment or decrement request from the matching module, it reads the reference count from the counting table based on the logical address obtained from the matching module, increments or decrements it, and writes it back. If a count increment and decrement request are received in the same clock cycle or in two consecutive clock cycles, and both are for the same logical address, no counting operation is required to avoid write-write and read-write conflicts. When the occupancy count of the counting module reaches the saturation threshold, a storage saturation signal is sent to the arbitration module, indicating that there are no more logical addresses available for allocation and that the controller needs to read a small amount of MR registration information. If the occupancy count does not reach the saturation threshold, the storage saturation signal is stopped.

[0079] Here, the counting table in this embodiment contains a 64-bit entry of "physical address valid bit + physical address + reference count", along with a 512-bit occupancy flag and a 9-bit occupancy count.

[0080] In this embodiment, the inspection module stores the queue numbers and MR registration information of the three request sources in the logical addresses of the storage module through a send queue mapping table, a receive queue mapping table, and an unpacking module mapping table, respectively. The inspection module records the validity of a logical address using a set of 512-bit valid bits. Each bit corresponds to a logical address; setting a valid bit to 1 indicates that the logical address is valid, and vice versa. When the inspection module receives a valid signal for a logical address from the storage module, it sets the valid bit to 1. When the inspection module receives a logical address release signal from the counting module, it sets the valid bit to 0.

[0081] Therefore, when the inspection module receives a set of logical addresses and source information from the matching module, it stores these logical addresses in the corresponding mapping table according to the source information and sets the corresponding flag bit to 1. When a set of flag bits is not all zero, meaning there is a queue number that needs to be checked for logical address validity, the logical address of the queue number with the flag bit set to 1 needs to be read through fair polling, and the validity of these logical addresses is checked according to the logical address validity bit. When all logical addresses of a queue number in the send queue mapping table or receive queue mapping table are valid, the queue number and source information are sent to the identification module, and the flag bit of that queue number is set to zero. When all logical addresses of a queue number in the unpacking module mapping table are valid, the queue number and source information are sent to the identification module. When the read and write pointers of a queue number in the unpacking module mapping table are the same, the flag bit of that queue number is set to zero.

[0082] Step S205: The identification module uses three sets of 256-bit identification bits to indicate whether the MR registration information of all queue numbers in the sending queue, receiving queue, and unpacking module is ready. If the registration information of all MRs required for a certain queue number is valid, the identification module marks the queue number as ready and notifies the controller.

[0083] Specifically, when the identification module receives the queue number and source information from the inspection module, it sets the corresponding flag to one. When the identification module receives the queue number and source information from the controller, it sets the corresponding flag to zero.

[0084] Step S206: The controller converts the physical address to a logical address through the matching module and reads the MR registration information from the storage module; the storage module adopts a 512×256bit BRAM memory design.

[0085] Specifically, when reading the registration information of MR, a count decrement request for the logical address is sent to the counting module.

[0086] The counting module records the number of MR registration information that is being occupied by allocating logical addresses and counting requests (a count greater than zero is considered occupied, which can be understood as irreplaceable). When the number of occupied items reaches the saturation threshold, the arbitration module is notified to stop receiving requests.

[0087] Additionally, when the cumulative number of applied MR registration information exceeds the maximum storage capacity of the storage module, but the number of occupied MR registration information is not saturated, the logical address allocated by the counting module may also be allocated to other MR registration information before this (provided that the MR registration information allocated before this was not occupied). In this case, it is necessary to release the previously allocated logical address, which can be achieved by clearing the logical address of the matching module and setting the valid address of the logical address in the checking module to zero.

[0088] Step S207: When the user ends RDMA communication, clear the logical address assigned by the MR registration information used by the user in the matching module.

[0089] This prevents the controller from reading incorrect MR registration information when the user restarts RDMA communication, as the MR registration information for the same physical address may differ between the two RDMA communications. Therefore, by releasing resources, the cached old MR registration information can be released promptly when the user ends RDMA communication, ensuring that the circuit will not read incorrect MR registration information due to the user restarting RDMA communication, effectively avoiding data consistency issues and ensuring data accuracy during reading.

[0090] Example 3

[0091] Based on the foregoing embodiments, this embodiment provides system verification. Using a 100G smart network card (hereinafter referred to as "self-developed network card") equipped with the RDMA registration memory information management circuit of the foregoing embodiments, this embodiment verifies whether the RDMA registration memory information management circuit of this embodiment meets the performance requirements.

[0092] The test environment was built using NVIDIA Mellanox ConnectX-6 smart network cards and self-developed network cards. The test topology is as follows: Figure 9 As shown, two CX-6 network cards and two self-developed network cards are used, connected to the server host through the PCIe3x16 interface. All of them support SR-IOV technology to create multiple virtual machines for communication. They are connected to the intermediate switch through 100G fiber to build a data transmission link.

[0093] The intelligent network interface card (NIC) testing platform built in this embodiment consists of a first server host equipped with a Xilinx development platform FPGA development board VCU118, which is connected to an FH-S6900 switch via optical fiber; a second server host equipped with a Mellanox CX6 NIC, which is connected to the FH-S6900 switch via cable; a third server host with the same configuration information and connection relationship as the first server host; and a fourth server host with the same configuration information and connection relationship as the second server host.

[0094] For performance testing, the third-party testing tool perftest was selected. perftest is an open-source testing tool for RDMA operations, which includes four types of RDMA operations: Send, Receive, RDMA Read, and RDMA Write. It also supports bandwidth and latency testing.

[0095] The most important metrics in RDMA communication are bandwidth and latency. perftest includes bandwidth and latency testing programs. During bandwidth testing, perftest issues a task, iterates through bytes 21-223, supports single-queue data transmission as well as multi-queue data transmission, and tests bandwidth for different queue numbers and different byte counts. During latency testing, taking RDMAWrite as an example, the sending end (perftest) fills the task payload with specified data. The receiving end compares the received payload content; if the data verification is correct, it notifies the sending end (perftest), calculates the latency of one RDMA Write operation, then reverses direction and continues sending. After multiple iterations, the average latency is taken to obtain the RDMA Write operation latency.

[0096] Different RDMA operations were tested using the perftest tool and compared with the Mellanox CX6 network card. The performance test results are as follows.

[0097] 1) Bandwidth test

[0098] like Figure 10 The chart shows a comparison of send / receive operation bandwidth tests. The self-developed network card achieved a rate of 80.12Gbps at 4K bytes and a maximum bandwidth of 88.54Gbps at 128K bytes. Because the data burst on the receiving side exceeded the QDMA data upload rate used in this design, the receive buffer was filled, causing flow control to activate, resulting in a bandwidth drop at 1M bytes.

[0099] like Figure 11 The chart shown is a comparison of bandwidth tests for RDMA Write operations. The self-developed network card achieves a rate of 86Gbps when transmitting 4K bytes, with a maximum bandwidth of 89Gbps.

[0100] like Figure 12 The chart shown is a comparison of bandwidth tests for RDMA Read operations. The self-developed network card achieved a speed of 80Gbps when transmitting 128K bytes. This is because the self-developed network card processes task requests received by the passive end one by one in a queue, so the speed is not as high as that of the RDMA Write operation that actively sends data.

[0101] 2) Latency Test

[0102] like Figure 13 The image shows a comparison of Send operation latency. It can be seen that the lowest average latency of the self-developed smart network card under Send operation is 5.22us, while the top-of-the-line Mellanox CX6 network card has a speed of 1.57us.

[0103] like Figure 14 The image shows a comparison of RDMA Write operation latency. Under RDMA Write operation, the minimum latency of Mellanox CX6 network card is 1.50us, and the minimum average latency of self-developed smart network card is 3.25us, which is smaller than the 5.22us latency of Send bilateral operation.

[0104] like Figure 15 The image shows a comparison of RDMA Read operation latency. Under RDMA Read operation, the lowest average latency of the self-developed smart network card is 5.77us, while the minimum latency of the Mellanox CX6 network card is 2.11us.

[0105] During the comparative testing of the self-developed network card and the Mellanox CX6 network card using the perftest testing tool, the self-developed network card was weaker than the CX6 network card in terms of both speed and latency. The fundamental reason is that the CX6 network card is designed using the Application Specific Integrated Circuit (ASIC) method, while the self-developed network card is implemented using FPGA, and there is still room for improvement in terms of clock frequency and storage speed.

[0106] The performance was tested using the perftest testing tool, and the test results were analyzed. The embodiments of this application meet the performance requirements of the self-developed network card.

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of circuits, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0108] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0109] It should be noted that although several modules or units of the device for performing actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed systems, modules, and methods can be implemented in other ways. For example, the module embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between modules or units, and may be electrical, mechanical, or other forms.

[0111] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. This application is not limited to the exact structures described above and illustrated in the accompanying drawings, and it should not be considered that the specific implementation of this application is limited to these descriptions. For those skilled in the art, various changes and modifications made without departing from the concept of this application should be considered to fall within the protection scope of this application.

Claims

1. A register memory information management circuit for an RDMA network card, comprising: The application relates to a register information management method and device. The arbitration module is used for responding to an MR registration information request corresponding to a target request source based on a preset arbitration strategy and sending the MR registration information request to the matching module. The matching module is used for analyzing a physical address in the MR registration information request and judging whether an effective logical address exists in an address mapping table based on the physical address; in the case that the effective logical address exists in the address mapping table, a first logical address corresponding to the physical address is read from the address mapping table, and the matching module sends a count-increment request to the counting module; in the case that the effective logical address does not exist in the address mapping table, the matching module applies for a second logical address from the counting module and applies for external MR registration information from an external memory based on the physical address, and then writes the external MR registration information into an MR registration information memory of the storage module by taking the second logical address as an index; wherein the second logical address is used for storing the external MR registration information in the storage module. The counting module is used for recording the reference times of the MR registration information buffered in the storage module, receiving the count-increment request of the matching module after the matching module reads the logical address corresponding to the physical address from the address mapping table, receiving the count-decrement request of the matching module after the controller reads the logical address corresponding to the physical address from the address mapping table, and updating a count table after the count-increment or the count-decrement is executed; wherein the count table saves the mapping relationship between the logical address and the physical address of the MR registration information and the reference count. The checking module is used for receiving a logical address group corresponding to a current queue number and identifying the current queue number as a ready state by using the identification module in the case that all logical addresses of the logical address group are effective, so that the controller acquires the logical address group by the matching module and reads the MR registration information corresponding to the logical address group from the storage module. In the case that the effective logical address does not exist in the address mapping table, the matching module sends a logical address application request to the counting module, the counting module allocates a second logical address for storing the MR registration information in the storage module to the matching module in response to the logical address application request, and the matching module updates the address mapping table based on the second logical address.

2. The register memory information management circuit for an RDMA network interface card of claim 1, wherein, The counting module sends a count-increment request to the matching module after the matching module reads the logical address corresponding to the physical address from the address mapping table, and the counting module receives the count-decrement request of the matching module after the controller reads the logical address corresponding to the physical address from the address mapping table. ​ ​ The matching module applies for the external MR registration information from the external memory based on the physical address, and writes the external MR registration information into the MR registration information storage of the storage module with the second logical address as an index; The depth of the address mapping table is the maximum number of MR registration information stored in the external memory, and the width of the address mapping table is the bit width of the first logical address or the second logical address.

3. The register memory information management circuit for an RDMA network card of claim 2, wherein, The count module saves the mapping relationship between the logical address and the physical address of the MR registration information and the reference count through a count table, and records whether the logical address is occupied through a group of 512-bit flag bits, wherein the flag bit is zero when the reference count is zero, and the flag bit is one when the reference count is not zero; When the count module receives the count-increment request or the count-decrement request from the matching module, the reference count in the count table is read according to the logical address determined by the matching module, and is written back after being incremented or decremented.

4. The register memory information management circuit for an RDMA network interface card of claim 1, wherein, The check module saves the queue number of the sending queue and the logical address of the corresponding MR registration information in the storage module through the sending queue mapping table; saves the queue number of the receiving queue and the logical address of the corresponding MR registration information in the storage module through the receiving queue mapping table; saves the queue number of the unpacking module and the logical address of the corresponding MR registration information in the storage module through the unpacking module mapping table; the sending queue mapping table corresponds to a group of 256-bit flag bits, the receiving queue mapping table corresponds to another group of 256-bit flag bits, and the unpacking module mapping table corresponds to another group of 256-bit flag bits, wherein each flag bit corresponds to a queue number; When the check module judges whether all logical addresses of the logical address group corresponding to the current queue number are valid, the check module determines whether the logical address group needs to be checked for validity based on the flag bit; wherein the flag bit corresponding to the current queue number is one, indicating that the current queue number needs to be checked for validity, and the flag bit corresponding to the current queue number is zero, indicating that the current queue number does not need to be checked for validity; In the case that the flag bit corresponding to the current queue number is one, all logical addresses of the logical address group corresponding to the current queue number are read through fair polling, and whether the all logical addresses are valid is checked according to the logical address validity; wherein the check module records whether each logical address is valid through a group of 512-bit logical address validity bits; wherein each bit of the logical address validity bit corresponds to a logical address, and the valid bit is one, indicating that the logical address is valid, and the valid bit is zero, indicating that the logical address is invalid; wherein when the check module receives a logical address validity signal from the storage module, the valid bit of the logical address is one; when the check module receives a logical address release signal from the count module, the valid bit of the logical address is zero.

5. The register memory information management circuit for an RDMA network interface card of claim 1, wherein, The arbitration module is further configured to, after receiving the arbitration enable signal from the matching module, respond to a MR registration information request from a target request source based on a preset arbitration strategy, and send the MR registration information request to the matching module. The arbitration strategy is to preferentially receive the MR registration information request from the unpacking module, and to perform fair polling on the sending queue and the receiving queue.

6. The register memory information management circuit for an RDMA network card of claim 5, wherein, The counting module is further configured to: In a case where the occupation count of the counting module reaches a saturation threshold, send a storage saturation signal to the arbitration module via the counting module, so that the arbitration module stops receiving any MR registration information request in response to the storage saturation signal; In a case where the occupation count indicates that the occupation status of the MR registration information is not saturated, execute the arbitration strategy.

7. The register memory information management circuit for an RDMA network card of claim 6, wherein, The identification module uses three groups of 256-bit identification bits to respectively indicate whether the MR registration information of all queue numbers of the sending queue, the receiving queue, and the unpacking module is ready.

8. The registered memory information management circuit for an RDMA network interface card of any of claims 1-7, wherein, The MR registration information storage of the storage module uses four BRAMs to store the registration information of the MRs. The checking module uses a first storage block to store the mapping relationship between the queue numbers of the sending queue and the corresponding logical addresses, uses a second storage block to store the mapping relationship between the queue numbers of the receiving queue and the corresponding logical addresses, and uses a third storage block to store the mapping relationship between the queue numbers of the unpacking module and the corresponding logical addresses; the third storage block is an Ultra-Scale RAM. The counting module uses a fourth storage block to store the mapping relationship between the logical addresses and the physical addresses of the MR registration information and the reference count.

9. The register memory information management circuit for an RDMA network card of claim 8, wherein, After the user ends the current RDMA communication, the matching module is sent the physical addresses of all MR registration information used by the user, and the matching module releases all corresponding logical addresses in the address mapping table based on the received physical addresses of all MR registration information corresponding to the current RDMA communication.

10. A method for managing registration memory information of an RDMA network card, applied to the circuit for managing registration memory information of an RDMA network card in any of claims 1-9, characterized in that, The arbitration module responds to a MR registration information request from a target request source based on a preset arbitration strategy, and sends the MR registration information request to the matching module; The matching module parses the physical address in the MR registration information request, and determines whether there is a valid logical address in the address mapping table based on the physical address, and the matching module sends a count-increment request to the counting module; In a case where the valid logical address exists in the address mapping table, a first logical address corresponding to the physical address is read from the address mapping table; in a case where the valid logical address does not exist in the address mapping table, the matching module applies for a second logical address from the counting module, and based on the physical address, applies for external MR registration information from an external storage, and writes the external MR registration information into the MR registration information storage of the storage module using the second logical address as an index; the second logical address is used to store the external MR registration information in the storage module; The arbitration module responds to a MR registration information request from a target request source based on a preset arbitration strategy, and sends the MR registration information request to the matching module; After the physical addresses of all MR registration information of the current queue number are obtained by the matching module, the matching module sends the corresponding logical address group of the current queue number to the checking module; When the checking module judges that all logical addresses of the logical address group are valid, the matching module identifies the current queue number as a ready state, so that the controller obtains the logical address group by the matching module and reads the corresponding MR registration information of the logical address group from the storage module.

Images