A queue pair state management method and system
By combining the two-stage storage method of BRAM and DRAM in the InfiniBand network card, the problem of limited number of network card queues is solved, and efficient QP state management and extended storage are achieved.
Patent Information
- Application Number
- CN202311667227.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-12-06
AI Technical Summary
In the prior art, the number of queue pairs (QPs) of the InfiniBand network card is limited by the Block RAM (BRAM) resource of the FPGA board, and the number of 16M connections cannot be realized, resulting in a relatively limited number of QPs.
By using a two-stage storage method combining BRAM and DRAM storage modules in the network card, first match the QP connection status address in the BRAM storage module. If it does not match, match it in the DRAM storage module. Use DRAM to expand storage and improve QP storage Quantity and query efficiency.
It realizes efficient storage and query of QP connection state, expands the number of QP storage, maintains query efficiency, and solves the problem of BRAM resource limitation.
Smart Images

Figure CN117873744B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information technology, and in particular to a queue pair state management method and system. Background Art
[0002] Remote Direct Memory Access (RDMA) is a high-performance network technology. The RoCEv2 RDMA network card implements zero-copy network transmission services that bypass the operating system kernel through hardware offload of the InfiniBand transport layer, while reducing the CPU occupancy rate on the host side. The InfiniBand transport layer uses queue pairs (QP) as communication nodes. In the connection-oriented reliable transmission service (RC), the local QP needs to establish a connection before communicating with the remote QP on the network. Each connection includes a series of status information. A typical RDMA network card uses 256 bytes to maintain the status of a connection.
[0003] The maximum number of QPs defined in the InfiniBand protocol is 16M. The network card uses the Block RAM (BRAM) of the FPGA board to store the status information of the QP connection and supports up to 256 QP connections. The typical scale of the BRAM resources of the FPGA board is tens of MB. Even if all BRAMs are used to store the QP status in implementation, 16M connections cannot be achieved. Therefore, the number of QPs in the existing technology is limited by the BRAM resources, and the number of QPs is relatively limited. Summary of the invention
[0004] In view of this, an embodiment of the present invention provides a queue pair state management method to eliminate or improve one or more defects existing in the prior art.
[0005] One aspect of the present invention provides a queue pair state management method, the method is applied to a network card, the network card includes a BRAM storage module and a DRAM storage module, the method comprises the following steps:
[0006] The network card obtains the QPID, and matches the QP connection state address in the first storage area of the BRAM storage module based on the QPID;
[0007] If the QPID does not match the QP connection state address in the first storage area of the BRAM storage module, matching the QPID with the QP connection state address in the DRAM storage module based on the cache controller;
[0008] matching a QP connection state in a second storage area of a BRAM storage module based on the QP connection state address;
[0009] If the QP connection state address does not match the QP connection state in the second storage area of the BRAM storage module, the QP connection state is matched in the DRAM storage module through the QP connection state address based on the cache controller.
[0010] By adopting the above scheme, on the one hand, the storage of QP connection status in this scheme further utilizes DRAM storage on the basis of using BRAM storage, thereby increasing the storage quantity of QP. On the other hand, this scheme adopts a two-stage storage method, first querying through the BRAM storage module, and then querying through the DRAM storage module. The BRAM storage module is used to store the QP connection status address and QP connection status with high utilization rate, which can improve the query efficiency. Then, the storage is expanded through the DRAM storage module, thereby expanding the storage quantity of QP while maintaining the query efficiency.
[0011] In some embodiments of the present invention, the QP connection state address is stored in the first storage area of the BRAM storage module corresponding to the QPID, and in the step of matching the QP connection state address in the first storage area of the BRAM storage module based on the QPID, the corresponding QP connection state address is queried based on the QPID.
[0012] In some embodiments of the present invention, a connection status address storage area for storing a QP connection status address is provided in the DRAM storage module. In the step of matching the QPID with the QP connection status address in the DRAM storage module based on the cache controller, a starting position of the connection status address storage area is obtained, a storage position of the QP connection status address in the connection status address storage area is calculated based on the starting position and the QPID, and a corresponding QP connection status address is obtained from the storage position.
[0013] In some embodiments of the present invention, in the step of calculating the storage position of the QP connection state address in the connection state address storage area based on the starting position and the QPID, the storage position of the QP connection state address in the connection state address storage area is calculated based on the following formula:
[0014] QPCSAP = Base_0 + QPID * 8;
[0015] Among them, QPCSAP represents the storage position of the QP connection state address in the connection state address storage area, and Base_0 represents the starting position.
[0016] In some embodiments of the present invention, the step of matching the QPID with the QP connection status address in the DRAM storage module based on the cache controller also includes, after obtaining the QP connection status address from the DRAM storage module, feeding back the obtained QP connection status address to the BRAM storage module and caching it in the first storage area of the BRAM storage module.
[0017] In some embodiments of the present invention, the step of matching the QP connection state in the DRAM memory module based on the cache controller through the QP connection state address includes acquiring the QP connection state at a corresponding position of the QP connection state address in the DRAM memory module.
[0018] In some embodiments of the present invention, the step of matching the QP connection state in the DRAM storage module based on the cache controller through the QP connection state address also includes, after obtaining the QP connection state from the DRAM storage module, feeding back the obtained QP connection state to the BRAM storage module and caching it in the second storage area of the BRAM storage module.
[0019] In some embodiments of the present invention, in the step of matching the QP connection state address in the first storage area of the BRAM storage module based on the QPID, the QPID includes a query tag and an index, and the query tag in the QPID is compared with the query tag in the first storage area of the BRAM storage module. If they are the same, the match is successful; in the step of matching the QP connection state in the second storage area of the BRAM storage module based on the QP connection state address, the QP connection state address includes a query tag and an index, and the query tag in the QP connection state address is compared with the query tag in the second storage area of the BRAM storage module. If they are the same, the match is successful.
[0020] In some embodiments of the present invention, in the steps of feeding back the acquired QP connection state address to the BRAM storage module and caching it in the first storage area of the BRAM storage module and feeding back the acquired QP connection state to the BRAM storage module and caching it in the second storage area of the BRAM storage module, the acquired QP connection state address or QP connection state replaces the QP connection state address or QP connection state corresponding to the same index in the first storage area or the second storage area, and the replaced QP connection state address or QP connection state is synchronized to the DRAM storage module.
[0021] The second aspect of the present invention also provides a queue pair state management system, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.
[0022] The third aspect of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the aforementioned queue pair state management method.
[0023] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention can be specifically pointed out and obtained in the specification and the accompanying drawings.
[0024] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.
[0026] Figure 1 A schematic diagram of an implementation of a queue pair state management method of the present invention;
[0027] Figure 2 A schematic diagram of the architecture of the queue pair state management method of the present invention;
[0028] Figure 3 The figure is a flow chart of querying the QP connection state address and the QP connection state in the BRAM storage module. DETAILED DESCRIPTION
[0029] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.
[0030] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.
[0031] The steps of the present invention specifically include:
[0032] like Figure 1 and 2 As shown, the present invention proposes a queue pair state management method, the method is applied to a network card, the network card includes a BRAM storage module and a DRAM storage module, and the steps of the method include:
[0033] Step S100, the network card obtains the QPID, and matches the QP connection state address in the first storage area of the BRAM storage module based on the QPID;
[0034] In some embodiments of the present invention, the network card is provided with a field programmable gate array chip (Field Programmable Gate Array, FPGA) and DDR4, the BRAM storage module is provided on the FPGA, and the DRAM storage module is provided on the DDR4.
[0035] In the specific implementation process, the access delay of the BRAM storage module on the FPGA is 1 clock cycle, and the access delay of the DRAM storage module on the DDR4 is 50 clock cycles.
[0036] In a specific implementation process, the QPID is the number of a queue pair (Queue Pair, QP).
[0037] In a specific implementation process, the network card parses the 24-bit QPID from the data packet header, the work queue element (WQE) or the QP connection management command.
[0038] In a specific implementation, the 24-bit QPID is used to search the QP connection state address cache.
[0039] Step S200, if the QPID is not matched to a QP connection state address in the first storage area of the BRAM storage module, then matching the QPID to a QP connection state address in the DRAM storage module based on a cache controller;
[0040] During the specific implementation process, the cache controller (Cache Controller) calculates the storage position (QP Connection State Address Pointer, QPCSAP) of the QP connection state address in the connection state address storage area based on the starting position and QPID of the connection state address storage area. Specifically, using a 64-bit DRAM storage module, QPCSAP=Base_0+QP_ID*8, Base_0 represents the starting position.
[0041] Step S300, matching the QP connection state in the second storage area of the BRAM storage module based on the QP connection state address;
[0042] Step S400: if the QP connection state address does not match the QP connection state in the second storage area of the BRAM storage module, then based on the cache controller, the QP connection state is matched in the DRAM storage module through the QP connection state address.
[0043] By adopting the above scheme, on the one hand, the storage of QP connection status in this scheme further utilizes DRAM storage on the basis of using BRAM storage, thereby increasing the storage quantity of QP. On the other hand, this scheme adopts a two-stage storage method, first querying through the BRAM storage module, and then querying through the DRAM storage module. The BRAM storage module is used to store the QP connection status address and QP connection status with high utilization rate, which can improve the query efficiency. Then, the storage is expanded through the DRAM storage module, thereby expanding the storage quantity of QP while maintaining the query efficiency.
[0044] In some embodiments of the present invention, the QP connection state address is stored in the first storage area of the BRAM storage module corresponding to the QPID, and in the step of matching the QP connection state address in the first storage area of the BRAM storage module based on the QPID, the corresponding QP connection state address is queried based on the QPID.
[0045] like Figure 3 As shown, in the specific implementation process, the transmission (Transmit) and reception (Receive) paths of the network card initiate a QP connection status query request based on the QPID. The query request includes a query tag (Tag) and an index (Index). The index is used as a read address to obtain the query tag from the first storage area. If the obtained query tag is equal to the query tag in the query request, it is a hit, and the QP connection status address is read out.
[0046] In some embodiments of the present invention, a connection status address storage area for storing a QP connection status address is provided in the DRAM storage module. In the step of matching the QPID with the QP connection status address in the DRAM storage module based on the cache controller, a starting position of the connection status address storage area is obtained, a storage position of the QP connection status address in the connection status address storage area is calculated based on the starting position and the QPID, and a corresponding QP connection status address is obtained from the storage position.
[0047] In some embodiments of the present invention, in the step of calculating the storage position of the QP connection state address in the connection state address storage area based on the starting position and the QPID, the storage position of the QP connection state address in the connection state address storage area is calculated based on the following formula:
[0048] QPCSAP = Base_0 + QPID * 8;
[0049] Among them, QPCSAP represents the storage position of the QP connection state address in the connection state address storage area, and Base_0 represents the starting position.
[0050] By adopting the above scheme, this scheme can quickly obtain the accurate storage address through formula calculation during the query process of the QP connection status address through a preset storage method. Compared with the search method of matching one by one, this scheme significantly improves the matching efficiency.
[0051] In some embodiments of the present invention, the step of matching the QPID with the QP connection status address in the DRAM storage module based on the cache controller also includes, after obtaining the QP connection status address from the DRAM storage module, feeding back the obtained QP connection status address to the BRAM storage module and caching it in the first storage area of the BRAM storage module.
[0052] In some embodiments of the present invention, the step of matching the QP connection state in the DRAM memory module based on the cache controller through the QP connection state address includes acquiring the QP connection state at a corresponding position of the QP connection state address in the DRAM memory module.
[0053] In some embodiments of the present invention, the step of matching the QP connection state in the DRAM storage module based on the cache controller through the QP connection state address also includes, after obtaining the QP connection state from the DRAM storage module, feeding back the obtained QP connection state to the BRAM storage module and caching it in the second storage area of the BRAM storage module.
[0054] like Figure 3As shown, in some embodiments of the present invention, in the step of matching the QP connection state address in the first storage area of the BRAM storage module based on the QPID, the QPID includes a query tag and an index, and the query tag in the QPID is compared with the query tag in the first storage area of the BRAM storage module. If they are the same, the match is successful; in the step of matching the QP connection state in the second storage area of the BRAM storage module based on the QP connection state address, the QP connection state address includes a query tag and an index, and the query tag in the QP connection state address is compared with the query tag in the second storage area of the BRAM storage module. If they are the same, the match is successful.
[0055] In a specific implementation process, a query tag (Tag) and an index (Index) are extracted based on the QP connection state address, and the query tag is matched based on the index. If a hit occurs, the QP connection state is determined; if a hit does not occur, the QP connection state address is sent to a cache controller, and the cache controller reads the corresponding QP connection state from the DRAM storage module.
[0056] In some embodiments of the present invention, in the steps of feeding back the acquired QP connection state address to the BRAM storage module and caching it in the first storage area of the BRAM storage module and feeding back the acquired QP connection state to the BRAM storage module and caching it in the second storage area of the BRAM storage module, the acquired QP connection state address or QP connection state replaces the QP connection state address or QP connection state corresponding to the same index in the first storage area or the second storage area, and the replaced QP connection state address or QP connection state is synchronized to the DRAM storage module.
[0057] In a specific implementation process, the replaced QP connection state address or QP connection state is synchronized to the DRAM storage module through the cache controller.
[0058] In the specific implementation process, the QP connection state can be classified into static members that are read-only on the request path, including the peer QPID, MAC address, and IP address, etc., and dynamic members that are read and written on the request path, including the packet sequence number PSN and the message sequence number MSN, etc., from the packet send / receive pipeline access behavior. The send / receive pipeline reads the entire state of a QP connection and updates the dynamic members after processing.
[0059] In the specific implementation process, for the QP connection state reading behavior, the QP connection state cache (QP Connection State Cache) is accessed based on the QP connection state address (QPCSA) obtained by the query, and the least recently used strategy (Least Recently Used, LRU) is adopted for replacement. Since the dynamic members to be written must exist in the cache, in order to reduce the latency overhead of updating the dynamic members, the write back strategy is adopted. Only when the corresponding cache item is replaced, the cache controller (Cache Controller) writes the updated data back to the DRAM storage module.
[0060] This solution takes into account the access delay difference of different types of storage media. The access delay of the BRAM storage module on the FPGA is 1 clock cycle, and the typical access delay of the DRAM storage module of DDR4 is 50 clock cycles. If the QP connection state is stored in DRAM, the RDMA transport layer protocol stack receives and processes the pipeline to obtain the state from DRAM each time. During this process, the data packet processing pipeline will be blocked until the state is retrieved from DRAM. The blocking of the data packet processing pipeline leads to an increase in the RDMA request processing delay, an increase in the data packet cache resource overhead, and idle pipeline logic resources. In order to reduce the blocking time of the pipeline, it is necessary to respond to the QP state access request as quickly as possible. Since the RDMA QP access has temporal locality, the QP referenced in a short time has a high probability of being accessed again. Therefore, the present invention constructs a QP connection state storage architecture based on the small-capacity, high-speed storage resource BRAM and the large-capacity, low-speed storage resource DRAM on the FPGA, realizes low-latency QP connection state access based on the cache mechanism, and realizes large-scale QP connection state storage based on DRAM storage.
[0061] An embodiment of the present invention also provides a queue pair state management system, which includes a computer device, wherein the computer device includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.
[0062] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps implemented by the aforementioned queue pair state management method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.
[0063] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.
[0064] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.
[0065] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.
[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A queue pair state management method, It is characterized in that The method is applied to a network card, wherein the network card includes a BRAM storage module and a DRAM storage module, and the steps of the method include: The network card obtains the QPID, and matches the QP connection state address in the first storage area of the BRAM storage module based on the QPID; If the QPID does not match the QP connection state address in the first storage area of the BRAM storage module, matching the QPID with the QP connection state address in the DRAM storage module based on the cache controller; matching a QP connection state in a second storage area of a BRAM storage module based on the QP connection state address; If the QP connection state address does not match the QP connection state in the second storage area of the BRAM storage module, the QP connection state is matched in the DRAM storage module through the QP connection state address based on the cache controller.
2. The queue pair state management method according to claim 1, It is characterized in that The QP connection state address is stored in the first storage area of the BRAM storage module corresponding to the QPID, and in the step of matching the QP connection state address in the first storage area of the BRAM storage module based on the QPID, the corresponding QP connection state address is queried based on the QPID.
3. The queue pair state management method according to claim 1, It is characterized in that The DRAM storage module is provided with a connection status address storage area for storing a QP connection status address. In the step of matching the QPID with the QP connection status address in the DRAM storage module based on the cache controller, the starting position of the connection status address storage area is obtained, the storage position of the QP connection status address in the connection status address storage area is calculated based on the starting position and the QPID, and the corresponding QP connection status address is obtained from the storage position.
4. The queue pair state management method according to claim 3, It is characterized in that In the step of calculating the storage position of the QP connection state address in the connection state address storage area based on the starting position and the QPID, the storage position of the QP connection state address in the connection state address storage area is calculated based on the following formula: QPCSAP = Base_0 + QPID * 8; Among them, QPCSAP represents the storage position of the QP connection state address in the connection state address storage area, and Base_0 represents the starting position.
5. The queue pair state management method according to claim 1, It is characterized in that The step of matching the QPID with the QP connection status address in the DRAM storage module based on the cache controller also includes, after acquiring the QP connection status address from the DRAM storage module, feeding back the acquired QP connection status address to the BRAM storage module and caching it in the first storage area of the BRAM storage module.
6. The queue pair state management method according to claim 1, It is characterized in that The step of matching the QP connection state in the DRAM memory module based on the cache controller through the QP connection state address includes obtaining the QP connection state at a corresponding position of the QP connection state address in the DRAM memory module.
7. The queue pair state management method according to claim 5, It is characterized in that The step of matching the QP connection state in the DRAM storage module based on the cache controller through the QP connection state address also includes, after acquiring the QP connection state from the DRAM storage module, feeding back the acquired QP connection state to the BRAM storage module and caching it in the second storage area of the BRAM storage module.
8. The queue pair state management method according to claim 7, It is characterized in that In the steps of feeding back the acquired QP connection state address to the BRAM storage module and caching it in the first storage area of the BRAM storage module and feeding back the acquired QP connection state to the BRAM storage module and caching it in the second storage area of the BRAM storage module, the acquired QP connection state address or QP connection state replaces the QP connection state address or QP connection state corresponding to the same index in the first storage area or the second storage area, and the replaced QP connection state address or QP connection state is synchronized to the DRAM storage module.
9. The queue pair state management method according to claim 8, It is characterized in that In the step of matching the QP connection state address in the first storage area of the BRAM storage module based on the QPID, the QPID includes a query tag and an index, and the query tag in the QPID is compared with the query tag in the first storage area of the BRAM storage module. If they are the same, the match is successful. In the step of matching the QP connection state in the second storage area of the BRAM storage module based on the QP connection state address, the QP connection state address includes a query tag and an index, and the query tag in the QP connection state address is compared with the query tag in the second storage area of the BRAM storage module. If they are the same, the match is successful.
10. A queue pair state management system, It is characterized in that The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions. The processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method as described in any one of claims 1 to 9.
