A UAR page allocation method and device for IB network card
By dividing the PCIE base address space on the IB network card and using global linked list to manage UAR pages, the race conditions caused by multiple queues to share pages are solved, and efficient concurrent operations and performance improvements are achieved.
Patent Information
- Application Number
- CN202410417373.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-08
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2044-04-08
AI Technical Summary
In a multi-threaded or multi-process environment, multiple queue pairs share the same UAR page, it is easy to cause race conditions and performance problems, especially when creating a large number of queue pairs in the same context, resulting in concurrent operations affecting performance.
By dividing the PCIE base address register space into UAR pages and managing these pages with a global linked list, ensuring that each queue pair gets independent free WQE space and avoiding race conditions. The specific steps include mapping the address of the UAR page to the host-side start address, initializing the global linked list, and allocating free WQE space from the linked list when creating a queue pair.
It effectively avoids race conditions and performance bottlenecks, ensures that each queue pair can independently access the BAR space, reduces CPU lock competition and resource waste, and improves software performance.
Smart Images
Figure CN118316778B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of network communication, and in particular relates to a UAR page allocation method and device for an IB network card. Background Art
[0002] RDMA (Remote Direct Memory Access) is an efficient network communication technology that allows computers to exchange memory data directly without going through the operating system. The key advantage of RDMA is that it reduces latency and CPU load during data transmission. BlueFlame is an important supplement to traditional RDMA technology. It improves the efficiency and performance of overall network communication by optimizing memory write operations. It allows the network card hardware to directly handle the operations of building and processing network packets, provides a more efficient way to handle RDMA memory write operations, reduces dependence on the CPU, and thus reduces latency and improves performance. The UAR (User Access Regin) page (UAR Page) is an area in memory used to store control information required for RDMA operations. The UAR page allows the HCA (high-performance network card) to efficiently access this control information to perform network operations. Each UAR page contains multiple UARs, each of which corresponds to a specific network operation or queue. BlueFlame and UAR pages work together to improve the efficiency of packet processing and ensure fast access to the control information required for the operation. Together, they optimize the RDMA process, reduce latency, and improve throughput.
[0003] However, Blueframe WQE (Work Queue Element, used to specify and control RDMA operations) is aligned to 64 bytes, so if it is submitted directly through UAR, the host side needs to make a copy. In a multi-threaded or multi-process environment, if multiple threads or processes share the same QP and UAR Page, they may try to write to the UAR Page at the same time, causing race conditions and performance issues.
[0004] The current Mellanox IB network card is designed with resource management and performance optimization in mind, and 16 UAR pages are allocated in each context, such as Figure 1 As shown in the figure, the context contains data structures and information associated with a specific IB device or resource. This can prevent race conditions from occurring as the number of contexts increases. However, when a large number of QPs (Queue Pairs) are created in the same context, the concurrency of these QPs will cause race conditions, thus affecting performance. Summary of the invention
[0005] The object of the present invention is to provide a UAR page allocation method and device for an IB network card, aiming to improve the network card performance under the concurrent situation of multiple QPs.
[0006] According to a first aspect of the present invention, a UAR page allocation method for an IB network card is provided, comprising:
[0007] Divide the PCIE base address register space into UAR pages, map the address of the UAR page to the corresponding host side start address using the offset of the PCIE base address, initialize the global linked list, and add the start address corresponding to each UAR page to the global linked list;
[0008] When creating a queue pair QP, a UAR page is taken out from the global linked list, and the free WQE space of the UAR page is allocated to the queue pair, so that each queue pair obtains an independent PCIE base address register space.
[0009] Preferably, the adding each UAR page to the global linked list further comprises:
[0010] The starting address corresponding to each UAR page is inserted into the tail of the global linked list in sequence.
[0011] The number of UAR pages is independent of the process or thread context.
[0012] The method further comprises: taking out the UAR page from the global linked list;
[0013] The current UAR page is taken out from the head of the global linked list, and the current UAR page is deleted from the head of the linked list, and then the current UAR page is added to the tail of the global linked list.
[0014] The allocating the free WQE space of the UAR page to the queue pair further comprises:
[0015] If there is free WQE space in the current UAR page, the free WQE space is allocated to the current queue pair;
[0016] If there is no free WQE space in the current UAR page, subsequent UAR pages are continuously taken out from the global linked list until it is determined that there is free WQE space in the subsequent UAR page, and the free WQE space is allocated to the current queue pair.
[0017] According to a second aspect of the present invention, there is provided a UAR page allocation device for an IB network card, comprising:
[0018] A linked list generation unit, used to divide the PCIE base address register space into UAR pages, map the address of the UAR page to the corresponding host side start address using the offset of the PCIE base address, initialize the global linked list, and add the start address corresponding to each UAR page to the global linked list;
[0019] The page allocation unit is used to take out the UAR page from the global linked list when creating a queue pair QP, and allocate the free WQE space of the UAR page to the queue pair, so that each queue pair obtains an independent PCIE base address register space.
[0020] Compared with the related art, the technical solution of the present invention has the following advantages:
[0021] Using a global linked list to manage UAR pages, removing the limit on the number of UAR pages in a process, and fully utilizing the entire BAR0 space can ensure that no race conditions will occur when all QPs in a context are concurrent, thereby ensuring software performance and avoiding access conflicts between multiple QPs to BAR. Each QP can access an independent BAR, and all QPs in a process or thread can monopolize BAR space resources, reducing BAR space resource waste. When all QPs in a process or thread perform concurrent operations, lock-free operations can be achieved at the software level, reducing CPU resource waste due to lock competition. Since all QPs monopolize the doorbell area in the UAR page, performance bottlenecks caused by contention for UAR page resources due to the concurrency of multiple QPs are avoided.
[0022] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures and processes indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. It is obvious that the drawings described below are certain embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0024] Figure 1 It is a schematic diagram of UAR page allocation according to the related art.
[0025] Figure 2 is a flow chart of a UAR page allocation method according to the present invention.
[0026] Figure 3 It is a schematic diagram of splitting a PCIE BAR into UAR pages according to the present invention.
[0027] Figure 4 It is a diagram of the UARpage global linked list structure according to the present invention.
[0028] Figure 5 It is a schematic diagram of the process of taking out and adding the UAR page global linked list according to the present invention.
[0029] Figure 6 FIG. 4 is a schematic diagram of the allocation status of the WQE space and the QP according to the present invention.
[0030] Figure 7 It is a schematic diagram of the UAR page allocation according to the present invention. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0032] Based on the above analysis, the present invention proposes a UAR page allocation method and device for an IB network card, which removes the restriction of PCIE BAR (Base Address Register) space on QP in the driver, does not classify the UAR page of the BAR space, so that all QPs in each context can use the doorbell area in a UAR page, namely, the WQE space, separately.
[0033] See also Figure 2 The UAR page allocation method for the IB network card provided by the present invention includes:
[0034] Step 101: Divide the PCIE base address register space into UAR pages, map the address of the UAR page to the corresponding host side start address using the offset of the PCIE base address, initialize the global linked list, and add the start address corresponding to each UAR page to the global linked list.
[0035] like Figure 3As shown, first, the 32M space of BAR0 is split into 4K UAR pages. Then a linked list UAR_page_list of global type UAR page is created on the host side, and the UAR page adopts a dynamic double-ended queue management strategy. This strategy adopts a double-ended queue (UAR_page_list) that can be dynamically resized, which supports efficient insertion and deletion operations at both ends. This feature of the double-ended queue allows it to be used as a stack or a queue, thereby providing great flexibility. In the present invention, a tail insertion strategy is adopted to add UAR pages, and new UAR pages are always added to the tail of the linked list. This method utilizes the dynamic characteristics of the linked list and can be expanded on demand without pre-allocating storage space, greatly improving storage efficiency. Tail insertion ensures the order of UARpage, so that the latest added UARpage is always located at the last position of the linked list. Maintenance and expansion can be achieved by utilizing the flexibility of the double-ended queue. Assuming that all UARs in the current BAR space are used, a tail deletion and head reuse strategy can be used to reuse a UAR page.
[0036] Convert the UAR page to an address that can be directly accessed by the host through address mapping, store the addresses of all UAR pages allocated in the BAR0 space in the global linked list UAR_page_list of the UAR page, and add them to UAR page 8192 in sequence starting from UARpage0, that is, insert the address of each UAR page into the end of the linked list in sequence, such as Figure 4 shown.
[0037] Since a global linked list is used to maintain all UAR pages, it is independent of the context of each thread or process.
[0038] Step 102: when creating a queue pair QP, taking out a UAR page from the global linked list, and allocating the free WQE space of the UAR page to the queue pair, so that each queue pair obtains an independent PCIE base address register space.
[0039] When a QP is first created, a UAR page is taken from the head of the linked list, the taken UAR page is assigned to the QP, and the taken UAR page is deleted from the head of the linked list and added to the tail of the linked list, such as Figure 5 As shown in the figure. When taking out the UAR page, the head extraction strategy is adopted. This strategy corresponds to the tail insertion. The head extraction strategy allows elements to be quickly removed from the head of the linked list. The time complexity of this operation is O(1), which can be completed in constant time, ensuring efficient access speed.
[0040] Each QP created by a context gets a UAR page from the linked list. There is no limit to the number of UAR pages in a context. One UAR page contains space for 4 WQEs, each of which can be used by a QP. Figure 6 shown.
[0041] In an optional embodiment, if there is no free WQE space in the current UAR page, it means that all WQEs in the current UAR page are allocated to the previous QP, then subsequent UAR pages are continuously taken out from the global linked list until free WQE space is found, and the free WQE space is allocated to the current QP.
[0042] Hence, see Figure 7 , up to 32K QPs applied for in a process can use the doorbell area in their own UAR page independently, which can effectively avoid the performance impact caused by sharing a UARpage between QPs in a process.
[0043] As mentioned above, in the related art, a process contains at most 16 UAR pages. If more than 64 QPs are concurrent in a process, a race condition will occur, affecting performance. Compared with the related art, the UAR page allocation method for the IB network card proposed in the present invention uses a global linked list to manage UAR pages, removes the number limit of UAR pages in a process, and fully utilizes the entire BAR0 space, which can ensure that no race conditions will occur when all QPs in a context are concurrent, thereby ensuring software performance and avoiding access conflicts of multiple QPs to BAR. Each QP can access an independent BAR, and all QPs in a process or thread can monopolize the resources of the BAR space, reducing the waste of BAR space resources. When all QPs in a process or thread perform concurrent operations, lock-free operations can be achieved at the software level, reducing the waste of CPU resources due to lock competition. Since all QPs monopolize the doorbell area in the UAR page, the performance bottleneck caused by the contention of UAR page resources caused by the concurrency of multiple QPs is avoided.
[0044] Accordingly, the present invention provides a UAR page allocation device for an IB network card in a second aspect, comprising:
[0045] A linked list generation unit, used to divide the PCIE base address register space into UAR pages, map the address of the UAR page to the corresponding host side start address using the offset of the PCIE base address, initialize the global linked list, and add the start address corresponding to each UAR page to the global linked list;
[0046] The page allocation unit is used to take out the UAR page from the global linked list when creating a queue pair QP, and allocate the free WQE space of the UAR page to the queue pair, so that each queue pair obtains an independent PCIE base address register space.
[0047] The above-mentioned device can be implemented by the UAR page allocation method for the IB network card provided by the embodiment of the first aspect above. The specific implementation method can refer to the description in the embodiment of the first aspect, which will not be repeated here.
[0048] It is understood that the storage structures, names and parameters described in the above embodiments are only examples. Those skilled in the art can also easily combine and adjust the structural features of the above embodiments according to the use requirements, and should not limit the concept of the present invention to the specific details of the above examples.
[0049] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A UAR page allocation method for an IB network card, characterized in that: include: Divide the PCIE base address register space into UAR pages, map the address of the UAR page to the corresponding host side start address using the offset of the PCIE base address, initialize the global linked list, and add the start address corresponding to each UAR page to the global linked list; The UAR page is managed by a dynamic double-ended queue that can be dynamically resized, and insertion and deletion operations are performed at both ends; When creating a queue pair QP, taking out a UAR page from the global linked list, and allocating the free WQE space of the UAR page to the queue pair, so that each queue pair obtains an independent PCIE base address register space; The allocating the free WQE space of the UAR page to the queue pair further comprises: If there is free WQE space in the current UAR page, the free WQE space is allocated to the current queue pair; If there is no free WQE space in the current UAR page, subsequent UAR pages are continuously taken out from the global linked list until it is determined that there is free WQE space in the subsequent UAR page, and the free WQE space is allocated to the current queue pair.
2. The UAR page allocation method for an IB network card according to claim 1, characterized in that: The step of adding the starting address corresponding to each UAR page to the global linked list further comprises: The starting address corresponding to each UAR page is inserted into the tail of the global linked list in sequence.
3. The UAR page allocation method for an IB network card according to claim 1, characterized in that: The number of UAR pages is independent of the process or thread context.
4. The UAR page allocation method for an IB network card according to claim 1, characterized in that: The method further comprises: taking out the UAR page from the global linked list; The current UAR page is taken out from the head of the global linked list, and the current UAR page is deleted from the head of the linked list, and then the current UAR page is added to the tail of the global linked list.
5. A UAR page allocation device for an IB network card, characterized in that: include: A linked list generation unit, used to divide the PCIE base address register space into UAR pages, map the address of the UAR page to the corresponding host side start address using the offset of the PCIE base address, initialize the global linked list, and add the start address corresponding to each UAR page to the global linked list; A page allocation unit, configured to take out a UAR page from the global linked list when creating a queue pair QP, and allocate an idle WQE space of the UAR page to the queue pair, so that each queue pair obtains an independent PCIE base address register space; The page allocation unit is further used for: If there is free WQE space in the current UAR page, the free WQE space is allocated to the current queue pair; If there is no free WQE space in the current UAR page, subsequent UAR pages are continuously taken out from the global linked list until it is determined that there is free WQE space in the subsequent UAR page, and the free WQE space is allocated to the current queue pair.
6. The UAR page allocation device for an IB network card according to claim 5, characterized in that: The linked list generating unit is further used for: The starting address corresponding to each UAR page is inserted into the tail of the global linked list in sequence.
7. The UAR page allocation device for an IB network card according to claim 5, characterized in that: The number of UAR pages is independent of the process or thread context.
8. The UAR page allocation device for an IB network card according to claim 7, characterized in that: The page allocation unit is further used for: The current UAR page is taken out from the head of the global linked list, and the current UAR page is deleted from the head of the linked list, and then the current UAR page is added to the tail of the global linked list.