PCIe ECAM address mapping optimization method and system

Abstract

The invention discloses a PCIe ECAM address mapping optimization method and system, and belongs to the technical field of operating system optimization, and the method comprises the steps: formulating a PCIe device region division rule, and dividing a PCIe device into a high-frequency access region device and a low-frequency access region device according to the device distribution condition in a PCIe bus topology based on the rule; the page table allocation comprises static page table pre-allocation and dynamic page table allocation adjustment, based on PCIe device area division, devices in a high-frequency access area are subjected to centralized memory mapping with a large page as the granularity, and devices in a low-frequency access area are subjected to independent memory mapping with a small page as the granularity; and setting an access frequency threshold, monitoring the access frequency of the PCIe equipment configuration space when the system runs, and merging or splitting the page table through the memory management unit. The number of page table items can be remarkably reduced, the hit rate of the TLB is increased, and the overall memory access efficiency of a system is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of operating system optimization, and in particular to a method and system for optimizing PCIe ECAM address mapping. Background Art

[0002] PCIe ECAM (Enhanced Configuration Access Mechanism) is a configuration space access mechanism defined by the PCIe specification. The configuration space of each PCIe device in the system is mapped to a specific memory address area in the ECAM. The driver configures and manages the PCIe device by accessing these memory areas.

[0003] In the PCIe ECAM mechanism, address mapping is the key to managing access to the PCIe device configuration space. Traditional ECAM mapping usually uses a uniform granularity, but this approach is not efficient enough in some cases, especially when dealing with a large number of PCIe devices, which will lead to an explosion of page table entries, resulting in a decrease in the system's TLB hit rate and memory access efficiency. Summary of the invention

[0004] The technical task of the present invention is to provide a PCIe ECAM address mapping optimization method and system to address the above shortcomings, which can significantly reduce the number of page table entries, improve the TLB hit rate, and enhance the overall memory access efficiency of the system.

[0005] The technical solution adopted by the present invention to solve its technical problem is: A method for optimizing PCIe ECAM address mapping, the implementation of which includes: PCIe device area division: formulate rules for PCIe device area division, and based on the rules, divide PCIe devices into high-frequency access area devices and low-frequency access area devices according to the device distribution in the PCIe bus topology; Page table allocation: including static pre-allocation of page tables and dynamic adjustment of page table allocation, The page table static pre-allocation: based on the PCIe device area division, the devices in the high-frequency access area are centrally memory mapped with large pages as the granularity, and the devices in the low-frequency access area are independently memory mapped with small pages as the granularity; The dynamic adjustment of page table allocation includes setting an access frequency threshold, monitoring the access frequency of the PCIe device configuration space when the system is running, and merging or splitting the page table through the memory management unit.

[0006] This method proposes a hybrid granularity PCIe ECAM address mapping method, which combines hybrid granularity mapping with PCIe configuration space management, improves memory access efficiency by flexibly adjusting the mapping granularity of different PCIe bus devices, divides PCIe devices into regions, monitors the access frequency of each device on the PCIe bus, and dynamically uses different sizes of page table granularity (such as 4KB, 64K, 2M) in ECAM address mapping to balance memory usage, TLB efficiency and memory access performance. Traditional fixed granularity mapping (such as full 4KB page) will cause an explosive growth of page table entries when the number of devices is large, while hybrid granularity can significantly reduce the number of page table entries by flexibly adjusting the page size, improve the TLB hit rate, and improve the overall memory access efficiency of the system.

[0007] Furthermore, the large page has a page table granularity of 64K or 2M; and the small page has a page table granularity of 4K.

[0008] Furthermore, the PCIe device area division is PCIe devices with two or more downstream devices are classified as high-frequency access area devices, and PCIe devices with less than two downstream devices are classified as low-frequency access area devices.

[0009] Furthermore, the centralized memory mapping refers to summing up the address spaces required to be mapped by the devices in each high-frequency access area, and using the summation result as a parameter and large pages as the granularity to perform memory space mapping.

[0010] Furthermore, the system monitors the access frequency of the PCIe device configuration space when it is running. The access frequency and continuity of the PCIe device configuration space are counted through performance counters (such as perf). When the device access frequency of the low-frequency access area exceeds the set frequency threshold, the page merge operation is triggered; when the device access frequency of the high-frequency access area is lower than the set frequency threshold, the page split operation is triggered.

[0011] Furthermore, the merged page table, If the devices in the low-frequency access area are frequently accessed and the access frequency exceeds the set threshold, they will be re-divided into the high-frequency access area, and the small page space allocated to them will be released, and memory mapping application will be made in the centrally allocated large page space.

[0012] Furthermore, the split page table, If fragmented access occurs to devices in the high-frequency access area and the access frequency is lower than the set threshold, the device is reclassified as a low-frequency access area device, and its memory mapping space in the large page is released and re-mapped independently with small pages as the granularity.

[0013] The present invention also claims a PCIe ECAM address mapping optimization system, including a PCIe device area division module and a page table allocation module, The PCIe device area division module is used to formulate a rule for PCIe device area division, and based on the rule, divide the PCIe devices into high-frequency access area devices and low-frequency access area devices according to the device distribution in the PCIe bus topology; The page table allocation module includes static pre-allocation of page tables and dynamic adjustment of page table allocation; The page table static pre-allocation: based on the PCIe device area division, the devices in the high-frequency access area are centrally memory mapped with large pages as the granularity, and the devices in the low-frequency access area are independently memory mapped with small pages as the granularity; The dynamic adjustment of page table allocation: setting an access frequency threshold, monitoring the access frequency of the PCIe device configuration space when the system is running, and merging or splitting the page table through the memory management unit; The system specifically implements the optimization of PCIe ECAM address mapping through the above method.

[0014] The present invention also claims a device for optimizing PCIe ECAM address mapping, comprising: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the above method.

[0015] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which implement the above method when executed by a processor.

[0016] Compared with the prior art, the PCIe ECAM address mapping optimization method and system of the present invention have the following beneficial effects: 1. Through the optimization method of mixed-granularity PCIe ECAM address mapping, the number of page table entries occupied by PCIe ECAM address mapping is reduced, and the page table memory usage is reduced.

[0017] 2. Through the optimization method of mixed-granularity PCIe ECAM address mapping, large pages enable a single TLB entry to cover a larger address range, thereby improving the TLB hit rate.

[0018] 3. Through the optimization method of mixed-granularity PCIe ECAM address mapping, large pages cover more device configuration space, reduce the page table level, and improve the efficiency of configuration register access. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a diagram of a method for optimizing PCIe ECAM address mapping provided by an embodiment of the present invention; Figure 2 This is a PCIe bus topology diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with specific embodiments.

[0021] The embodiment of the present invention provides a method for optimizing PCIe ECAM address mapping, which divides PCIe devices into regions, monitors the access frequency of each device on the PCIe bus, and dynamically uses page table granularities of different sizes (such as 4KB, 64K, 2M) in ECAM address mapping to balance memory usage, TLB efficiency, and memory access performance. Traditional fixed-granularity mapping (such as full 4KB pages) will cause an explosive growth of page table entries when the number of devices is large, while hybrid granularity can significantly reduce the number of page table entries by flexibly adjusting the page size, improve the hit rate of TLB, and improve the overall memory access efficiency of the system.

[0022] The specific implementation of the mixed-granularity PCIe ECAM address mapping method is as follows: 1. PCIe device area division: A rule for dividing PCIe device regions is formulated, and based on the rule, PCIe devices are divided into high-frequency access region devices and low-frequency access region devices according to device distribution in the PCIe bus topology.

[0023] 2. Page table allocation: including static pre-allocation of page tables and dynamic adjustment of page table allocation.

[0024] The page table is statically pre-allocated: based on the PCIe device area division, the devices in the high-frequency access area are centrally memory mapped with large pages (64K / 2M) as the granularity, and the devices in the low-frequency access area are independently memory mapped with small pages (4K) as the granularity.

[0025] The dynamic adjustment of page table allocation: setting an access frequency threshold, monitoring the access frequency of the PCIe device configuration space when the system is running, and merging or splitting the page table through the memory management unit.

[0026] In order to enable those skilled in the art to better understand and implement the present method, the following Figure 1 , Figure 2 , the implementation process of this method is described in detail. The implementation method of this method includes the following steps: S1: PCIe device area division.

[0027] Formulate rules for device area division, and divide PCIe devices into high-frequency access area devices and low-frequency access area devices according to the device distribution in the PCIe bus topology.

[0028] As attached Figure 2 The device distribution in the PCIe bus topology shown in the figure, based on the preset rules, divides PCIe devices with more than 2 downstream devices into high-frequency access area devices, and divides PCIe devices with less than 2 downstream devices into low-density area devices. The device area division results are as follows: High frequency access area equipment: A, B, D, H.

[0029] Low-frequency access area equipment: E, F, G, L, M, N, I, J, P, Q, C, K.

[0030] S2: Page table static pre-allocation.

[0031] Based on the device area division result described in step S1, the devices A, B, D, and H in the high-frequency access area are centrally memory mapped with large pages (64K / 2M) as the granularity. The centralized memory mapping refers to summing up the address spaces required to be mapped for the four devices A, B, D, and H, and using the summation result as a parameter to map the memory space with large pages as the granularity.

[0032] Devices E, F, G, L, M, N, I, J, P, Q, C, and K in the low-frequency access area are independently memory mapped with small pages (4K) as the granularity.

[0033] S3: Dynamically adjust page table allocation.

[0034] An access frequency threshold is set. When the system is running, the monitoring module monitors the access frequency of the PCIe device configuration space in real time and merges or splits the page table through the memory management unit in the Linux operating system kernel.

[0035] Monitoring module: Use performance counters (such as perf) to count the access frequency and continuity of the PCIe device configuration space. When the device access frequency of the low-frequency access area exceeds the set frequency threshold, the page merge operation is triggered; when the device access frequency of the high-frequency access area is lower than the set frequency threshold, the page split operation is triggered.

[0036] Page merging: If devices C and K are frequently accessed continuously and the access frequency exceeds the set threshold, they will be re-divided into high-frequency access areas, and the small page space allocated to them will be released, and memory mapping applications will be made in the centrally allocated large page space.

[0037] Page splitting: If fragmented access occurs to H devices in the large page area and the access frequency is lower than the set threshold, the H devices are re-divided into low-frequency access area devices, released from the memory mapping space in the large page, and re-mapped to independent memory with small pages as the granularity.

[0038] This method is suitable for scenarios with high-frequency access to PCIe devices (such as data center GPUs and NVMe storage), and can significantly improve the overall system throughput and response speed.

[0039] The embodiment of the present invention also provides a PCIe ECAM address mapping optimization system, including a PCIe device area division module and a page table allocation module. The PCIe device area division module is used to formulate a rule for PCIe device area division, and based on the rule, divide the PCIe devices into high-frequency access area devices and low-frequency access area devices according to the device distribution in the PCIe bus topology; The page table allocation module includes static pre-allocation of page tables and dynamic adjustment of page table allocation; The page table static pre-allocation: based on the PCIe device area division, the devices in the high-frequency access area are centrally memory mapped with large pages as the granularity, and the devices in the low-frequency access area are independently memory mapped with small pages as the granularity; The dynamic adjustment of page table allocation: setting an access frequency threshold, monitoring the access frequency of the PCIe device configuration space when the system is running, and merging or splitting the page table through the memory management unit; The system specifically optimizes the PCIe ECAM address mapping through the PCIe ECAM address mapping optimization method described in the above embodiment.

[0040] The embodiment of the present invention also provides a PCIe ECAM address mapping optimization device, comprising: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the PCIe ECAM address mapping optimization method described in the above embodiment.

[0041] The embodiment of the present invention further provides a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the processor executes the PCIeECAM address mapping optimization method described in the above embodiment. Specifically, a system or device equipped with a storage medium can be provided, on which software program codes for implementing the functions of any of the above embodiments are stored, and a computer (or CPU or MPU) of the system or device reads out and executes the program code stored in the storage medium.

[0042] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0043] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0044] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0045] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0046] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A method for optimizing PCIe ECAM address mapping, characterized in that: The implementation of this method includes: PCIe device area division: formulate rules for PCIe device area division, and based on the rules, divide PCIe devices into high-frequency access area devices and low-frequency access area devices according to the device distribution in the PCIe bus topology; Page table allocation: including static pre-allocation of page tables and dynamic adjustment of page table allocation, The page table static pre-allocation: based on the PCIe device area division, the devices in the high-frequency access area are centrally memory mapped with large pages as the granularity, and the devices in the low-frequency access area are independently memory mapped with small pages as the granularity; The dynamic adjustment of page table allocation: setting an access frequency threshold, monitoring the access frequency of the PCIe device configuration space when the system is running, and merging or splitting the page table through the memory management unit.

2. The PCIe ECAM address mapping optimization method according to claim 1, characterized in that: The large page has a page table granularity of 64K or 2M; the small page has a page table granularity of 4K.

3. The PCIe ECAM address mapping optimization method according to claim 1, characterized in that: The PCIe device area division, PCIe devices with two or more downstream devices are classified as high-frequency access area devices, and PCIe devices with less than two downstream devices are classified as low-frequency access area devices.

4. The PCIe ECAM address mapping optimization method according to claim 1, characterized in that: The centralized memory mapping refers to summing up the address spaces required to be mapped by the devices in each high-frequency access area, and using the summation result as a parameter and large pages as the granularity to perform memory space mapping.

5. The PCIe ECAM address mapping optimization method according to claim 1, characterized in that: The system monitors the access frequency of the PCIe device configuration space when it is running, The access frequency and continuity of the PCIe device configuration space are counted through performance counters. When the device access frequency of the low-frequency access area exceeds the set frequency threshold, a page merge operation is triggered; when the device access frequency of the high-frequency access area is lower than the set frequency threshold, a page split operation is triggered.

6. A PCIe ECAM address mapping optimization method according to claim 1 or 5, characterized in that: The merged page table, If the devices in the low-frequency access area are frequently accessed and the access frequency exceeds the set threshold, they will be re-divided into the high-frequency access area, and the small page space allocated to them will be released, and memory mapping application will be made in the centrally allocated large page space.

7. A PCIe ECAM address mapping optimization method according to claim 1 or 5, characterized in that: The split page table, If fragmented access occurs to devices in the high-frequency access area and the access frequency is lower than the set threshold, the device is reclassified as a low-frequency access area device, and its memory mapping space in the large page is released and re-mapped independently with small pages as the granularity.

8. A PCIe ECAM address mapping optimization system, characterized in that: Including PCIe device area division module and page table allocation module, The PCIe device area division module is used to formulate a rule for PCIe device area division, and based on the rule, divide the PCIe devices into high-frequency access area devices and low-frequency access area devices according to the device distribution in the PCIe bus topology; The page table allocation module includes static pre-allocation of page tables and dynamic adjustment of page table allocation; The page table static pre-allocation: based on the PCIe device area division, the devices in the high-frequency access area are centrally memory mapped with large pages as the granularity, and the devices in the low-frequency access area are independently memory mapped with small pages as the granularity; The dynamic adjustment of page table allocation: setting an access frequency threshold, monitoring the access frequency of the PCIe device configuration space when the system is running, and merging or splitting the page table through the memory management unit; The system specifically implements the optimization of PCIe ECAM address mapping through any method described in claims 1 to 7.

9. A PCIe ECAM address mapping optimization device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the method described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images