An Optimization Method and System for PCIe ECAM Address Mapping

Abstract

The present invention discloses an optimization method and system for PCIe ECAM address mapping, belonging to the technical field of operating system optimization, including: formulating rules for PCIe device area division, and based on the rules, dividing PCIe devices into high-frequency access area devices and low-frequency access area devices according to the device distribution in the PCIe bus topology; page table allocation: including static pre-allocation of page tables and dynamic adjustment of page table allocation. Based on the above-mentioned PCIe device area division, devices in the high-frequency access area are subjected to centralized memory mapping with large pages as the granularity, and devices in the low-frequency access area are subjected to independent memory mapping with small pages as the granularity; setting an access frequency threshold, monitoring the access frequency of the PCIe device configuration space during system operation, and merging or splitting page tables through the memory management unit. The present invention can significantly reduce the number of page table entries, improve the hit rate of the TLB, and enhance the overall memory access efficiency of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of operating system optimization, and specifically to an optimization method and system for PCIe ECAM address mapping. Background Art

[0002] PCIe ECAM (Enhanced Configuration Access Mechanism) is a configuration space access mechanism defined by the PCIe specification. In the system, the configuration space of each PCIe device is mapped to a specific memory address area in the ECAM, and the driver configures and manages the PCIe device by accessing these memory areas.

[0003] In the PCIe ECAM mechanism, address mapping is the key to managing the access to the PCIe device configuration space. Traditional ECAM mapping usually adopts a unified granularity, but this approach is not efficient enough in some cases. Especially when dealing with a large number of PCIe devices, it will cause the page table entries to explode, resulting in problems such as a decrease in the TLB hit rate and a decrease in memory access efficiency of the system. Summary of the Invention

[0004] The technical task of the present invention is to provide an optimization method and system for PCIe ECAM address mapping to significantly reduce the number of page table entries, improve the TLB hit rate, and enhance the overall memory access efficiency of the system in view of the above deficiencies.

[0005] The technical solution adopted by the present invention to solve its technical problems is as follows:

[0006] An optimization method for PCIe ECAM address mapping, the implementation of this method includes:

[0007] PCIe device area division: Formulate rules for PCIe device area division. Based on these rules, according to the device distribution in the PCIe bus topology, divide PCIe devices into high-frequency access area devices and low-frequency access area devices;

[0008] Page table allocation: including static pre-allocation of the page table and dynamic adjustment of page table allocation,

[0009] The static pre-allocation of the page table: Based on the above-mentioned PCIe device area division, map the devices in the high-frequency access area to the memory in a large page granularity, and map the devices in the low-frequency access area to the memory independently in a small page granularity;

[0010] The dynamic adjustment of page table allocation: Set an access frequency threshold, monitor the access frequency of the PCIe device configuration space during system operation, and merge or split the page table through the memory management unit.

[0011] This method proposes a hybrid granularity PCIe ECAM address mapping method, which combines hybrid granularity mapping with PCIe configuration space management, improves memory access efficiency by flexibly adjusting the mapping granularity of different PCIe bus devices, divides PCIe devices into regions, monitors the access frequency of each device on the PCIe bus, and dynamically uses different sizes of page table granularity (such as 4KB, 64K, 2M) in ECAM address mapping to balance memory usage, TLB efficiency and memory access performance. Traditional fixed granularity mapping (such as full 4KB page) will cause an explosive growth of page table entries when the number of devices is large, while hybrid granularity can significantly reduce the number of page table entries by flexibly adjusting the page size, improve the TLB hit rate, and improve the overall memory access efficiency of the system.

[0012] Furthermore, the large page has a page table granularity of 64K or 2M; and the small page has a page table granularity of 4K.

[0013] Furthermore, the PCIe device area division is:

[0014] PCIe devices with two or more downstream devices are classified as high-frequency access area devices, and PCIe devices with less than two downstream devices are classified as low-frequency access area devices.

[0015] Furthermore, the centralized memory mapping refers to summing up the address spaces required to be mapped by the devices in each high-frequency access area, and using the summation result as a parameter and large pages as the granularity to perform memory space mapping.

[0016] Furthermore, the system monitors the access frequency of the PCIe device configuration space when it is running.

[0017] The access frequency and continuity of the PCIe device configuration space are counted through performance counters (such as perf). When the device access frequency of the low-frequency access area exceeds the set frequency threshold, the page merge operation is triggered; when the device access frequency of the high-frequency access area is lower than the set frequency threshold, the page split operation is triggered.

[0018] Furthermore, the merged page table,

[0019] If the devices in the low-frequency access area are frequently accessed and the access frequency exceeds the set threshold, they will be re-divided into the high-frequency access area, and the small page space allocated to them will be released, and memory mapping application will be made in the centrally allocated large page space.

[0020] Furthermore, the split page table,

[0021] For devices in high-frequency access areas with fragmented access and an access frequency lower than the set threshold, the device is reclassified as a low-frequency access area device, and its memory mapping space within the large page is released, and independent memory mapping is performed again with small pages as the granularity.

[0022] The present invention also claims to protect an optimized system for PCIe ECAM address mapping, including a PCIe device area division module and a page table allocation module.

[0023] The PCIe device area division module is used to formulate rules for PCIe device area division. Based on the rules, according to the device distribution in the PCIe bus topology, the PCIe devices are divided into high-frequency access area devices and low-frequency access area devices.

[0024] The page table allocation module includes static pre-allocation of the page table and dynamic adjustment of page table allocation.

[0025] The static pre-allocation of the page table: Based on the above-mentioned PCIe device area division, the devices in the high-frequency access area are centrally memory-mapped with large pages as the granularity, and the devices in the low-frequency access area are independently memory-mapped with small pages as the granularity.

[0026] The dynamic adjustment of page table allocation: Set an access frequency threshold, monitor the access frequency of the PCIe device configuration space during system operation, and merge or split the page table through the memory management unit.

[0027] This system specifically realizes the optimization of PCIe ECAM address mapping through the above method.

[0028] The present invention also claims to protect an optimized device for PCIe ECAM address mapping, including: at least one memory and at least one processor.

[0029] The at least one memory is used to store machine-readable programs.

[0030] The at least one processor is used to call the machine-readable program to implement the above method.

[0031] The present invention also claims to protect a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the above method is implemented.

[0032] Compared with the prior art, an optimized method and system for PCIe ECAM address mapping of the present invention have the following beneficial effects:

[0033] 1. Through the optimized method of PCIe ECAM address mapping with a mixed granularity, the number of page table entries occupied by PCIe ECAM address mapping is reduced, and the page table memory occupancy is reduced.

[0034] 2. Optimization method for PCIe ECAM address mapping with mixed granularity. Large pages enable a single TLB entry to cover a larger address range, improving the TLB hit rate.

[0035] 3. Optimization method for PCIe ECAM address mapping with mixed granularity. Large pages cover more device configuration space, reducing the page table hierarchy and improving the efficiency of accessing configuration registers. Description of the Drawings

[0036] Figure 1 is a diagram showing the optimization method for PCIe ECAM address mapping provided by an embodiment of the present invention;

[0037] Figure 2 is a PCIe bus topology diagram provided by an embodiment of the present invention. Detailed Embodiments

[0038] The present invention will be further described below in conjunction with specific embodiments.

[0039] The embodiment of the present invention provides an optimization method for PCIe ECAM address mapping, which divides PCIe devices into regions, monitors the access frequencies of devices on the PCIe bus, and dynamically uses page table granularities of different sizes (such as 4KB, 64K, 2M) in the ECAM address mapping to balance memory occupancy, TLB efficiency, and memory access performance. Traditional fixed-granularity mapping (such as all 4KB pages) will cause an explosive growth of page table entries when the number of devices is large, while mixed granularity significantly reduces the number of page table entries by flexibly adjusting the page size, improves the TLB hit rate, and enhances the overall memory access efficiency of the system.

[0040] The specific implementation of this PCIe ECAM address mapping method with mixed granularity is as follows:

[0041] 1. PCIe device region division:

[0042] Formulate rules for PCIe device region division. Based on these rules, according to the device distribution in the PCIe bus topology, divide PCIe devices into high-frequency access region devices and low-frequency access region devices.

[0043] 2. Page table allocation: including static pre-allocation of the page table and dynamic adjustment of page table allocation.

[0044] The static pre-allocation of the page table: Based on the above-mentioned PCIe device region division, map the devices in the high-frequency access region to memory in a concentrated manner with large pages (64K / 2M) as the granularity, and map the devices in the low-frequency access region to memory independently with small pages (4K) as the granularity.

[0045] The dynamic adjustment of page table allocation: Set an access frequency threshold. When the system is running, monitor the access frequency of the PCIe device configuration space, and merge or split the page table through the memory management unit.

[0046] To enable those skilled in the art to better understand and implement this method, the following will Figure 1 、 Figure 2 be used to detail the implementation process of this method. The implementation manner of this method includes the following steps:

[0047] S1: PCIe device area division.

[0048] Formulate the rules for device area division. According to the device distribution in the PCIe bus topology, divide the PCIe devices into high-frequency access area devices and low-frequency access area devices.

[0049] As shown in the Figure 2 device distribution in the PCIe bus topology, based on the preset rules, divide the PCIe devices with more than 2 downstream devices into high-frequency access area devices, and divide the PCIe devices with less than 2 downstream devices into low-density area devices. Then the device area division results are as follows:

[0050] High-frequency access area devices: A, B, D, H.

[0051] Low-frequency access area devices: E, F, G, L, M, N, I, J, P, Q, C, K.

[0052] S2: Static pre-allocation of page tables.

[0053] Based on the device area division results described in step S1, perform centralized memory mapping on the devices A, B, D, H in the high-frequency access area with large pages (64K / 2M) as the granularity. The so-called centralized memory mapping means summing up the address spaces required to be mapped by the four devices A, B, D, H, and using the summation result as a parameter to perform memory space mapping with large pages as the granularity.

[0054] Perform independent memory mapping on the devices E, F, G, L, M, N, I, J, P, Q, C, K in the low-frequency access area with small pages (4K) as the granularity.

[0055] S3: Dynamic adjustment of page table allocation.

[0056] Set an access frequency threshold. When the system is running, the monitoring module monitors the access frequency of the PCIe device configuration space in real time, and merges or splits the page table through the memory management unit in the linux operating system kernel.

[0057] Monitoring module: Statistically analyze the access frequency and continuity of the PCIe device configuration space through performance counters (such as perf). When the access frequency of a device in the low-frequency access area exceeds the set frequency threshold, a page merging operation is triggered. When the access frequency of a device in the high-frequency access area is lower than the set frequency threshold, a page splitting operation is triggered.

[0058] Page merging: If devices C and K are continuously accessed frequently and the access frequency exceeds the set threshold, they are reclassified into the high-frequency access area, and the small page space allocated to them is released. A memory mapping application is then performed again in the large page space allocated centrally.

[0059] Page splitting: If device H in the large page area has fragmented access and the access frequency is lower than the set threshold, device H is reclassified as a device in the low-frequency access area, and its memory mapping space in the large page is released. Independent memory mapping is then performed again in units of small pages.

[0060] This method is applicable to scenarios where PCIe devices are accessed frequently (such as GPUs in data centers and NVMe storage), and can significantly improve the overall system throughput and response speed.

[0061] The embodiment of the present invention also provides an optimized system for PCIe ECAM address mapping, including a PCIe device area division module and a page table allocation module.

[0062] The PCIe device area division module is used to formulate rules for dividing PCIe device areas. Based on these rules, according to the device distribution in the PCIe bus topology, PCIe devices are divided into high-frequency access area devices and low-frequency access area devices.

[0063] The page table allocation module includes static pre-allocation of the page table and dynamic adjustment of page table allocation.

[0064] The static pre-allocation of the page table: Based on the division of the PCIe device area, devices in the high-frequency access area are centrally memory-mapped in units of large pages, and devices in the low-frequency access area are independently memory-mapped in units of small pages.

[0065] The dynamic adjustment of page table allocation: Set an access frequency threshold, monitor the access frequency of the PCIe device configuration space during system operation, and merge or split the page table through the memory management unit.

[0066] This system specifically realizes the optimization of PCIe ECAM address mapping through the optimized method for PCIe ECAM address mapping described in the above embodiment.

[0067] The embodiment of the present invention also provides an optimized device for PCIe ECAM address mapping, including: at least one memory and at least one processor;

[0068] The at least one memory is configured to store machine-readable programs.

[0069] The at least one processor is configured to call the machine-readable programs to implement the optimization method for PCIe ECAM address mapping described in the above embodiments.

[0070] An embodiment of the present invention further provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor is caused to execute the optimization method for PCIe ECAM address mapping described in the above embodiments. Specifically, a system or device equipped with a storage medium can be provided, on which software program codes for implementing the functions of any one of the above embodiments are stored, and the computer (or CPU or MPU) of the system or device is caused to read and execute the program codes stored in the storage medium.

[0071] In this case, the program codes read from the storage medium itself can implement the functions of any one of the above embodiments. Therefore, the program codes and the storage medium storing the program codes constitute a part of the present invention.

[0072] Embodiments of the storage medium for providing program codes include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program codes can be downloaded from a server computer via a communication network.

[0073] Furthermore, it should be clear that not only can the functions of any one of the above embodiments be implemented by executing the program codes read by a computer, but also by causing an operating system or the like operating on the computer based on the instructions of the program codes to complete part or all of the actual operations.

[0074] Furthermore, it can be understood that the program codes read from the storage medium are written into a memory provided in an expansion board inserted into the computer or a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program codes, a CPU or the like installed on the expansion board or the expansion unit is caused to execute part and all of the actual operations, thereby implementing the functions of any one of the above embodiments.

[0075] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that more embodiments of the present invention can be obtained by combining the code review means in the above different embodiments, and these embodiments are also within the protection scope of the present invention.

Claims

1. An optimization method for PCIe ECAM address mapping, characterized in that, The implementation of this method includes: PCIe device area division: formulating rules for PCIe device area division, and based on these rules, dividing PCIe devices into high-frequency access area devices and low-frequency access area devices according to the device distribution in the PCIe bus topology; Page table allocation: including static pre-allocation of page tables and dynamic adjustment of page table allocation, The static pre-allocation of page tables: based on the above-mentioned PCIe device area division, performing centralized memory mapping for devices in the high-frequency access area with large pages as the granularity, and performing independent memory mapping for devices in the low-frequency access area with small pages as the granularity; The dynamic adjustment of page table allocation: setting an access frequency threshold, monitoring the access frequency of the PCIe device configuration space during system operation, and merging or splitting page tables through the memory management unit; The PCIe device area division, dividing PCIe devices with 2 or more downstream devices into high-frequency access area devices, and dividing PCIe devices with less than 2 downstream devices into low-frequency access area devices; The centralized memory mapping refers to summing up the address spaces required for mapping each high-frequency access area device, and using the summation result as a parameter to perform memory space mapping with large pages as the granularity; During system operation, monitoring the access frequency of the PCIe device configuration space, and statistically analyzing the access frequency and continuity of the PCIe device configuration space through performance counters. When it is detected that the access frequency of a device in the low-frequency access area exceeds the set frequency threshold, a page merging operation is triggered; when it is detected that the access frequency of a device in the high-frequency access area is lower than the set frequency threshold, a page splitting operation is triggered.

2. The optimized method for PCIe ECAM address mapping according to claim 1, characterized in that The large page is a page table granularity of 64K or 2M; the small page is a page table granularity of 4K.

3. An optimization method for PCIe ECAM address mapping according to claim 1, characterized in that The merging of page tables, For devices in the low-frequency access area with frequent consecutive accesses and an access frequency exceeding the set threshold, re-divide them into the high-frequency access area, release the small page space allocated to them, and re-apply for memory mapping in the centrally allocated large page space.

4. An optimization method for PCIe ECAM address mapping according to claim 1, characterized in that, The splitting of page tables, For devices in the high-frequency access area with fragmented accesses and an access frequency lower than the set threshold, re-divide the device into a low-frequency access area device, release its memory mapping space within the large page, and re-perform independent memory mapping with small pages as the granularity.

5. An optimized system for PCIe ECAM address mapping, characterized in that, It includes a PCIe device area division module and a page table allocation module, The PCIe device area division module is used to formulate rules for PCIe device area division, and based on these rules, divide PCIe devices into high-frequency access area devices and low-frequency access area devices according to the device distribution in the PCIe bus topology; The page table allocation module includes static pre-allocation of page tables and dynamic adjustment of page table allocation; The static pre-allocation of page tables: based on the above-mentioned PCIe device area division, performing centralized memory mapping for devices in the high-frequency access area with large pages as the granularity, and performing independent memory mapping for devices in the low-frequency access area with small pages as the granularity; The dynamic adjustment of page table allocation: setting an access frequency threshold, monitoring the access frequency of the PCIe device configuration space during system operation, and merging or splitting page tables through the memory management unit; The system specifically optimizes the PCIe ECAM address mapping through the method described in any one of claims 1 to 4.

6. An optimized device for PCIe ECAM address mapping, characterized in that, It includes: At least one memory and at least one processor; The at least one memory is used to store machine-readable programs; The at least one processor is used to call the machine-readable program to implement the method described in any one of claims 1 to 4.

7. A computer-readable medium, characterized in that, Computer instructions are stored on the computer-readable medium, and when the computer instructions are executed by the processor, the method described in any one of claims 1 to 4 is implemented.

Images