A virtual machine page table tracking method and system based on PML

Abstract

The present invention relates to a virtual machine page table tracking method and system based on PML. The method comprises: in a virtual machine, obtaining the page table pages used by the virtual machine load by scanning processes with a resident set greater than 1GB and transmitting them back to the hypervisor to obtain a monitoring target; periodically clearing the Dirty bit in the EPT of the virtual machine page table page and the Access bit and Dirty bit of the virtual machine page table entries contained therein, utilizing Intel PML technology to efficiently obtain the page table page entries where write operations have occurred, further scanning the virtual machine page table entries therein to obtain the pages where read and write operations have occurred within this cycle; and proposing a multi-level queue algorithm to accelerate tracking efficiency. The solution proposed by the present invention makes full use of the hardware characteristics of Intel PML to track the page tables used by the client, alleviating the overhead of the traditional page table scanning method that requires traversing all the page tables of the virtual machine; using the multi-level queue algorithm to screen the tracking pages, greatly reducing the number of pages that need to be set, and further reducing the overall tracking overhead.

Description

Technical Field

[0001] The present invention relates to the field of memory virtualization, and in particular to a method and system for tracing a virtual machine page table based on Intel PML. Background Art

[0002] Virtualization is a foundational technology in cloud computing. It involves running one or more virtual computers (virtual machines, or clients) on a single physical computer. These virtual machines have independent virtual hardware and operating environments, but effectively share the physical computer's resources, such as the CPU, memory, and network interface. Essentially, virtualization involves a software layer residing on the physical machine providing the virtual machine with an interface that is identical to the operating environment it requires. This abstracted virtual software or hardware interface enables the upper-layer virtual machine to run directly in the virtual environment. This software layer residing on the physical machine is the virtual machine monitor (or virtual machine monitor, VMM, or hypervisor), which controls the virtual machine operating system's access to physical machine resources and provides abstraction for the virtual machine. The core of virtualization technology lies in three key areas: I / O virtualization, CPU virtualization, and memory virtualization. With the emergence of memory-intensive applications in current cloud services and the ever-increasing memory requirements, the memory access efficiency provided by memory virtualization has become a key performance bottleneck.

[0003] The primary purpose of memory virtualization is to provide virtual machines with a continuous physical memory space starting from zero, and to effectively isolate, schedule, and manage shared memory resources between virtual machines. To achieve these two goals, the VMM introduces a new address space, the guest physical address space. This space, distinct from the actual physical address space in a physical machine, is an abstraction of physical addresses provided to virtual machines. Therefore, memory virtualization requires two-dimensional address translation: from the guest virtual address (GVA) where the application resides to the guest physical address (GPA), and finally to the host physical address (HPA). The former translation is performed by the client's operating system, while the latter requires additional software or hardware support from the physical machine.

[0004] Memory virtualization schemes in mainstream architectures fall into two categories: software-based shadow page tables and hardware-based nested page tables. Currently, most architectures have introduced hardware-assisted virtualization support, such as Intel Virtualization Technology (Intel VT). Intel VT provides an Extended Page Table (EPT) to store the mapping between GPA and HPA, with physical hardware performing secondary address translation. Each virtual machine's EPT is maintained by the VMM. Beyond address mapping information, it is similar to a regular page table. EPT page table entries also contain permission bits for the corresponding physical page. Bits 8 and 9 of an EPT entry are the access bit and dirty bit, respectively. When a virtual machine accesses the GPA corresponding to an EPT entry, the access bit is set by the hardware. Similarly, when a write occurs, the dirty bit of the corresponding EPT page table entry is set.

[0005] Intel PML technology is designed to optimize dirty page handling during live migration of virtual machines. During live migration, the virtual machine's memory accesses are not interrupted, so additional tracking and retransmission of dirty pages generated during this process is required. Previously, all pages of the virtual machine were write-protected, so any writes would trigger an expensive virtual machine exit (VM-exit) to retrieve dirty pages during migration. Intel PML technology, however, uses hardware to directly track dirty pages within the virtual machine, eliminating the need for a VM-exit. Its key principle is to capture the GPA generated by memory accesses, which sets the dirty page bit for EPT page table entries. Specifically, before a virtual machine performs a memory access, the processor determines whether the access requires setting the dirty page bit for the corresponding EPT page table entry. If Intel PML technology is enabled, the GPA of the current memory access is recorded by the MMU (Memory Management Unit) in a PML buffer. The virtual machine's CPU is abstracted by the VCPU structure, with each VCPU maintaining a 512-entry PML buffer. When the buffer is full, a VM-exit is generated, returning the structure to the VMM for further processing. Furthermore, when a dirty page needs to be retrieved again, it is only necessary to clear the dirty bit of the set EPT entry and refresh the translation lookaside buffer (TLB) to proceed to the next round of processing.

[0006] Page table tracking technology aims to capture virtual machine page accesses, providing fundamental information for analyzing virtual machine memory access behavior and optimizing performance. Page table tracking technology is now widely used in heterogeneous memory management systems. Its page monitoring can identify application memory access hotspots, enabling efficient management of page placement within heterogeneous memory. Specifically, in non-virtualized environments, page table tracking primarily tracks process page tables, periodically setting and checking the Access and Dirty bits in page table entries. After multiple cycles, the page access frequency during that period can be determined. In virtualized environments, due to the existence of two-level address translation, current page table tracking technology primarily periodically sets and checks the virtual machine's EPT page table, typically requiring a complete traversal of the virtual machine's EPT page table. With the increasing number of large working set programs, cloud services are requiring increasingly larger virtual machine memory. Therefore, for virtual machines with larger allocated memory, even if they are currently idle and unloaded, enabling page table tracking will result in a greater number of EPT page table entries being traversed per cycle, resulting in increased overhead. Make full use of virtualization features and rich hardware support, such as using Intel PML technology designed for virtual machine hot migration to capture information about pages being used and avoid scanning the entire page table. This can also provide a more flexible solution for page table tracking technology. Summary of the Invention

[0007] The purpose of the present invention is to fully utilize the Intel PML mechanism in a virtualized environment to implement efficient virtual machine page table tracking technology. Specifically, the present invention fully utilizes the rich hardware support in the virtualized environment and utilizes the characteristics of Intel PML to efficiently capture dirty pages of virtual machines, thereby realizing a page table tracking solution with smaller page table scan volume and lower overhead. The core idea of ​​the present invention is to utilize the Intel PML mechanism to efficiently capture and record writes to virtual machine page table pages, periodically traverse the captured page table pages, check the Access bit and Dirty bit of the page table entries, record and clear the set pages, and propose a multi-machine queue algorithm to further filter the pages that need to be set to avoid frequent setting overhead, and ultimately be able to obtain the memory access status of the page with low page scan volume and low overhead.

[0008] The technical solution adopted in the present invention is as follows:

[0009] A virtual machine page table tracing method based on PML includes the following steps:

[0010] When page table tracking technology is enabled, the virtual machine page table is scanned first, traversing the virtual machine's process page table;

[0011] Obtain the virtual machine's last-level GPT pointer (i.e., page table page address) through page table traversal and save it to the last-level page table pointer buffer (LL-gPTP buffer) in the virtual machine. After the scan is completed, the LL-gPTP buffer is transferred back to the hypervisor.

[0012] The hypervisor receives the LL-gPTP buffer from the virtual machine, traverses the page table entries in the page table page, clears the access bit and dirty bit of these data page table entries in the GPT (guest page table), clears the dirty bit of the page table page in the EPT, and flushes the TLB, then starts a loop to monitor the pages that are accessed.

[0013] Furthermore, the traversing of the process page table of the virtual machine is to select the process page table with a resident set larger than 1GB for page table traversal.

[0014] Furthermore, after the scanning is completed, hypercall communication is called to transmit the last-level page table pointer buffer of the virtual machine back to the hypervisor.

[0015] Furthermore, in each of the cycles:

[0016] The PML monitor in the hypervisor captures the PML logs generated by the hardware from the PML buffer in real time, filters out the page table pages corresponding to the LL-gPTP buffer, and clears the PML buffer.

[0017] Furthermore, for the page table page PML log captured in the current cycle, the page table entries of the GPT pages contained therein are scanned, the Access bit and Dirty bit of the page table entries of these data pages are recorded, and at the same time, some GPT page table entries that have been set are filtered out using a multi-level queue algorithm;

[0018] Clear the Dirty bit of the captured page table page in the EPT and refresh the TLB to enter the next cycle.

[0019] Furthermore, the multi-level queue algorithm process is as follows:

[0020] At the beginning, all pages are at level 0. These set pages will immediately clear the Access bit and Dirty bit in the current cycle;

[0021] If a page has just had its Access bit and Dirty bit cleared in the previous cycle and is captured again in this cycle, the page level will be increased by 1; the page with level k will be in 2 k-1After a cycle, the Access bit and Dirty bit are cleared; pages that have not yet reached the cleaning cycle will remain set to avoid overhead; if a page is not captured again immediately after being cleaned, the level of the page will be reduced by 1;

[0022] Finally, after the set multiple cycle loops are completed, the number of read and write times for each page in the virtual machine has been successfully counted.

[0023] A PML-based virtual machine page table tracking system using the above method includes:

[0024] The virtual machine page table scanning module is located inside the virtual machine and is used to scan the virtual machine page table when the page table tracking function is enabled, and transfer the page table pages to the hypervisor for processing;

[0025] The page table cycle monitoring module is located at the Hypervisor layer and is used to receive the virtual machine page table page address, periodically clear the Dirty bit of the EPT table entry of the page table page, periodically clear the Access bit and Dirty bit of the virtual machine page table entry in the page table page and refresh the TLB. During the cycle, it reads the record results of the PML to obtain the virtual machine page table access status.

[0026] The proposed PML-based virtual machine page table tracking technology leverages the Intel PML hardware features, specifically for efficiently capturing dirty pages within virtual machines. On the one hand, the present invention builds on traditional page table scanning by utilizing Intel PML technology to track only the pages currently in use by the virtual machine, avoiding the drawback of traditional technologies that require scanning all virtual machine pages. This significantly reduces the number of page scans required for page table tracking, thereby lowering overhead. On the other hand, the present invention proposes a multi-level queue algorithm that leverages the temporal locality of memory accesses to filter captured pages, avoiding frequent page table resets. This significantly reduces page table tracking overhead while ensuring accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a framework diagram of the virtual machine page table tracking technology based on PML.

[0028] Figure 2 This is a test result chart of the page scan amount and performance loss of a virtual machine loaded with a 16GB random read program using PML-based virtual machine page table tracking technology, where (a) is the page scan amount and (b) is the page tracking performance loss. DETAILED DESCRIPTION

[0029] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to specific embodiments and accompanying drawings.

[0030] The "PML" mentioned in the present invention refers to the additional hardware support provided by Intel for virtual machine live migration. It is a mechanism for efficiently obtaining dirty pages of virtual machines by checking the Dirty bit of EPT page table entries through hardware.

[0031] "Page table tracking," as described in this invention, refers to a technique used by the virtual machine monitoring layer in a virtualized environment to obtain information about virtual machine page accesses, specifically the number of page reads and writes within a specific time period. This is a fundamental technology for virtual machine memory access analysis and optimization, and a key technology for heterogeneous memory management systems.

[0032] The "multi-level queue" algorithm described in this invention is designed to further reduce the overhead of page table tracking. Page table tracking is typically implemented by periodically clearing and setting the access and dirty bits. Setting these bits for all pages is costly. Therefore, the multi-level queue leverages the locality of page access to upgrade frequently accessed pages, skipping the setting process for these pages and accumulating counts. Since clearing and setting the permission bits of page table entries and TLB flushes are the main sources of page table tracking overhead, the multi-level queue can significantly reduce page table tracking overhead.

[0033] Kernel-based Virtual Machine (KVM) is an open-source, efficient, native Linux full virtualization solution that implements virtual machines as regular Linux processes within a physical machine. It is widely used in the virtualization field. This paper implements PML-based virtual machine page table tracking technology on Intel Optane servers using KVM as the virtualization solution. Figure 1 The PML-based virtual machine page table tracking technology framework is shown. Hypercall represents a communication mechanism from a virtual machine to a physical machine, OS represents an operating system, A / D bits represent access / dirty bits, respectively, the dotted box in the host OS represents the KVM virtualization solution, the EPT walker in the hardware part represents the hardware page table traversal process, PMLbuffer represents the buffer for recording dirty pages provided by the Intel PML mechanism, TLB represents the translation bypass buffer, AccessedGPT buffer represents the virtual machine page that has been accessed scanned from the page table page, LL-gPTP buffer represents the last-level page table address buffer, GPT represents the client page table, and EPT represents the extended page table. In the present invention, the page table tracking technology mainly ensures that the virtual machine page access situation can be tracked efficiently and accurately, which can reduce the amount of page scanning and reduce the setting overhead.

[0034] 1. Page table tracking process:

[0035] The task of the PML-based virtual machine page table tracking technology is to efficiently track the virtual machine page access situation.

[0036] 1) Initialization. After the page table tracking technology is turned on, first enter the virtual machine operating system to perform page table scanning ( Figure 1 The page table scanning module in the virtual machine reads and traverses the process page table of the virtual machine, selects the process page table with a resident set larger than 1GB to perform a four-level page table traversal, obtains the last level page table pointer (i.e., the page table page address), and writes it into the last level page table pointer buffer ( Figure 1 The hypercall function is called to notify the host OS to read the page table page address. After the host OS obtains the virtual machine page table page address, it clears the access bit and dirty bit of the GPT page table entry contained in the page table page ( Figure 1 The GPT is cleaned up by the A / D bit operation in the GPT), and the dirty bit of the EPT entry corresponding to the page table page is cleared ( Figure 1 At the same time, the TLB is refreshed so that the PML can hardware-fetch the modified pages.

[0037] 2) Intel PML records dirty pages. When a virtual machine page is read or written, the address translation hardware automatically fills in the A / D bits of the corresponding data page table entry in the page table page that have been cleared, that is, a write occurs to the page table page. At this time, when the EPT page table is traversed for the page table page, the cleared EPT dirty bit will be set, and Intel PML will obtain this setting information and record the corresponding page table page in the buffer ( Figure 1 PML buffer in PBS).

[0038] 3) Periodically scan the permission bits. For each cycle, the hypervisor reads the PML buffer to obtain the page table pages that have been modified in this cycle. Traverse the GPT data page table entries contained in the page table page and check the A / D bits of these page table entries. If it is set, it means that the page corresponding to this page table entry has been read or modified, and the access status counter of the page is accumulated accordingly. After all pages are counted, the access bit and dirty bit of the GPT page table entry contained in the page table page are cleared again, and the dirty bit of the EPT table entry corresponding to the page table page is cleared. At the same time, the TLB is refreshed so that the PML hardware can capture the modified page.

[0039] 4) Multi-level queue filtering sets the page. Each page has an additional queue level, which represents the frequency of access to the current page. If the page is set again in this cycle after the A / D bits were cleared in the previous monitoring cycle, it means that the page is frequently accessed and its queue level is increased by 1. If it is set two cycles or more after the A / D bits were cleared, it means that the page is not frequently accessed and its queue level is reduced by 1. For a page with queue level i, its A / D bits will be set at 2 i-1 Pages with a level of 0 are cleared in this cycle. Frequently accessed pages are set by default in cycles where they are not cleared, and no clearing and resetting overhead is incurred. More frequently accessed pages have a higher queue level and lower overhead.

[0040] In this embodiment, when the virtual machine is idle or operating with a low memory load, only a small number of virtual machine pages are modified, resulting in fewer pages being captured by the PML mechanism, and thus a lower page scan rate. When the virtual machine load increases, all pages are initially ranked at level 0. When a page is accessed again, its level is gradually increased, thereby reducing the reset overhead.

[0041] 2. Experimental evaluation:

[0042] In order to verify that the present invention can reduce the amount of page scans and reduce the overall overhead, a virtual machine with a total memory of 64GB and a load of 10GB random read program was used to test the page scan amount. By adjusting the size of the hot page set of the random read program, it is possible to abstract the running conditions of different programs. After the virtual machine is preheated, the virtual machine will establish a total of 64GB of memory page table mappings. The overall overhead of the present invention was verified using 429.mcf in the SPEC CPU 2006 test tool launched by the Standard Performance Evaluation Corporation, the classic memory database Redis and the classic graph computing test suite graph500. In particular, graph500 was configured for 8 iterations of BFS and SSSP, Redis was configured for 4k key-value pairs, the read-write ratio was 50%, and 80% of the accesses were concentrated on 20% of the key-value pairs. The memory access characteristics and application scenarios of the above tests are different, ensuring that the test results can more comprehensively reflect the performance of the present invention.

[0043] Figure 2 It is the page table scan amount obtained by using the PML-based page table tracking technology to perform page table access on the virtual machine ( Figure 2 (a)) and the resulting performance loss graph ( Figure 2 (b)). For comparison, Figure 2In (a), the X-axis represents the percentage of pages that 90% of program accesses are concentrated on (i.e., the hot page ratio), and the Y-axis represents the amount of page table scans performed by the page table tracking technique on the virtual machine. This indicates that, regardless of the virtual machine's operating conditions, the amount of page table scanned by the traditional method is related to the virtual machine's memory size, always scanning page table entries for 64GB pages. However, the amount of page table scanned by the PML-based page table tracking method is related to the amount of memory used by the running program in the virtual machine. When the virtual machine load is lighter or access is more concentrated, the amount of page table scans is also smaller. Figure 2 In (b), the shaded area represents the performance loss caused by this example, while the unshaded area represents the performance loss caused by the traditional method. The monitoring window indicates the periodic interval for clearing permission bits. It can be seen that for the traditional method, the smaller the interval, the higher the frequency of setting and clearing, resulting in greater performance loss. In the 429.mcf test, the traditional method even experienced a performance loss of up to 20%. Other tests also generally experienced high performance losses. However, the PML-based virtual machine page table tracking technology was able to maintain a performance loss below 5% across different tests and time intervals.

[0044] Experimental results show that the page table tracking technology of the present invention performs well across a variety of applications, maintaining performance loss below 5% across various monitoring period selections. Specifically, for the 429.mcf test program, the present invention reduces performance loss by nearly 8 times compared to traditional methods. For programs with diverse characteristics, the average performance loss is approximately 2 times lower than that of traditional methods.

[0045] Based on the same inventive concept, another embodiment of the present invention provides a PML-based virtual machine page table tracking system using the above method, comprising:

[0046] The virtual machine page table scanning module is used to initialize when the page table tracking function is turned on, scan the virtual machine page table, and obtain the page table page address.

[0047] The page table cycle monitoring module is used to read the virtual machine page table page address, periodically clear the Dirty bit of the page table page EPT table entry, periodically clear the Access bit and Dirty bit of the virtual machine page table entry in the page table page and refresh the TLB, read the PML record results within the cycle, and obtain the virtual machine page table access status.

[0048] Based on the same inventive concept, another embodiment of the present invention provides a computer device (computer, server, etc.), which includes the above-mentioned PML-based virtual machine page table tracking system.

[0049] The specific embodiments and drawings of the present invention disclosed above are intended to facilitate understanding of the present invention and its implementation. Those skilled in the art will appreciate that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the embodiments and drawings disclosed in this specification. The scope of protection of the present invention shall be determined by the scope defined in the claims.

Claims

1. A virtual machine page table tracking method based on PML, characterized in that: The following steps are involved: When page table tracing is enabled, the virtual machine page table is scanned to traverse the virtual machine's process page table; Obtain the virtual machine's last-level GPT pointer through page table traversal and save it to the virtual machine's last-level page table pointer buffer. After the scan is complete, the virtual machine's last-level page table pointer buffer is transferred back to the hypervisor. The hypervisor receives the last-level page table pointer buffer from the virtual machine, traverses the page table entries in the page table page, clears the access bit and dirty bit of the page table entry in the GPT, clears the dirty bit of the page table page in the EPT, and flushes the TLB. Then, it starts a loop to monitor the pages that are accessed. Among them, PML is a mechanism that efficiently obtains the dirty pages of the virtual machine by checking the Dirty bit of the EPT page table entry through hardware; GPT is the guest page table; EPT is the extended page table; In each of these cycles: The PML monitor in the hypervisor captures the PML logs generated by the hardware from the PML buffer in real time, filters out the page table pages in the last-level page table pointer buffer corresponding to the virtual machine, and clears the PML buffer at the same time; In each of these cycles: For the page table page PML log captured in the current cycle, scan the page table entries of the GPT pages contained therein, record the Access bit and Dirty bit of these data page page table entries, and use a multi-level queue algorithm to filter and clear the GPT page table entries that have been set; Clear the Dirty bit of the captured page table page in the EPT and refresh the TLB, entering the next cycle; The multi-stage queue algorithm includes the following steps: At the beginning, all pages are at level 0, and the set pages immediately clear the Access bit and Dirty bit in the current cycle; If a page has just had its Access bit and Dirty bit cleared in the previous cycle and is captured again in this cycle, the level of the page is increased by 1; the page with level k is increased by 2. k-1 The Access bit and Dirty bit are cleared after a cycle; pages that have not yet reached the cleaning cycle remain set to avoid overhead; if a page is not captured again immediately after being cleaned, the level of the page is reduced by 1; After the set multiple cycle cycles are completed, the number of read and write times of each page in the virtual machine has been successfully counted.

2. The method according to claim 1, characterized in that After the scan is completed, a hypercall communication is called to transmit the last-level page table pointer buffer of the virtual machine back to the hypervisor.

3. A virtual machine page table tracking system based on PML, characterized in that: include: The virtual machine page table scanning module is located inside the virtual machine and is used to scan the virtual machine page table when the page table tracking function is enabled, and transfer the page table pages to the hypervisor for processing; The page table cycle monitoring module, located at the hypervisor layer, is used to receive the virtual machine page table page address, periodically clear the dirty bit of the EPT table entry of the page table page, periodically clear the access bit and dirty bit of the virtual machine page table entry in the page table page, and refresh the TLB. During the cycle, it reads the PML record results to obtain the virtual machine page table access status. Among them, PML is a mechanism for efficiently obtaining virtual machine dirty pages through hardware checking the dirty bit of the EPT page table entry. EPT is an extended page table. The page table cycle monitoring module: Capture the PML logs generated by the hardware in real time from the PML buffer, filter out the page table pages in the last-level page table pointer buffer corresponding to the virtual machine, and clear the PML buffer at the same time; For the page table page PML log captured in the current cycle, scan the page table entries of the GPT pages contained in it, record the Access bit and Dirty bit of these data page page table entries, and use a multi-level queue algorithm to filter and clear the GPT page table entries that have been set. GPT is the client page table. Clear the Dirty bit of the captured page table page in the EPT and refresh the TLB, entering the next cycle; The multi-stage queue algorithm includes the following steps: At the beginning, all pages are at level 0, and the set pages immediately clear the Access bit and Dirty bit in the current cycle; If a page has just had its Access bit and Dirty bit cleared in the previous cycle and is captured again in this cycle, the level of the page is increased by 1; the page with level k is increased by 2. k-1 The Access bit and Dirty bit are cleared after a cycle; pages that have not yet reached the cleaning cycle remain set to avoid overhead; if a page is not captured again immediately after being cleaned, the level of the page is reduced by 1; After the set multiple cycle cycles are completed, the number of read and write times of each page in the virtual machine has been successfully counted.

4. The system according to claim 3, characterized in that When page table tracing is turned on, the virtual machine page table scanning module traverses the virtual machine's process page table and selects the process page table with a resident set larger than 1GB for page table traversal; obtains the virtual machine's last-level GPT pointer through page table traversal and saves it to the virtual machine's last-level page table pointer buffer. After the scan is completed, hypercall communication is called to transfer the virtual machine's last-level page table pointer buffer back to the hypervisor.

5. A computer device, characterized in that: The invention comprises the PML-based virtual machine page table tracking system as described in claim 3 or 4.

Images