Method and apparatus for using dram as cache for byte addressable memory

Abstract

This application relates to methods and apparatus for using DRAM as a cache for byte-addressable memory. Various embodiments generally relate to virtualization systems. A first guest memory page in a page table can be identified based at least in part on a number of accesses to a page table entry for the first guest memory page by an application executing in a virtual machine (VM) on a processor, the first guest memory page corresponding to a first byte-addressable memory. Execution of the VM and the application on the processor can be paused. The first guest memory page can be migrated to a target memory page in a second byte-addressable memory, the target memory page comprising one of a target host memory page and a target guest memory page, the second byte-addressable memory having a faster access speed than an access speed of the first byte-addressable memory.

Description

[0001] This application is a divisional application of the invention patent application filed on September 28, 2018, with application number 201880093976.4 (PCT international application number PCT / CN2018 / 108206) and entitled "Method and apparatus for using DRAM as a cache for a slow byte-addressable memory for efficient cloud applications". Technical Field

[0002] The embodiments described herein relate to the field of computing devices. More specifically, the embodiments described herein relate to using DRAM as a cache for slow, byte-addressable memory for efficient cloud applications. Background Technology

[0003] Some computing systems use large-capacity, byte-addressable persistent memory (PMEM). PMEM's high density and low cost make it an ideal memory type for cloud computing platforms, which use large amounts of memory to host multiple applications and / or virtual machines (VMs). However, in cloud platforms, most applications and / or VMs are typically idle (e.g., unused). PMEM is slower than some other memory types, such as dynamic random access memory (DRAM). PMEM is typically exposed as a system component as a storage device, not as memory. Therefore, conventional systems cannot directly use PMEM. Previous attempts to allow the operating system to use PMEM directly required application modifications, were often inefficient, and could be inaccurate in systems with both DRAM and PMEM installed. Some solutions may expose PMEM directly to the system as slow, byte-addressable memory. Solutions for non-uniform memory access (NUMA) mechanisms used by the operating system (OS) are insufficient for this, as NUMA is implemented to address only the speed difference between local and remote memory (which can have latency differences on the order of approximately 1-1.5X). However, PMEM can be much slower than DRAM, by about 10X-50X. Summary of the Invention

[0004] According to one aspect of this application, a processor is provided, comprising: a multi-chip package including a plurality of integrated circuit chips, the integrated circuit chips including: a plurality of cores for executing instructions; interface circuitry for coupling the plurality of cores to a plurality of memories, the plurality of memories including a first memory and a second memory, the first memory being associated with a faster access speed compared to the second memory; one or more of the plurality of cores for: translating a plurality of client virtual addresses into a plurality of corresponding client physical addresses based on a first set of page tables, and translating a plurality of client physical addresses into a plurality of corresponding host physical addresses based on a second set of page tables, a first portion of the plurality of host physical addresses being associated with a corresponding first plurality of memory pages in the first memory, and a second portion of the plurality of host physical addresses being associated with a corresponding second plurality of memory pages in the second memory; a translation look-ahead buffer (TLB) for caching translations associated with the corresponding plurality of host physical addresses; and circuitry at least partially operable according to executable code for: migrating a set of memory pages from the second plurality of memory pages from the second memory to the first memory, and clearing one or more translations corresponding to the migrated set of memory pages from the TLB.

[0005] According to another aspect of this application, a machine-readable medium is provided having program code stored thereon that, when executed by a machine, causes the machine to perform operations including: executing instructions on multiple cores coupled to multiple memories via interface circuitry, the multiple memories including a first memory and a second memory, the first memory being associated with faster access speeds compared to the second memory; translating multiple client virtual addresses into corresponding multiple client physical addresses based on a first set of page tables; translating multiple client physical addresses into corresponding multiple host physical addresses based on a second set of page tables, a first portion of the multiple host physical addresses being associated with corresponding first multiple memory pages in the first memory, and a second portion of the multiple host physical addresses being associated with corresponding second multiple memory pages in the second memory; caching translations associated with the corresponding multiple host physical addresses in a translation lookahead buffer (TLB); identifying a set of memory pages based on the number of accesses made by at least one guest operating system or virtual machine to a set of memory pages in the second multiple memory pages; migrating the set of memory pages from the second memory to the first memory; and clearing one or more translations corresponding to the migrated set of memory pages from the TLB.

[0006] According to another aspect of this application, a method is provided, comprising: initializing an access counter value for each memory page in a memory; traversing a second-dimensional page table that maps client physical memory addresses to host physical memory addresses; clearing each access bit in the second-dimensional page table to zero, and after waiting for a predefined amount of time, traversing the second-dimensional page table again and incrementing a traversal counter; incrementing the access counter value of the client memory page when it is determined that an access bit for the client memory page in the second-dimensional page table is set; determining whether the value of the traversal counter exceeds a traversal threshold, and continuing to traverse the second-dimensional page table when it is determined that the value of the traversal counter does not exceed the traversal threshold; and identifying at least one hot client memory page based on the access counter value. Attached Figure Description

[0007] Figure 1 The figure illustrates an embodiment of the system.

[0008] Figures 2A to 2B The figure shows an example of migrating memory pages.

[0009] Figure 3 The figure illustrates an embodiment of the first logic flow.

[0010] Figure 4 The figure illustrates an embodiment of the second logic flow.

[0011] Figure 5 The figure illustrates an embodiment of the third logic flow.

[0012] Figure 6 The figure illustrates an embodiment of the fourth logic flow.

[0013] Figure 7 The figure illustrates an embodiment of the fifth logic flow.

[0014] Figure 8 The figure illustrates an embodiment of the sixth logic flow.

[0015] Figure 9 The figure illustrates an embodiment of the seventh logic flow.

[0016] Figure 10 The figure illustrates an embodiment of the storage medium.

[0017] Figure 11 The figure illustrates an embodiment of the system. Detailed Implementation

[0018] The embodiments disclosed herein provide a framework that allows guest operating systems and associated applications to directly use large-capacity, byte-addressable memory (such as PMEM) within a virtualized environment. The embodiments disclosed herein generally use a Basic Input / Output System (BIOS) and / or operating system reporting mechanisms (such as Advanced Configuration and Power Interface (ACPI)) to present PMEM as slower DRAM to guest components, and also implement a caching mechanism to move frequently used (or “hot”) guest memory pages from PMEM to DRAM to reduce memory load / flush costs.

[0019] More specifically, the embodiments disclosed herein provide a two-level Virtual Non-Uniform Memory Access (NUMA) mechanism, wherein the first level relates to the distance from memory cells and / or corresponding memory pages to the processor (CPU). For example, the first level may distinguish between local CPU memory and remote memory cells (e.g., DRAM connected to a remote CPU socket), where access to local CPU memory is faster than access to remote memory cells. The second level may relate to speed differences between different memory cells (e.g., PMEM is slower than DRAM) and / or speed differences between corresponding memory pages. Generally, the embodiments disclosed herein divide these two levels of memory pages into two memory regions, referred to herein as “fast memory pages” and “slow memory pages,” because the primary contributor to system latency is the speed difference between PMEM and DRAM cells (e.g., in some examples, access to DRAM is 10 times faster than access to PMEM).

[0020] Furthermore, the embodiments disclosed herein provide a caching mechanism that identifies frequently used client memory pages based on access bits in corresponding page tables using second-dimensional page tables (such as Extended Page Tables (EPT), Nested Page Tables (NPT), Second-Level Address Translation (SLAT), and / or client page tables). Once identified, the caching mechanism can migrate frequently used client memory pages to maximize application and / or VM performance by utilizing fast memory pages in DRAM. Doing so may require migrating the original memory pages being replaced in DRAM to slower memory pages in PMEM. Further, the embodiments disclosed herein can divide the client memory into multiple regions and implement the caching mechanism in each region to save the cost of identifying frequently accessed client memory pages in the super-client memory space.

[0021] Advantageously, the embodiments disclosed herein provide application-transparent solutions that allow applications executing on a computing platform to directly utilize PMEM and use it for a variety of different workloads. Furthermore, the embodiments disclosed herein improve the performance of two-level NUMA, and consequently improve the performance of systems incorporating PMEM.

[0022] Referring generally to the symbols and terminology used herein, one or more parts of the detailed description below can be presented as procedural processes executed on a computer or computer network. Those skilled in the art use these processes to describe and represent them in order to most effectively communicate the essence of their work to others skilled in the art. Processes herein and generally are conceived as a self-consistent series of operations that produce desired results. These operations are those that require physical manipulation of physical quantities. Typically (but not necessarily), these quantities take the form of electrical, magnetic, or optical signals that can be stored, transferred, combined, compared, and otherwise manipulated. It is sometimes convenient to refer to these signals as bits, values, elements, symbols, letters, terms, numbers, etc., primarily due to common usage. However, it should be noted that these and similar terms will be associated with appropriate physical quantities and are merely convenient notations applied to those quantities.

[0023] Furthermore, these manipulations are often referred to by terms such as addition or comparison, which are generally associated with mental operations performed by a human operator. However, in any of the operations described herein that form part of one or more embodiments, such capability of a human operator is not necessary or desirable in most cases. Instead, these operations are machine operations. Useful machines for performing the operations of the various embodiments include general-purpose digital computers selectively activated or configured by computer programs stored therein, written according to the teachings herein, and / or include means specifically constructed for the desired purpose. The various embodiments also relate to means or systems for performing these operations. These means may be specifically constructed for the desired purpose or may include general-purpose computers. The required structure of these various machines will be apparent from the given description.

[0024] Referring now to the accompanying drawings, in which the same reference numerals are used throughout to refer to the same elements. In the following description, numerous specific details are set forth for illustrative purposes in order to provide a thorough understanding of the invention. However, it will be apparent that new embodiments may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form for ease of description. It is intended to cover all modifications, equivalents, and alternatives within the scope of the claims.

[0025] Figure 1The figure illustrates an embodiment of computing system 100. System 100 represents any number and type of computing system, such as servers, server farms, blade servers, compute nodes, workstations, desktop computers, tablet computers, netbook computers, laptop computers, and so on. Embodiments are not limited thereto. As shown, computing system 100 includes hardware resources 110, a virtual machine monitor (VMM) 108, and one or more virtual machines (VMs) 106-1, 106-2, and 106-n, where “n” is any positive integer. VMM 108 (also referred to as a hypervisor) is the application and / or logic that controls the creation and operation of VMs 106-1, 106-2, and 106-n (collectively referred to as VMs 106) and serves as the control and translation system between VMs 106 and hardware resources 110. This allows the operating system (OS) 120 and applications 122 of VMs 106 to access hardware resources 110 in a virtualized environment (e.g., a cloud computing platform).

[0026] As shown in the figure, hardware resource 110 includes at least one or more processors (CPUs) 112, a first memory 113-1, and a second memory 113-n, where "n" is any positive integer. Hardware resource 110 includes other elements not depicted for clarity (e.g., buses, persistent storage devices, network interfaces, etc.). Processor 112 represents any number and type of computing elements, such as microprocessors, processors, central processing units, digital signal processing units, dual-core processors, mobile device processors, desktop processors, single-core processors, system-on-a-chip (SoC) devices, complex instruction set computing (CISC) microprocessors, reduced instruction set computing (RISC) microprocessors, very long instruction word (VLIW) microprocessors, or any other type of processor or processing circuitry on a single chip or integrated circuit.

[0027] Memory 113-1, 113-n represent any number and type of memory elements, and Figure 1The number of memory elements depicted herein should not be considered as limiting the scope of this disclosure. However, system 100 includes at least memory 113-1 and memory 113-n, which can be different types of memory. For example, the first memory 113-1 can be a DRAM DIMM, while the second memory 113-n can be a byte-addressable PMEM. Memory 113-1 and 113-n can be any large-capacity byte-addressable memory, and the use of DRAM or PMEM herein should not be considered as limiting the scope of this disclosure. However, DRAM 113-1 is generally faster than PMEM 113-n. In one embodiment, memory 113-n is implemented as a non-volatile DIMM (NVDIMM). Thus, although DRAM 113-1 and PMEM 113-n can be physically connected to the same type of DIMM slot in system 100, the physical architecture of DRAM 113-1 and PMEM 113-n imparts different speed characteristics to each memory type. For example, DRAM 113-1 is faster than PMEM 113-n. In some embodiments, DRAM 113-1 is ten times faster than PMEM 113-n. Therefore, processor 112 accesses data in DRAM 113-1 faster than accessing data in PMEM 113-n. Furthermore, the distance from DRAM 113-1 and / or PMEM 113-n to processor 112 may further contribute to the data access time. Additionally, in terms of cost per memory cell, the economic cost of DRAM 113-1 is greater than that of PMEM 113-1. Therefore, system 100 may include more PMEM 113-n than DRAM 113-1 (e.g., in gigabytes, terabytes, etc.). As shown, DRAM 113-1 and PMEM 113-n respectively house host page tables 116-1 and 116-2 accessible by VMM 108.

[0028] However, advantageously, the embodiments disclosed herein configure memories 113-1, 113-n as a two-level virtual NUMA to VM 106 (and / or OS 120 for each VM 106). More specifically, system 100 presents PMEM 113-n as slower memory via BIOS / OS reporting mechanisms such as virtual ACPI and utilizes DRAM 113-1 as a faster cache for PMEM 113-n. Subsequently, application 122 of VM 106 can identify DRAM 113-1 as relatively faster memory and PMEM 113-n as relatively slower memory. The cache components 115 of VMM 108 (including entities 115-1, 115-2, 115-3, 115-n, where “n” is any positive integer) and VM 106 can typically identify “hot” (or frequently used) client pages of memory 113-n and migrate the identified hot client pages to DRAM 113-1. The guest memory page is the memory page used by VM 106 and / or the corresponding OS 120 and / or the application 122 executed therein.

[0029] To identify hot client pages, the caching component 115 of VMM 108 can monitor access bits of entries in the Second Dimension Page Table (SDPT) 118 and / or the client page table 121 of VM 106. Although the client page table 121 is depicted as part of OS 120, it can also be a component corresponding to VM 106. The client page table 121 typically includes multiple entries, each mapping one or more client virtual memory addresses to client physical memory addresses for memory address translation. Furthermore, the client page table 121 may include multiple layers (e.g., a 4-layer 64-bit page table). Some entries in the client page table 121 (e.g., leaf entries) may include access bits indicating whether a memory translation has been accessed. SDPT 118 includes multiple entries, some of which (e.g., leaf entries) map one or more client physical memory addresses to one or more host physical addresses. Each entry in SDPT 118 similarly includes access bits indicating whether the corresponding memory translation has been accessed. VMM 108 provides a different SDPT 118 for each VM 106-1 through 106-n. SDPT 118 represents any type of second-dimensional page table, such as nested page tables (NPT), second-level address translation (SLAT) tables, etc.

[0030] Typically, application 122 can use virtual addresses {VA1, VA2, ..., VAn}, where "n" is a positive integer. Some of these virtual addresses can map to the same memory page, where application 122 uses memory pages {PA1, PA2, ..., PAm}, where "m" is any positive integer. When application 122 accesses memory page X, this may result in cache loading and / or cache flushing. For a given memory page X, application 122 uses a frequency appropriate for load-read operations. Trigger cache loading, and at a frequency used for flush-write operations. Trigger cache clearing. System 100 can use it. N T One memory page, namely DRAM 113-1 N d Page or page { D 0 , D 1 ,…, D Nd-1} and PMEM 113-n N p Page or page { P 0 , P 1 ,…, P Np-1},in N T = N d + N p .

[0031] Furthermore, system 100 can expose two or more topologies for DRAM 113-1 and PMEM 113-n to VM 106 (and / or its components), namely exclusive topology and containment topology. In exclusive topology, the total customer memory pages N T = N d + N p This means that DRAM 113-1 is directly used as a client memory page with a dedicated client physical address. In an inclusive topology, the total number of client memory pages N T = N pThis means that DRAM 113-1 is not exposed as a client memory page, but is only used as a cache for certain PMEM pages (in this case, the cached DRAM has host-side physical page addresses but not client-side physical page addresses). Furthermore, a hybrid mode combining exclusive and inclusive modes is possible, where the system can include exclusive DRAM. and included DRAM .

[0032] The performance overhead T in the system A This is attributable to memory loads and / or writes caused by CPU cache loading and / or flushing. In other words, the total overhead... T A = K 1 ·( In this context, the DRAM read latency is normalized to 1, and L1 corresponds to the PMEM read / write latency ratio (which can be on the order of 10X-30X). To achieve optimal system performance, cache component 115 reduces latency by identifying hot memory pages in PMEM 113-n. T A The cost is reduced, and DRAM 113-1 is used to support the identified hot memory pages.

[0033] In some embodiments, cache component 115 may divide memory 113-n into different regions (e.g., 10 regions, 100 regions, etc.) to identify hot memory pages within them. For example, each of the n different regions may include 1 / n of the size of PMEM 113-n and 1 / n of the size of DRAM 113-1. This saves time in identifying hot memory pages, as traversing the entire SDPT 118 and / or client page table 121 could require significant time and computational resources. In other embodiments, cache component 115 does not divide the PMEM memory 113-n and identifies hot memory pages across the entire PMEM memory 113-n.

[0034] Typically, to identify hot memory pages, cache component 115 clears the access bit A of each entry in SDPT 118 of VM 106 (or a subset of entries in SDPT 118 corresponding to a partitioned memory region). Cache component 115 may then wait for a predefined amount of time (e.g., 1 microsecond, 10 microseconds, etc.) and re-traverse SDPT 118 to identify entries with an access bit A that is "set" (e.g., a value indicating a transition and thus indicating that the corresponding memory page has been accessed). If the access bit of a given entry is set, cache component 115 increments the access counter value for that entry and / or the corresponding memory page (e.g., ...). The caching component 115 can repeat the process a predetermined number of times (e.g., 1000 times, 10000 times, etc.). Once repeated, the maximum access counter value... This reflects the most frequently accessed and therefore "hot" pages in the customer memory. Hot pages may be in PMEM 113-n, which is not optimal, and these hot pages can be considered candidates for migration to DRAM 113-1. In some embodiments, an absolute threshold (e.g., 10, 100, 200, etc.) can define which pages are "hot," for example, based on access counter values ​​exceeding the threshold. Those pages. In some embodiments, a relative threshold (e.g., top 100, top 200, or top 10% / top 20%, etc.) can define which pages are "hot," for example, those with the highest access counter value. Those pages. In some embodiments, the cache component 115 considers only the hot memory pages in PMEM 113-n, and not DRAM 113-1, as candidates for migration, because the hot memory pages in DRAM 113-1 do not need to be moved for the purpose of optimizing system performance.

[0035] Similarly, the caching component 115 of OS 120 (and / or VM 106) can identify hot guest memory pages by clearing and monitoring the access bits of guest page tables 121. Typically, OS 120 uses guest virtual addresses as indexes for guest page tables 121 (which may include multiple guest page tables 121 corresponding to OS 120 within a specific time period). However, the caching component 115 considers the physical address of the virtual address from entries in guest page tables 121 to determine which guest physical memory pages are being accessed most frequently. Typically, if a memory page is accessed by two or more applications using two or more guest page tables, the set of access bits for any entry in page table 121 is ORed to indicate the access to the corresponding guest memory page. When guest page tables 121 are used, the caching component 115 of OS 120 determines the corresponding guest physical address X and increments the access bit if the access bit in any guest page table entry (pointing to guest physical address X) is set. This allows the cache component 115 to monitor client memory pages over time (e.g., over 1000 iterations) and use the determined... The value is used to identify the hottest customer memory page in PMEM 113-1. The migration of the hottest customer memory page in PMEM 113-1 to DRAM 113-1 can then be triggered after cache component 115.

[0036] Cache component 115 can also implement a migration strategy that selects a predefined number of hot pages in PMEM 113-n for migration to DRAM memory 113-1. In one embodiment, cache component 115 can use an access counter value Sort from largest to smallest to identify the most frequently accessed memory pages in PMEM 113-n.

[0037] According to a migration strategy, cache component 115 selects the component with the largest access counter value. N pages, where N equals the number of DRAM pages. N d And obtain the page set T d ={ T 0 , T 1 ,…, T Nd-1}. Cache component 115 uses PMEM 113-n and is in the page set T d Any page in the DRAM can be used to migrate M pages (where M≤N). In other words, if a DRAM page is in T d If the page is not present, it will not be migrated. T d If the page is moved to PMEM 113-n, the hot memory page will be moved from PMEM 113-n to DRAM 113-1. Typically, PMEM 113-n contains the maximum access counter value. The page has a lower access counter value in PMEM 113-n. The pages were previously migrated. In some embodiments, system 100 can share memory pages between VMs 106. Therefore, the caching component can use host-side memory pages in PMEM 113-n to migrate page sets. T d The page in the middle.

[0038] According to the second migration strategy, cache component 115 can be based on the access counter value. In candidate page set T d The second strategy allows for the selection of more popular pages for migration. For example, it can specify selecting 1.5 times more pages than the first migration strategy. Therefore, the cache component 115 selects a set of pages. T d The maximum access counter value is in 1.5* N d Pages. In one embodiment, if a client memory page is mapped from a DRAM page.R d exist T d If the page is inside the DRAM, it will not be migrated. Otherwise, the DRAM page... R d Pages mapped by PMEM are available R p To migrate and not in the page set T d In the second migration strategy, if the previous pages mapped by the DRAM are not too hot (but not hotter than the customer memory pages), the migration effort can be saved by not migrating all the hottest pages. In some embodiments, a low access counter value is used. The “cold” memory pages can be migrated from DRAM 113-1 to PMEM 113-n.

[0039] In some embodiments, system 100 can share memory pages among VMs 106. Therefore, the caching component can use host-side memory pages in PMEM 113-n to migrate pages. T x In some embodiments, system 100 can share memory pages among VMs 106. Therefore, the caching component can use host-side memory pages in PMEM 113-n to migrate pages. T x .

[0040] In some embodiments, the cache component 115-1 of VMM 108 can treat multiple VMs 106 as a bundle to move hot memory pages from PMEM 113-n to DRAM 113-1. In such an embodiment, cache component 115-1 considers the maximum access counter value across the entire memory space of the system. The hottest memory page.

[0041] To migrate pages, in one embodiment, the cache component 115-1 of VMM 108 can migrate the hot client memory page GPN-X in PMEM 113-n using the target client memory page GPN-Y in DRAM 113-1. In such an example, SDPT 118 maps the hot client memory page GPN-X to the host memory page HPN-X and maps the target client memory page GPN-Y using the host memory page HPN-Y. In such an example, the cache component 115-1 of VMM 108 temporarily suspends the execution of the corresponding VM 106 (and any associated OS 120 and / or application 122) and identifies the temporary page Z in DRAM 113-1. Subsequently, the cache component 115-1 of VMM 108 can copy the contents of HPN-X to DRAM page Z. Then, the cache component 115-1 of VMM 108 can copy the contents of HPN-Y to HPN-X, and then copy the contents of DRAM page Z to HPN-Y. The caching component 115-1 of VMM 108 can then modify SDPT 118 for VM 106 to map GPN-X to HPN-X and GPN-Y to HPN-X. The caching component 115-1 of VMM 108 can then clear the translation cache of SDPT 118 (including at least the existing entries for GPN-X and GPN-Y) and restore the execution of VM 106.

[0042] In another embodiment, the cache component 115-1 of VMM 108 can migrate a hot client memory page GPN-X (whose host page number is HPN-X) in PMEM 113-n using a target DRAM page DDD (where the DDD corresponds to a host page number and does not have a client physical page number), which is currently caching a client PMEM page GPN-Y (whose host page number is HPN-Y) in an inclusive topology. In such an example, the cache component 115-1 of VMM 108 temporarily suspends the execution of the corresponding VM 106 (and any associated OS 120 and / or application 122) and clears the contents of the DRAM page DDD to the client PMEM GPN-Y by copying the contents of the page DDD to the page HPN-Y. The cache component 115-1 then loads the contents of GPN-x in PMEM into the DRAM DDD. This can be achieved by copying the contents of the page HPN-X to the page DDD. Then, the cache component 115-1 of VMM 108 can modify SDPT 118 for VM 106 to map GPN-X to DDD and GPN-Y to HPN-Y. Subsequently, the cache component 115 of VMM 108 can clear the translation cache of SDPT 118 (including at least the existing entries for GPN-X and GPN-Y) and resume execution of VM 106. This is helpful in some cases because it saves the data copying work that occurs during page migration time compared to an exclusive topology. In such an embodiment, a fixed-size DRAM 113-n is used for caching.

[0043] In another embodiment, the additional DRAM page is used for caching. More specifically, the caching component 115-1 of VMM 108 can migrate a hot client memory page GPN-X (whose host page number is HPN-X) in PMEM 113-n using a free target host memory page DDD in DRAM 113-1. In such an embodiment, the SDPT 118 for the corresponding VM 106 includes a translation entry mapping GPN-X to HPN-X and does not include an entry for the host memory page DDD. In such an example, the caching component 115-1 of VMM 108 temporarily suspends the execution of the corresponding VM 106 (and any associated OS 120 and / or application 122). The caching component 115-1 of VMM 108 can then copy the contents of HPN-X to the target host memory page DDD. Then, the cache component 115-1 of VMM 108 can modify SDPT 118 for VM 106 to map GPN-X to the target host memory page DDD in DRAM 113-1. The cache component 115-1 of VMM 108 can then clear the translation cache of SDPT 118 (including any existing entries for GPN-X) and release HPN-X. The cache component 115-1 of VMM 108 can then resume the execution of the suspended VM 106.

[0044] In another embodiment, the cache component 115 of VM 106 (and / or the corresponding OS 120) can migrate pages in PMEM 113-n on the client side. For example, the cache component 115-2 of VM 106-1 can migrate a hot client memory page GPN-X in PMEM 113-n using a target client memory page GPN-Y in DRAM 113-1. In such an embodiment, the client page table 121 of OS 120 of VM 106-1 can map virtual address VA-X1 (and optionally, virtual addresses {VA-X2, VA-X3,...,VA-Xn}) to GPN-X and virtual address VA-Y1 (and optionally, virtual addresses {VA-Y2, VA-Y3,...,VA-Yn}) to GPN-Y. The cache component 115-2 of VM 106-1 can temporarily suspend the execution of any application 120 using VA-X1 and VA-Y1. The cache component 115-2 of VM 106-1 can allocate page Z (which is the client physical page number, i.e., GPN) in DRAM 113-1 and copy the contents of GPN-X to Z. Then, the cache component 115-2 of VM 106-1 can copy the contents of page GPN-Y to page GPN-X. The cache component 115-2 of VM 106-1 can then copy the contents of DRAM page Z to page GPN-Y. Subsequently, the cache component 115-2 of VM 106-1 can modify the client page table 121 to map {VA-X2, VA-X3,...,VA-Xn} to GPN-Y and {VA-Y2, VA-Y3,..., VA-Yn} to GPN-X. The cache component 115-2 of VM 106 can then clear the translation lookahead buffer (TLB) for at least any existing entries for VA-X1 and VA-Y1 and resume the execution of the suspended application.

[0045] In another embodiment suitable for a dedicated topology, the cache component 115 of VM 106 (and / or the corresponding OS 120) can migrate the hot client page GPN-X in PMEM 113-n using the free target client page GPN-Y in DRAM 113-1. For example, the client page table 121 of OS 120 of VM 106-1 can map virtual address VA-X1 (and optionally, virtual addresses {VA-X2, VA-X3,..., VA-Xn}) to GPN-X. The cache component 115-2 of VM 106-1 can then temporarily suspend the execution of application 122 using VA-X1 and allocate GPN-Y in DRAM 113-1. The cache component 115-2 of VM 106-1 can then copy the contents of GPN-X to GPN-Y. The caching component 115-2 of VM 106-1 can then modify the client page table 121 to map {VA-X2, VA-X3,..., VA-Xn} to GPN-Y, release page GPN-X, clear the TLB for any entries for VA-X1, and resume the execution of the paused application 122.

[0046] Although cache components 115-2, 115-3, and 115-n are depicted as residing in OS 120 of VM 106, cache component 115 can be a component corresponding to VM 106. Regardless of the specific configuration, cache component 115 in VM 106 can migrate memory pages based on the page tables managed by VM 106 as described above.

[0047] Figure 2A This is a more detailed schematic diagram 200 showing the components of system 100. As shown, schematic diagram 200 depicts two example CPU sockets 201 and 202. As shown, CPU socket 201 includes an example processor, namely CPU 112-1, DRAM 203, and PMEM 204. Similarly, CPU socket 202 includes CPU 112-2, DRAM 213, and PMEM 214. Typically, DRAM 203 and 213 can correspond to... Figure 1 The DRAM 113-1, while PMEM 204 and 214 can correspond to Figure 1 The PMEM 113-n. However, as shown in the figure, DRAM 203, 213 and PMEM 204, 214 are different memory elements.

[0048] As described above, DRAMs 203 and 213 can have faster access speeds than PMEMs 204 and 214. Furthermore, the location of DRAMs 203 and 213 and / or PMEMs 204 and 214 may introduce additional latency to the access times of CPUs 112-1 and 112-2. Typically, when the memory and CPU are in the same socket, CPUs 112-1 and 112-2 can access data in DRAMs 203 and 213 and PMEMs 204 and 214 faster. Therefore, CPU 112-1 may cause additional latency when accessing DRAM 213 and / or PMEM 214 relative to DRAM 203 and / or PMEM 204. Similarly, CPU 112-2 may cause additional latency when accessing DRAM 203 and / or PMEM 204 relative to accessing DRAM 213 and / or PMEM 214.

[0049] As shown in the figure, PMEM 204 includes an example guest memory page 210. As described above, cache component 115 can determine that guest memory page 210 is a "hot" memory page by monitoring the access bits of entries in SDPT 118 and / or guest page table 121, which includes transitions for guest memory page 210. As described above, cache component 115 can clear the access bits of entries in SDPT 118 and / or guest page table 121, wait for a predefined time period, and check to see if the access bit of the entry has been set. If the access bit has been set, cache component 115 increments the access counter value of guest page table 210. After a predefined number of repetitions, cache component 115 can determine that guest page table 210 is a hot or frequently accessed guest memory page based on the access counter value. For example, cache component 115 can determine that the access counter value exceeds a hot threshold, or that the access counter value is the maximum access counter value across all memory pages, and / or that the access counter value is one of N maximum access counter values. Therefore, cache component 115 can determine to migrate customer memory page 210 to DRAM 203 and / or DRAM 213.

[0050] Figure 2B The migration of customer memory page 210 from PMEM 204 to DRAM 203 is depicted. Cache component 115 can perform the migration as described above. Figures 2A to 2B This is for illustrative purposes only and should not be construed as limiting this disclosure. As described above, cache component 115 may need to evict pages in DRAM 203 when migrating guest memory page 210, and may further update the transitions in SDPT 118 and / or guest page table 121 to reflect the new location of guest memory page 210.

[0051] Figure 3The figure illustrates an embodiment of logic flow 300. Logic flow 300 may represent some or all of the operations performed by one or more embodiments described herein. The embodiments are not limited thereto. For example, cache component 115 of system 100 may perform the operations in logic flow 300 to identify hot guest memory pages in PMEM and migrate the identified pages to faster DRAM.

[0052] As shown in the figure, logic flow 300 begins at block 310, where system 100 presents a two-level virtual NUMA to OS 120 and / or VM 106. The two-level virtual NUMA includes providing OS 120 and / or VM 106 with direct access to DRAM 113-1 and PMEM 113-n. As mentioned above, DRAM 113-1 and PMEM 113-n can have different physical characteristics, which make DRAM 113-1 faster than PMEM 113-n in operation. Similarly, the farther DRAM 113-1 and / or PMEM 113-n are from a given processor, the more additional access latency is introduced.

[0053] At block 320, cache component 115 selects a migration strategy for migrating memory pages. As described above, multiple migration strategies can be provided. A given migration strategy can be selected based on monitored performance improvements (which are achieved based on a given strategy). In another example, a system administrator can select the strategy. At block 330, cache component 115 can optionally partition the memory of system 100 into multiple regions. Doing so allows cache component 115 to perform PMEM DRAM support at the partition level, rather than across the entire memory space of system 100, which can provide improved system performance. At block 340, cache component 115 identifies one or more hot (or frequently accessed) guest memory pages in PMEM 113-n. As described above, in some embodiments, hot guest pages can be identified in regions of PMEM 113-n. As described above, cache component 115 monitors the access bits in SDPT 118 and / or the guest page table 121 corresponding to the translation entries for a given memory page. Over time, cache component 115 determines the access counter value for each guest memory page. The caching component 115 can select one or more hot client pages based on a determined maximum access counter value.

[0054] At block 350, cache component 115 identifies a target memory page in DRAM 113-1. In at least one embodiment, cache component 115 randomly selects an available memory page in DRAM 113-1 as the target memory page. If no memory page is freely available, cache component 115 may determine to select the least recently used memory page. At block 360, cache component 115 migrates the hot client memory page identified at block 340 to the target memory page identified at block 350. In this way, the performance of system 100 is improved by moving more frequently accessed memory pages from the slower PMEM 113-n to the faster DRAM 113-1.

[0055] Figure 4 The figure illustrates an embodiment of logic flow 400. Logic flow 400 may represent some or all of the operations performed by one or more embodiments described herein. Embodiments are not limited thereto. For example, cache component 115 may perform logic flow 400 based on SDPT 118 to identify hot memory pages in PMEM 113-1. Although SDPT 118 is used as an example, cache component 115 of OS 120 may use client page table 121 to identify hot pages in PMEM 113-n.

[0056] At block 410, cache component 115 initializes the access counter value (e.g., sets it to zero) for each guest memory page in PMEM 113-n (which has an entry in SDPT 118). Cache component 115 may also initialize a traversal counter that counts the number of traversals of SDPT 118 performed by cache component 115. At block 420, cache component 115 traverses SDPT 118. As described above, in some embodiments, cache component 115 traverses the portion of SDPT 118 corresponding to the region in PMEM 113-n. At block 430, cache component 115 clears each access bit in SDPT 118 (e.g., sets it to zero). At block 440, cache component 115 waits for a predefined amount of time. At block 450, cache component 115 traverses SDPT 118 again and increments the traversal counter. Typically, during traversal of SDPT 118, cache component 115 determines whether the access bit of each entry in SDPT 118 has been set to 1. At block 460, cache component 115 increments the access counter value of the guest memory pages in PMEM 113-n when it determines that the access bit for the guest memory page in SDPT 118 is set. At block 470, cache component 115 determines whether the traversal counter value exceeds a traversal threshold (e.g., 1000). If the traversal counter does not exceed the threshold, cache component 115 returns to block 420 (e.g., clears the access bit in SDPT 118 and monitors it). Otherwise, cache component 115 proceeds to block 480, where cache component 115 identifies at least one hot guest memory page based on the access counter value. As mentioned above, the larger the access counter value, the more frequently the associated memory pages in PMEM 113-n are accessed. Therefore, cache component 115 identifies the most frequently used or "hot" guest memory pages in PMEM 113-n based on access counter values. Thus, for example, if guest memory page X has an access counter value of 300 and guest memory page Y has an access counter value of 200, cache component 115 can determine that guest memory page X is a hot memory page. However, in some embodiments, cache component 115 can also determine that guest memory page Y is also a hot memory page. At block 490, cache component 115 triggers a migration of the hot memory page identified at block 480 based on the access counter value.

[0057] Figure 5 The figure illustrates an embodiment of logic flow 500. Logic flow 500 may represent some or all of the operations performed by one or more embodiments described herein. The embodiments are not limited thereto. For example, cache component 115 may execute logic flow 500 that identifies frequently used memory pages for migration.

[0058] As shown in the figure, at block 510, cache component 115 classifies customer memory pages based on access counter values ​​determined for each customer memory page (e.g., determined according to logic flow 400). At block 520, cache component 115 optionally applies a first migration strategy to identify a set of candidate memory pages in PMEM 113-n by identifying each memory page with an access counter value exceeding a hot access counter threshold. For example, if the hot access counter threshold is 250, cache component 115 determines to migrate each customer memory page with an access counter value exceeding 250. This ensures that each hot memory page is migrated from PMEM 113-n to DRAM 113-1. As another example, the first migration strategy may specify the migration of N memory pages in PMEM 113-1 with the largest access counter values.

[0059] At block 530, cache component 115 identifies candidate pages by applying a second migration strategy. The second migration strategy can specify the migration of all hot memory pages and additional memory pages. In some embodiments, if a hot memory page is in DRAM 113-1, that page is not migrated. For example, if the hot access counter threshold is 250, the second access counter threshold can be 100. Therefore, cache component 115 can migrate all (or a predefined number) of customer memory pages with access counter values ​​exceeding 100 from PMEM 113-n. As another example, the second migration strategy can specify the migration of N+M (or N multiplied by 1.5 or some other multiplier) memory pages in PMEM 113-1 with the largest access counter value.

[0060] At block 540, cache component 115 may optionally apply a third migration strategy that specifies migrating guest memory pages with free host memory pages. At block 550, cache component 115 may optionally treat multiple VMs 106 as a bundle. For example, at blocks 520 and / or 530, cache component 115 may consider the highest access counter value across all VMs 106 (rather than for a single VM 106).

[0061] Figure 6 The figure illustrates an embodiment of logic flow 600. Logic flow 600 may represent some or all of the operations performed by one or more embodiments described herein. The embodiments are not limited thereto. For example, cache component 115-1 of VMM 108 may execute logic flow 600, which migrates a hot client memory page GPN-X in PMEM 113-n to a target client memory page GPN-Y in DRAM 113-1.

[0062] At block 610, cache component 115 determines that VM 106's SDPT 118 maps hot guest memory page GPN-X to host memory page HPN-X, and maps target guest memory page GPN-Y to host memory page HPN-Y. At block 620, VMM 108's cache component 115-1 temporarily suspends execution of the corresponding VM 106 (and any associated OS 120 and / or application 122) and identifies temporary page Z in DRAM 113-1. At block 630, VMM 108's cache component 115-1 copies the contents of HPN-X to DRAM page Z. At block 640, VMM 108's cache component 115-1 copies the contents of HPN-Y to HPN-X. At block 650, VMM 108's cache component 115-1 copies the contents of DRAM page Z to HPN-Y. At box 660, VMM 108's caching component 115-1 modifies SDPT 118 for VM 106 to map GPN-X to HPN-X and GPN-Y to HPN-X. At box 670, VMM 108's caching component 115-1 can then clear the translation cache of SDPT 118 (including at least any existing entries for GPN-X and GPN-Y). At box 680, VMM 108's caching component 115-1 resumes execution of VM 106.

[0063] Figure 7 The figure illustrates an embodiment of logic flow 700. Logic flow 700 may represent some or all of the operations performed by one or more embodiments described herein. The embodiments are not limited thereto. For example, cache component 115-1 of VMM 108 may execute logic flow 700, which migrates a hot client memory page GPN-X in PMEM 113-n to a target host memory page HPN-Y in DRAM 113-1.

[0064] At block 710, the cache component 115-1 of VMM 108 determines that the SDPT 118 for the corresponding VM 106 includes a translation entry mapping GPN-X to HPN-X, and does not include an entry for the host memory page HPN-Y. At block 720, the cache component 115-1 of VMM 108 temporarily suspends the execution of the corresponding VM 106 (and any associated OS 120 and / or application 122). At block 730, the cache component 115-1 of VMM 108 copies the contents of HPN-X to the target host memory page HPN-Y. At block 740, the cache component 115-1 of VMM 108 modifies the SDPT 118 for VM 106 to map GPN-X to the target host memory page HPN-Y in DRAM 113-1. At box 750, VMM 108's caching component 115-1 can clear the translation cache of SDPT 118 (including any existing entries for GPN-X) and release HPN-X. At box 760, VMM 108's caching component 115-1 can resume the execution of a suspended VM 106.

[0065] Figure 8 The figure illustrates an embodiment of logic flow 800. Logic flow 800 may represent some or all of the operations performed by one or more embodiments described herein. The embodiments are not limited thereto. For example, cache component 115-n of VM 106-n (and / or OS 120) may execute logic flow 800, which migrates a hot client memory page GPN-X in PMEM 113-n to a target client memory page GPN-Y in DRAM 113-1.

[0066] At block 810, cache component 115-n of VM 106 determines that the client page table 121 of OS 120 maps virtual address VA-X1 (and optionally, virtual addresses {VA-X2, VA-X3, ..., VA-Xn}) to GPN-X, and virtual address VA-Y1 (and optionally, virtual addresses {VA-Y2, VA-Y3, ..., VA-Yn}) to GPN-Y. At block 820, cache component 115-n of VM 106-n temporarily suspends the execution of any application 120 using VA-X1 and VA-Y1, and allocates page Z in DRAM 113-1. At block 830, cache component 115-n of VM 106-n copies the contents of GPN-X to DRAM page Z. At block 840, cache component 115-n of VM 106-n copies the contents of GPN-Y to host page HPN-X. At block 850, the cache component 115-n of VM 106-n can then copy the contents of DRAM page Z to GPN-Y. At block 860, the cache component 115-n of VM 106-n can modify the client page table 121 to map {VA-X2, VA-X3,..., VA-Xn} to GPN-Y and {VA-Y2, VA-Y3,..., VA-Yn} to GPN-X. At block 870, the cache component 115-n of VM 106-n can clear the translation lookahead buffer (TLB) for at least any existing entries for VA-X1 and VA-Y1. At block 880, the cache component 115-n of VM 106-n can resume the execution of the suspended application.

[0067] Figure 9 The figure illustrates an embodiment of logic flow 900. Logic flow 900 may represent some or all of the operations performed by one or more embodiments described herein. The embodiments are not limited thereto. For example, a cache component 115-n of VM 106-n (and / or OS 120) may execute logic flow 900, which migrates GPN-X in PMEM 113-n to an idle target client page GPN-Y in DRAM 113-1.

[0068] At block 910, the cache component 115-n of VM 106-n can determine that the client page table 121 maps virtual address VA-X1 (and optionally, virtual addresses {VA-X2, VA-X3,..., VA-Xn}) to GPN-X. At block 920, the cache component 115-n of VM 106-n can temporarily suspend the execution of application 122 using VA-X1 and allocate GPN-Y in DRAM 113-1. At block 930, the cache component 115-n of VM 106-n can copy the contents of GPN-X to GPN-Y. At block 940, the cache component 115-n of VM 106-n can modify the client page table 121 to map {VA-X2, VA-X3,..., VA-Xn} to GPN-Y. At box 950, the caching component 115-n of VM 106-n can clear the TLB for any entry of VA-X1 and release GPN-X. At box 960, the caching component 115-n of VM 106-n can resume the execution of the suspended application 122.

[0069] Figure 10 The figure illustrates an embodiment of storage medium 1000. Storage medium 1000 may include any non-transitory computer-readable or machine-readable storage medium, such as optical, magnetic, or semiconductor storage media. In various embodiments, storage medium 1000 may include articles of manufacture. In some embodiments, storage medium 1000 may store computer-executable instructions, such as those for implementing instructions described herein, such as those relating to… Figures 3 to 9 The storage medium 1000 may also store one or more computer-executable instructions from the logical flow or operation described in 300, 400, 500, 600, 700, 800, and 900. The storage medium 1000 may also store the computer-executable instructions for caching component 115. Examples of computer-readable or machine-readable storage media may include any tangible medium capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. Examples of computer-executable instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, etc. Embodiments are not limited thereto.

[0070] Figure 11 The figures illustrate embodiments of an exemplary computing architecture 1100 that may be adapted to implement the various embodiments described above. In various embodiments, the computing architecture 1100 may include or be implemented as part of an electronic device. In some embodiments, the computing architecture 1100 may represent, for example, implementing... Figure 1 and Figures 2A to 2BThe operating environment is a computer system with one or more components of 100, 200. Examples are not limited thereto.

[0071] As used herein, the terms “system,” “component,” and “module” are intended to refer to computer-related entities, or hardware, a combination of hardware and software, software, or software in execution, examples of which are provided by the exemplary computing architecture 1100. For example, a component can be, but is not limited to, a process running on a processor, a processor, a hard disk drive, multiple storage drives (of optical and / or magnetic storage media), an object, an executable file, an execution thread, a program, and / or a computer. In an interpretable manner, an application running on a server and a server can both be components. One or more components may reside within a process and / or an execution thread, and components may reside on a single computer and / or be distributed across two or more computers. Furthermore, components may communicatively couple with each other to coordinate operation via various types of communication media. Coordination may involve one-way or two-way information exchange. For example, components may transmit information in the form of signals transmitted over a communication medium. This information may be implemented as signals assigned to various signal lines. In such an assignment, individual messages are signals. However, alternative embodiments may employ data messages. Such data messages can be sent across various connections. Exemplary connections include parallel interfaces, serial interfaces, and bus interfaces.

[0072] The computing architecture 1100 includes various common computing elements, such as one or more processors, multi-core processors, coprocessors, memory units, chipsets, controllers, peripherals, interfaces, oscillators, timing devices, video cards, audio cards, multimedia input / output (I / O) components, power supplies, etc. However, embodiments are not limited to implementations of the computing architecture 1100.

[0073] like Figure 11 As shown, the computing architecture 1100 includes a processing unit 1104, a system memory 1106, and a system bus 1108. The processing unit 1104 can be any of a variety of commercially available processors, including but not limited to AMD®, Athlon®, Duron®, and Opteron® processors; ARM® application, embedded, and security processors; IBM®, Motorola® DragonBall®, and PowerPC® processors; IBM and Sony® Cell processors; Intel® Celeron®, Core(2) Duo®, Itanium®, Pentium®, Xeon®, and XScale® processors; and similar processors. Dual-microprocessors, multi-core processors, and other multiprocessor architectures can also be used as the processing unit 1104.

[0074] System bus 1108 provides interfaces for system components, including but not limited to system memory 1106 to processing unit 1104. System bus 1108 can be any of several types of bus architectures, which can also interconnect to memory buses (with or without memory controllers), peripheral buses, and local buses using any of a variety of commercially available bus architectures. Interface adapters can be connected to system bus 1108 via slot architectures. Example slot architectures may include, but are not limited to, Accelerated Graphics Port (AGP), Card Bus, (Extended) Industry Standard Architecture ((E)ISA), Micro Channel Architecture (MCA), Network User Bus (NuBus), Peripheral Component Interconnect (Extended) (PCI(X)), PCI Express, PCMCIA, etc.

[0075] System memory 1106 may include various types of computer-readable storage media in the form of one or more high-speed storage cells, such as read-only memory (ROM), random access memory (RAM), dynamic RAM (DRAM), double data rate DRAM (DDRAM), synchronous DRAM (SDRAM), mass-capacity byte-addressable persistent memory (PMEM), static RAM (SRAM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory (e.g., one or more flash memory arrays), polymer memory such as ferroelectric polymer memory, austenite memory, phase-change or ferroelectric memory, silicon oxynitride (SONOS) memory, magnetic cards or optical cards, device arrays such as redundant array of independent disks (RAID) drives, solid-state storage devices (e.g., USB memory, solid-state drives (SSDs)), and any other type of storage media suitable for storing information. Figure 11 In the illustrated embodiment, system memory 1106 may include non-volatile memory 1110 and / or volatile memory 1112. The basic input / output system (BIOS) may be stored in non-volatile memory 1110.

[0076] Computer 1102 may include various types of computer-readable storage media in the form of one or more low-speed storage units, including internal (or external) hard disk drive (HDD) 1114, floppy disk drive (FDD) 1116 for reading from or writing to removable disk 1118, and optical disk drive 1120 for reading from or writing to removable optical disk 1122 (e.g., CD-ROM or DVD). HDD 1114, FDD 1116, and optical disk drive 1120 may be connected to system bus 1108 via HDD interface 1124, FDD interface 1126, and optical disk drive interface 1128, respectively. HDD interface 1124 for external drive implementations may include at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

[0077] The drive and associated computer-readable medium provide volatile and / or non-volatile storage for data, data structures, computer-executable instructions, etc. For example, multiple program modules may be stored in the drive and storage units 1110, 1112, including an operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. In one embodiment, one or more application programs 1132, other program modules 1134, and program data 1136 may include various application and / or components and / or other logic described herein, such as cache component 115.

[0078] Users can input commands and information into computer 1102 through one or more wired / wireless input devices, such as keyboard 1138 and pointing devices such as mouse 1140. Other input devices may include microphone, infrared (IR) remote control, radio frequency (RF) remote control, game controller, stylus, card reader, dongle, fingerprint reader, gloves, drawing tablet, joystick, keyboard, retinal reader, touch screen (e.g., capacitive, resistive, etc.), trackball, trackpad, sensor, stylus, etc. These and other input devices are typically connected to processing unit 1104 via input device interface 1142 coupled to system bus 1108, but may also be connected via other interfaces such as parallel port, IEEE 1394 serial port, game port, USB port, IR interface, etc.

[0079] Monitor 1144 or other types of display devices are also connected to system bus 1108 via an interface such as video adapter 1146. Monitor 1144 can be internal or external to computer 1102. In addition to monitor 1144, computers typically include other peripheral output devices such as speakers, printers, etc.

[0080] Computer 1102 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer 1148, via wired and / or wireless communications. In various embodiments, one or more migrations can occur via the networked environment. Remote computer 1148 can be a workstation, server computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer-to-peer device, or other common network node, and typically includes many or all of the elements described with respect to computer 1102, but for simplicity, only memory / storage device 1150 is illustrated. The depicted logical connections include wired / wireless connections to a local area network (LAN) 1152 and / or a larger network, such as a wide area network (WAN) 1154. Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks such as intranets, all of which can connect to global communication networks such as the Internet.

[0081] When used in a LAN networking environment, computer 1102 is connected to LAN 1152 via a wired and / or wireless communication network interface or adapter 1156. Adapter 1156 facilitates wired and / or wireless communication to LAN 1152, which may also include a wireless access point disposed thereon for communicating with the wireless functionality of adapter 1156.

[0082] When used in a WAN networking environment, computer 1102 may include modem 1158, or a communication server connected to WAN 1154, or other means for establishing communication over WAN 1154 (such as via the Internet). Modem 1158, which may be internal or external and may be a wired and / or wireless device, is connected to system bus 1108 via input device interface 1142. In a networking environment, the program modules or portions thereof described with respect to computer 1102 may be stored in remote memory / storage device 1150. It should be understood that the network connections shown are exemplary, and other means for establishing communication links between computers may be used.

[0083] Computer 1102 is operable to communicate with wired and wireless devices or entities using the IEEE 802 family of standards, such as wireless devices operable as wireless communication devices (e.g., IEEE 802.16 air modulation technology). This includes, among other things, at least Wi-Fi (or Wireless Fidelity), WiMax, and Bluetooth wireless technologies. Therefore, communication can be a predefined structure with a conventional network, or simply self-organized communication between at least two devices. Wi-Fi networks use radio technologies known as IEEE 802.11x (a, b, g, n, etc.) to provide secure, reliable, and fast wireless connectivity. Wi-Fi networks can be used to connect computers to each other, connect to the Internet, and connect to wired networks (which use the media and functions related to IEEE 802.3).

[0084] One or more aspects of at least one example can be implemented by representative instructions stored on at least one machine-readable medium, which represent various logics within a processor that, when read by a machine, computing device, or system, cause the machine, computing device, or system to manufacture logic that performs the techniques described herein. Such a representation, referred to as an "IP core," can be stored on a tangible machine-readable medium and supplied to various customers or manufacturing facilities for loading into manufacturing machines that manufacture logic or processors.

[0085] Various examples can be implemented using hardware components, software components, or a combination of both. In some examples, hardware components may include devices, components, processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), memory cells, logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc. In some examples, software components may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application programming interfaces (APIs), instruction sets, computational code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. The determination of whether to use hardware components and / or software components to implement an example can vary based on any number of factors, such as desired computational speed, power levels, thermal tolerance, processing cycle budget, input data rate, output data rate, storage resources, data bus speed, and other design or performance constraints as desired for a given implementation.

[0086] Some examples may include an article of writing or at least one computer-readable medium. A computer-readable medium may include a non-transitory storage medium that stores logic. In some examples, a non-transitory storage medium may include one or more types of computer-readable storage media capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. In some examples, logic may include various software elements, such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, APIs, instruction sets, computational code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof.

[0087] According to some examples, a computer-readable medium may include a non-transitory storage medium that stores or maintains instructions that, when executed by a machine, computing device, or system, cause the machine, computing device, or system to perform methods and / or operations according to the described examples. These instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, etc. These instructions may be implemented according to predefined computer languages, methods, or syntaxes to instruct the machine, computing device, or system to perform specific functions. These instructions may be implemented using any suitable high-level, low-level, object-oriented, visual, compiled, and / or interpreted programming languages.

[0088] Some examples may be described using the expressions "in one example" or "example" and their derivatives. These terms imply that a particular feature, structure, or property described in connection with the example is included in at least one example. The phrase "in one example" appearing in various places in the specification does not necessarily refer to the same example in all cases.

[0089] Some examples can be described using the expressions “coupling” and “connection” and their derivatives. These terms are not necessarily intended to be synonyms with each other. For example, descriptions using the terms “connection” and / or “coupling” can indicate that two or more elements are in direct physical or electrical contact with each other. However, the term “coupling” can also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0090] Furthermore, in the foregoing specific embodiments, various features are grouped together and disclosed in a simplified version in a single example. This method of disclosure should not be construed as reflecting an intention that the claimed example requires more features than those explicitly recited in each claim. Rather, as reflected in the appended claims, the subject matter of the invention lies in fewer than all features in a single disclosed example. Accordingly, the appended claims are incorporated into the specific embodiments, wherein each claim is an independent, separate example. In the appended claims, the terms “including” and “in which” are used as concise English equivalents to the corresponding terms “comprising” and “wherein”, respectively. Moreover, the terms “first,” “second,” “third,” etc., are used only as designations and are not intended to impose numerical requirements on their contents.

[0091] Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as examples of implementing the claims.

[0092] A data processing system suitable for storing and / or executing program code will include at least one processor directly or indirectly coupled to a storage element via a system bus. The storage element may include local memory, mass storage, and cache memory used during the actual execution of the program code. The cache memory provides temporary storage for at least some of the program code to reduce the number of times code must be retrieved from mass storage during execution. The term "code" covers a wide range of software components and constructs, including applications, drivers, processes, routines, methods, modules, firmware, microcode, and subroutines. Therefore, the term "code" can be used to refer to any set of instructions that, when executed by a processing system, are expected to perform one or more operations.

[0093] The logic circuits, devices, and interfaces described herein can perform functions implemented in hardware and can be implemented using code that executes on one or more processors. A logic circuit refers to hardware or hardware and code that implement one or more logical functions. A circuit is hardware and can refer to one or more circuits. Each circuit can perform a specific function. The circuitry of a circuit system can include discrete electrical components interconnected with one or more conductors, integrated circuits, chip packages, chipsets, memories, etc. Integrated circuits include circuits created on a substrate such as a silicon wafer and can include components. Furthermore, integrated circuits, processor packages, chip packages, and chipsets can include one or more processors.

[0094] A processor can receive signals such as instructions and / or data at one or more inputs and process these signals to generate at least one output. During code execution, the code alters the physical states and characteristics of the transistors that make up the processor pipeline. The physical states of the transistors are translated into logical bits of one and zero stored in registers within the processor. The processor can transfer the physical states of transistors into registers and to other storage media.

[0095] A processor may include circuitry that performs one or more sub-functions, which are implemented to perform the overall function of the processor. An example of a processor is a state machine or application-specific integrated circuit (ASIC) that includes at least one input and at least one output. A state machine can manipulate at least one input to generate at least one output by performing a predetermined series of serial and / or parallel manipulations or transformations on at least one input.

[0096] The logic described above can be part of the design of an integrated circuit chip. Chip designs are created using a graphical computer programming language and stored on computer storage media or data storage media (such as disks, tapes, physical hard drives, or virtual hard drives such as in a storage access network). If the designer does not manufacture the chip or the photomask used to manufacture it, the designer transfers the obtained design directly or indirectly to such an entity by physical means (e.g., by providing a copy of the storage medium containing the design) or electronically (e.g., via the Internet). The stored design is then converted into a suitable format for manufacturing (e.g., GDSII).

[0097] Manufacturers may distribute the resulting integrated circuit chips in the form of raw wafers (i.e., as a single wafer with multiple unpackaged chips), as bare dies, or in packages. In the latter case, the chips are mounted in a single-chip package (such as a plastic carrier with leads that are attached to a motherboard or other more advanced carrier) or a multi-chip package (such as a ceramic carrier with one or both of surface-mount or buried interconnects). In any case, the chips are then integrated with other chips, discrete circuit elements, and / or other signal processing devices as part of (a) an intermediate product (such as a processor board, server platform, or motherboard) or (b) a final product.

[0098] The following examples relate to other embodiments, and many permutations and configurations will be apparent from these examples.

[0099] Example 1 is an apparatus including a processor and a memory storing instructions that, when executed by the processor, cause the processor to: identify a first customer memory page based at least in part on the number of page table entries of a first customer memory page accessed by an application executing in a virtual machine (VM) on the processor, the first customer memory page residing in a first byte-addressable memory of the apparatus; suspend the execution of the VM and the application on the processor; and migrate the first customer memory page to a target memory page in a second byte-addressable memory of the apparatus, the target memory page including either a target host memory page or a target customer memory page, the second byte-addressable memory having a faster access speed than the first byte-addressable memory.

[0100] Example 2 includes the subject of Example 1, where the page table includes a second-dimensional page table (SDPT) that maps client physical memory addresses to host physical memory addresses. Instructions for identifying a first client memory page include instructions that, when executed by the processor, cause the processor to: clear the access bit of each of a plurality of entries in the SDPT, each entry in the SDPT corresponding to a corresponding client memory page among a plurality of client memory pages, including the first client memory page; wait for a predefined amount of time; traverse the SDPT; determine that the access bit of the entry in the SDPT corresponding to the first client memory page has been set; and increment the access counter value of the first client memory page, the access counter value reflecting the number of accesses to the page table entries of the first client memory page.

[0101] Example 3 includes the subject of Example 2, wherein the memory stores instructions that, when executed by the processor, cause the processor to: divide the memory space of the VM into multiple parts, a first customer memory page being in the first part of the multiple parts of the memory space of the VM, and instructions for identifying the first customer memory page being executed for the first part of the memory space of the VM.

[0102] Example 4 includes the subject of Example 3, wherein the first customer memory page is further identified based on a migration policy that specifies the migration of multiple customer memory pages, including the first customer memory page. The instructions that identify the first customer memory page include instructions that, when executed by the processor, cause the processor to perform the following operations: determine the access counter value of each of the multiple customer memory pages; and determine the multiple customer memory pages based on the multiple customer memory pages having the largest relative access counter value.

[0103] Example 5 includes the subject of Example 2, wherein the target memory page includes a target customer memory page in a second byte-addressable memory, and the instructions for migrating the first customer memory page include instructions for a virtual machine monitor (VMM) that, when executed by the processor, cause the processor to: determine that the SDPT maps the first customer memory page to a first host memory page; determine that the SDPT maps the target customer memory page to a second host memory page; copy the first host memory page to a temporary memory page in the second byte-addressable memory; copy the second host memory page to the first host memory page; copy the temporary memory page in the second byte-addressable memory to the second host memory page; modify the SDPT to (i) map the first customer memory page to the first host memory page and (ii) map the target customer memory page to the second host memory page; clear the translation cache of the SDPT; and resume execution of the VM and the application.

[0104] Example 6 includes the subject of Example 2, wherein the target memory page includes a target host memory page in a second byte-addressable memory, and the instructions for migrating the first client memory page include instructions for the virtual machine monitor (VMM) that, when executed by the processor, cause the processor to: determine that the SDPT maps the first client memory page to the first host memory page; copy the first client memory page to the target host memory page; modify the SDPT to map the first client memory page to the target host memory page; release the first host memory page; clear the translation cache of the SDPT; and resume execution of the VM and the application.

[0105] Example 7 includes the subject of Example 1, where the page table includes a guest page table of a VM, the target memory page includes a target guest memory page in second byte-addressable memory, and the VM includes instructions for migrating a first guest memory page. These instructions are executable by a processor to cause the processor to: determine that the guest page table maps a first virtual address to the first guest memory page and maps a second virtual address to the target guest memory page; allocate a temporary memory page in the second byte-addressable memory; copy the first guest memory page to the temporary memory page; copy the target guest memory page to the first host memory page; copy the temporary memory page to the target guest memory page; modify the guest page table to map the first virtual address to the target guest memory page; release the first guest memory page; clear the translation look-ahead buffer (TLB) for the entry of the first guest memory page; and resume execution of the VM and the application.

[0106] Example 8 includes the subject of Example 1, the page table includes a guest page table of a VM, the target memory page includes a target guest memory page in a second byte-addressable memory, the VM includes instructions for migrating a first guest memory page, the instructions for migrating the first guest memory page being executable by a processor such that the processor: determines that the guest page table maps a first virtual address to the first guest memory page; allocates the target guest memory page in the second byte-addressable memory; copies the first guest memory page to the target guest memory page; modifies the guest page table to map the first virtual address to the target guest memory page; clears the translation look-ahead buffer (TLB) for the entry of the first guest memory page; and resumes execution of the VM and the application.

[0107] Example 9 includes the subject matter of Examples 1 through 8, wherein the first byte-addressable memory includes persistent memory (PMEM) implemented in a non-volatile dual in-line memory module, and the second byte-addressable memory includes dynamic random access memory (DRAM), wherein the access speed of PMEM and DRAM is relative to the processor.

[0108] Example 10 is a non-transitory computer-readable storage medium including instructions that, when executed by a processor of a computing device, cause the processor to: identify a first customer memory page based at least in part on the number of page table entries of a first customer memory page accessed by an application executing in a virtual machine (VM) on the processor, the first customer memory page residing in a first byte-addressable memory of the computing device; suspend the execution of the VM and the application on the processor; and migrate the first customer memory page to a target memory page in a second byte-addressable memory of the computing device, the target memory page including one of a target host memory page and a target customer memory page, the second byte-addressable memory having a faster access speed than the first byte-addressable memory.

[0109] Example 11 includes the subject of Example 10, wherein the page table includes a second-dimensional page table (SDPT) that maps client physical memory addresses to host physical memory addresses. The computer-readable storage medium also includes instructions executable by a processor to cause the processor to: clear the access bit of each of a plurality of entries in the SDPT, each entry in the SDPT corresponding to a corresponding client memory page among a plurality of client memory pages, including a first client memory page; wait for a predefined amount of time; traverse the SDPT; determine that the access bit of the entry in the SDPT corresponding to the first client memory page has been set; and increment the access counter of the first client memory page, incrementing the access counter value of the first client memory page, the access counter value reflecting the number of accesses to the page table entries of the first client memory page.

[0110] Example 12 includes the subject matter of Example 11, and the computer-readable storage medium further includes instructions that can be executed by a processor to cause the processor to: divide the memory space of the VM into multiple portions, with a first customer memory page in a first portion of the multiple portions of the memory space of the VM, and to execute instructions for identifying the first customer memory page for the first portion of the memory space of the VM.

[0111] Example 13 includes the subject matter of Example 12, wherein the first customer memory page is further identified based on a migration policy that specifies the migration of multiple customer memory pages, including the first customer memory page. The computer-readable storage medium also includes instructions executable by a processor to cause the processor to: determine an access counter value for each of the multiple customer memory pages; and determine the multiple customer memory pages based on the multiple customer memory pages having the largest relative access counter value.

[0112] Example 14 includes the subject matter of Example 11, wherein the target memory page includes a target client memory page in a second byte-addressable memory, and the computer-readable storage medium also includes instructions for a virtual machine monitor (VMM) that can be executed by a processor to cause the processor to: determine that the SDPT maps a first client memory page to a first host memory page; determine that the SDPT maps a target client memory page to a second host memory page; copy the first host memory page to a temporary memory page in the second byte-addressable memory; copy the second host memory page to the first host memory page; copy the temporary memory page in the second byte-addressable memory to the second host memory page; modify the SDPT to (i) map the first client memory page to the first host memory page, and (ii) map the target client memory page to the second host memory page; clear the translation cache of the SDPT; and resume execution of the VM and the application.

[0113] Example 15 includes the subject matter of Example 11, wherein the target memory page includes a target host memory page in a second byte-addressable memory, and the computer-readable storage medium also includes instructions for a virtual machine monitor (VMM) that can be executed by a processor to cause the processor to: determine that the SDPT maps a first client memory page to a first host memory page; copy the first client memory page to the target host memory page; modify the SDPT to map the first client memory page to the target host memory page; release the first host memory page; clear the translation cache of the SDPT; and resume execution of the VM and the application.

[0114] Example 16 includes the subject of Example 10, where the page table includes a guest page table of a VM, the target memory page includes a target guest memory page in a second byte-addressable memory, and the VM includes instructions for migrating a first guest memory page. These instructions are executable by a processor to cause the processor to: determine that the guest page table maps a first virtual address to the first guest memory page and maps a second virtual address to the target guest memory page; allocate a temporary memory page in the second byte-addressable memory; copy the first guest memory page to the temporary memory page; copy the target guest memory page to a first host memory page; copy the temporary memory page to the target guest memory page; modify the guest page table to map the first virtual address to the target guest memory page; release the first guest memory page; clear the translation look-ahead buffer (TLB) for the entry of the first guest memory page; and resume execution of the VM and the application.

[0115] Example 17 includes the subject of Example 10, where the page table includes a guest page table of a VM, the target memory page includes a target guest memory page in a second byte-addressable memory, and the VM includes instructions for migrating a first guest memory page. These instructions are executable by a processor to cause the processor to: determine that the guest page table maps a first virtual address to the first guest memory page; allocate the target guest memory page in the second byte-addressable memory; copy the first guest memory page to the target guest memory page; modify the guest page table to map the first virtual address to the target guest memory page; clear the translation look-ahead buffer (TLB) for the entry of the first guest memory page; and resume execution of the VM and the application.

[0116] Example 18 includes the subject matter of Examples 10 through 17, wherein the first byte-addressable memory includes persistent memory (PMEM) implemented in a non-volatile dual in-line memory module, and the second byte-addressable memory includes dynamic random access memory (DRAM), wherein the access speed of PMEM and DRAM is relative to the processor.

[0117] Example 19 is a method comprising: identifying a first customer memory page based at least in part on the number of page table entries of a first customer memory page accessed by an application executing in a virtual machine (VM) on a processor, the first customer memory page residing in a first byte-addressable memory; suspending the VM and the execution of the application on the processor; and migrating the first customer memory page to a target memory page in a second byte-addressable memory, the target memory page including either a target host memory page or a target customer memory page, the second byte-addressable memory having a faster access speed than the first byte-addressable memory.

[0118] Example 20 includes the subject of Example 19, where the page table includes a second-dimensional page table (SDPT) that maps client physical memory addresses to host physical memory addresses. The method further includes: clearing the access bit of each of a plurality of entries in the SDPT, each entry in the SDPT corresponding to a corresponding client memory page among a plurality of client memory pages, including a first client memory page; waiting for a predefined amount of time; traversing the SDPT; determining that the access bit of the entry in the SDPT corresponding to the first client memory page has been set; and incrementing the access counter of the first client memory page, the access counter value reflecting the number of times the first client memory page has been accessed.

[0119] Example 21 includes the subject of Example 20, wherein the first customer memory page is further identified based on a migration policy that specifies the migration of multiple customer memory pages, including the first customer memory page, and the method further includes: dividing the memory space of the VM into multiple parts, the first customer memory page being in a first part of the multiple parts of the memory space of the VM, and instructions for identifying the first customer memory page being executed for the first part of the memory space of the VM.

[0120] Example 22 includes the subject of Example 21, wherein the first client memory page is also identified based on a migration strategy that specifies the migration of multiple client memory pages, including the first client memory page, and the method further includes: determining an access counter value for each of the multiple client memory pages; and determining the multiple client memory pages based on the multiple client memory pages having the largest relative access counter value.

[0121] Example 23 includes the subject of Example 20, wherein the target memory page includes a target client memory page in a second byte-addressable memory, and the method further includes: determining, by a virtual machine monitor (VMM) executing on the processor, that the SDPT maps a first client memory page to a first host memory page; determining, by the VMM, that the SDPT maps a target client memory page to a second host memory page; copying the first host memory page to a temporary memory page in the second byte-addressable memory by the VMM; copying the second host memory page to the first host memory page by the VMM; copying the temporary memory page in the second byte-addressable memory to the second host memory page by the VMM; modifying the SDPT by the VMM to (i) map the first client memory page to the first host memory page and (ii) map the target client memory page to the second host memory page; clearing the translation cache of the SDPT by the VMM; and resuming the execution of the VM and the application by the VMM.

[0122] Example 24 includes the subject of Example 20, wherein the target memory page includes a target host memory page in a second byte-addressable memory, and the method further includes: determining, by a virtual machine monitor (VMM) executing on the processor, that the SDPT maps a first client memory page to a first host memory page; copying the first client memory page to the target host memory page by the VMM; modifying the SDPT by the VMM to map the first client memory page to the target host memory page; releasing the first host memory page by the VMM; clearing the translation cache of the SDPT by the VMM; and resuming the execution of the VM and the application by the VMM.

[0123] Example 25 includes the subject of Example 19, the page table includes a VM's client page table, the target memory page includes a target client memory page in a second byte-addressable memory, and the method further includes: the VM determining that the client page table maps a first virtual address to a first client memory page and maps a second virtual address to a target client memory page; the VM allocating a temporary memory page in the second byte-addressable memory; the VM copying the first client memory page to the temporary memory page; the VM copying the target client memory page to a first host memory page; the VM copying the temporary memory page to the target client memory page; the VM modifying the client page table to map the first virtual address to the target client memory page; the VM releasing the first client memory page; the VM clearing the Translation Lookahead Buffer (TLB) for the entry used for the first client memory page; and the VM resuming execution of the VM and the application.

[0124] Example 26 includes the subject of Example 19, where the page table includes a VM's client page table, the target memory page includes a target client memory page in a second byte-addressable memory, and the method further includes: the VM determining that the client page table maps a first virtual address to a first client memory page; the VM allocating a target client memory page in the second byte-addressable memory; the VM copying the first client memory page to the target client memory page; the VM modifying the client page table to map the first virtual address to the target client memory page; the VM clearing the translation lookahead buffer (TLB) for the entry used for the first client memory page; and the VM resuming execution of the VM and the application.

[0125] Example 27 includes the subject matter of Examples 19 through 26, wherein the first byte-addressable memory includes persistent memory (PMEM) implemented in a non-volatile dual in-line memory module, and the second byte-addressable memory includes dynamic random access memory (DRAM), wherein the access speed of PMEM and DRAM is relative to the processor.

[0126] Example 28 is an apparatus comprising: means for identifying a first client memory page in a page table based at least in part on the number of times an application accesses the first client memory page in the page table, the application executing in a virtual machine (VM) on a processor, the first client memory page corresponding to a first byte-addressable memory of the apparatus; means for suspending the VM and the execution of the application on the processor; and means for migrating the first client memory page to a target memory page in a second byte-addressable memory corresponding to the apparatus, the target memory page including one of a target host memory page and a target client memory page, the second byte-addressable memory having a faster access speed than the first byte-addressable memory.

[0127] Example 29 includes the subject of Example 28, where the page table includes a second-dimensional page table (SDPT) that maps client physical memory addresses to host physical memory addresses. The apparatus further includes: means for clearing the access bit of each of a plurality of entries in the SDPT, each entry in the SDPT corresponding to a corresponding client memory page among a plurality of client memory pages, including a first client memory page; means for waiting for a predefined amount of time; means for traversing the SDPT; means for determining that the access bit of the entry in the SDPT corresponding to the first client memory page has been set; and means for incrementing an access counter of the first client memory page, the access counter reflecting the number of accesses to the first client memory page.

[0128] Example 30 includes the subject of Example 29, wherein the first client memory page is further identified based on a migration policy that specifies the migration of multiple client memory pages, including the first client memory page, and the apparatus further includes: means for dividing the memory space of the VM into multiple portions, the first client memory page being in a first portion of the multiple portions of the memory space of the VM, and instructions for identifying the first client memory page being executed for the first portion of the memory space of the VM.

[0129] Example 31 includes the subject of Example 30, wherein the first client memory page is further identified based on a migration policy that specifies the migration of a plurality of client memory pages, including the first client memory page, and the apparatus further includes: means for determining an access counter value for each of the plurality of client memory pages; and means for determining the plurality of client memory pages based on the plurality of client memory pages having the largest relative access counter value.

[0130] Example 32 includes the subject of Example 29, wherein the target memory page includes a target guest memory page in a second byte-addressable memory, and the apparatus further includes: means for determining, by a virtual machine monitor (VMM) executing on the processor, that the SDPT maps a first guest memory page to a first host memory page; means for determining, by the VMM, that the SDPT maps a target guest memory page to a second host memory page; means for copying, by the VMM, the first host memory page to a temporary memory page in the second byte-addressable memory; means for copying, by the VMM, the second host memory page to the first host memory page; means for copying, by the VMM, the temporary memory page in the second byte-addressable memory to the second host memory page; means for modifying, by the VMM, the SDPT to (i) map the first guest memory page to the first host memory page and (ii) map the target guest memory page to the second host memory page; means for clearing the translation cache of the SDPT by the VMM; and means for resuming the execution of the VM and the application by the VMM.

[0131] Example 33 includes the subject of Example 29, wherein the target memory page includes a target host memory page in a second byte-addressable memory, and the apparatus further includes: means for determining, by a virtual machine monitor (VMM) executing on the processor, that the SDPT maps the first client memory page to the first host memory page; means for copying the first client memory page to the target host memory page by the VMM; means for modifying the SDPT by the VMM to map the first client memory page to the target host memory page; means for releasing the first host memory page by the VMM; means for clearing the translation cache of the SDPT by the VMM; and means for resuming the execution of the VM and the application by the VMM.

[0132] Example 34 includes the subject matter of Example 28, the page table including a VM's client page table, the target memory page including a target client memory page in second byte-addressable memory, and the apparatus further includes: means for the VM to determine that the client page table maps a first virtual address to a first client memory page and maps a second virtual address to a target client memory page; means for the VM to allocate a temporary memory page in the second byte-addressable memory; means for the VM to copy the first client memory page to the temporary memory page; means for the VM to copy the target client memory page to a first host memory page; means for the VM to copy the temporary memory page to the target client memory page; means for the VM to modify the client page table to map the first virtual address to the target client memory page; means for the VM to release the first client memory page; means for the VM to clear the Translation Look-Ahead Buffer (TLB) for the entry of the first client memory page; and means for the VM to resume execution of the VM and the application.

[0133] Example 35 includes the subject of Example 28, the page table includes a VM's client page table, the target memory page includes a target client memory page in a second byte-addressable memory, and the apparatus further includes: means for the VM to determine that the client page table maps a first virtual address to a first client memory page; means for the VM to allocate the target client memory page in the second byte-addressable memory; means for the VM to copy the first client memory page to the target client memory page; means for the VM to modify the client page table to map the first virtual address to the target client memory page; means for the VM to clear a translation look-ahead buffer (TLB) for entries for the first client memory page; and means for the VM to resume execution of the VM and the application.

[0134] Example 36 includes the subject matter of Examples 28 to 35, wherein the first byte-addressable memory includes persistent memory (PMEM) implemented in a non-volatile dual in-line memory module, and the second byte-addressable memory includes dynamic random access memory (DRAM), wherein the access speed of PMEM and DRAM is relative to the processor.

[0135] For purposes of explanation and description, the above description of exemplary embodiments has been provided. This description is not intended to be exhaustive or to limit this disclosure to the precise forms disclosed. Many modifications and variations are possible based on this disclosure. It is anticipated that the scope of this disclosure is not limited by this specific embodiment but rather by the appended claims. Future applications claiming priority to this application may claim the disclosed subject matter in different ways and may generally include any set of one or more limitations as disclosed or otherwise shown herein.

Claims

1. A processor, comprising: Multi-chip package, comprising multiple integrated circuit chips, including: Multiple cores used to execute instructions; An interface circuit is provided for coupling the plurality of cores to a plurality of memories, the plurality of memories including a first memory and a second memory, wherein the first memory is associated with a faster access speed compared to the second memory; One or more of the plurality of cores are used to: convert a plurality of client virtual addresses into a plurality of corresponding client physical addresses based on a first set of page tables, and convert the plurality of client physical addresses into a plurality of corresponding host physical addresses based on a second set of page tables, wherein a first part of the plurality of host physical addresses is associated with a first plurality of memory pages in the first memory, and a second part of the plurality of host physical addresses is associated with a second plurality of memory pages in the second memory; A Translation Lookout Buffer (TLB) is used to cache translations associated with the respective plurality of host physical addresses; and A circuit, at least in part according to executable code, operable to: migrate a set of memory pages from the second memory to the first memory, and clear one or more transitions from the TLB corresponding to the migrated set of memory pages.

2. The processor according to claim 1, wherein, The set of memory pages is selected for migration based on the number of times the set of memory pages are accessed by at least one guest operating system or virtual machine.

3. The processor according to claim 1, wherein, The corresponding entries in the second set of page tables will be updated with the host physical address from the first part of the plurality of host physical addresses to indicate the update location of the set of memory pages in the first memory.

4. The processor according to claim 1, wherein, The first memory includes dynamic random access memory (DRAM), and the second memory includes persistent memory that is addressable by bytes.

5. The processor according to claim 1, wherein, When the set of memory pages is migrated, access to the set of memory pages will be suspended.

6. The processor according to claim 1, wherein, The set of memory pages in the second plurality of memory pages will migrate from the second memory to the first memory in response to the virtual machine monitor (VMM).

7. The processor according to claim 6, wherein, The plurality of client physical addresses will be translated into the corresponding plurality of host physical addresses in response to the VMM.

8. The processor according to claim 2, wherein, The number of accesses to the set of memory pages is determined by performing a series of operations, including: clearing the access bit of each of the multiple entries in the second set of page tables to zero; waiting for a predefined amount of time; identifying the entries whose access bits have been set; and incrementing the corresponding access counter value.

9. The processor according to claim 1, wherein, Another set of memory pages in the first plurality of memory pages will be migrated from the first memory to the second memory, so as to free up space in the first memory for the set of memory pages to be migrated in the second plurality of memory pages.

10. The processor according to claim 9, wherein, The set of memory pages in the first plurality of memory pages is identified based on the number of times the set of memory pages is accessed, and the number of times the set of memory pages in the first plurality of memory pages includes a number less than the number of times the set of memory pages in the second plurality of memory pages is accessed.

11. The processor according to claim 10, wherein, The corresponding entries in the second set of page tables will be updated with the host physical address in the second part of the plurality of host physical addresses to indicate the update position of the set of memory pages in the second memory.

12. A machine-readable medium having program code stored thereon, the program code causing the machine to perform operations when executed by a machine, the operations including: Instructions are executed on multiple cores, which are coupled to multiple memories via interface circuits. The multiple memories include a first memory and a second memory, with the first memory associated with faster access speeds compared to the second memory. Based on the first set of page tables, multiple client virtual addresses are translated into corresponding multiple client physical addresses; Based on the second set of page tables, the plurality of client physical addresses are converted into the corresponding plurality of host physical addresses. The first part of the plurality of host physical addresses is associated with the corresponding first plurality of memory pages in the first memory, and the second part of the plurality of host physical addresses is associated with the corresponding second plurality of memory pages in the second memory. The translations associated with the corresponding multiple host physical addresses are cached in the Translation Look-Ahead Buffer (TLB); The set of memory pages is identified based on the number of accesses made by at least one guest operating system or virtual machine to a set of memory pages in the plurality of memory pages; Migrate the set of memory pages from the second memory to the first memory; as well as Clear one or more transitions from the TLB that correspond to the set of memory pages that were migrated.

13. The machine-readable medium of claim 12, further comprising: The corresponding entries in the second set of page tables are updated with the host physical addresses from the first portion of the plurality of host physical addresses to indicate the update location of the set of memory pages in the first memory.

14. The machine-readable medium according to claim 12, wherein, The first memory includes dynamic random access memory (DRAM), and the second memory includes persistent memory that is addressable by bytes.

15. The machine-readable medium according to claim 12, wherein, When the set of memory pages is migrated, access to the set of memory pages will be suspended.

16. The machine-readable medium of claim 12, wherein, The set of memory pages in the second plurality of memory pages will migrate from the second memory to the first memory in response to the virtual machine monitor (VMM).

17. The machine-readable medium of claim 16, wherein, The plurality of client physical addresses will be translated into the respective plurality of host physical addresses in response to the VMM.

18. The machine-readable medium of claim 13, wherein, The number of accesses to the set of memory pages is determined by performing additional operations in response to the program code. These additional operations include: clearing the access bit of each of the plurality of entries in the second set of page tables to zero; waiting for a predefined amount of time; identifying the entries among the plurality of entries whose access bits have been set; and incrementing the corresponding access counter value.

19. The machine-readable medium according to claim 12, wherein, Another set of memory pages in the first plurality of memory pages will be migrated from the first memory to the second memory, so as to free up space in the first memory for the set of memory pages to be migrated in the second plurality of memory pages.

20. The machine-readable medium of claim 19, wherein, The set of memory pages in the first plurality of memory pages is identified based on the number of times the set of memory pages is accessed, and the number of times the set of memory pages in the first plurality of memory pages includes a number less than the number of times the set of memory pages in the second plurality of memory pages is accessed.

21. The machine-readable medium of claim 20, wherein, The corresponding entries in the second set of page tables will be updated with the host physical address in the second part of the plurality of host physical addresses to indicate the update position of the set of memory pages in the second memory.

Images