Memory access tracking using peripheral devices

Abstract

This disclosure relates to memory access tracking using peripheral devices. A compute node includes memory, a processor, and peripheral devices. The memory stores memory pages. The processor runs software that accesses the memory and identifies one or more first memory pages accessed by the software. The peripheral devices directly access one or more second memory pages in the compute node's memory using direct memory access (DMA) and inform the processor of the second memory pages accessed using DMA. The processor also maintains a data structure that tracks both (i) the first memory pages identified by the processor and (ii) the second memory pages informed by the peripheral devices.

Description

Field of the Invention

[0001] This invention relates generally to computing systems, and more particularly to methods and systems for memory management. Background of the Invention

[0002] Computing systems typically track hardware or software accesses to memory pages. Depending on the application or use case, a computing system may track only write accesses (i.e., only memory access operations that modify memory pages) or both read and write accesses. Memory page tracking is useful in a wide variety of use cases, to name just a few, such as virtual machine (VM) migration between physical compute nodes, page swapping between memory and storage devices, and various caching schemes. Invention Overview

[0003] The embodiments of the invention described herein provide a computing node including memory, a processor, and peripheral devices. The memory stores memory pages. The processor runs software that accesses the memory and identifies one or more first memory pages accessed by the software in the memory. The peripheral devices directly access one or more second memory pages in the computing node's memory using direct memory access (DMA) and inform the processor of the second memory pages accessed using DMA. The processor also maintains a data structure that tracks both (i) the first memory pages identified by the processor and (ii) the second memory pages informed by the peripheral devices.

[0004] In some embodiments, the peripheral device is a network adapter, and the second memory page includes data transferred between the processor and the network by the network adapter.

[0005] In some embodiments, the software includes a guest hosted on a compute node, the guest having memory space allocated in memory, and a processor configured to track dirty pages belonging to the guest's memory space in a data structure, the dirty pages including (i) a first dirty page written by the guest and (ii) a second dirty page written by a peripheral device. In an example embodiment, the processor is configured to perform a process of cooperating with another compute node and using the data structure tracking the first and second dirty pages to migrate the guest to another compute node. In one embodiment, the peripheral device is configured to limit the rate at which the guest is migrated.

[0006] In the disclosed embodiments, the processor is used to perform a process of exchanging some memory pages with a storage device using a data structure that tracks first and second memory pages.

[0007] In some embodiments, the peripheral device is used to merge indications of access to a second memory page and to inform the processor by sending the merged indications. In an example embodiment, the peripheral device is used to merge indications by caching the indications in a merge buffer. In another embodiment, the peripheral device is used to merge indications by using a bit representing the location of the second memory page in a bitmap.

[0008] In yet another embodiment, the peripheral device informs the processor of the second memory page by directly writing to a data structure maintained by the processor. In yet another embodiment, the peripheral device informs the processor of the second memory page by reporting the corresponding address of the second memory page.

[0009] According to embodiments of the present invention, a method is also provided, comprising using a processor to run software that accesses memory and identifying one or more first memory pages in memory accessed by the software. One or more second memory pages in memory are directly accessed using direct memory access (DMA) via a peripheral device, and the processor is informed of the second memory pages accessed using DMA. A data structure is maintained by the processor that tracks both: (i) the first memory pages identified by the processor and (ii) the second memory pages informed of by the peripheral device.

[0010] According to embodiments of the present invention, a method is also provided for jointly tracking (i) memory pages accessed by software of the processor and (ii) memory pages accessed by peripheral devices independent of the processor. The method includes running software that accesses memory on the processor and identifying one or more memory pages accessed by the software. One or more second memory pages are directly accessed by the peripheral device using Direct Memory Access (DMA), and the processor is informed of the memory pages accessed using DMA. The processor maintains a data structure that tracks both: (i) the memory pages accessed by the software as identified by the processor, and (ii) the memory pages accessed by the peripheral device as informed by the peripheral device.

[0011] The invention will be more fully understood from the following detailed description of embodiments thereof, taken in conjunction with the accompanying drawings, in which: Brief description of the attached diagram

[0012] Figure 1 This is a block diagram schematically illustrating a computing node performing memory page tracking with the assistance of a peripheral device according to an embodiment of the present invention;

[0013] Figure 2 This is a flowchart illustrating, schematically, a method for tracking memory pages according to an embodiment of the present invention; and

[0014] Figure 3 This is a block diagram illustrating the migration of virtual machines (VMs) between compute nodes according to an embodiment of the present invention. Detailed Implementation

[0015] Overview

[0016] The embodiments of the invention described herein provide improved methods and systems for tracking memory page accesses in a computing node. In this context, accessed memory pages are also referred to as “dirty pages,” and the two terms are used interchangeably herein. The exact definition of a “dirty page,” and when a page is no longer considered a dirty page, may vary from application to application.

[0017] In some applications, a page becomes dirty when it is modified and is considered dirty until some level of consistency is reached, for example, when the modified page has been committed to the storage device. In other applications (such as those tracking Least Recently Used (LRU) or Most Recently Used (MRU) pages), reading a page is sufficient to make it dirty. Overall, the disclosed techniques can be used in any suitable application involving tracking access to memory pages. This document describes example use cases related to virtual machine (VM) migration and page swapping.

[0018] In some embodiments, a compute node includes memory and a processor. The processor runs software that, among other tasks, accesses (e.g., reads and / or writes) memory pages in memory. The software may include, for example, a hypervisor or an operating system. Additionally, the compute node includes peripheral devices that use direct memory access (DMA) to access memory.

[0019] In one example, the peripheral device is a network adapter that receives data from the network and writes it directly to the memory of the compute node. In another example, the peripheral device is a storage device, such as a solid-state drive (SSD) that stores data for the CPU of the compute node. When a read operation is performed for the CPU, data is retrieved from the storage device and written to the CPU memory using DMA. These DMA operations can transfer large amounts of data to the CPU memory without CPU intervention.

[0020] In some implementations, the processor (including the software running thereon) is unaware of memory access operations performed by peripheral devices because these operations are performed using DMA. Therefore, the processor can identify and track memory pages accessed by software (“dirty”), rather than those accessed by peripheral devices. This partial tracking of dirty pages is problematic and can lead to data inconsistencies and suboptimal performance.

[0021] To address this challenge, in some embodiments of the invention, the peripheral device informs the processor which memory pages it has accessed using DMA. The processor maintains a data structure that tracks both: (i) memory pages accessed by software (identified by the processor) and (ii) memory pages accessed by the peripheral device (informed by the peripheral device). In this way, the processor can fully track all dirty pages in memory, regardless of which entity made those pages dirty.

[0022] In some embodiments, the memory page tracking scheme described above is implemented entirely in hardware. Peripheral devices can use various protocols and formats to report the memory pages they access. Several examples are described herein.

[0023] System Description

[0024] Figure 1 This is a block diagram schematically illustrating a computing node 20 for tracking dirty pages according to an embodiment of the present invention. Figure 1 It is relatively universal; in a sense, it is not dedicated and is applicable to various types of compute nodes and peripheral devices. Several specific examples will be described further below.

[0025] Computing node 20 may include, for example, a server, workstation, or any other suitable type of computing node in a data center. Node 20 includes processor 24 and memory 28. In this example, processor 24 includes a central processing unit (CPU), but the processor may alternatively include a graphics processing unit (GPU) or any other suitable type of processor. In this example, memory 28 includes random access memory (RAM), but may also include any other suitable type of memory.

[0026] Node 20 also includes a peripheral device 32 that communicates with processor 24 and memory 28 via peripheral interface 36. Peripheral device 32 may include, for example, a network adapter such as an Ethernet network interface controller (NIC), an unlimited bandwidth host channel adapter (HCA), a smart NIC, a data processing unit (DPU), a network-enabled GPU, or any other suitable type of network adapter. As another example, peripheral device 32 may include a storage device such as a solid-state drive (SSD) or a small computer system interface (SCSI) disk. Alternatively, peripheral device 32 may include an accelerator implemented in a GPU or field-programmable gate array (FPGA) or any other suitable type of peripheral device. Peripheral interface 36 may include, for example, a peripheral component interconnect high-speed (PCIe) bus.

[0027] In some embodiments, processor 24 runs software 40, which accesses memory pages 44 in memory 28, etc. Software 40 may include, for example, a hypervisor, one or more virtual machines (VMs), one or more applications (“apps”), an operating system, and / or any other suitable type of software. Software 40 may appropriately read and / or write to memory pages 44. In the following description, for the sake of brevity, memory page 44 is also simply referred to as a “page”.

[0028] In addition to memory accesses (reads and / or writes) performed by software 40, peripheral device 32 also uses DMA to access (reads and / or writes) memory page 44. Peripheral device 32 may access the same memory page as software 40, partially overlapping memory pages, or different memory pages. Typically, accesses to memory page 44 by software 40 and peripheral device 32 are uncoordinated. Specifically, processor 24 is typically unaware of which memory pages 44 have been accessed by peripheral device 32 (“dirty”).

[0029] In order to perform a complete (i.e., impartial) tracking of dirty pages, peripheral device 32 reports the pages it is accessing to processor 24. Peripheral device 32 typically (e.g., periodically) sends such report messages through interface 36. Several examples of efficient report formats are described below.

[0030] Based on reports from peripheral device 32, processor 24 maintains a dirty page data structure 48 that tracks both: (i) memory pages accessed by software 40 (recognized by processor 24) and (ii) memory pages accessed by peripheral device 32 (reported by peripheral device). In some embodiments, the dirty page data structure 48 includes a bitmap, where each bit in the bitmap corresponds to a corresponding memory page. A set bit in the bitmap indicates that the corresponding page is dirty, and vice versa.

[0031] Processor 24 can identify pages accessed (“dirty”) by software 40 in various ways. For example, the processor can maintain a page table with multiple page table entries (PTEs). When software 40 dirties a memory page, processor 24 can set the corresponding bit in the corresponding PTE to indicate that the page is dirty.

[0032] Figure 2 This is a flowchart schematically illustrating a method for tracking memory pages in a computing node 20 according to an embodiment of the present invention. In this example, (i) software 40 is a hypervisor for a managed VM, (ii) peripheral device 32 is a NIC, (iii) peripheral interface 36 is a PCIe bus, and (iv) dirty page is defined as a page that has been written to (i.e. modified).

[0033] As shown in the figure, memory 28 is written to by the hypervisor and the NIC, typically without mutual coordination or awareness. In hypervisor access operation 50, the hypervisor writes some memory pages 44 on behalf of the VM. In NIC access operation 54, the NIC receives data from the network and uses DMA to write the data to some memory pages 44.

[0034] In this example, the NIC operates in pass-through mode using single root input / output virtualization (SR-IOV). In this mode, the NIC exposes a corresponding virtual NIC (vNIC) for each VM. Each vNIC is configured to use DMA to write data directly to the memory space of the corresponding VM. The hypervisor is unaware of these DMA operations and cannot track the pages written by the NIC.

[0035] To achieve impartial dirty page tracking, during record operation 58, the NIC records the address of the memory page being written. Typically, the NIC tracks and records addresses based on the physical memory space defined for the VM (i.e., the passenger physical address - GPA).

[0036] In reporting operation 62, the NIC reports the recorded addresses to the hypervisor. In update operation 66, the hypervisor updates the dirty page bitmap 48 to reflect (in operation 50) the pages dirtyed by the hypervisor and (in operation 54) the pages dirtyed by the NIC. In this example, the hypervisor maintains a separate bitmap 48 for each VM. In a given bitmap 48, the address of a memory page is the GPA in the VM's guest physical address space.

[0037] Figure 2 The method described herein is an example method and is depicted solely for conceptual clarity purposes. In alternative embodiments, any other suitable method may be used.

[0038] Typically, peripheral device 32 (NIC in this example) receives a request from software 40 (hypervisor in this example) to begin tracking dirty pages for a specified guest machine (e.g., VM). In response, device 32 begins tracking writes to the memory of the specified guest machine (and / or, appropriately, other accesses). Dirty page tracking is performed according to the GPA. In some embodiments, dirty page tracking is performed according to the GPA even if the write is actually published to the machine address (MA) (e.g., when peripheral device 32 supports Address Translation Service (ATS)). Dirty page tracking is typically performed in packets. Processing a given packet may require marking multiple pages as dirty. Peripheral device 32 typically writes the addresses of dirty pages (e.g., pages that have been written) to an internal database. If multiple packets are written to the same physical memory page, device 32 can reduce reporting to software 40 by reporting the page as dirty only once. This improvement can be done thoroughly or as a best-effort approach. Software 40 then updates the dirty page data structure 48 based on the addresses reported by device 32.

[0039] Real-time migration use cases

[0040] One possible use case for the disclosed dirty page tracing technique is migrating a VM from one compute node to another. In a typical live migration process, both the VM image (e.g., the memory used by the kernel and various drives, including I / O drives) and the VM's memory space are transferred from the source compute node to the target compute node.

[0041] Some migration processes (called "pre-copy" migrations) first transfer the VM's storage space, then temporarily suspend the VM, transfer the VM image, and then restore the VM on the target compute node. Other migration processes (called "post-copy" migrations) begin by suspending the VM and transferring the VM image to the target compute node, and then transferring the VM's storage space. Hybrid approaches are also possible. For most practical applications, a simple solution of freezing the VM and then copying its entire storage to the target compute node is not feasible because it would result in unacceptably long offline times.

[0042] During any migration process, the VM's memory pages may become dirty after the migration begins. Therefore, the migration process typically includes tracking dirty pages in the VM's address space. Any such tracking can be performed using the disclosed techniques.

[0043] Figure 3This is a schematic block diagram illustrating the migration of VMs between compute nodes according to an embodiment of the present invention. The top of the diagram shows two compute nodes, denoted as 70A and 70B, hosting multiple VMs. Each compute node includes one or more CPUs 74 and a hypervisor (HV) 78. The compute nodes are connected to a network 86 via a corresponding DMANIC 82. It is assumed that the system operates using SR-IOV.

[0044] Initially, node 70A hosts two VMs, VM-A and VM-B, and node 70B hosts two other VMs, VM-C and VM-D. At some point, a decision is made to migrate VM-B from node 70A to node 70B. During this migration, compute node 70A acts as the source compute node, and compute node 70B acts as the target compute node. In the following description, HV 78 of node 70A is referred to as the "source HV," and HV 78 of node 70B is referred to as the "target HV."

[0045] In this example, VM-B is being migrated during a pre-replication migration process jointly executed by the source and target hypervisors. Typically, a pre-replication migration process includes the following sequence of operations:

[0046] ■ Source HV begins tracing memory pages accessed by the VM.

[0047] ■ The source HV copies the entire memory space (dirty pages or others) of the VM to the target node.

[0048] ■ From this perspective, the source HV only copies the dirty pages (relative to the version that has already been transferred).

[0049] ■ Once the stopping conditions are met (e.g., the number of dirty pages drops below a certain small threshold, or the number of dirty pages does not decrease quickly enough due to high access rates, or other conditions), the source HV will temporarily suspend the VM.

[0050] ■ At this point, the source HV transfers the VM image to the target node.

[0051] ■ The network was reconfigured to redirect subsequent traffic destined for the VM to the target node.

[0052] ■ The target VM is restored from the source VM in the same state as the suspended VM.

[0053] As stated above, when using SR-IOV, the HV 78 on the source node is unaware of the pages in the VM memory space that have been soiled by the NIC 82. Unless the disclosed techniques are used to account for these pages, the source HV is likely unable to track all soiled pages, leading to data corruption during migration.

[0054] According to the example embodiment, Figure 3 The two illustrations at the middle and bottom of the diagram show the internal structure of the HV 78 and NIC 82 of the source node 70A. The illustrations focus on the NIC and HV components associated with dirty page tracking. In this example, NIC 82 and HV 78 track the dirty pages of both VM-A and VM-B.

[0055] As shown in the bottom illustration, NIC 82 includes packet processing hardware 90 for transmitting packets to and receiving packets from network 86. The NIC also includes a Translation and Protection Table (TPT) 94, NIC memory 98, and distributed hardware 110. TPT 94 stores address translations between the virtual address specified in the packet and the GPA in the VM memory. In some embodiments, TPT 94 also includes a dirty page database 102. In response to recognizing an incoming packet from the network guaranteeing a write to the memory space of VM-A or VM-B (labeled "Passenger 0" and "Passenger 1" in the figure), the NIC writes the GPA of that write operation to the dirty page database 102 in TPT 94. Note that dirty page tracking using TPT is only one example. In alternative embodiments, NIC 82 may track its soiled memory pages in any other suitable manner.

[0056] The GPA of the dirty page is copied to the dirty page address buffer 106 in the NIC memory 98. The distribution hardware 110 then separates the GPA of the dirty page for VM-A and the GPA of the dirty page for VM-B, and delivers them to HV 78 respectively. In one embodiment, the distribution hardware 110 delivers the GPA of the dirty page on demand, for example, in response to a request from HV 78 for the currently buffered dirty pages of a given VM.

[0057] HV 78 (see) Figure 3 The central illustration includes dirty page memory 114, which stores separate dirty page address buffers for different VMs. In this example, dirty page address buffers 118A and 118B cache the GPAs of dirty pages for VM-A and VM-B, respectively, provided by distributed hardware 110 of NIC 82.

[0058] Based on the GPA cached in buffer 118A, HV 78 updates the dirty page bitmap 122A of VM-A. Dirty page bitmap 122A thus tracks dirty pages of VM-A that have been soiled by HV itself and dirty pages of VM-A that have been soiled by NIC 82. Similarly, HV78 updates the dirty page bitmap 122B of VM-B based on the GPA cached in buffer 118B. Dirty page bitmap 122B tracks dirty pages of VM-B that have been soiled by HV78 and dirty pages of VM-B that have been soiled by NIC 82.

[0059] In some embodiments, the source HV 78 performs the migration process described above while simultaneously tracking dirty pages of VM-B in bitmap 122B. In this way, dirty pages are not missed despite the use of pass-through virtualization (e.g., SR-IOV).

[0060] The migration process described above is an example process, presented purely for clarity. In alternative embodiments, any other suitable process may be used. For example, the disclosed techniques are not limited to pre-copy migration and may be used for other migration processes. As another example, the above description relates to the migration of VMs, but the disclosed techniques can be used for the migration of other types of machines (e.g., containers or processes).

[0061] An alternative solution, instead of using the disclosed page-tracking technology, is to replace the pass-through peripherals (e.g., SR-IOV NICs) with fully software-virtualized devices before migration. However, this solution is costly in terms of performance. More importantly, it exposes the migration process to the VM itself, which is generally unacceptable.

[0062] In some embodiments, the NIC 82 in the source compute node 70A can help accelerate the migration process by limiting the rate at which VMs (or other passengers) are migrated. Limiting the rate at which VMs are migrated, in turn, reduces the rate at which VM pages become soiled, thereby reducing overall migration time. In example embodiments, the NIC 82 can limit the inbound packet rate of the migrated VMs by applying a regulator to lossy traffic (e.g., TCP) or by changing congestion control parameters for lossless traffic (e.g., InfiniBand or lossless RoCE). This regulation can be static (enforcing a static maximum rate) or dynamic.

[0063] During dynamic monitoring, NIC 82 can adjust the permitted inbound rate of passenger aircraft based on one or two of the following parameters:

[0064] 1. Dirty Page Rate. The NIC can monitor the rate at which each passenger marks a dirty page. If the rate exceeds a defined threshold, the NIC may change its inbound packet rate policy to further reduce the dirty page rate.

[0065] 2. Number of Pre-Replication Iterations. The NIC can monitor the number of pre-replication iterations performed by the hypervisor (e.g., by sending an alert to the NIC for each pre-replication iteration via a software API). If too many iterations occur (e.g., when the number of pre-replication iterations exceeds a defined number), the NIC can reduce the inbound traffic rate to decrease the rate of soiling.

[0066] Other use case examples

[0067] Another possible use case for the disclosed dirty page tracking technique is page-swapping applications. In the example page-swapping application, processor 24 ( Figure 1 Page-swapping software (e.g., as part of the OS) runs between memory 28 and a storage device (e.g., an SSD) to exchange memory pages. Typically, page-swapping software is designed to retain frequently accessed pages (or recently used pages) in memory and export rarely used pages (or pages accessed long ago) to the storage device. However, page-swapping performance may not be optimal if some memory page accesses are performed transparently to the processor 24 by a peripheral device 32 (e.g., a NIC).

[0068] Therefore, in some embodiments, processor 24 may use the disclosed techniques to track dirty pages in order to also account for pages contaminated by peripheral device 32. Page switching based on such dirty page tracking will perform well, even when some page accesses are transparent to processor 24.

[0069] Another possible use case is tracking reordered accesses to dynamic random access memory (DRAM) to save power. Another use case is collecting statistics on memory access operations. Furthermore, alternatively, the disclosed techniques can be used to track accesses to memory pages in any other suitable use case.

[0070] Tracking and merging in peripheral devices

[0071] In some embodiments, before writing to the memory of a passenger machine (e.g., a VM), peripheral device 32 checks the functions it writes to that should be traced, such as Bus:Device.Function(BDF), or a BDF combined with a process address space ID (PASID). If device 32 identifies the function as one that requires dirty page tracing (e.g., a passenger machine in migration), then device 32 records the accessed GPA[64:12] and the function.

[0072] To reduce software processing, device 32 can perform merging while tracking and logging dirty pages. In an example embodiment, device 32 maintains an intermediate merge buffer of a specific size (e.g., 32, 64, or 128 entries). Each entry in the merge buffer contains a pair of {function, GPA}. Each time a {function, GPA} is inserted into the merge buffer, device 32 checks whether the detected combination of {function, GPA} already exists in the merge buffer. If so, no further action is required. If not, the device inserts the new {function, GPA} into the merge buffer. When the merge buffer becomes full, or at the end of a pre-copy migration iteration, device 32 pops the merge buffer and reports all entries to software 40.

[0073] In an alternative embodiment, peripheral device 32 performs merging by maintaining a dirty page bitmap for each aircraft (e.g., in...). Figure 3 (In the example). The bitmap represents the aircraft's physical address (GPA) space, with each bit corresponding to a memory page. A set bit in the bitmap means the corresponding page is dirty, and vice versa. When a page is detected to be dirty, device 32 sets the corresponding bit. Multiple writes to the same page will set the same bit, and therefore device 32 will only report the page as dirty to software 40 once.

[0074] In reality, the bitmap for each passenger aircraft can be very large. For example, for a 4KB page per bit and a 64-bit address range, the bitmap would require 2... 49 Bytes of memory. In this case, additional optimizations can be implemented, where software (e.g., a device driver) informs device 32 of selected regions in the passenger's memory that require dirty page tracking. With this improvement, device 32 can retain only a portion (or portions) of the bitmap required for the total storage space of each passenger. Device 32 can report dirty bits at any suitable time interval or based on any suitable event (e.g., whenever each pre-copy iteration (in the pre-copy migration use case) and / or as requested by software 40).

[0075] In the example embodiment, 128 GB of passenger-side RAM space would require a total of 4 MB for compressing the bitmap. Device 32 may allocate memory in memory 28 (host memory) to maintain multiple bitmaps. In such an embodiment, writing a bitmap by device 32 may require fetching entries from host memory (in order to modify it and write it back). In severe cases (e.g., multiple fetches and high latency on interface 36), back pressure may be applied to the device. To avoid deadlock, care should be taken to avoid the risk of back pressure when fetching data from host memory.

[0076] Dirty Page Reporting Solution

[0077] In various embodiments, the peripheral device 32 may use various schemes and formats to report the dirty pages it tracks to the software 40.

[0078] In some embodiments, peripheral device 32 directly writes to the dirty page data structure 48 of processor 24 (e.g., writes to a dirty page bitmap maintained by a hypervisor). When using PCIe, this technique may require read-modify-write operations over the PCIe bus because PCIe writes are performed at the byte level. In one embodiment, to reduce the overhead of read-modify-write operations over the PCIe bus, peripheral device 32 uses PCIe TLP byte enable to write single bytes instead of double words. This mode allows direct writing to the dirty page data structure without read-modify-write. This technique provides optimal performance for byte-map-based tracing (where entire bytes in data structure 48 represent corresponding memory pages). For bitmap-based tracing, this technique introduces some false positives (i.e., marking some unwritten pages as dirty pages).

[0079] In an alternative embodiment, peripheral device 32 reports the addresses and functions it accesses, allowing software 40 (e.g., a device driver) to update dirty page data structure 48 accordingly. This technique allows for a more flexible interface between device 32 and software 40, but on the other hand, it may consume more PCIe bandwidth than directly writing to data structure 48.

[0080] Alternatively, peripheral device 32 can use any other suitable reporting scheme to report dirty pages to software 40.

[0081] Figure 1 and Figure 3 The configurations shown (e.g., system configuration, compute node configuration, NIC configuration, and HV configuration) are example configurations described purely for conceptual clarity. Any other suitable configuration may be used in alternative embodiments. The various system and compute node elements described herein may be implemented using software, suitable hardware (such as hardware in one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs), or a combination of software and hardware elements.

[0082] Certain system components (e.g., CPUs 24 and 74) and / or the processor in the NIC 82 may include a general-purpose processor that is programmed in software to perform the functions described herein. For example, the software may be downloaded to the computer electronically via a network, or the software may be optionally or additionally provided and / or stored on a non-transitory tangible medium, such as magnetic storage, optical storage, or electronic storage.

[0083] Although the embodiments described herein primarily address dirty page tracking in compute nodes, the methods and systems described herein can also be used for a variety of other applications.

[0084] Therefore, it will be understood that the embodiments described above are by way of example, and the invention is not limited to what has been specifically shown and described above. Rather, the scope of the invention includes combinations and sub-combinations of the various features described above, as well as variations and modifications of the invention that would arise to those skilled in the art upon reading the foregoing description and that are not disclosed in the prior art. Documents incorporated herein by reference shall be considered an integral part of this application, and the definitions in this specification shall be considered only unless any term is defined in these incorporated documents in a manner that conflicts to some extent with the definitions expressly or implicitly made in this specification.

Claims

1. A computing node, comprising: The memory is used to store memory pages; Peripheral bus; A processor, the processor being configured to run software that accesses the memory, and to identify one or more first memory pages in the memory accessed by the software; and Peripheral devices, the peripheral devices being used for: Direct Memory Access (DMA) is used to directly access one or more second memory pages in the memory of the compute node; and The processor is informed of the second memory page that has been directly accessed using Direct Memory Access (DMA). The processor also maintains a data structure that tracks two things: (i) a first memory page identified by the processor and (ii) a second memory page informed by the peripheral device, wherein the peripheral device is configured to inform the processor of the second memory page by directly writing to the data structure via the peripheral bus.

2. The computing node according to claim 1, wherein, The peripheral device is a network adapter, and the second memory page includes data transferred between the processor and the network by the network adapter.

3. The computing node according to claim 1, wherein, The software includes a passenger plane hosted on the computing node, the passenger plane having memory space allocated in the memory, and wherein the processor is configured to track dirty pages belonging to the memory space of the passenger plane in the data structure, the dirty pages including both: (i) a first dirty page written by the passenger plane and (ii) a second dirty page written by the peripheral device.

4. The computing node according to claim 3, wherein, The processor is used to perform a process of migrating the passenger aircraft to the other computing node in cooperation with another computing node and using the data structure that tracks the first dirty page and the second dirty page.

5. The computing node according to claim 4, wherein, The peripheral device is used to limit the flow rate at which the passenger aircraft is relocated.

6. The computing node according to claim 1, wherein, The processor is used to perform a process of exchanging some memory pages with a storage device using the data structure that tracks the first memory page and the second memory page.

7. The computing node according to claim 1, wherein, The peripheral device informs the processor of the second memory page by reporting the corresponding address of the second memory page.

8. The computing node according to claim 1, wherein, The peripheral device is used to merge instructions for accessing the second memory page and to inform the processor of the second memory page by sending a merge instruction.

9. The computing node according to claim 8, wherein, The peripheral device is used to merge the indication by caching the indication in a merge buffer residing in the peripheral device.

10. The computing node according to claim 8, wherein, The peripheral device is used to incorporate the indication by using a location bit representing the second memory page in a bitmap residing in the peripheral device.

11. A method for memory management, comprising: The processor runs software that accesses memory and identifies one or more first memory pages in the memory that are accessed by the software. Use peripheral devices that communicate with the processor via a peripheral bus: Direct memory access (DMA) is used to directly access one or more second memory pages of the memory; Inform the processor of the second memory page that has been directly accessed using Direct Memory Access (DMA); and The processor maintains a data structure that tracks both: (i) the first memory page identified by the processor and (ii) the second memory page informed by the peripheral device. In this context, informing the processor about the second memory page includes writing directly to the data structure via the peripheral bus.

12. The method according to claim 11, wherein, The peripheral device is a network adapter, and the second memory page includes data transferred between the processor and the network by the network adapter.

13. The method according to claim 11, wherein, The software includes a passenger plane hosted on a computing node, the passenger plane having memory space allocated in the memory, and wherein maintaining the data structure includes tracking dirty pages belonging to the memory space of the passenger plane in the data structure, the dirty pages including both: (i) a first dirty page written by the passenger plane and (ii) a second dirty page written by the peripheral device.

14. The method of claim 13, further comprising the processor performing a process of cooperating with another computing node and using the data structure tracking the first dirty page and the second dirty page to migrate the passenger aircraft to the other computing node.

15. The method of claim 14, further comprising using the peripheral device to limit the flow rate at which the passenger aircraft is diverted.

16. The method of claim 11, further comprising the processor using the data structure that tracks the first memory page and the second memory page to perform a process of exchanging some memory pages with the storage device.

17. The method according to claim 11, wherein, Informing the processor about the second memory page includes reporting the corresponding address of the second memory page.

18. The method according to claim 11, wherein, Informing the processor about the second memory page includes an instruction to merge access to the second memory page, and informing the processor by sending a merge instruction.

19. The method according to claim 18, wherein, The merge instruction includes caching the instruction in a merge buffer residing in the peripheral device.

20. The method according to claim 18, wherein, The merged instruction includes the location bit representing the second memory page in the bitmap residing in the peripheral device.

21. A method for jointly tracking both: (i) memory pages accessed by software of a processor and (ii) memory pages accessed by a peripheral device independent of the processor, the method comprising: Software that accesses memory runs on the processor, and one or more memory pages accessed by the software are identified; The peripheral device directly accesses one or more memory pages using Direct Memory Access (DMA); Inform the processor of memory pages that have been directly accessed by the peripheral device; and The processor maintains a data structure that tracks both: (i) memory pages accessed by the software and identified by the processor, and (ii) memory pages accessed by the peripheral device and disclosed by the peripheral device. In this context, informing the processor of the memory pages accessed by the peripheral device includes writing directly to the data structure via the peripheral bus.

Images