Memory migration in a multi-host data processing environment

Abstract

The data processing system includes a system fabric, a system memory, a memory controller, and a link controller configured to be communicatively coupled to the system fabric and to a destination host that is non-coherent to the source host via a communication link. The multiple processing units are configured to execute the logical partition and migrate the logical partition to the destination host via the communication link. Migrating the logical partition includes migrating a data set of the logical partition executing on the source host from a system memory of the source host to a system memory of the destination host via the communication link. After migrating at least a portion of the data set, migrating a state of the logical partition from the source host to the destination host via the communication link such that the logical partition executes on the destination host.

Description

[Technical Field]

[0001] The present invention relates generally to data processing, and more particularly to memory migration in a multi-host data processing environment. [Background technology]

[0002] Generally, cloud computing refers to a computational model in which data processing, data storage, network resources, software, and data are accessible to remote data processing systems, and the details of the underlying information technology (IT) infrastructure providing such resources are transparent to consumers of cloud services. In various implementations, the IT infrastructure can be on-premises or off-premises (or a hybrid of both) with respect to the cloud consumer. Furthermore, cloud computing resources can be (but need not be) widely distributed geographically and / or topologically.

[0003] Cloud computing is facilitated by easy access to remote computing websites (such as the Internet or private corporate networks), often in the form of web-based resources, tools, or applications that cloud customers can access and use through a web browser as if the resources, tools, or applications were local programs installed on the cloud customer's computer system. Commercial cloud implementations are generally expected to meet cloud customers' quality of service (QoS) requirements, which may be specified in a service level agreement (SLA). In a typical cloud implementation, cloud customers consume computational resources as a service and pay only for the resources they use.

[0004] The acceptance of cloud computing has been fueled by the widespread adoption of virtualization, which creates virtual instances (as opposed to actual instances) of computing resources such as operating systems, servers, storage devices, and network resources. For example, a virtual machine (VM), also known as a logical partition (LPAR), is a software implementation of a physical machine (e.g., a computer system) that executes instructions like a physical machine. VMs are sometimes classified as system VMs and process VMs. System VMs provide a complete system platform that supports the execution of an entire operating system (OS), such as Windows, Linux, or Android, and its associated applications. Process VMs, on the other hand, are typically designed to run a single program and support a single process. In either case, application software running on a VM is limited to the resources and abstractions provided by that VM. As a result, the actual resources provided by a typical IT infrastructure can be efficiently managed and utilized through the deployment of multiple VMs, possibly from multiple different cloud computing customers. The virtualization of the actual IT resources and management of the VMs are typically provided by software called a VM monitor (VMM) or hypervisor.

[0005] In a typical virtualized computing environment, VMs and VMMs can communicate with each other and with physical entities in the computing environment's IT infrastructure using traditional input / output (I / O) and networking protocols. As known in the art, traditional networking protocols generally assume the well-known seven-layer Open Systems Interconnection (OSI) model, including (in ascending order) the physical layer, data link, network, transport, session, presentation, and application layers. In some implementations, VMs and VMMs use virtual network connections in place of traditional physical layer connections, enabling the VMs and VMMs to communicate with other network entities as if they were physical network elements. This traditional virtualized I / O and network infrastructure is referred to in the art as a virtual input / output server (VIOS).

[0006] In cloud computing environments such as those described, computational workloads can generally be characterized as including two components: a workload state maintained in the registers and caches of the physical hosts that execute the workload, and a workload dataset residing in the data storage of the physical hosts. These computational workloads are frequently transferred between physical hosts for a variety of reasons, including, for example, data processing system maintenance and upgrades, load balancing, regulatory compliance, security, and resource optimization.

[0007] According to one conventional technique for transferring a running workload from a source host to a destination host, the operating systems or hypervisors of the source and destination hosts first coordinate the copying of the workload dataset, memory page by memory page, from the source host's memory to the destination host's memory via the VIOS. After the workload dataset is successfully transferred, the operating systems or hypervisors of the source and destination hosts coordinate the transfer of the workload state from the source host to the destination host via the VIOS. During the transfer of the workload from the source host to the destination host, the workload continues to run on the source host, so the workload typically continues both reading from and writing to the workload dataset. Using this technique, the time required to migrate the workload can be unpredictably long because each time the source host updates (writes) a memory page that has already been transferred to the destination host, the updated memory page must be transferred a second time.

[0008] In the alternative technique, the operating systems or hypervisors on the source and destination hosts first coordinate the transfer of workload state from the source host to the destination host via the VIOS, begin execution of the workload on the destination host, and then migrate memory pages from the source host to the destination host based on demand paging. Thus, whenever a workload running on the destination host generates a request to access data in a memory page residing on the source host, a software page fault occurs, and the operating system or hypervisor handles the page fault by copying the requested memory page from the source host to the destination host via the VIOS. While this alternative technique transfers memory pages only once from the source host to the destination host, the significant latency required to process each demand paging request results in poor responsiveness throughout the workload migration period.

[0009] All of the conventional workload migration techniques described herein utilize VIOS to transfer workloads between non-coherent hosts and are characterized by high communication overhead and latency due to the many protocol layers that must be traversed to communicate workload data sets and workload states between hosts. Summary of the Invention

[0010] Various embodiments of the present invention support live migration of workloads between non-coherent hosts while avoiding the communication overhead and latency associated with the VIOS (or external network) during workload state and workload data set transfer.

[0011] In at least one embodiment, a data processing system serving a source host to which a logical partition is migrated may include a system fabric, a system memory, a memory controller for the system memory communicatively coupled to the system fabric, and a link controller communicatively coupled to the system fabric and configured to be communicatively coupled to a destination host that is non-coherent with respect to the source host via a communication link. The source host further includes a plurality of processing units communicatively coupled to the system fabric. The plurality of processing units are configured, for example, via a hypervisor or virtual machine monitor, to execute a logical partition having a dataset residing in the system memory and to migrate the logical partition to the destination host via the communication link. Migrating the logical partition includes migrating, via the communication link, a dataset of the logical partition executing on the source host from system memory of the source host to system memory of the destination host. After migrating at least a portion of the dataset, migrating the state of the logical partition from the source host to the destination host via the communication link so that the logical partition executes on the destination host. This migration technique reduces migration latency and improves jitter for the logical partition.

[0012] Aspects of the present invention may also be implemented as a data processing method. In one example, the method may include migrating, via a communications link, a data set of a logical partition executing on a source host from a system memory of the source host to a system memory of a destination host that is non-coherent with respect to the source host. After migrating at least a portion of the data set, a state of the logical partition is migrated from the source host to the destination host via the communications link. After migrating the state of the logical partition to the destination host, the logical partition is executed on the destination host.

[0013] Aspects of the present invention may also be embodied as a program product. In one example, the program product includes a computer-readable storage device and program code stored in the computer-readable storage device and executed by a data processing system functioning as a source host, the program code causing the data processing system to migrate, via a communications link, a dataset of a logical partition executing on the source host from a system memory of the source host to a system memory of a destination host that is non-coherent with respect to the source host. The program code also causes the data processing system to migrate, via the communications link, a state of the logical partition from the source host to the destination host such that, after migrating at least a portion of the dataset, the logical partition executes on the destination host.

[0014] In some examples or operating scenarios, migrating the dataset includes migrating at least a portion of the dataset of the logical partition from the source host to the destination host after the logical partition begins execution on the destination host.

[0015] In some examples or operating scenarios, migrating a data set includes the source host copying one or more memory pages of the data set to a system memory of the destination host based on a request of the destination host.

[0016] In some examples or operating scenarios, migrating a data set involves a source host initiating a copy of one or more memory pages of the data set to a system memory of a destination host.

[0017] In some examples or operating scenarios, the data set includes multiple memory pages each having a respective associated page table entry that defines the virtual-to-real address translation for the associated memory page, and the source host tracks in a data structure which page table entries of the multiple memory pages have been migrated to the destination host.

[0018] In some examples or operating scenarios, migrating a data set includes a source host issuing a store request on a system fabric that specifies a real address in a system memory of a destination host, and a link controller in the source host receiving the store request and transmitting the store request over a communication link to a memory controller in the destination host based on the real address.

[0019] In some examples, the method of claim 1 includes migrating the dataset by the source host flushing all memory pages in the dataset that are subject to update by the source host from the source host's cache to system memory.

[0020] In some examples, the source host utilizes communication over a network connection, such as a Virtual Input Output Server (VIOS), to coordinate the migration of logical partitions from the source host to the destination host, but refrains from migrating the dataset and logical partition state over the network connection. [Brief explanation of the drawings]

[0021] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1] FIG. 1 illustrates a high-level block diagram of an exemplary host data processing system, according to one embodiment. [Figure 2] FIG. 2 is a more detailed block diagram of an exemplary processing unit of a host data processing system, according to one embodiment. [Figure 3] 1 illustrates an exemplary data processing system environment in which multiple hosts are coupled by non-coherent communication links, according to one embodiment. [Figure 4] 1 illustrates a logical view of a hypervisor and logical partitions of an exemplary host data processing system, according to one embodiment. [Figure 5] 1 illustrates an exemplary page table entry (PTE), according to one embodiment. [Figure 6] 3 illustrates various memory access requests of an exemplary host data processing system, according to one embodiment. [Figure 7] 1 illustrates exemplary write primitives that two host data processing systems can communicate, according to one embodiment. [Figure 8] 1 illustrates an exemplary read primitive by which two host data processing systems can communicate, according to one embodiment. [Figure 9] 8 is a high-level logical flowchart of an exemplary store operation in the write primitive of FIG. 7, according to one embodiment. [Figure 10] 9 is a high-level logical flowchart of an exemplary load operation in the read primitive of FIG. 8, according to one embodiment. [Figure 11A] 11A and 11B together form a high-level logical flowchart of an exemplary method for a source host to migrate a logical partition to a destination host, according to one embodiment. [Figure 11B] 11B together form a high-level logical flowchart of an exemplary method for a source host to migrate a logical partition to a destination host, according to one embodiment. [Figure 12] 1 is a high-level logical flowchart of an exemplary method for a source host to handle page protection updates during logical partition migration, according to one embodiment. [Figure 13]1 depicts a high-level logical flowchart of an exemplary method for a source host to copy memory pages of a logical partition to a destination host, according to one embodiment. [Figure 14] 1 depicts a high-level logical flowchart of an exemplary method for a destination host to receive and activate a migrating logical partition, according to one embodiment. [Figure 15] 4 is a high-level logical flowchart of an exemplary method by which a destination host handles page faults for a migrating logical partition, according to one embodiment. [Figure 16] 4 depicts a high-level logical flowchart of an exemplary method by which a destination host "pulls" memory pages of a migrating logical partition from a source host, according to one embodiment. [Figure 17] 4 depicts a high-level logical flowchart of an exemplary method by which a source host processes a "pull" request for memory pages of a migrating logical partition from a destination host, according to one embodiment. [Figure 18] 1 is a high-level logical flowchart of an exemplary method by which a source host "pushes" memory pages of a migrating logical partition to a destination host, according to one embodiment. [Figure 19] 4 depicts a high-level logical flowchart of an exemplary method by which a destination host processes a "push" request for memory pages of a migrating logical partition from a source host, according to one embodiment. [Figure 20] FIG. 1 is a data flow diagram illustrating a design process, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] Referring now to the figures, wherein like reference numerals refer to like and corresponding parts throughout, and with particular reference to FIG. 1 , there is depicted a high-level block diagram depicting an exemplary data processing system 100 in accordance with one embodiment. In various use cases and topologies, data processing systems such as data processing system 100, which include hardware components and may additionally include software and / or firmware components, are sometimes referred to in the art as “hosts” or “host data processing systems.” In various operating scenarios, any given host may be a source host from which a workload (e.g., a logical partition) is migrated, or a destination host to which a workload is migrated.

[0023] In the depicted embodiment, host data processing system 100 is a cache-coherent multiprocessor (MP) data processing system that includes multiple processing nodes 102 for processing data and instructions. The processing nodes 102 are coupled to a system interconnect 110 for communicating address, data, and control information. System interconnect 110 may be implemented, for example, as a bused interconnect, a switched interconnect, or a hybrid interconnect.

[0024] In the depicted embodiment, each processing node 102 is implemented as a multi-chip module (MCM) that includes one or more (e.g., four) processing units 104a-104d, each preferably implemented as a respective integrated circuit. The processing units 104 in each processing node 102 are communicatively coupled to each other and to the system interconnect 110 by a local interconnect 114, which may be implemented, for example, by one or more buses and / or switches, as well as the system interconnect 110. The system interconnect 110 and the local interconnect 114 together form a system fabric. In at least some preferred embodiments, communications on the system fabric conform to a so-called host bus protocol, which defines, among other things, a predetermined set of legal requests, responses, and control information communicated between entities (e.g., caches, memory controllers, etc.) communicating over the system fabric.

[0025] 2, in some embodiments, one or more of the processing units 104 (and possibly all of the processing units 104) each include a memory controller 106 coupled to a local interconnect 114 to provide an interface to a respective system memory 108. Data and instructions residing in the system memory 108 may generally be accessed, cached, and modified by processor cores within any processing unit 104 of any processing node 102 in the data processing system 100. In other embodiments, one or more memory controllers 106 (and system memory 108) may be directly or indirectly coupled (e.g., via a switch) to the system interconnect 110 rather than to the local interconnect 114.

[0026] Those skilled in the art will appreciate that data processing system 100 of FIG. 1 may include many additional, not-shown, components, such as interconnect bridges, non-volatile storage, ports for connecting to networks or attached devices, etc. Such additional components are not necessary for an understanding of the described embodiments and are therefore not shown in FIG. 1 or discussed further herein. However, it should also be understood that the supplemental information described herein is applicable to data processing systems of a variety of architectures and is in no way limited to the generalized data processing system architecture illustrated in FIG. 1.

[0027] 2, a more detailed block diagram of an exemplary processing unit 104 and system memory 108 is depicted, according to one embodiment. In the depicted embodiment, each processing unit 104 is an integrated circuit that includes one or more processor cores 200 for processing instructions and data. In the depicted example, processor core 200 includes one or more execution units 202 that can execute instructions from one or more concurrently executing hardware threads.

[0028] Processor core 200 further includes a memory management unit (MMU) 204 responsible for translating effective addresses determined by execution of memory-referencing instructions in execution unit 202 into real addresses within the real address space referenced by all processing units 104 in data processing system 100. MMU 204 performs effective-to-real address translations by reference to one or more translation structures 206, such as a translation lookaside buffer (TLB), an effective-to-real address translation (ERAT) cache, or a segment lookaside buffer (SLB). The number and / or type of these address translation structures may vary between implementations and architectures. Address translation structures 206 reduce latency associated with address translations by buffering local copies of selected address translations, which may be retrieved from system memory 108, as discussed further below.

[0029] The operation of each processor core 200 is supported by a multi-level memory hierarchy with a composite system memory at the bottom provided by various system memories 108 and made accessible via memory controllers 106. The real address ranges for which each memory controller 106 is responsible can be defined, for example, by a hypervisor and / or operating system software through appropriate configuration of one or more base address registers (BARs) 216 within the memory controller 106. As shown, the system memory 108 stores multiple memory pages 209 that, among other things, provide storage for various workload data sets (also referred to as "logical partitions" (LPARs)). Additionally, one or more system memories 108 store a page frame table 210 that includes multiple page table entries (PTEs) 212, each PTE 212 specifying effective-to-real address translations for a respective corresponding memory page 209 residing in one of the system memories 108. The PTEs 212 further specify access protection (e.g., read-only, read / write (R / W), etc.) for the different memory pages. PTEs 212 accessed by MMU 204 from page frame table 210 may be cached by MMU 204 for subsequent access, for example, in address translation structure 206. Page frame table 210 may be established, maintained, and updated, for example, by operating system and / or hypervisor software executing within data processing system 100.

[0030] The multi-level memory hierarchy of each processor core 200 further includes one or more levels of cache memory, which in the illustrated embodiment includes a private store-through level 1 (L1) cache 208 within each processor core 200 and a respective store-in level 2 (L2) cache 230 for each processor core 200. While the illustrated cache hierarchy includes only two levels of cache, those skilled in the art will appreciate that other embodiments may include additional levels (e.g., L3, L4) of on-chip or off-chip, private or shared, in-line or look-aside cache, which may fully, partially, or not encompass the contents of the higher levels of cache.

[0031] In the depicted embodiment, each processing unit 104 further includes an integrated and distributed fabric controller 214 that is responsible for controlling the flow of operations on the system fabric in accordance with the host bus protocol and for implementing the coherency communications necessary to implement the desired cache coherency protocol. Processing unit 104 may further include an integrated I / O (input / output) controller 218 that supports the attachment of one or more I / O devices and / or I / O channels (not shown).

[0032] In the depicted example, processing unit 104 also includes an attached non-coherent (NC) link controller 220 that, in at least one operating mode, supports the attachment of other host data processing systems 100 to host data processing system 100 via non-coherent communication links. For example, Figure 3 shows an exemplary data processing system environment 300 in which host data processing system 100a and host data processing system 100b are coupled together for communication by non-coherent communication links 302 (data processing system environment 300 may optionally include one or more additional hosts 100 coupled to hosts 100a and / or 100b by additional non-coherent communication links 302). In such an arrangement, each of host data processing systems 100a, 100b maintains its own respective real address space that is non-coherent with the other hosts. The real address range for which the NC link controller 220 in each host 100a, 100b is responsible can be defined by the hypervisor and / or operating system software through appropriate configuration of, for example, one or more base address registers (BARs) 224 in the NC link controller 220. Once configured, memory access requests transmitted over the system fabric of a host 100 that specify a real address within the real address range defined by the BAR 216 of a local memory controller 106 can be serviced by that memory controller 106 by reference to the associated system memory 108. Memory access requests over the system fabric that specify a real address within the real address range defined by the BAR 224 of an NC link controller 220 are transmitted by the NC link controller 220 over the NC communication link 302 to the other host 100a or 100b. Inbound memory access requests received by the NC link controller 220 over the non-coherent communication link 302 are transmitted by the NC link controller 220 over the system fabric of the receiving host 100.

[0033] 2 , processing unit 104 further includes a nest memory management unit (NMMU) 228, which provides address translation to other communicating entities, such as NC link controller 220, upon request via the system fabric. It should be understood that in other embodiments, NMMU 228 may be communicatively coupled to provide address translation to communicating entities, including NC link controller 220, in an alternative or additional manner, for example, by being coupled to system interconnect 110 rather than local interconnect 114.

[0034] 4 illustrates a logical view of the software configuration of host 100a or 100b of FIGS. 1-3 according to one embodiment. In this example, the software configuration 400 of each host 100 includes a hypervisor (or VMM) 402 that manages the hardware and software resources of host 100. Hypervisor 402 virtualizes many of the underlying resources of host 100 through the implementation of one or more logical partitions (LPARs) 404. In this example, each LPAR 404 includes a designated number of hardware threads 406 within one or more processor cores 200 of host 100, a collection of memory pages 408 (from memory pages 209) at designated real memory addresses for storing the LPAR's 404 dataset, an instance of an operating system 410 (e.g., Linux, Windows, Android, iOS, etc.), and a collection of applications 412 that run on OS 410. The hypervisor 402 further supports network connections between LPARs 404 and between the hypervisor 402 and other hosts 100 via a virtual input / output server (VIOS) 420 that virtualizes the physical network connections to which the hosts 100 are coupled via the I / O controllers 218.

[0035] As described above, hypervisor 402 may determine, for any of a variety of reasons, that it is desirable or necessary to migrate a live workload, e.g., one of LPARs 404, from its own host 100 to another host 100. In accordance with embodiments described herein, the migration preferably copies the dataset and state of the migrating LPAR 404 from the host 100 on which the LPAR 404 was originally running (referred to herein as the “source host”) to the other host 100 on which the LPAR 404 will continue its execution (referred to herein as the “destination host”) using NC communication 302 rather than a virtual network connection supported by VIOS 420 (as is conventional). Employing NC communication link 302 rather than VIOS 420 for migrating the LPAR dataset has the advantage of reducing or eliminating duplicate transmissions of memory pages that make up the LPAR dataset, thereby accelerating the LPAR migration. Employing NC communication link 302 also has the advantage of providing predictable response times for the migrating LPAR's applications 412.

[0036] 5, an exemplary page table entry (PTE) 212 is illustrated according to one embodiment. In this example, the PTE 212 includes a valid field 502 that indicates whether the contents of the PTE 212 are valid and therefore usable in translating virtual addresses to real addresses in an associated one of the memory pages 209. Additionally, the PTE 212 includes a virtual address (VA) field 504 and a real address (RA) field 506 that specify the virtual address and real address, respectively, assigned to the base storage location of the associated memory page 209. The PTE 212 also includes one or more page protection fields that indicate one or more attributes of the associated memory page 209. For example, in the illustrated embodiment, the page protection fields include a read (R) field 508 and a read / write (RW) field 510, respectively, that alternatively indicate whether the associated memory page 209 is read-only or is subject to both read and write accesses. Hereinafter, a field value of R=1, RW=0 indicates that the associated memory page 209 is a read-only memory page, and a field value of R=0, RW=1 indicates that the associated memory page 209 is a read-write memory page.

[0037] In the illustrated embodiment, PTE 212 additionally includes one or more migration-related fields that may be utilized by hypervisor 402 to manage the migration of the associated memory page 209 from system memory 108 of the source host 100 to system memory 108 of the destination host 100. In this embodiment, these migration-related fields include a copy (CP) field 512 that indicates whether the associated memory page 209 has been copied from the source host 100 to the destination host 100, a clean (CL) field 514 that indicates whether all data within the associated memory page 209 has been cleaned (i.e., all modified data has been written from cache memory to system memory 108 so that the image of the memory page in system memory 108 matches any cached data), and a migrate (MI) field 516 that indicates whether PTE 212 has been migrated (i.e., is present in system memory 108 of the destination host 100) but has not been authenticated. It should be understood that the implementation of migration-related fields 512-516 in PTE 212 is a design choice, and that other embodiments may omit these fields from PTE 212 and instead utilize one or more other data structures to manage the migration of LPAR datasets.

[0038] Referring now to Figure 6, various memory access requests of an exemplary host data processing system 100 are depicted. In Figure 6, a core 200 of the host data processing system 100 (e.g., host data processing system 100a) executes user-level instructions of, for example, an application 412 in a first hardware thread and executes instructions of a hypervisor 402 in a second hardware thread. Execution of the user-level instructions in the first hardware thread generates memory access requests 602, each of which specifies a respective target virtual (logical) address (e.g., virtual address 610 or 612) to be accessed in a virtual address space 606. As described above, the MMU 204 of the core 200 translates each of the target virtual addresses 610, 612 to a respective real address 614 or 616 in the real address space 608 based on the address translation provided by the PTE 212. In this example, the real address 614 identifies a storage location in the system memory 108 of the host data processing system 100a for which the local memory controller 106 is responsible. In contrast, real address 616 identifies a storage location within system memory 108 of host data processing system 100b for which NC link controller 220 is responsible. Thus, a memory access request transmitted over the system fabric of host data processing system 100a may initiate an access to a storage location within local system memory 108 or a storage location within remote system memory 108 of another host data processing system 100b (non-coherent with host data processing system 100a), depending on the real address assignment by hypervisor 402 of host data processing systems 100a, 100b.

[0039] In at least some embodiments, memory access requests of hypervisor 402 need not undergo the same address translation applied to the target addresses of user-level memory access requests 602. For example, FIG. 6 further illustrates that an instruction of hypervisor 402 executing in a second hardware thread of core 200 may generate memory access request 604. In this case, however, memory access request 604 directly specifies real addresses 614, 616 in real address space 608. As a result, hypervisor 402 of host data processing system 100a can access storage locations in local system memory 108 of host data processing system 100a and in remote system memory 108 of host data processing system 100b without using the virtual-to-real address translation recorded in PTE 212.

[0040] 7, an exemplary write primitive 700 is illustrated that allows one of the host data processing systems 100a, 100b to write data to the system memory 108 of the other host 100a or 100b over the non-coherent communication link 302, according to one embodiment. The write primitive 700 may form part of an instruction sequence for the hypervisor 402, the OS 410, or the application 412.

[0041] As shown, write primitive 700 includes a first store operation 702 that stores data granules in memory. FIG. 9 provides a high-level logical flowchart of an exemplary embodiment of this store operation. In this exemplary embodiment, the store operation shown in FIG. 9 begins at block 900 and then proceeds to block 902, which depicts core 200 of host 100 (hereinafter assumed to be host 100a) executing a store instruction that specifies storing data granules in memory. After applicable virtual-to-real address translation, core 200 stores the data granules in its cache hierarchy (e.g., in its L2 cache 230) in association with a target real address assigned to system memory 108 in another host 100 (i.e., host 100b). Because hosts 100a, 100b are non-coherent and NC communication link 302 does not transmit coherence messaging between hosts 100a, 100b, simply storing the data granule in the cache memory of host 100a is insufficient to make the association between the data granule and the target real address visible to the hardware thread of host 100b. Therefore, the store operation further includes core 200 executing a cache clean instruction to force a copy of the data granule from the cache hierarchy of core 200 to the associated system memory 108 of host 100b (block 904). The process of FIG. 9 then terminates at block 906.

[0042] 7, following the program-order store operation 702, the write primitive 700 includes a barrier operation 704 that commands the execution of all operations preceding the program-order barrier operation 704 prior to the execution of all operations following the program-order barrier operation 704. Following the barrier operation 704 in program order, the write primitive 700 includes a second store operation 706. The store operation 706 updates (e.g., to a value of 1) a flag variable (which may be polled by a process executing on host 100b) present in the system memory 108 of either host 100a or host 100b to indicate that a new data granule has been written to the system memory 108 of host 100b. If the flag is present on host 100b, the store operation 706 may be executed as described above with reference to FIG. 9.

[0043] 8, an exemplary read primitive 800 is depicted that allows one of the host data processing systems 100a, 100b to read data from the system memory 108 of the other host 100a, 100b over the non-coherent communication link 302, according to one embodiment. The read primitive 800 may form part of an instruction sequence for the hypervisor 402, the OS 410, or the application 412.

[0044] As shown, read primitive 800 includes a first load operation 802 that loads a flag from memory. FIG. 10 provides a high-level logical flowchart of an exemplary embodiment of this load operation. In this embodiment, the load operation begins at block 1000 and then proceeds to block 1002, which depicts core 200 of host 100 (hereinafter assumed to be host 100a) executing a flush instruction specifying that any cached data in host 100a associated with the real address of the data granule (e.g., the flag) being loaded be removed from the cache of host 100a and written back to the associated memory controller 106 of host 100b. The load operation then further includes core 200 executing a load instruction that loads the data granule from the associated system memory 108 of host 100b (block 1004). The processing of FIG. 10 then terminates at block 1006.

[0045] 8, following load operation 802, load primitive 800 includes a compare operation 804 that compares the value of the flag loaded from host 100b with a predetermined value (e.g., 1) to determine whether the granule of load data is ready to be read. If compare operation 804 determines that the value of the flag indicates that the granule of load data is ready to be read, conditional branch operation 806 causes execution of load primitive 800 to continue with load operation 810. Load operation 810, which loads the granule of data from system memory 108 of host 100b, may be performed as described above with reference to FIG. 10. If compare operation 804 determines that the value of the flag indicates that the granule of load data is not ready to be read, branch operation 808 causes execution of load primitive 800 to branch back to load operation 802, previously described.

[0046] Those skilled in the art will appreciate that the semaphore communication described above with reference to Figures 7-10 is only one of several alternative techniques for supporting read and write communications between host data processing systems that are non-coherent with respect to one another.

[0047] 11A-11B, a high-level logical flowchart of an exemplary method for a source host 100 to migrate an LPAR 404 from the source host 100 to a destination host 100 is depicted, according to one embodiment. Processing begins at block 1100 and proceeds to block 1102, which illustrates the hypervisors 402 of the source and destination hosts 100 (hereafter referred to as hosts 100a and 100b, respectively) communicating with each other to negotiate the allocation of resources (e.g., hardware threads 406, real memory addresses, memory pages 209, etc.) of the hosts 100a, 100b to the LPAR 404 and to establish specific memory pages 209 to use for communication between the hosts 100a, 100b. In at least some embodiments, the hypervisors 402 of the source host 100a and the destination host 100b communicate at block 1102 via the VIOS 420.

[0048] Processing proceeds from block 1102 to block 1104, which illustrates the hypervisor 402 of the source host 100a determining whether it has completed copying all of the memory pages 408 comprising the data set of the migrating LPAR 404 from host 100a to host 100b's system memory 108. If so, processing proceeds to block 1112, described below. However, if the hypervisor 402 of the source host 100a determines in block 1104 that it has not completed copying the memory pages 408 comprising the data set of the migrating LPAR 404 from the source host 100a to the destination host 100b, processing continues at block 1106. Block 1106 illustrates the hypervisor 402 of the source host 100a determining whether to initiate a state migration of the migrating LPAR 404 from the source host 100a to the destination host 100b. It should be noted that hypervisor 402 may determine to migrate the state of migrating LPAR 404 before all or a portion of the data set is migrated, or may wait to migrate the state of migrating LPAR 404 until all of the data set of migrating LPAR 404 has been copied to destination host 100b. As an example, hypervisor 402 may determine at block 1106 to migrate the state of migrating LPAR 404 after all memory pages 408 in a subset of memory pages 408 that are the most frequently (or most recently) accessed memory pages 408 have been migrated. In response to a positive determination at block 1106, processing proceeds from block 1106 to block 1112. In response to a negative determination at block 1106, processing proceeds to block 1108.

[0049] In block 1108, the hypervisor 402 of the source host 100a copies one or more memory pages 408 in the dataset of the migrating LPAR 404 to at least one system memory 108 of the destination host 100b. In at least some embodiments, the copying of the memory pages 408 in block 1108 may be performed according to the process described below with reference to FIG. 13. It should be understood that copying the memory pages 408 may require the execution of multiple store primitives to a contiguous sequence of real addresses. For example, the memory pages 408 may be 4 KB, while the size of each data granule copied by the store primitive may be 128 B. In this example, copying the memory pages 408 requires the execution of 32 store primitives. For each memory page 408 that is completely copied from the source host 100a to the destination host 100b, the hypervisor 402 of the source host 100a sets the CP field 512 of the associated PTE 212 in the system memory 108 of the host 100a to indicate that the memory page 408 has been copied to the destination host 100b (block 1110). Processing of Figure 11A returns from block 1110 to block 1104.

[0050] At block 1112 and following blocks, the hypervisor 402 of the source host 100a copies the state of the migrating LPAR 404 from the source host 100a to the destination host 100b. With specific reference to block 1112, the hypervisor 402 of the source host 100a places the hardware threads 406 of the migrating LPAR 404 under its control. Additionally, the hypervisor 402 saves the state of each hardware thread 406 of the migrating LPAR 404 (e.g., the contents of data and control registers associated with the hardware thread 406 in the execution cores 200) and writes each such hardware thread state of the migrating LPAR 404 to the system memory 108 in the destination host 100b. The copying of the state of the hardware threads of the migrating LPAR 404, as illustrated in block 1114, may be performed via the NC communication link 302 utilizing the write primitive 700, as described above with reference to FIGS. 7 and 9. At block 1116, the hypervisor 402 of the source host 100a also updates each PTE 212 associated with a memory page 408 in the dataset of the migrating LPAR 404 by invalidating the PTE 212 (e.g., resetting the valid field 502 to 0) and setting the migration (MI) field 516 (e.g., to 1). Block 1116 further indicates that the hypervisor 402 invalidates any entries in the address translation structure 206 of the source host 100a that correspond to the invalidated PTE 212 and waits for any in-flight operations in the source host 100a that reference real addresses translated by the invalidated PTEs to be drained from the cores 200 of the source host 100a. Processing then passes through page connector A to block 1120 of FIG. 11B.

[0051] As indicated by the dashed line, block 1120 is an optional step in which the hypervisor 402 of the source host 100a determines whether to flush the read-write memory pages 408 of the migrating LPAR 404 (i.e., those identified as read-write memory pages by the RW fields 510 of their associated PTEs 212) from the cache of the source host 100a. In response to a negative determination at block 1120, or if the source host 100a does not support the ability to efficiently flush the cache of the source host 100a, processing proceeds to block 1126, described below. However, if the hypervisor 402 determines at block 1120 to flush the read-write memory pages 408 of the migrating LPAR 404 from the cache of the source host 100a, the hypervisor 402 performs a procedure to flush all modified data belonging to the dataset of the migrating LPAR 404 from the cache of the source host 100a (block 1122). In some embodiments, the eviction depicted at block 1122 may be performed by a bulk flush of all modified data from the cache of the source host 100a. Note that, depending on whether the underlying memory pages 408 have been migrated, the flushing operation depicted at block 1122 may involve writing the modified data back to the local system memory 108 of the source host 100a or to the system memory 108 of the destination host 100b. Further, at block 1124, the hypervisor 402 of the source host 100a updates the PTEs 212 on the source host 100a as cleaning (e.g., by setting the CL field 514 to 1) for all read-write memory pages 408 of the migrating LPAR 404 that have already been migrated to the destination host 100b. Performing a bulk flush of the read-write memory pages 408 of the logical partition 404 as shown in blocks 1120-1124 has the advantage of reducing or eliminating duplicate copies of the read-write memory pages 408 and reducing jitter after the migrating LPAR 404 begins execution on the destination host 100b.Additionally, flushing the read-write memory pages 408 of a migrating LPAR 404 en masse may provide improved performance over solutions that handle the flushing of each read-write memory page 408 of a migrating LPAR 404 individually.

[0052] Referring now to block 1126, the hypervisor 402 of the source host 100a copies PTEs 212 that translate addresses within memory pages 408 of the data set of the migrating LPAR 404 from the source host 100a to the destination host 100b. The hypervisor 402 then resets the MI field 516 in the PTE 212 of each read-write memory page in the data set of the migrating LPAR 404 identified by PTE fields 512-514 as both copy and clean, and resets the MI field 516 in the PTE 212 of each read-only memory page in the data set of the migrating LPAR 404 identified by PTE field 512 as copy (block 1128). Following block 1128, the hypervisor 402 of the source host 100a begins monitoring for a "pull" request from the hypervisor 402 of the destination host 100b by which the destination host 100b initiates a migration of one or more previously unmigrated memory pages 408 in the dataset of the LPAR 404 being migrated (block 1130). An exemplary process by which the hypervisor 402 of the destination host 100b "pull" memory pages in the dataset of the LPAR 404 being migrated from the source host 100a to the system memory 108 of the destination host 100b, and the hypervisor 402 of the source host 100a responds to the "pull" request of the destination host 100b, is described below with reference to Figures 16 and 17, respectively.

[0053] As indicated by optional block 1132, the hypervisor 402 of the source host 100a may alternatively or additionally initiate a process in which the source host 100a “pushes” any remaining unmigrated and migrated-but-not-clean memory pages 408 in the dataset of the LPAR 404 being migrated from the source host 100a to the system memory 108 of the destination host 100b. Exemplary processes in which the hypervisor 402 of the source host 100a “pushes” the memory pages 408 in the dataset of the LPAR 404 being migrated from the source host 100a to the system memory 108 of the destination host 100b, and in which the hypervisor 402 of the destination host 100b responds to the source host 100a’s “push” request, are described below with reference to Figures 18 and 19, respectively. Following block 1132 (or following block 1130 if block 1132 is omitted), the process of Figure 11B ends at block 1134.

[0054] 12, a high-level logical flowchart of an exemplary method for source host 100a to process page protection updates during migration of LPAR 404 is depicted, according to one embodiment. The process of FIG. 12 begins at block 1200 and then proceeds to block 1202, which illustrates hypervisor 402 of source host 100a determining whether page protection updates to PTEs 212 of memory pages 408 of the migrating LPAR 404 are requested. If not, the process repeats block 1202. However, if hypervisor 402 determines that page protection updates to PTEs 212 of memory pages 408 of the migrating LPAR 404 are requested, the process proceeds to block 1204.

[0055] Block 1204 illustrates hypervisor 402 determining whether the PTE 212 to be updated currently indicates that the associated memory page 408 is a read-write page (e.g., RW field 510 is set with RW=1). If not (e.g., R field 508 is set with R=1 to indicate a read-only memory page), processing proceeds to block 1212, which illustrates hypervisor 402 performing other processing. Processing of FIG. 12 then ends at block 1214.

[0056] Returning to block 1204, in response to the hypervisor 402 determining that the to-be-updated PTE 212 currently indicates that the associated memory page 408 is a read-write memory page, the hypervisor 402 additionally determines whether the associated memory page 408 has already been migrated to the system memory 108 of the destination host 100b by referencing the CP field 512 of the to-be-updated PTE 212. If so, the hypervisor 402 executes one or more instructions that cause any modified data for the memory page 408 to be flushed from the cache of the source host 100a to the system memory 108 of the destination host 100b (block 1208), and then updates the page protection information of the memory page 408 from read-write to read-only, for example, by resetting the R field 508 and setting the RW field 510 (block 1210). By flushing any cached modifications to the date of the memory page 408 for which protection is to be updated, an additional copy of the memory page 408 from the source host 100a to the destination host 100b may be avoided. If the hypervisor 402 determines in block 1206 that the associated memory page 408 has not been migrated to the system memory 108 of the destination host, the hypervisor 402 simply updates the page protection information as shown in block 1210 without flushing the memory page 408. Following block 1210, the processing of FIG. 12 ends in block 1214.

[0057] 13, a high-level logical flowchart of an exemplary method by which a source host 100a copies memory pages 408 of a migrating LPAR 404 to a system memory 108 of a destination host 100b is depicted, according to one embodiment. In at least some embodiments, the depicted process may be utilized to copy, for example, one of the memory pages 408 copied from the source host 100a to the destination host 100b in block 1108 of FIG. 11A.

[0058] 13 begins at block 1300 and then proceeds to block 1302, which illustrates the hypervisor 402 of the source host 100a invalidating the PTE 212 associated with the memory page 408 to be migrated, for example, by resetting the valid field 502 to 0. Invalidating the PTE 212 makes the PTE 212 temporarily inaccessible for address translation, thus making the associated memory page 408 accessible only to the hypervisor 402. Block 1302 further illustrates the hypervisor 402 invalidating any entries in the address translation structure 206 of the source host 100a that correspond to the invalidated PTE 212 and waiting for any in-flight operations that reference real addresses translated by the invalidated PTE to be drained from the core 200 of the source host 100a.

[0059] The hypervisor 402 then causes one or more cores 200 of the source host 100a to execute instructions that cause the memory page 408 to be copied to a real address assigned to the system memory 108 of the destination host 100b (block 1304). As part of this copy process, the hypervisor 402 loads an existing image of the memory page 408 from the system memory 108 of the source host 100a (which resides in the source real address range) and stores the image of the memory page 408 in a destination real address range that specifies a storage location within the system memory 108 of the destination host 100b. As an optimization, some embodiments allocate cache lines corresponding to the destination real address range in the cache memory of the source host 100a using a Data Cache Block Zero (DCBZ), Data Cache Block Allocate (DCBA), or similar cache line allocation instruction, which creates a cache line in the cache memory of the source host 100a (and optionally initializes the newly created cache line to a predetermined value (e.g., zero)) without fetching any associated data. These cache line allocation instructions may be conveniently implemented in an execution loop that iterates through each cache line in the migrating memory page 408. In such an execution loop, each cache line allocation instruction is preferably constrained to execute before and closely in time with a corresponding store instruction that stores data from the existing image of the migrating memory page 408 into the newly allocated cache line. This execution constraint increases the likelihood that a store instruction that updates a cache line in the destination real address range will avoid a cache miss and the attendant latency of fetching the cache line from the destination host 100b. In other words, the described optimization has the advantage of avoiding the high latency of fetching a copy of each cache line of data in the destination real address range from the destination host 100b only to overwrite the cache line with data from the migrating memory page 408.

[0060] At block 1308, the hypervisor 402 determines whether the memory page 408 is a read-write memory page, for example, by examining the RW field 510 of the associated PTE 212. In response to determining at block 1308 that the memory page 408 is a read-write memory page, processing proceeds to block 1312, described below. However, if the hypervisor 402 determines that the memory page 408 is a read-only memory page (e.g., the R field 508 is set to R=1), the hypervisor 402 causes one or more clean or flush instructions to be executed by one or more cores 200 of the source host 100a such that data in the read-only memory page 408 is written from the cache of the source host 100a to the system memory 108 of the destination host 100b via the NC communication link 302 (block 1310). Following block 1308 or block 1310, the hypervisor 402 of the source host 100a updates the real address (RA) specified for the memory page 408 in the RA field 506 of the PTE 212 to specify the new real address for the memory page 408 in the system memory 108 of the destination host 100b, and then updates the valid field 502 of the PTE 212 on the source host 100a to a valid state (e.g., V=1) to make the PTE 212 accessible again for address translation (block 1312). The process of FIG. 13 then ends at block 1314.

[0061] 14, a high-level logical flowchart of an exemplary method for destination host 100b to receive and activate migrating LPAR 404 is depicted, according to one embodiment. The process of FIG. 14 cooperates with the previously described process of FIGS. 11A-11B, which is executed on source host 100a.

[0062] 14 begins at block 1400 and then proceeds to block 1402, which depicts the hypervisor 402 of the destination host 100b communicating with the hypervisor 402 executing on the source host 100a to negotiate the allocation of resources (e.g., hardware threads 406, real memory addresses, memory pages 209, etc.) of the hosts 100a, 100b to the LPARs 404 and to establish specific memory pages 209 for use in communications between the hosts 100a, 100b. In at least some embodiments, the hypervisors 402 of the source host 100a and the destination host 100b communicate via the VIOS 420 at block 1402.

[0063] Processing proceeds from block 1402 to block 1404, which illustrates the hypervisor 402 of the destination host 100b receiving and buffering the state of the migrating LPAR 404 from the source host 100a via the NC communication link 302. The LPAR state was sent by the source host 100a at block 1114 of FIG. 11A. Further, at block 1406, the hypervisor 402 of the destination host 100b receives and buffers PTEs 212 for the memory pages 408 of the migrating LPAR 404 from the source host 100a via the NC communication link 302. These PTEs 212 are sent by the source host 100a at block 1126 of FIG. 11B.

[0064] Block 1408 depicts the hypervisor 402 of the destination host 100b establishing a PTE 212 associated with the memory page 408 of the migrating LPAR 404 in the page frame table 210 in the system memory 108 of the destination host 100b. For each read-only memory page 408 indicated by its associated PTE 212 as already copied to destination host 100b (e.g., R field 508 is set to R=1 and CP field 512 is set to CP=1), and for each read-write memory page 408 indicated by its associated PTE 212 as copied to destination host 100b and cleaned (e.g., RW field 510 is set to RW=1 and CL field 514 is set to CL=1), the hypervisor 402 of destination host 100b updates the real address field 506 to reflect the storage location of the memory page 408 within system memory 108 of destination host 100b, resets the migration field 516 (e.g., MI=0), and sets the valid field 502 (e.g., V=1). The hypervisor 402 of destination host 100b makes these PTEs 212 available for virtual-to-real address translation on destination host 100b.

[0065] At block 1410, the hypervisor 402 of the destination host 100b initiates a process to "pull" any remaining, unmigrated memory pages 408 of the migrating LPAR 404 from the source host 100a to the system memory 108 of the destination host 100b via the NC communication link 302. An exemplary process by which the destination host 100b pulls such memory pages 408 from the source host 100a is described below with reference to FIG. 16. Further, at block 1412, the hypervisor 402 of the destination host 100b selectively initiates a process by which the hypervisor 402 of the destination host 100b receives, via the NC communication link 302, any remaining, unmigrated memory pages 408 of the migrating LPAR 404 that were "pushed" from the source host 100a to the system memory 108 of the destination host 100b. An exemplary process by which the destination host 100b receives such memory pages 408 from the source host 100a is described below with reference to Figure 18. The hypervisor 402 of the destination host 100b begins execution of the migrating LPAR 404 on the destination host 100b by loading the state of the various hardware threads 406 of the migrating LPAR 404 onto one or more cores 200 of the destination host 100b (block 1414) and releasing the hardware threads 406 of the migrating LPAR 404 to begin execution (block 1416). The process of Figure 14 then ends at block 1416.

[0066] Referring now to FIG. 15, a high-level logical flowchart of an exemplary method for destination host 100b to handle a page fault during migration of a migrating LPAR 404 is depicted, according to one embodiment. The processing of FIG. 15 begins at block 1500 in response to a memory access request from destination host 100b that generates a page fault. In response to the page fault, processing proceeds from block 1500 to block 1502, which illustrates a page fault handler routine executing on destination host 100b that references PTE 212 of memory page 209 containing the target real address of the memory access request to determine whether the migration field 516 is set (e.g., MI=1), indicating that memory page 209 is one of memory pages 408 of migrating LPAR 404. If not (e.g., MI=0), the page fault handler processes the page fault using other, potentially conventional, processing, as depicted at block 1504. Following block 1504, the processing of FIG. 15 terminates at block 1518.

[0067] Referring again to block 1502, in response to determining that the migration field 516 is set, the hypervisor 402 of the destination host 100b acquires a lock on the memory page 408 on which the page fault occurred (block 1506). The hypervisor 402 of the destination host 100b then determines, in block 1508, whether the migration field 516 of the PTE 212 of the memory page 408 was reset (e.g., MI=0) during the process of acquiring the lock. If so, processing proceeds to block 1516, described below. However, if the migration field 516 was not reset, the hypervisor 402 issues one or more memory access requests to the source host 100a via the NC communication link 302 to pull the memory page 408 from the system memory 108 of the source host 100a to the system memory 108 of the destination host 100b (block 1512). In the PTE 212 of the memory page 408, the hypervisor 402 of the destination host 100b sets the valid field 502 (V=1), resets the migration field 516 (MI=0), sets the appropriate page protection in the read field 508 and the read / write field 510, and updates the real address field 506 to the new real address of the memory page 408 in the system memory 108 of the destination host 100b (block 1514). Following block 1514, the hypervisor 402 of the destination host 100b releases the lock on the memory page 408 (block 1516). Processing then proceeds to block 1504, which indicates that the page fault handler completes processing the page fault with other, potentially conventional, processing. The processing of FIG. 15 then ends at block 1518.

[0068] 16, a high-level logical flowchart of an exemplary method for a destination host 100b to "pull" memory pages of a migrating LPAR 404 from a source host 100a, according to one embodiment. This process may be performed, for example, at block 1410 of FIG. 14.

[0069] The processing of Figure 16 begins at block 1600 and then proceeds to block 1602, which illustrates the hypervisor 402 of the destination host 100b determining whether the migration field 516 is reset (e.g., MI = 0) in the PTEs 212 of all memory pages 408 of the migrating LPAR 404. If so, the migration of the data set of the migrating LPAR 404 from the source host 100a to the destination host 100b is complete. Thus, the processing of Figure 16 ends at block 1604.

[0070] However, if the hypervisor 402 of the destination host 100b determines in block 1602 that at least one memory page 408 remains to be migrated to the destination host 100b, the hypervisor 402 of the destination host 100b selects one of the unmigrated memory pages 408 in the data set of the migrating LPAR 404 (as indicated by the migration field 516 of its PTE 212 being set) and acquires a lock on the memory page 408 (block 1606). The hypervisor 402 of the destination host 100b then determines in block 1608 whether the migration field 516 of the PTE 212 of the selected memory page 408 was reset (e.g., MI=0) during the process of acquiring the lock. If so, processing proceeds to block 1610, described below. However, if the migration field 516 has not been reset, the hypervisor 402 issues one or more memory access requests to the source host 100a via the NC communication link 302 to pull the memory page 408 from the system memory 108 of the source host 100a to the system memory 108 of the destination host 100b (block 1612). In the PTE 212 for the memory page 408 on the destination host 100b, the hypervisor 402 of the destination host 100b sets the valid field 502 (e.g., V=1), resets the migration field 516 (e.g., MI=0), sets the appropriate page protection for the memory page 408 in the read field 508 and the read / write field 510, and updates the real address field 506 to the new real address of the memory page 408 in the system memory 108 of the destination host 100b (block 1614). Following block 1614, the hypervisor 402 of the destination host 100b releases the lock on the memory page 408 (block 1610). The process of Figure 16 then returns to block 1602 as described above.

[0071] 17, a high-level logical flowchart of an exemplary method for a source host 100a to process a pull request for memory pages 408 of a migrating LPAR 408 from a destination host 100b is depicted, according to one embodiment. The depicted process may be performed, for example, at block 1130 of FIG. 11B.

[0072] The processing of Figure 17 begins at block 1700 and then proceeds to block 1702, which illustrates the hypervisor 402 of the source host 100a determining whether the migration field 516 has been reset (e.g., MI = 0) in the PTE 212 on the source host 100a for all memory pages 408 in the dataset of the LPAR 404 being migrated. If so, the migration of the dataset of the LPAR 404 being migrated from the source host 100a to the destination host 100b is complete. Accordingly, the processing of Figure 17 ends at block 1704.

[0073] However, if the hypervisor 402 of the source host 100a determines in block 1702 that at least one memory page 408 remains to be migrated from the source host 100a to the destination host 100b, then the hypervisor 402 of the source host 100a determines in block 1706 whether a pull request for the memory page 408 in the data set of the migrating LPAR 404 has been received from the destination host 100b via the NC communication link 302. If not, the process of FIG. 17 returns to block 1702 as described above. However, if a pull request for the memory page 408 in the data set of the migrating LPAR 404 has been received, then the hypervisor 402 of the source host 100a responds to the pull request of the destination host 100b by sending the requested memory page 408 to the system memory 108 of the destination host 100b via the NC communication link 302 (block 1708).

[0074] At block 1710, the hypervisor 402 of the source host 100a determines whether the memory page 408 migrated at block 1708 is identified as a read-write memory page in the read-write field 510 of the associated PTE 212 on the source host 100a. If not (e.g., RW=0), processing proceeds to block 1714, described below. However, if the read-write field 510 of the PTE 212 for the memory page 408 is set to indicate a read-write page (e.g., RW=1), the hypervisor 402 of the source host 100a flushes all granules for the memory page 408 from the cache of the source host 100a (block 1712). The source host 100a additionally resets (e.g., MI=0) the migration field 516 in the PTE 212 of the memory page 408 in the system memory 108 of the source host 100a (block 1714). Following block 1714, the hypervisor 402 of the source host 100a signals the destination host 100b that the memory page 408 has been migrated to the system memory 108 of the destination host 100b, for example, by setting a flag (block 1716). The process of Figure 17 then returns to block 1702 as described above.

[0075] 18, a high-level logical flowchart of an exemplary method for a source host 100a to "push" memory pages 408 of a migrating LPAR 404 to a destination host 100b is depicted, according to one embodiment. This process may be performed, for example, at block 1132 of FIG. 11B.

[0076] The processing of Figure 18 begins at block 1800 and then proceeds to block 1802, which depicts the hypervisor 402 of the source host 100a determining whether the migration field 516 has been reset (e.g., MI = 0) in the PTEs 212 of all memory pages 408 of the migrating LPAR 404 on the source host 100a. If so, the migration of the data set of the migrating LPAR 404 from the source host 100a to the destination host 100b is complete. Thus, the processing of Figure 18 ends at block 1804.

[0077] However, if the hypervisor 402 of the source host 100a determines, in block 1802, that at least one memory page 408 remains to be migrated to the destination host 100b, the hypervisor 402 of the source host 100a selects one of the unmigrated memory pages 408 in the dataset of the migrating LPAR 404 (as indicated by the migration field 516 of its PTE 212 being set) and acquires a lock on the memory page 408 (block 1806). The hypervisor 402 of the source host 100a then determines, in block 1808, whether the migration field 516 of the PTE 212 of the selected memory page 408 was reset (e.g., MI=0) while the lock was acquired. If so, processing proceeds to block 1810, described below. However, if the migration field 516 has not been reset, the hypervisor 402 of the source host 100a issues one or more memory access requests to the destination host 100b via the NC communication link 302 to push the memory page 408 from the system memory 108 of the source host 100a to the system memory 108 of the destination host 100b (block 1812).

[0078] At block 1814, the hypervisor 402 of the source host 100a determines whether the memory page 408 migrated at block 1812 is identified as a read-write memory page in the read-write field 510 of the associated PTE 212 of the page frame table 210 in the system memory 108 of the source host 100a. If not (e.g., RW=0), processing proceeds to block 1818, described below. However, if the read-write field 510 of the PTE 212 of the memory page 408 is set to indicate a read-write page (e.g., RW=1), the hypervisor 402 of the source host 100a flushes all granules of the memory page 408 from the cache of the source host 100a to the system memory 108 of the destination host 100b (block 1816). The source host 100a additionally resets (eg, MI=0) the migration field 516 in the PTE 212 of the memory page 408 in the system memory 108 of the source host 100a (block 1818).

[0079] At block 1820, the hypervisor 402 of the source host 100a sends a request to the destination host 100b over the NC communication link 302 to update the PTE 212 of the memory page 408 on the destination host 100b. In response to the PTE update request, the hypervisor 402 of the destination host 100b sets the valid field 502 (e.g., V=1), resets the migration field 516 (e.g., MI=0), and updates the real address field 506 to the new real address of the memory page 408 in the system memory 108 of the destination host 100b. Following block 1820, the hypervisor 402 of the source host 100a monitors for a confirmation message from the destination host 100b over the NC communication link 302 confirming the update of the PTE 212 (block 1822). In response to receiving the confirmation message, the source host 100a releases (block 1810) the lock on the memory page 408. The process of Figure 18 then returns to block 1802 as described above.

[0080] 19, a high-level logical flowchart of an exemplary method for processing a push request for a memory page 408 of a migrating LPAR 404 received from a source host 100a by a destination host 100b is depicted, according to one embodiment. The depicted process may be performed, for example, at block 1412 of FIG. 14.

[0081] The processing of Figure 19 begins at block 1900 and then proceeds to block 1902, which depicts the hypervisor 402 of the destination host 100b determining whether the migration field 516 is reset (e.g., MI = 0) in the PTE 212 on the source host 100a for all memory pages 408 in the dataset of the LPAR 404 being migrated. If so, the migration of the dataset of the LPAR 404 being migrated from the source host 100a to the destination host 100b is complete. Thus, the processing of Figure 19 ends at block 1904.

[0082] However, if the hypervisor 402 of the destination host 100b determines in block 1902 that at least one memory page 408 remains to be migrated from the source host 100a to the destination host 100b, then the hypervisor 402 of the destination host 100b determines in block 1906 whether a PTE update request has been received from the source host 100a via the NC communication link 302 requesting an update to the PTE 212 of the memory page 408 in the dataset of the migrating LPAR 404. If not, the process of FIG. However, if a request to update the PTE is received, the hypervisor 402 of the destination host 100b responds to the request to update the PTE by setting the valid field 502 (e.g., V=1), resetting the migration field 516 (e.g., MI=0), and updating the real address field 506 to the new real address of the memory page 408 in the system memory 108 of the destination host 100b (block 1908). The hypervisor 402 of the destination host 100b sends a confirmation message to the source host 100a confirming the update of the PTE 212 (block 1910). The process of FIG. 19 then returns to block 1902 as described above.

[0083] Referring now to FIG. 20 , a block diagram of an exemplary design flow 2000 for use, for example, in the logic design, simulation, testing, layout, and manufacture of semiconductor ICs is depicted. The design flow 2000 includes a process, machine, or mechanism, or a combination thereof, for processing a design structure or device to generate a logically or other functionally equivalent representation of the design structure or device, or both, described herein. The design structure processed and / or generated by the design flow 2000 may be encoded on a machine-readable transmission or storage medium to include data and / or instructions that, when executed on a data processing system or otherwise processed, generate a logically, structurally, mechanically, or other functionally equivalent representation of a hardware component, circuit, device, or system. The machine includes, but is not limited to, any machine used in an IC design process, such as the design, manufacture, or simulation of a circuit, component, device, or system. For example, a machine may include a lithography apparatus, a machine or apparatus, or both, for generating a mask (such as an electron beam writing apparatus), a computer or apparatus for simulating a design structure, any apparatus used in a manufacturing or testing process, or any machine for programming a functionally equivalent representation of a design structure into any medium (such as a machine for programming a programmable gate array).

[0084] The design flow 2000 may differ depending on the type of representation being designed. For example, a design flow 2000 for building an application specific integrated circuit (ASIC) may differ from a design flow 2000 for designing a standard component, or may differ from a design flow 2000 for simplifying a design to a programmable array, such as a programmable gate array (PGA) or field programmable gate array (FPGA) offered by Altera® or Xilinx®.

[0085] FIG. 20 illustrates multiple such design structures, preferably including an input design structure 2020, processed by design process 2000. Design structure 2020 may be a logic simulation design structure generated and processed by design process 2000 to generate a logically equivalent functional representation of a hardware device. Design structure 2020 may also, or alternatively, include data and / or program instructions that, when processed by design process 2000, generate a functional representation of the physical structure of a hardware device. Whether representing functional or structural or both design features, design structure 2020 may be generated using electronic computer-aided design (ECAD) as implemented by a core developer or designer. Once encoded onto a machine-readable data transmission, gate array, or storage medium, design structure 2020 may be accessed and processed by one or more hardware and / or software modules in design process 2000 to simulate or otherwise functionally represent an electronic component, circuit, electronic or logical module, apparatus, device, or system as described herein. Thus, design structure 2020 may include files or other data structures containing human- or machine-readable source code, compiled structures, and computer-executable code structures, or combinations thereof, that, when processed by a design or simulation data processing system, functionally simulate or otherwise represent a circuit or other level of hardware logic design. Such data structures may include hardware description language (HDL) design entities or other data structures that conform to and / or are compatible with low-level HDL design languages, such as Verilog or VHDL, or high-level design languages, such as C or C++.

[0086] Design process 2000 preferably employs and incorporates hardware and / or software modules that synthesize, translate, or otherwise process design or simulation functional equivalents of components, circuits, devices, or logic structures shown herein to generate netlist 2080, which may include design structures such as design structure 2020. Netlist 2080 may include, for example, a compiled or otherwise processed data structure representing a list of wires, discrete components, logic gates, control circuits, I / O devices, models, etc., that describe connections to other elements and circuits in an integrated circuit design. Netlist 2080 may be synthesized using an iterative process in which netlist 2080 is resynthesized one or more times according to device design specifications and parameters. As with the other design structure types described herein, netlist 2080 may be recorded on a machine-readable storage medium or programmed into a programmable gate array. The medium may be a non-volatile storage medium such as a magnetic or optical disk drive, a programmable gate array, a CompactFlash, or other flash memory. Additionally or alternatively, the medium may be system or cache memory, or buffer space.

[0087] Design process 2000 may include hardware and software modules for processing various input data structure types, including netlist 2080. Such data structure types may exist, for example, in library elements 2030 and include sets of commonly used elements, circuits, and devices, including models, layouts, and symbolic representations, for a given manufacturing technology (e.g., different technology nodes, 32 nm, 45 nm, 90 nm, etc.). Data structure types may further include design specifications 2040, characterization data 2050, verification data 2060, design rules 2070, and test data files 2085, which may include input test patterns, output test results, and other test information. Design process 2000 may further include standard mechanical design processes, such as stress analysis, thermal analysis, mechanical event simulation, and process simulation for operations such as casting, molding, and die pressing. Those of ordinary skill in the art of mechanical design will appreciate the range of possible mechanical design tools and applications that may be used in design process 2000 without departing from the scope and spirit of the present invention. The design process 2000 may also include modules for performing standard circuit design processes such as timing analysis, verification, design rule checking, and place-and-route operations.

[0088] The design process 2000 employs and incorporates logical and physical design tools, such as HDL compilers and simulation model building tools, to process the design structure 2020 along with some or all of the depicted supporting data structures, along with any additional mechanical design or data (if applicable), to generate a second design structure 2090. The design structure 2090 resides on a storage medium or programmable gate array in a data format used for the exchange of mechanical device and structure data (e.g., information stored in IGES, DXF, Parasolid XT, JT, DRG, or any other suitable format for storing or rendering such mechanical design structures). Like the design structure 2020, the design structure 2090 preferably resides on a transmission or data storage medium and includes one or more files, data structures, or other computer-encoded data or instructions that, when processed by an ECAD system, generate a logically or otherwise functionally equivalent form of one or more embodiments of the present invention. In one embodiment, the design structure 2090 may include a compiled, executable HDL simulation model that functionally simulates the devices shown herein.

[0089] The design structure 2090 may also employ a data format used for the exchange of integrated circuit layout data and / or symbolic data formats (e.g., information stored in GDSII (GDS2), GL1, OASIS, map files, or any other suitable format for storing such design data structures). The design structure 2090 may include information such as, for example, symbolic data, map files, test data files, design content files, manufacturing data, layout parameters, wires, metal levels, vias, shapes, data for routing through a manufacturing line, and any other data a manufacturer or other designer or developer needs to manufacture the devices or structures described above and herein. The design structure 2090 can then proceed to stage 2095, for example, where the design structure 2090 proceeds to tapeout, is released to manufacturing, is released to a mask house, is sent to another design house, is sent back to the customer, etc.

[0090] As described above, in at least one embodiment, a data processing system serving a source host to which a logical partition is migrated may include a system fabric, a system memory, a memory controller for the system memory communicatively coupled to the system fabric, and a link controller communicatively coupled to the system fabric and configured to be communicatively coupled to a destination host that is non-coherent with respect to the source host via a communication link. The source host further includes a plurality of processing units communicatively coupled to the system fabric. The plurality of processing units are configured, for example, via a hypervisor or virtual machine monitor, to execute a logical partition having a dataset residing in the system memory and to migrate the logical partition to the destination host via the communication link. Migrating the logical partition includes migrating, via the communication link, a dataset of the logical partition executing on the source host from system memory of the source host to system memory of the destination host. After migrating at least a portion of the dataset, migrating the state of the logical partition from the source host to the destination host via the communication link so that the logical partition executes on the destination host. This migration technique reduces migration latency and improves jitter for the logical partition.

[0091] In the example described above, the source host uses a real address to identify a storage location in the system memory of the destination host to which the memory page should be copied. Those skilled in the art will appreciate that in other embodiments, the storage location in the system memory of the destination host may alternatively be specified using an effective address or a virtual address that undergoes one or more layers of address translation to obtain a real address within the real address space of the destination host.

[0092] While various embodiments have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and detail can be made therein without departing from the scope of the appended claims, and that all such alternative embodiments fall within the scope of the appended claims.

[0093] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a special-purpose hardware-based system that performs the specified functions or acts or executes a combination of special-purpose hardware and computer instructions.

[0094] While aspects have been described with respect to a computer system executing program code that directs the functions of the present invention, it should be understood that the present invention may alternatively be embodied as a program product including a computer-readable storage device that stores program code that, when processed by a processor of the data processing system, causes the data processing system to perform the functions described. The computer-readable storage device may include volatile or non-volatile memory, optical or magnetic disks, etc., but excludes non-statutory subject matter such as propagated signals per se, transmission media per se, and forms of energy per se.

[0095] As an example, a program product may include data and / or instructions that, when executed or otherwise processed on a data processing system, generate a logical, structural, or other functionally equivalent representation (including a simulation model) of a hardware component, circuit, device, or system disclosed herein. Such data and / or instructions may include Hardware Description Language (HDL) design entities or other data structures conforming to or compatible with low-level HDL design languages such as Verilog or VHDL, and / or high-level design languages such as C or C++. Furthermore, the data and / or instructions may employ data formats and / or symbolic data formats used for the exchange of integrated circuit layout data (e.g., information stored in GDSII (GDS2), GL1, OASIS, map files, or any other suitable format for storing such design data structures).

Claims

1. A method of data processing in a data processing environment including first and second data processing systems, the first data processing system being a source host and the second data processing system being a destination host, the method comprising: executing, by the source host, one or more hardware threads of a logical partition on one or more processor cores of the source host, each hardware thread of the one or more hardware threads having a respective thread state including the contents of registers in a processor core executing the respective hardware thread; the source host coordinating, via communication over a network connection, the migration of the logical partition from the source host to the destination host; performing a live migration of a data set of the logical partition from a system memory of the source host to a system memory of the destination host via a link controller in the source host and a communication link connecting the source host and the destination host while the one or more hardware threads of the logical partition are executing on the source host, wherein the source host is a separate data processing system from the destination host and is non-coherent with respect to the destination host, and performing the live migration includes refraining from migrating the data set from the source host to the destination host using the network connection; After migrating at least a portion of the data set, the source host migrates a state of the logical partition from the source host to the destination host via the link controller and the communication link, the state of the logical partition including a thread state of each of the one or more hardware threads of the logical partition; and after migrating the state of the logical partition to the destination host, the destination host executes the one or more hardware threads of the logical partition on the destination host.

2. 2. The method of claim 1, wherein performing a live migration of the dataset comprises migrating at least a portion of the dataset of the logical partition from the source host to the destination host after the one or more hardware threads of the logical partition begin execution on the destination host.

3. 2. The method of claim 1, wherein performing a live migration of the data set includes the source host copying one or more memory pages of the data set to a system memory of the destination host based on a request of the destination host.

4. 2. The method of claim 1, wherein performing a live migration of the data set includes the source host initiating a copying of one or more memory pages of the data set to a system memory of the destination host.

5. the data set includes a plurality of memory pages each having a respective associated page table entry defining a virtual-to-real address translation for the associated memory page; 2. The method of claim 1, further comprising the source host tracking in a data structure which page table entries of the plurality of memory pages have been migrated to the destination host.

6. performing a live migration of the dataset, the source host issuing a store request over a system fabric specifying an address in a system memory of the destination host; the link controller in the source host receiving the store request and transmitting the store request to a memory controller in the destination host over the communication link based on the address.

7. 2. The method of claim 1, wherein performing a live migration of the dataset includes the source host flushing all memory pages in the dataset that are subject to update by the source host from a cache of the source host to a system memory.

8. A data processing system that is a source host, said data processing system comprising: a system fabric; System memory; a memory controller of the system memory, communicatively coupled to the system fabric and the system memory; a link controller configured to be communicatively coupled to the system fabric and communicatively coupled via a communication link to a destination host that is a separate data processing system from the source host, the source host being non-coherent with respect to the destination host; a plurality of processor cores communicatively coupled to the system fabric, each processor core of the plurality of processor cores processing one or more hardware threads of execution, each processor core having a respective thread state including contents of registers within that processor core, the plurality of processor cores configured to execute one or more hardware threads of a logical partition having a data set residing in the system memory, the plurality of processor cores comprising: Coordinating the migration of the logical partition from the source host to the destination host using communication over a network connection; migrating the dataset of the logical partition from the system memory of the source host to the system memory of the destination host via the link controller and the communication link while the one or more hardware threads of the logical partition are executing on the source host, wherein migrating the dataset includes refraining from migrating the dataset using the network connection; and migrating a state of the logical partition from the source host to the destination host via the link controller and the communication link, after migrating at least a portion of the data set, such that the one or more hardware threads of the logical partition subsequently execute on the destination host, the state of the logical partition including a thread state for each of the one or more hardware threads of the logical partition; a plurality of processor cores configured to perform live migration of the logical partition to the destination host via the link controller and the communication link while the one or more hardware threads of the logical partition are executing on the source host.

9. 9. The data processing system of claim 8, wherein migrating the dataset comprises migrating at least a portion of the dataset of the logical partition from the source host to the destination host after the one or more hardware threads of the logical partition begin execution on the destination host.

10. 9. The data processing system of claim 8, wherein migrating the data set includes the source host copying one or more memory pages of the data set to a system memory of the destination host based on a request of the destination host.

11. 9. The data processing system of claim 8, wherein migrating the data set includes the source host initiating a copying of one or more memory pages of the data set to a system memory of the destination host.

12. the data set includes a plurality of memory pages each having a respective associated page table entry defining a virtual-to-real address translation for the associated memory page; 9. The data processing system of claim 8, wherein the plurality of processor cores are further configured to perform tracking in a data structure which page table entries of the plurality of memory pages have been migrated to the destination host.

13. Migrating the dataset includes: the source host issuing a store request on the system fabric specifying an address in a system memory of the destination host; 9. The data processing system of claim 8, further comprising: the link controller of the source host receiving the store request and transmitting the store request to a memory controller in the destination host over the communication link based on the address.

14. 9. The data processing system of claim 8, wherein migrating the dataset includes the source host flushing all memory pages in the dataset that are subject to updating by the source host from a cache of the source host to a system memory.

15. A method for managing a data processing system, comprising: program code that, when executed by a first data processing system functioning as a source host, causes the first data processing system to perform a live migration of a logical partition to a second data processing system functioning as a destination host; the live migration of the logical partition comprising: executing, on one or more processor cores of the source host, one or more hardware threads of the logical partition, each hardware thread of the one or more hardware threads having a respective thread state including the contents of registers in a processor core executing the respective hardware thread; Coordinating the migration of the logical partition from the source host to the destination host using communication over a network connection; performing a live migration of a dataset of the logical partition from a system memory of the source host to a system memory of the destination host via a link controller in the source host and a communication link connecting the source host and the destination host while the one or more hardware threads of the logical partition are executing on the source host, wherein the source host is a separate data processing system and is non-coherent with respect to the destination host, and performing the live migration includes refraining from migrating the dataset from the source host to the destination host using the network connection; migrating a state of the logical partition from the source host to the destination host via the link controller and the communication link after migrating at least a portion of the data set, the state of the logical partition including a thread state of each of the one or more hardware threads of the logical partition; executing the one or more hardware threads of the logical partition on the destination host after migrating the state of the logical partition to the destination host.

16. 16. The program of claim 15, wherein performing live migration of the dataset comprises migrating at least a portion of the dataset of the logical partition from the source host to the destination host after the one or more hardware threads of the logical partition begin execution on the destination host.

17. 16. The program of claim 15, wherein performing the live migration of the dataset includes the source host copying one or more memory pages of the dataset to a system memory of the destination host based on a request of the destination host.

18. 16. The program of claim 15, wherein performing a live migration of the data set includes the source host initiating a copying of one or more memory pages of the data set to a system memory of the destination host.

19. the data set includes a plurality of memory pages each having a respective associated page table entry defining a virtual-to-real address translation for the associated memory page; 16. The program of claim 15, wherein the program code further causes the source host to track in a data structure which page table entries of the plurality of memory pages have been migrated to the destination host.

20. performing a live migration of the dataset, the source host issuing a store request over a system fabric specifying an address in a system memory of the destination host; 16. The program of claim 15, further comprising: the link controller in the source host receiving the store request and transmitting the store request to a memory controller in the destination host over the communication link based on the address.

21. 16. The program of claim 15, wherein performing the live migration of the dataset includes the source host flushing all memory pages in the dataset that are subject to update by the source host from a cache of the source host to a system memory.

Images