Memory migration in multi-host data processing environments

Abstract

A data processing system includes a system fabric, a system memory, a memory controller, and a link controller communicatively coupled to the system fabric and configured to be communicatively coupled to a source host and a destination host non-coherent thereto via a communication link. A plurality of processing units are configured to execute a logical partition and migrate the logical partition to the destination host via the communication link. The migration of the logical partition includes migrating a data set of the logical partition executing on the source host from the system memory of the source host to the system memory of the destination host via the communication link. After migrating at least a portion of the data set, a state of the logical partition is migrated from the source host to the destination host via the communication link such that the logical partition thereafter executes on the destination host.

Description

Background Technology

[0001] This invention generally relates to data processing, and more particularly to memory migration in a multi-host data processing environment.

[0002] Generally, cloud computing refers to a computing model in which data processing, data storage and network resources, software and data are accessible to remote data processing systems, and the details of the underlying information technology (IT) infrastructure providing such resources are transparent to consumers of cloud services. In various implementations, the IT infrastructure can be on-premises or off-premises (or a hybrid of both) relative to the cloud consumer. Furthermore, cloud computing resources can (but are not required to) be widely distributed geographically and / or topologically.

[0003] Cloud computing is facilitated by easy access to remote computing sites (e.g., via the Internet or private corporate networks) and often takes the form of web-based resources, tools, or applications that cloud consumers can access and use through a web browser, as if the resources, tools, or applications were local programs installed on the cloud consumer's computer system. Commercial cloud implementations are typically expected to meet the Quality of Service (QoS) requirements of cloud consumers, which can be specified in a Service Level Agreement (SLA). In a typical cloud implementation, cloud consumers consume computing resources as a service and only pay for the resources they use.

[0004] The widespread use of virtualization has facilitated the adoption of cloud computing. Virtualization creates virtual (not physical) instances of computing resources, such as operating systems, servers, storage devices, and network resources. For example, a virtual machine (VM), also known as a logical partition (LPAR), is a software implementation of a physical machine (e.g., a computer system) that executes instructions like a physical machine. VMs can be categorized as system VMs or process VMs. System VMs provide a complete system platform that supports the execution of full operating systems (OS) such as Windows, Linux, and Android, along with their associated applications. On the other hand, process VMs are typically designed to run a single program and support a single process. In either case, any application software running on a VM is limited by the resources and abstractions provided by that VM. Therefore, by deploying multiple VMs, which may come from multiple different cloud computing customers, it is possible to effectively manage and utilize the actual resources provided by a public IT infrastructure. The virtualization of actual IT resources and the management of VMs are typically provided by software called a VM monitor (VMM) or hypervisor.

[0005] In a typical virtualized computing environment, VMs and VMMs can communicate with each other and with physical entities within the computing environment's IT infrastructure using traditional input / output (I / O) and networking protocols. As is well known in the art, traditional network protocols typically presuppose the well-known seven-layer Open Systems Interconnection (OSI) model, which includes (in ascending order) the physical layer, data link layer, network layer, transport layer, session layer, presentation layer, and application layer. In some implementations, by replacing traditional physical layer connections with virtual network connections, VMs and VMMs can communicate with other network entities as if they were physical network components. This traditional virtualized I / O and networking infrastructure is known in the art as a Virtual Input / Output Server (VIOS).

[0006] In a cloud computing environment as described, a computing workload can generally be characterized by two components: the workload state maintained in the registers and caches of the physical host executing the workload, and the workload dataset residing in the data storage devices of the physical host. These computing workloads are frequently moved between physical hosts for various reasons, including, for example, data processing system maintenance and upgrades, load balancing, regulatory compliance, security, and resource optimization.

[0007] A common technique for transferring workloads from a source host to a destination host involves the operating systems or hypervisors of both the source and destination hosts first coordinating the copying of the workload dataset from the source host's memory to the destination host's memory on a page-by-page basis via VIOS. After a successful transfer of the workload dataset, the operating systems or hypervisors of both the source and destination hosts coordinate the transfer of the workload state from the source host to the destination host via VIOS. While the workload continues to run on the source host during the transfer, it typically continues to read from and write to the workload dataset. Using this technique, each update (write) by the source host to a memory page that has already been transferred to the destination host requires a second transfer of the now-updated memory page, thus unpredictably extending the time required for workload migration.

[0008] In one alternative technique, the operating systems or hypervisors of the source and destination hosts first coordinate the transfer of workload states from the source host to the destination host via VIOS, begin executing the workload on the destination host, and subsequently migrate memory pages from the source host to the destination host based on paging requests. Therefore, whenever a workload executing on the destination host generates an access request for data in a memory page residing on the source host, a software page fault is generated, and the operating system or hypervisor handles the page fault by copying the requested memory page from the source host to the destination host via VIOS. Using this alternative technique, memory pages are transferred from the source host to the destination host only once; however, the workload suffers from poor responsiveness throughout the workload migration due to the significant latency required to service each paging request.

[0009] Both traditional workload migration techniques described in this paper utilize VIOS to transfer workloads between non-coherent hosts. VIOS is characterized by high communication overhead and latency, partly due to the numerous protocol layers that must be traversed to transfer workload datasets and workload states between hosts. Summary of the Invention

[0010] Various embodiments of the present invention provide support for dynamic migration of workloads between incoherent hosts while avoiding the communication overhead and latency associated with VIOS (or external networks), and transmitting workload status and workload datasets.

[0011] In at least one embodiment, a data processing system serving a source host from which logical partitions are migrated includes a system architecture, system memory, a memory controller communicatively coupled to the system memory, and a link controller communicatively coupled to the system architecture and configured to communicatively couple to a destination host via a communication link, wherein the source host and the destination host are incoherent. The source host further includes multiple processing units communicatively coupled to the system architecture. These processing units, for example by a hypervisor or virtual machine monitor, are configured to execute logical partitions having datasets residing in system memory and to migrate the logical partitions to the destination host via the communication link. The migration of logical partitions includes migrating the dataset of the logical partitions executed on the source host from the source host's system memory to the destination host's system memory via the communication link. After migrating at least a portion of the dataset, the state of the logical partitions is migrated from the source host to the destination host via the communication link, such that the logical partitions are subsequently executed on the destination host. This migration technique provides reduced latency for migration and improved jitter for logical partitions.

[0012] Various aspects of the present invention can also be implemented as a data processing method. In one example, the method may include migrating a dataset of logical partitions executed on a source host from the system memory of the source host to the system memory of a destination host that is incoherent with respect to the source host via a communication link. After migrating at least a portion of the dataset, the state of the logical partitions is migrated from the source host to the destination host via the communication link. After the state of the logical partitions is migrated to the destination host, the logical partitions are executed on the destination host.

[0013] Various aspects of the present invention can also be implemented as a program product. In one example, the program product includes a computer-readable storage device and program code stored within the computer-readable storage device, which, when executed by a data processing system acting as a source host, causes the data processing system to migrate, via a communication link, a dataset of a logical partition executed on the source host from the system memory of the source host to the system memory of a destination host incoherent with respect to the source host. The program code also causes the data processing system, after migrating at least a portion of the dataset, to migrate the state of the logical partition from the source host to the destination host via a communication link, such that the logical partition is executed on the destination host.

[0014] In some examples or operational scenarios, migrating a dataset involves migrating at least some of the datasets of a logical partition from the source host to the destination host after the logical partition begins execution on the destination host.

[0015] In some examples or operational scenarios, migrating a dataset involves the source host copying one or more memory pages of the dataset to the system memory of the destination host based on a request from the destination host.

[0016] In some examples or operational scenarios, migrating a dataset involves the source host initiating the process of copying one or more memory pages of the dataset to the system memory of the destination host.

[0017] In some examples or operational scenarios, the dataset consists of multiple memory pages, each with a corresponding associated page table entry that defines the virtual-to-real address translation of the associated memory page, and the source host tracks in the data structure which page table entries of the multiple memory pages have been migrated to the destination host.

[0018] In some examples or operational scenarios, the migration of a dataset involves the source host issuing a storage request at a specified real address in the system memory of the destination host. The link controller in the source host receives the storage request and, based on the real address, transmits the storage request to the memory controller in the destination host via a communication link.

[0019] In some examples, migrating the dataset involves the source host performing a batch flush of all memory pages in the dataset that have undergone updates by the source host from the source host's cache to system memory.

[0020] In some examples, the source host uses communication via a network connection, such as a Virtual Input / Output Server (VIOS), to coordinate the migration of logical partitions from the source host to the destination host, but prevents the migration of datasets and the state of logical partitions via the network connection. Attached Figure Description

[0021] Embodiments of the invention will now be described by way of example only with reference to the accompanying drawings, in which:

[0022] Figure 1 This is a high-level block diagram of an exemplary host data processing system according to one embodiment;

[0023] Figure 2 This is a more detailed block diagram of an exemplary processing unit of a host data processing system according to one embodiment;

[0024] Figure 3 An exemplary data processing system environment according to one embodiment is shown, wherein multiple hosts are coupled together via incoherent communication links;

[0025] Figure 4 A logical view of the management program and logical partitions of an exemplary host data processing system according to one embodiment is shown;

[0026] Figure 5 An exemplary page table entry (PTE) according to one embodiment is shown;

[0027] Figure 6 Various memory access requests of an exemplary host data processing system according to one embodiment are described;

[0028] Figure 7 An exemplary write primitive according to one embodiment is shown, through which two host data processing systems can communicate;

[0029] Figure 8 An exemplary read primitive is shown that two host data processing systems, according to one embodiment, can communicate.

[0030] Figure 9 According to one embodiment Figure 7 A high-level logic flowchart of an exemplary storage operation in the write primitive;

[0031] Figure 10 According to one embodiment Figure 8A high-level logic flowchart of an exemplary load operation in a read primitive;

[0032] Figures 11A to 11B Together, they form a high-level logic flowchart of an exemplary method according to one embodiment, in which a source host migrates a logical partition to a destination host;

[0033] Figure 12 This is a high-level logic flowchart of an exemplary method according to one embodiment, through which the source host processes page protection updates during the migration of a logical partition;

[0034] Figure 13 This is a high-level logic flowchart of an exemplary method according to one embodiment, in which a source host copies memory pages of a logical partition to a destination host.

[0035] Figure 14 This is a high-level logic flowchart of an exemplary method according to one embodiment, through which the destination host receives and activates a migration logical partition;

[0036] Figure 15 This is a high-level logic flowchart of an exemplary method according to one embodiment, through which the destination host handles page errors of a migration logical partition;

[0037] Figure 16 This is a high-level logic flowchart of an exemplary method according to one embodiment, through which a destination host “pulls” memory pages of a migration logical partition from a source host.

[0038] Figure 17 This is a high-level logic flowchart of an exemplary method according to one embodiment, in which a source host processes a "pull" request from a destination host for memory pages of a migration logical partition.

[0039] Figure 18 This is a high-level logic flowchart of an exemplary method according to one embodiment, in which a source host “push” memory pages of a migration logical partition to a destination host.

[0040] Figure 19 This is a high-level logic flowchart of an exemplary method according to one embodiment, through which the destination host processes a "push" request from the source host for memory pages of a migration logical partition; and

[0041] Figure 20 This is a data flow diagram illustrating the design process according to one embodiment. Detailed Implementation

[0042] Now refer to the accompanying drawings, in which the same reference numerals always denote the same and corresponding parts, and especially refer to... Figure 1 This illustration shows a high-level block diagram describing an exemplary data processing system 100 according to one embodiment. In various use cases and topologies, a data processing system such as data processing system 100, which includes hardware components and may additionally include software and / or firmware components, may be referred to in the art as a "host" or "host data processing system". In various operating scenarios, any given host can be a source host from which workloads (e.g., logical partitions) can be migrated, or a destination host to which workloads can be migrated.

[0043] In the described embodiment, the host data processing system 100 is a cache-coherent multiprocessor (MP) data processing system that includes multiple processing nodes 102 for processing data and instructions. The processing nodes 102 are coupled to a system interconnect 110 for transmitting address, data, and control information. The system interconnect 110 can be implemented as, for example, a bus interconnect, a switched interconnect, or a hybrid interconnect.

[0044] In the described embodiments, each processing node 102 is implemented as a multi-chip module (MCM) comprising one or more (e.g., four) processing units 104a-104d, each processing unit preferably implemented as a corresponding integrated circuit. The processing units 104 within each processing node 102 are coupled via a local interconnect 114 for communication with each other and with a system interconnect 110, which, similar to the system interconnect 110, may be implemented using one or more buses and / or switches. The system interconnect 110 and the local interconnect 114 together form the system architecture. In at least some preferred embodiments, communication on the system architecture conforms to a so-called host bus protocol, which specifically defines a predetermined set of legitimate request, response, and control information transmitted between communication participants (e.g., caches, memory controllers, etc.) via the system architecture.

[0045] For reference below Figure 2 More specifically, in some embodiments, one or more processing units 104 (and possibly all processing units 104) each include a memory controller 106 coupled to a local interconnect 114 to provide an interface to a corresponding system memory 108. Data and instructions residing in the system memory 108 can generally be accessed, cached, and modified by the processor core in any processing unit 104 of any processing node 102 within the data processing system 100. In alternative embodiments, one or more memory controllers 106 (and system memory 108) may be directly or indirectly coupled (e.g., via a switch) to the system interconnect 110 instead of the local interconnect 114.

[0046] Those skilled in the art will understand that Figure 1 The data processing system 100 may include a number of additional components, not shown, such as interconnect bridges, non-volatile storage devices, ports for connecting to a network or attached devices, etc. Because these additional components are not essential for understanding the described embodiments, they are not listed here. Figure 1 The enhancements are shown or discussed further in this paper. However, it should also be understood that the enhancements described herein are applicable to data processing systems with different architectures and are by no means limited to them. Figure 1 The generalized data processing system architecture shown is illustrated.

[0047] Now for reference Figure 2 A more detailed block diagram of an exemplary processing unit 104 and system memory 108 according to one embodiment is described. In the described embodiment, each processing unit 104 is an integrated circuit including one or more processor cores 200 for processing instructions and data. In the depicted example, the processor core 200 includes one or more execution units 202 capable of executing instructions from one or more concurrent hardware execution threads.

[0048] Processor core 200 also includes a memory management unit (MMU) 204, which is responsible for translating the effective address determined by the execution of memory reference instructions in execution unit 202 into a real address within the real address space referenced by all processing units 104 within data processing system 100. MMU 204 performs effective-to-real address translation by referencing one or more translation structures 206, such as a translation back buffer (TLB), an effective-to-real address translation (ERAT) cache, a segment back buffer (SLB), etc. The number and / or type of these address translation structures can vary between implementations and architectures. Address translation structures 206 reduce latency associated with address translation by buffering local copies of selected address translations that can be retrieved from system memory 108, as discussed further below.

[0049] The operation of each processor core 200 is supported by a multi-level memory hierarchy, which has a composite system memory at its lowest level provided by various system memories 108 and accessible via a memory controller 106. For example, the individual real address ranges handled by the memory controller 106 can be defined by the hypervisor and / or operating system software through the appropriate configuration of one or more base address registers (BARs) 216 within the memory controller 106. As shown, the system memory 108 stores multiple memory pages 209, providing storage for datasets (also referred to as “logical partitions” (LPARs) of various workloads. Additionally, one or more system memories 108 store a page frame table 210 containing multiple page table entries (PTEs) 212, where each PTE 212 specifies a valid-to-real address translation for a corresponding memory page 209 residing in one of the system memories 108. The PTEs 212 also specify access protections (e.g., read-only, read / write (R / W), etc.) for different memory pages. PTE 212 accessed by MMU 204 from page frame table 210 can be cached by MMU 204 for subsequent access, for example, in address translation structure 206. Page frame table 210 can be created, maintained, and updated, for example, by operating system and / or hypervisor software executing within data processing system 100.

[0050] Each processor core 200's multi-level memory hierarchy further includes one or more levels of cache memory, which in the illustrative embodiment includes a pass-through Level 1 (L1) cache 208 dedicated to each processor core 200 and a corresponding store-in Level 2 (L2) cache 230 for each processor core 200. Although the cache hierarchy shown includes only two levels of cache, those skilled in the art will understand that alternative embodiments may include additional levels (L3, L4, etc.) of on-chip or off-chip, private or shared, embedded or backup caches that may fully include, partially include, or exclude the contents of higher-level caches.

[0051] In the described embodiments, each processing unit 104 further includes an integrated and distributed architecture controller 214, which is responsible for controlling the operational flow on the system architecture according to the host bus protocol and for implementing the coherent communication required to achieve the desired cache coherency protocol. The processing unit 104 may also include an attached integrated I / O (input / output) controller 218 supporting one or more I / O devices and / or I / O channels (not shown).

[0052] In the depicted example, processing unit 104 also includes an additional incoherent (NC) link controller 220, which supports the host data processing system 100 being attached to another host data processing system 100 via an incoherent communication link in at least one operating mode. For example, Figure 3 An exemplary data processing system environment 300 is illustrated, in which host data processing system 100a and host data processing system 100b are coupled together via an incoherent communication link 302 for communication. The data processing system environment 300 may optionally include one or more additional hosts 100 coupled to hosts 100a and / or 100b via additional incoherent communication links 302. In this arrangement, each of the host data processing systems 100a, 100b maintains its own corresponding real address space incoherent with the other hosts. The real address range handled by the NC link controller 220 in each host 100a, 100b may be defined, for example, by a hypervisor and / or operating system software through appropriate configuration of one or more base address registers (BARs) 224 within the NC link controller 220. Once configured, memory access requests transmitted on the system architecture of host 100 specifying real addresses within the real address range defined by the BAR 216 of the local memory controller 106 can be serviced by the memory controller 106 by referencing the associated system memory 108. The NC link controller 220 transmits a memory access request, specifying a real address within the real address range defined by the BAR 224 of the NC link controller 220, to another host 100a or 100b via the NC communication link 106. Inbound memory access requests received by the NC link controller 220 via the incoherent communication link 302 are sent by the NC link controller 220 on the system architecture of the receiving host 100.

[0053] Refer again Figure 2 The processing unit 104 also includes a nested memory management unit (NMMU) 228, which provides address translation to other communication participants, such as the NC link controller 220, upon request via system architecture. It should be understood that in other implementations, the NMMU 228 may be communicatively coupled to provide address translation to communication participants including the NC link controller 220 in an alternative or additional manner, for example, by coupling to system interconnect 110 instead of local interconnect 114.

[0054] Figure 4 Depicting according to one embodiment Figures 1 to 3This is a logical view of the software configuration of host 100a or 100b. In this example, the software configuration 400 of each host 100 includes a hypervisor (or VMM) 402 that manages the hardware and software resources of host 100. The hypervisor 402 virtualizes many of the underlying resources of host 100 by implementing one or more logical partitions (LPARs) 404. In this example, each LPAR 404 includes a specified number of hardware threads 406 within one or more processor cores 200 of host 100, a collection of memory pages 408 (from memory page 209) at a specified real memory address for storing the dataset of LPAR 404, and an instance of operating system 410 (e.g., ...). (etc.) and a collection of applications 412 running on OS 410. Hypervisor 402 also supports network connectivity between LPARs 404 and between hypervisor 402 and other hosts 100 via Virtual Input / Output Server (VIOS) 420, wherein VIOS virtualization hosts 100 are connected to physical network connections via I / O controller 218.

[0055] As described above, the hypervisor 402 can determine that, for any reason, it desires or needs to migrate an active workload, such as LPAR 404, from its own host 100 to another host 100. According to the embodiments described herein, the migration preferably replicates the dataset and state of the LPAR 404 from the host 100 on which the LPAR 404 was initially executed (referred to herein as the “source host”) to another host 100 on which the LPAR 404 continues its execution (referred to herein as the “destination host”) via NC communication link 302 instead of a virtualized network connection supported by VIOS 420. Using NC communication link 302 instead of VIOS 420 to migrate the LPAR dataset has the advantage of reducing or eliminating redundant transfers of memory pages containing the LPAR dataset, thereby accelerating the LPAR migration. Using NC communication link 302 also has the advantage of providing predictable response times for the application(s) 412 migrating the LPAR(s).

[0056] Now for reference Figure 5An exemplary page table entry (PTE) 212 according to one embodiment is shown. In this example, PTE 212 includes a validity field 502, which indicates whether the contents of PTE 212 are valid and can therefore be used when translating a virtual address in one of the associated memory pages 209 to a physical address. Additionally, PTE 212 includes a virtual address (VA) field 504 and a physical address (RA) field 506, which specify the virtual and physical addresses, respectively, of the underlying storage location assigned to the associated memory page 209. PTE 212 also includes one or more page protection fields indicating one or more attributes of the associated memory page 209. For example, in the illustrated embodiment, the page protection fields include a read (R) field 508 and a read-write (RW) field 510, and alternatively indicate whether the associated memory page 209 is read-only or subject to both read and write access. Hereafter, it will be assumed that the field values ​​R=1 and RW=0 indicate that the associated memory page 209 is a read-only memory page, and that the field values ​​R=0 and RW=1 indicate that the associated memory page 209 is a read-write memory page.

[0057] In the illustrated embodiment, PTE 212 further includes one or more migration-related fields that can be used by hypervisor 402 to manage the migration of associated memory pages 209 from system memory 108 of source host 100 to system memory 108 of destination host 100. In this embodiment, these migration-related fields include a replication (CP) field 512 indicating whether the associated memory page 209 has been copied from source host 100 to destination host 100, a clear (CL) field 514 indicating whether all data in the associated memory page 209 has been cleared (i.e., all modified data has been written from cache memory to system memory 108 so that the image of the memory page in system memory 108 is consistent with any cached data), and a migration (MI) field 516 indicating whether PTE 212 has been migrated (i.e., resides in system memory 108 of destination host 100) but has not been verified. It should be understood that the implementation of migration-related fields 512-516 in PTE 212 is a design choice, and other embodiments may omit these fields from PTE 212 and instead utilize one or more other data structures to manage the migration of the LPAR dataset.

[0058] Now for reference Figure 6 This describes various memory access requests of an exemplary host data processing system 100. Figure 6In this configuration, the core 200 of the host data processing system 100 (e.g., host data processing system 100a) executes user-level instructions, such as application 412, in a first hardware thread and executes instructions of the hypervisor 402 in a second hardware thread. The execution of the user-level instructions in the first hardware thread generates memory access requests 602, each specifying a corresponding target virtual (logical) address (e.g., virtual address 610 or 612) to be accessed in the virtual address space 606. As described above, the MMU 204 of the core 200, based on the address translation provided by PTE 212, translates each of the target virtual addresses 610, 612 into a corresponding real address 614 or 616 in the real address space 608. In this example, real address 614 identifies a storage location in the system memory 108 of the host data processing system 100a, which is managed by the local memory controller 106. Conversely, real address 616 identifies a storage location in the system memory 108 of the host data processing system 100b, which is managed by the NC link controller 220. Therefore, a memory access request transmitted on the system architecture of host data processing system 100a can initiate access to a storage location in the local system memory 108 or a storage location in the remote system memory 108 of another host data processing system 100b (host data processing system 100a is not related to it), depending on the allocation of real addresses by the management program 402 of host data processing systems 100a and 100b.

[0059] In at least some embodiments, the memory access request of the hypervisor 402 does not need to undergo the same address translation applied to the target address of the user-level memory access request 602. For example, Figure 6 It is further shown that the instructions of the hypervisor 402 executing in the second hardware thread of core 200 can also generate memory access request 604. However, in this case, memory access request 604 directly specifies real addresses 614, 616 in real address space 608. Therefore, the hypervisor 402 of host data processing system 100a can access storage locations in local system memory 108 of host data processing system 100a and remote system memory 108 of host data processing system 100b without using the virtual-to-real address translation recorded in PTE 212.

[0060] Now for reference Figure 7 The illustration shows an exemplary write primitive 700 according to one embodiment, through which one of the host data processing systems 100a and 100b can write data to the system memory 108 of the other host 100a or 100b via an incoherent communication link 302. The write primitive 700 may form part of a sequence of instructions of a hypervisor 402, an OS 410, or an application 412.

[0061] As shown in the figure, the write primitive 700 includes a first storage operation 702 that stores data granules into memory. Figure 9 A high-level logic flowchart of an exemplary embodiment of this storage operation is provided. In the embodiments described herein, Figure 9 The storage operation described herein begins at box 900 and proceeds to box 902, which describes how core 200 of host 100 (hereinafter, assumed to be host 100a) executes a storage instruction specifying that a data block be stored in memory. After any applicable virtual-to-real address translation, core 200 stores the data block in its cache hierarchy associated with the target real address (e.g., in its L2 cache 230), which is allocated to system memory 108 in another host 100 (i.e., host 100b). Because hosts 100a and 100b are incoherent and NC communication link 302 does not transmit consistency messages between hosts 100a and 100b, simply storing the data block in the cache memory of host 100a is insufficient to provide visibility of the association between the data block and the target real address to the hardware threads of host 100b. Therefore, the memory operation also includes core 200 executing cache clear instructions to force data blocks from the cache hierarchy of core 200 to the relevant system memory 108 (box 904) of host 100b. Afterwards, Figure 9 The process ends at box 906.

[0062] Back Figure 7 Following storage operation 702, write primitive 700 includes a blocking operation 704, which sequentially orders the execution of all operations preceding blocking operation 704 before all operations following blocking operation 704. Following blocking operation 704 in the programmed sequential write primitive 700 is a second storage operation 706. Storage operation 706 updates (e.g., updates to the value 1) a flag variable (which may be polled by a procedure executed on host 100b) residing in system memory 108 of host 100a or host 100b to indicate that a new data block has been written to system memory 108 of host 100b. If the flag resides on host 100b, storage operation 706 may be referenced as above. Figure 9 To be executed as described.

[0063] Now for reference Figure 8 The illustration shows an exemplary read primitive 800 according to one embodiment, through which one of the host data processing systems 100a, 100b can read data from the system memory 108 of the other host 100a, 100b via an incoherent communication link 302. The read primitive 800 may form part of a sequence of instructions of a hypervisor 402, an OS 410, or an application 412.

[0064] As shown in the figure, the read primitive 800 includes a first load operation 802 that loads flags from memory. Figure 10 A high-level logic flowchart of an exemplary embodiment of the load operation is provided. In this embodiment, the load operation begins at block 1000 and proceeds to block 1002, which describes that core 200 of host 100 (hereinafter, assumed to be host 100a) executes a flush instruction that specifies the removal of any cached data (e.g., flags) associated with the real address of the data block to be loaded from the cache memory of host 100a and writing it back to the associated memory controller 106 of host 100b. The load operation also includes core 200 subsequently executing a load instruction to load the data block from the associated system memory 108 of host 100b (block 1004). Thereafter, Figure 10 The process ends at box 1006.

[0065] Return to Figure 8 Following load operation 802, load primitive 800 includes a comparison operation 804, which compares the value of a flag loaded from host 100b with a predetermined value (e.g., 1) to determine whether the data block is ready to be read. If comparison operation 804 determines that the value of the flag indicates that the data block is ready to be read, then conditional branch operation 806 causes the execution of load primitive 800 to continue to load operation 810. Load operation 810, which loads a data block from system memory 108 of host 100b, can be referenced as above. Figure 10 If the comparison operation 804 determines the value of the flag indicating that the loaded data block is not ready to be read, then the branch operation 808 causes the execution branch of the load primitive 800 to return to the load operation 802, which has already been described.

[0066] Those skilled in the art should understand that the above references Figures 7 to 10 The semaphore communication described is just one of many alternative technologies used to support read and write communication between mutually incoherent host data processing systems.

[0067] Now for reference Figures 11A to 11BThis diagram illustrates a high-level logic flowchart of an exemplary method according to one embodiment, in which a source host 100 migrates an LPAR 404 from a source host 100 to a destination host 100. The process begins at block 1100 and proceeds to block 1102, which shows the hypervisors 402 of the source and destination hosts 100 (hereafter assumed to be hosts 100a and 100b, respectively) communicating with each other to negotiate the allocation of resources (e.g., hardware threads 406, real memory addresses, memory pages 209, etc.) of hosts 100a and 100b to the LPAR 404 and to establish certain memory pages 209 for use in communication between hosts 100a and 100b. In at least some embodiments, the hypervisors 402 of the source host 100a and destination host 100b communicate at block 1102 via VIOS 420.

[0068] Processing proceeds from box 1102 to box 1104, which shows the source host 100a's hypervisor 402 determining whether it has completed copying all memory pages 408 of the dataset including the migrated LPAR 404 from host 100a to system memory 108 in host 100b. If so, the process proceeds to box 1112, described below. However, if the source host 100a's hypervisor 402 determines in box 1104 that the copying of the memory pages 408 of the dataset including the migrated LPAR from source host 100a to destination host 100b has not been completed, the process continues in box 1106. Box 1106 shows the source host 100a's hypervisor 402 determining whether to initiate the migration of the LPAR 404 from source host 100a to destination host 100b. It should be noted that the hypervisor 402 may determine the status of migrating LPAR 404 before migrating all or some of the datasets, or it may wait for the status of migrating LPAR 404 until all datasets for migrating LPAR 404 have been copied to the destination host 100b. As an example, the hypervisor 402 may determine the status of migrating LPAR 404 at box 1106 after all memory pages 408 that are a subset of the most frequently (or most recently) accessed memory pages 408 have been migrated. In response to a positive determination at box 1106, the process proceeds from box 1106 to box 1112. In response to a negative determination at box 1106, the process proceeds to box 1108.

[0069] At box 1108, the hypervisor 402 of source host 100a copies one or more memory pages 408 from the dataset of migrating LPAR 404 to at least one system memory 108 in destination host 100b. In at least some embodiments, copying memory pages 408 at box 1108 may be performed according to the following references. Figure 13The described process is performed in a manner that, as should be understood, may require the execution of multiple storage primitives over a contiguous sequence of real addresses when copying memory page 408. For example, memory page 408 may be 4KB, while the size of individual data blocks copied by storage primitives may be 128B. In this example, copying memory page 408 would require the execution of 32 storage primitives. For each memory page 408 completely copied from source host 100a to destination host 100b, the hypervisor 402 of source host 100a sets the CP field 512 of the associated PTE 212 in system memory 108 of host 100a to indicate that memory page 408 has been copied to destination host 100b (box 1110). Figure 11A The process returns from box 1110 to box 1104.

[0070] In boxes 1112 and following boxes, the hypervisor 402 of the source host 110a copies the state of the migrated LPAR 404 from the source host 100a to the destination host 100b. Specifically referring to box 1112, the hypervisor 402 of the source host 100a brings the hardware threads 406 of the migrated LPAR 404 under its control. Furthermore, the hypervisor 402 saves the state of each hardware thread 406 of the migrated LPAR 404 (e.g., the contents of data and control registers associated with the hardware thread 406 in the execution core 200) and writes the state of each such hardware thread of the migrated LPAR 404 to system memory 108 in the destination host 100b. The copying of the state of the hardware threads of the migrated LPAR 404, as shown in box 1114, can be performed via NC communication link 302 using write primitive 700, as referred to above. Figure 7 and Figure 9 As described above, at block 1116, the hypervisor 402 of source host 110a also updates each PTE 212 associated with memory pages 408 in the dataset of migrated LPAR 404 by invalidating PTE 212 (e.g., resetting valid field 502 to 0) and setting migration (MI) field 516 (e.g., setting it to 1). Block 1116 also shows that hypervisor 402 invalidates any entries in address translation structure 206 of source host 100a corresponding to invalidated PTE 212 and waits for any and all ongoing operations in source host 100a referencing the real address translated by the invalidated PTE to be discharged from core 200 of source host 100a. The process then proceeds via page joiner A. Figure 11B The frame is 1120.

[0071] As indicated by the dashed line, box 1120 is an optional step in which the hypervisor 402 of the source host 100a determines whether to flush the read-write memory pages 408 of the migrated LPAR 404 (i.e., those identified as read-write memory pages by their associated PTE 212 RW field 510) from the cache of the source host 100a. In response to a negative determination at box 1120, or if the source host 100a does not support the ability to effectively flush the cache of the source host 100a, the process proceeds to box 1126 described below. However, if the hypervisor 402 determines at box 1120 to flush the read-write memory pages 408 of the migrated LPAR 404 from the cache of the source host 100a, the hypervisor 402 performs the process of flushing all modified data belonging to the dataset of the migrated LPAR 404 from the cache of the source host 100a (box 1122). In some embodiments, the eviction depicted at box 1122 can be performed by bulk refreshing all modified data from the cache of source host 100a. It should be noted that, depending on whether the underlying memory pages 408 have been migrated, the refresh process depicted at box 1122 may require writing the modified data back to the local system memory 108 of source host 100a or the system memory 108 of destination host 100b. Furthermore, at box 1124, the hypervisor 402 of source host 110a updates the PTE 212 on source host 110a (e.g., by setting the CL field 514 to 1) for all read / write memory pages 408 of the migration LPAR 404 that have been migrated to the purged destination host 100b. As shown in boxes 1120-1124, performing a bulk refresh of the read / write memory pages 408 of migration partition 404 provides the advantage of reducing or eliminating duplicate copying of read / write memory pages 408 and reducing jitter after migration LPAR 404 begins execution on destination host 100b. Furthermore, batch refreshing of the read / write memory pages 408 of the migrated LPAR 404 can provide improved performance compared to a solution that handles refreshing each read / write memory page 408 of the migrated LPAR 404 individually.

[0072] Referring now to box 1126, the hypervisor 402 of source host 100a replicates PTE 212, which translates the addresses in memory pages 408 of the dataset of migrated LPAR 404 from source host 100a to destination host 100b. Then, hypervisor 402 resets the MI field 516 in PTE 212 for each read-write memory page in the migrated LPAR 404 dataset, identified by PTE fields 512-514 as replicated and cleared, and resets the MI field 516 in PTE 212 for each read-only memory page in the migrated LPAR 404 dataset, identified by PTE field 512 as replicated (box 1128). Following box 1128, the hypervisor 402 of source host 100a begins monitoring for "pull" requests from the hypervisor 402 of destination host 100b. Through these "pull" requests, destination host 100b initiates the migration of one or more previously unmigrated memory pages 408 in the dataset migrating LPAR 404 (box 1130). See below for further details. Figure 16 and Figure 17 The description describes an exemplary process in which the hypervisor 402 of destination host 100b “pulls” storage pages from the dataset of migration LPAR 404 from source host 100a to system memory 108 of destination host 100b, and the hypervisor 402 of source host 100a responds to the “pull” request of destination host 100b.

[0073] As shown in optional box 1132, the hypervisor 402 of source host 100a may alternatively or additionally initiate a process by which source host 100a “push” any remaining unmigrated and migrated, but not cleared, memory pages 408 in the dataset of migrating LPAR 404 from source host 100a to system memory 108 in destination host 100b. See below for reference. Figure 18 and Figure 19 This describes an exemplary process in which the hypervisor 402 of source host 100a “push” memory pages 408 from the dataset of migrated LPAR 404 from source host 100a to system memory 108 of destination host 100b, and the hypervisor 402 of destination host 100b responds to the “push” request from source host 100a. Following box 1132 (or box 1130 if box 1132 is omitted), Figure 11B The process ends at box 1134.

[0074] Now for reference Figure 12 This describes a high-level logic flowchart of an exemplary method according to one embodiment, by which source host 100a processes page protection updates during the migration of LPAR 404. Figure 12The process begins at box 1200 and proceeds to box 1202, where the hypervisor 402 of the source host 100a determines whether a page protection update for the PTE 212 of the memory page 408 of the migrated LPAR 404 is requested. If not, the process repeats at box 1202. However, if the hypervisor 402 determines that a page protection update for the PTE 212 of the memory page 408 of the migrated LPAR 404 is requested, the process proceeds to box 1204.

[0075] Box 1204 shows the hypervisor 402 determining whether the PTE 212 currently being updated indicates that the associated memory page 408 is a read-write page (e.g., RW field 510 is set to RW=1). If not (e.g., R field 508 is set to R=1 to indicate a read-only memory page), the process proceeds to box 1212, which shows the hypervisor 402 performing further processing. Thereafter, Figure 12 The process ends at box 1214.

[0076] Returning to box 1204, in response to hypervisor 402 determining that the PTE 212 to be updated currently indicates that the associated memory page 408 is a read-write memory page, hypervisor 402 further determines, by referring to the CP field 512 of the PTE 212 to be updated, whether the associated memory page 408 has already been migrated to the system memory 108 of the destination server 100b. If so, hypervisor 402 executes one or more instructions that cause any modified data of memory page 408 to be flushed from the cache of source host 100a to the system memory 108 of destination host 100b (box 1208), and then, for example, by resetting the R field 508 and setting the RW field 510, updates the page protection information of memory page 408 from read-write to read-only (box 1210). By flushing any cached modifications to the date of the memory page 408 to be updated for protection, additional copying of memory page 408 from source host 100a to destination host 100b can be avoided. If hypervisor 402 determines in box 1206 that the associated memory page 408 has not yet been migrated to the destination host's system memory 108, then hypervisor 402 simply updates the page protection information, as shown in box 1210, without refreshing memory page 408. Following box 1210, Figure 12 The process ends at box 1214.

[0077] Now for reference Figure 13 A high-level logic flowchart of an exemplary method according to one embodiment is shown, in which source host 100a copies memory pages 408 of migration LPAR 404 to system memory 108 of destination host 100b. In at least some embodiments, the illustrated processing can be utilized, for example, to... Figure 11AAt box 1108, one of the (one or more) memory pages 408 copied from the source host 100a is copied to the destination host 100b.

[0078] Figure 13 The process begins at box 1300 and proceeds to box 1302, which shows that the hypervisor 402 of the source host 100a invalidates, for example, the PTE 212 associated with the memory page 408 to be migrated by resetting the valid field 502 to 0. Invalidating PTE 212 makes PTE 212 temporarily inaccessible for address translation, and thus makes the associated memory page 408 accessible only by the hypervisor 402. Box 1302 also shows that the hypervisor 402 invalidates any entries in the address translation structure 206 of the source host 100a corresponding to the invalid PTE 212, and awaits any and all ongoing operations referencing the real address translated by the invalid PTE to be ejected from the core 200 of the source host 100a.

[0079] Then, hypervisor 402 causes one or more cores 200 of source host 100a to execute instructions to copy memory page 408 to a real address (box 1304) allocated to system memory 108 in destination host 100b. As part of this copying process, hypervisor 402 loads an existing image of memory page 408 from system memory 108 in source host 100a (which resides in the source real address range) and stores the image of memory page 408 to the destination real address range of the specified storage location in system memory 108 of destination host 100b. As an optimization, some embodiments use similar cache line allocation instructions, such as Data Cache Block Zero (DCBZ), Data Cache Block Allocation (DCBA), or creating a cache line in cache memory of source host 100a (and optionally initializing the newly created cache line to a predetermined value (e.g., zero)), to allocate a cache line corresponding to the destination real address range in cache memory of source host 100a without fetching the associated data. These cache line allocation instructions can be conveniently implemented in an execution loop that iterates through each cache line in the memory page 408 to be migrated. In such an execution loop, preferably, each cache line allocation instruction is constrained to be executed before and close in time to the corresponding store instruction that stores data from the existing image of the migrated memory page 408 into the newly allocated cache line. Under this execution constraint, the store instructions that update cache lines within the destination real address range will avoid a high probability of cache misses and the associated latency of fetching cache lines from the destination host 100b. In other words, the described optimization has the advantage of avoiding the high latency of fetching a copy of each cache line containing data within the destination real address range from the destination host 100b to rewrite the cache line with data from the migrated memory page 408.

[0080] In box 1308, hypervisor 402 determines, for example, whether memory page 408 is a read-write memory page by checking the RW field 510 of the associated PTE 212. In response to determining at box 1308 that memory page 408 is a read-write memory page, the process proceeds to box 1312, which will be described below. However, if hypervisor 402 determines that memory page 408 is a read-only memory page (e.g., R field 508 is set to R=1), hypervisor 402 causes one or more clear or flush instructions to be executed by one or more cores 200 of source host 100a to ensure that data of read-only memory page 408 is written from the cache of source host 100a to system memory 108 of destination host 100b via NC communication link 302 (box 1310). Following box 1308 or box 1310, the hypervisor 402 of source host 100a updates the real address (RA) specified for memory page 408 in the RA field 506 of PTE 212 to specify the new real address of memory page 408 in system memory 108 of destination host 100b, and then updates the validity field 502 of PTE 212 on source host 100a to a valid state (e.g., V=1) to make PTE 212 accessible again for address translation (box 1312). After this, Figure 13 The process ends at box 1314.

[0081] Now for reference Figure 14 This describes a high-level logic flowchart of an exemplary method according to one embodiment, by which a destination host 100b receives and activates a migration LPAR 404. Figure 14 The process is the same as the one described earlier, executed on source host 100a. Figures 11A to 11B Collaboration during the process.

[0082] Figure 14 The process begins at block 1400 and proceeds to block 1402, which illustrates the communication between the hypervisor 402 of the destination host 100b and the hypervisor 402 executing on the source host 100a to negotiate the allocation of resources (e.g., hardware threads 406, real memory addresses, memory pages 209, etc.) from hosts 100a and 100b to LPAR 404, and to establish certain memory pages 209 for use in communication between hosts 100a and 100b. In at least some embodiments, the hypervisors 402 of the source host 100a and the destination host 100b communicate at block 1402 via VIOS 420.

[0083] Processing proceeds from box 1402 to box 1404, which shows that the destination host 100b's management program 402 receives the status of the migrated LPAR 404 from the source host 100a via the NC communication link 302 and buffers the LPAR status. The LPAR status is determined by the source host 100a in... Figure 11A In box 1114, the data is transmitted. Additionally, in box 1406, the destination host 100b's management program 402 receives and buffers PTEs 212 from the source host 100a via NC communication link 302 for migrating memory pages 408 of the LPAR 404. These PTEs 212 are transmitted by the source host 100a in... Figure 11B Send at frame 1126.

[0084] Box 1408 shows that the hypervisor 402 of the destination host 100b creates a PTE 212 in the page frame table 210 in the system memory 108 of the destination server 100b that is associated with the memory page 408 of the migrated LPAR 404. For each read-only memory page 408 indicated by its associated PTE 212 as having been copied to destination host 100b (e.g., R field 508 is set to R=1 and CP field 512 is set to CP=1), and for each read-write memory page 408 indicated by its associated PTE 212 as having been copied to destination host 100b and cleared (e.g., RW field 510 is set to RW=1 and CL field 514 is set to CL=1), the hypervisor 402 of destination host 100b updates the real address field 506 to reflect the storage location of the memory page 408 in the system memory 108 of destination host 100b, resets the migration field 516 (e.g., MI=0), and sets the valid field 502 (e.g., V=1). The hypervisor 402 of destination host 100b thus makes these PTEs 212 available for virtual-to-real address translation on destination host 100b.

[0085] In box 1410, the hypervisor 402 of the destination host 100b initiates a process by which the hypervisor 402 of the destination server 100b “pull” any remaining unmigrated memory pages 408 of the migrated LPAR 404 from the source host 100a via the NC communication link 302 into the system memory 108 of the destination host 100b. See below for reference. Figure 16 The exemplary process described below involves destination host 100b pulling such memory pages 408 from source host 100a. Additionally, in block 1412, the hypervisor 402 of destination host 100b optionally initiates a process by which the hypervisor 402 of destination server 100b receives any remaining unmigrated memory pages 408 of the migrated LPAR 404 of system memory 108 “pushed” from source host 100a to destination host 100b via NC communication link 302. References below... Figure 18The exemplary process described is as follows: the destination host 100b receives such a memory page 408 from the source host 100a. The hypervisor 402 of the destination host 100b initiates the execution of the migration LPAR 404 on the destination host 100b by loading the state of the various hardware threads 406 of the migration LPAR 404 into one or more cores 200 of the destination host 100b (box 1414) and releasing the hardware threads 406 of the migration LPAR 404 to begin execution (box 1416). Figure 14 The process then ends in box 1416.

[0086] Now for reference Figure 15 The diagram illustrates a high-level logic flowchart of an exemplary method according to one embodiment, by which destination host 100b handles page errors during the migration of LPAR 404. Figure 15 The process begins at box 1500 in response to a memory access request from the destination host 100b that caused the page fault. In response to the page fault, processing proceeds from box 1500 to box 1502, which shows the page fault processor routine executed on the destination host 100b determining, by referring to the PTE 212 of the memory page 209 containing the target real address of the memory access request, whether the migration field 516 is set (e.g., MI = 1) to indicate that memory page 209 is one of the memory pages 408 migrating LPAR 404. If not (e.g., MI = 0), then as shown in box 1504, the page fault processor handles the page fault using other, possibly conventional, processing. After box 1504, Figure 15 The process ends at box 1518.

[0087] Referring again to box 1502, in response to determining that migration field 516 is set, the hypervisor 402 of destination host 100b acquires a lock on the memory page 408 on which a page fault has occurred (box 1506). The hypervisor 402 of destination host 100b then determines in box 1508 whether migration field 516 of the PTE 212 of memory page 408 was reset during the lock acquisition process (e.g., MI = 0). If so, the process proceeds to box 1516 described below. However, if migration field 516 was not reset, the hypervisor 402 issues one or more memory access requests to source host 100a via NC communication link 302 to pull memory page 408 from system memory 108 of source host 100a to system memory 108 of destination host 100b (box 1512). In PTE 212 of memory page 408, the hypervisor 402 of destination host 100b sets the valid field 502 (V=1), resets the migration field 516 (MI=0), sets appropriate page protection in the read field 508 and read / write field 510, and updates the real address field 506 with the new real address of memory page 408 in system memory 108 of destination host 100b (box 1514). After box 1514, the hypervisor 402 of destination host 100b releases the lock on memory page 408 (box 1516). The process then proceeds to box 1504, which shows that the page fault processor has completed handling the page fault, while other potential routine processing is performed. Figure 15 The process then ends at box 1518.

[0088] Now for reference Figure 16 This is a high-level logic flowchart of an exemplary method according to one embodiment, by which destination host 100b “pulls” the migrated memory pages of LPAR 404 from source host 100a. For example, it can be done in... Figure 14 The procedure is executed at box 1410.

[0089] Figure 16 The process begins at box 1600 and proceeds to box 1602, which shows the destination host 100b's hypervisor 402 determining whether the migration field 516 has been reset in the PTE 212 of all memory pages 408 of the migration LPAR 404 (e.g., MI = 0). If so, the migration of the dataset of the migration LPAR 404 from the source host 100a to the destination host 100b is completed. Therefore, Figure 16 The process ends in box 1604.

[0090] However, if the hypervisor 402 of destination host 100b determines in box 1602 that at least one memory page 408 still needs to be migrated to destination host 100b, then the hypervisor 402 of destination host 100b selects one of the unmigrated memory pages 408 in the dataset of migrated LPAR 404 (as indicated by the migration field 516 of its PTE 212 being set) and acquires a lock on that memory page 408 (box 1606). The hypervisor 402 of destination host 100b then determines in box 1608 whether the migration field 516 of the PTE 212 of the selected memory page 408 was reset during the lock acquisition process (e.g., MI = 0). If so, the process proceeds to box 1610, which will be described below. However, if migration field 516 is not reset, the hypervisor 402 issues one or more memory access requests to the source host 100a via NC communication link 302 to pull memory page 408 from system memory 108 of source host 100a to system memory 108 of destination host 100b (box 1612). In PTE 212 for memory page 408 on destination host 100b, the hypervisor 402 of destination host 100b sets the validity field 502 (e.g., V=1), resets migration field 516 (e.g., MI=0), sets appropriate page protection for memory page 408 in read field 508 and read / write field 510, and updates the real address field 506 with the new real address of memory page 408 in system memory 108 of destination host 100b (box 1614). After box 1614, the hypervisor 402 of destination host 100b releases the lock on memory page 408 (box 1610). Thereafter, Figure 16 The process returns to the already described box 1602.

[0091] Now for reference Figure 17 A high-level logic flowchart of an exemplary method according to one embodiment is shown, in which source host 100a processes a pull request from destination host 100b for memory pages 408 of migrated LPAR 408. For example, it can be done in... Figure 11B The procedure shown is executed at box 1130.

[0092] Figure 17 The processing begins at box 1700 and proceeds to box 1702, which shows that the hypervisor 402 of source host 100a determines whether the migration field 516 in PTE 212 on source host 100a has been reset (e.g., MI = 0) for all memory pages 408 in the dataset for migrating LPAR 404. If so, the migration of the dataset for migrating LPAR 404 from source host 100a to destination host 100b is completed. Therefore, Figure 17 The process ends at box 1704.

[0093] However, if the hypervisor 402 of source host 100a determines in block 1702 that at least one memory page 408 still needs to be migrated from source host 100a to destination host 100b, then the hypervisor 402 of source host 100a determines in block 1706 whether a pull request for memory page 408 in the dataset for migrating LPAR 404 has been received from destination host 100b via NC communication link 302. If not, then Figure 17 The process returns to box 1702, which has already been described. However, if a pull request for memory page 408 in the dataset of the migrated LPAR 404 is received, the hypervisor 402 of the source host 100a responds to the pull request of the destination host 100b by sending the requested memory page 408 to the system memory 108 of the destination host 100b via the NC communication link 302.

[0094] In box 1710, the hypervisor 402 of source host 100a determines whether the memory page 408 migrated in box 1708 is identified as a read-write memory page in the read-write field 510 of the associated PTE 212 on source host 100a. If not (e.g., RW=0), the process proceeds to box 1714, which will be described below. However, if the read-write field 510 of the PTE 212 of memory page 408 is set to indicate a read-write page (e.g., RW=1), the hypervisor 402 of source host 100a refreshes all extents of memory page 408 from the cache of source host 100a (box 1712). Source host 100a also resets the migration field 516 (e.g., MI=0) in the PTE 212 of memory page 408 in system memory 108 of source host 100a (box 1714). Following box 1714, the source host 100a's management program 402, for example, signals to the destination host 100b that memory page 408 has been migrated to the destination host 100b's system memory 108 by setting a flag (box 1716). Afterwards, Figure 17 The process returns to the already described box 1702.

[0095] Now for reference Figure 18 This describes a high-level logic flowchart of an exemplary method according to one embodiment, by which source host 100a “pushes” memory pages 408 of migrated LPAR 404 to destination host 100b. For example, it can be done in... Figure 11B The procedure is executed at box 1132.

[0096] Figure 18The process begins at box 1800 and proceeds to box 1802, which shows the source host 100a's hypervisor 402 determining whether the migration field 516 in the PTE 212 of all memory pages 408 of the migration LPAR 404 on the source host 1004a has been reset (e.g., MI = 0). If so, the migration of the dataset of the migration LPAR 404 from the source host 100a to the destination host 100b is completed. Therefore, Figure 18 The process ends at box 1804.

[0097] However, if the hypervisor 402 of source host 100a determines in box 1802 that at least one memory page 408 still needs to be migrated to destination host 100b, then the hypervisor 402 of source host 100a selects an unmigrated memory page 408 from the dataset of migrated LPAR 404 (as indicated by the migration field 516 of its PTE 212 being set) and acquires a lock on that memory page 408 (box 1806). The hypervisor 402 of source host 100a then determines in box 1808 whether the migration field 516 of the PTE 212 of the selected memory page 408 has been reset (e.g., MI = 0) when the lock was acquired. If so, the process proceeds to box 1810, which will be described below. However, if the migration field 516 is not reset, the hypervisor 402 of the source host 100a sends one or more memory access requests to the destination host 100b via the NC communication link 302 to push memory page 408 from the system memory 108 of the source host 100a to the system memory 108 of the destination host 100b (box 1812).

[0098] In box 1814, the hypervisor 402 of source host 100a determines whether the memory page 408 migrated in box 1812 is identified as a read-write memory page in the read-write field 510 of the associated PTE 212 in the page frame table 210 of system memory 108 of source host 100a. If not (e.g., RW=0), the process proceeds to box 1818, which will be described below. However, if the read-write field 510 of the PTE 212 of memory page 408 is set to indicate a read-write page (e.g., RW=1), the hypervisor 402 of source host 100a flushes all sectors of memory page 408 from the cache of source host 100a to the system memory 108 of destination host 100b (box 1816). Source host 100a also resets the migration field 516 in the PTE 212 of memory page 408 in system memory 108 of source host 100a (e.g., MI=0) (box 1818).

[0099] In box 1820, the hypervisor 402 of source host 100a sends a request to destination host 100b via NC communication link 302 to update the PTE 212 of memory page 408 on destination host 100b. In response to the PTE update request, the hypervisor 402 of destination host 100b sets the validity field 502 (e.g., V=1), resets the migration field 516 (e.g., MI=0), and updates the real address field 506 with the new real address of memory page 408 in system memory 108 of destination host 100b. After box 1820, the hypervisor 402 of source host 100a monitors for an acknowledgment message from destination host 100b via NC communication link 302, which confirms the update of PTE 212 (box 1822). In response to receiving the acknowledgment message, source host 100a releases the lock on memory page 408 (box 1810). Thereafter, Figure 18 The process returns to the already described box 1802.

[0100] Now for reference Figure 19 A high-level logic flowchart of an exemplary method according to one embodiment is shown, in which destination host 100b processes a push request received from source host 100a for a memory page 408 of migrated LPAR 404. For example, it can be... Figure 14 The procedure shown is executed at box 1412.

[0101] Figure 19 The process begins at box 1900 and proceeds to box 1902, which shows the destination host 100b's hypervisor 402 determining whether the migration field 516 in the PTE 212 on the source host 100a has been reset (e.g., MI = 0) for all memory pages 408 in the dataset for migrating LPAR 404. If so, the migration of the dataset for LPAR 404 from the source host 100a to the destination host 100b is completed. Therefore, Figure 19 The process ends at box 1904.

[0102] However, if the hypervisor 402 of destination host 100b determines in block 1902 that at least one memory page 408 still needs to be migrated from source host 100a to destination host 100b, then the hypervisor 402 of destination host 100b determines in block 1906 whether it has received a PTE update request from source host 100a via NC communication link 302 requesting an update of the PTE 212 of memory page 408 in the dataset for migrating LPAR 404. If not, then Figure 19The process returns to box 1902, which has already been described. However, if a PTE update request has been received, the hypervisor 402 of destination host 100b responds to the PTE update request by setting the valid field 502 (e.g., V=1), resetting the migration field 516 (e.g., MI=0), and updating the real address field 506 using the new real address of the memory page 408 in the system memory 108 of destination host 100b (box 1908). The hypervisor 402 of destination host 100b sends an acknowledgment message to source host 100a confirming the update of PTE 212 (box 1910). Afterwards, Figure 19 The process returns to the already described box 1902.

[0103] Now for reference Figure 20 This document describes a block diagram of an exemplary design flow 2000 used, for example, in semiconductor IC logic design, simulation, testing, placement, and manufacturing. Design flow 2000 includes processes, machines, and / or mechanisms for processing design structures or devices to generate logically or otherwise functionally equivalent representations of the design structures and / or devices described herein. Design structures processed and / or generated by design flow 2000 may be encoded on a machine-readable transmission or storage medium to include data and / or instructions that, when executed on a data processing system or otherwise processed, generate logically, structurally, mechanically, or otherwise functionally equivalent representations of hardware components, circuits, devices, or systems. Machines include, but are not limited to, any machine used in the IC design process, such as for designing, manufacturing, or simulating circuits, components, devices, or systems. For example, machines may include: lithography machines, machines and / or devices for generating masks (e.g., electron beam writers), computers or devices for simulating design structures, any means for manufacturing or testing processes, or any machine for programming a functionally equivalent representation of a design structure to any medium (e.g., a machine for programming programmable gate arrays).

[0104] Design flow 2000 can vary depending on the type of representation being designed. For example, design flow 2000 for building application-specific integrated circuits (ASICs) may differ from design flow 2000 for designing standard components or design flow 2000 for instantiating designs into programmable arrays, such as programmable gate arrays (PGAs) or field-programmable gate arrays (FPGAs) provided by Altera or Xilinx.

[0105] Figure 20Several such design structures are illustrated, including an input design structure 2020 preferably processed by design process 2000. Design structure 2020 can be a logical simulation design structure generated and processed by design process 2000 to produce a logically equivalent functional representation of a hardware device. Design structure 2020 may also, or alternatively, include data and / or program instructions that, when processed by design process 2000, generate a functional representation of the physical structure of the hardware device. Whether representing functional and / or structural design features, design structure 2020 can be generated using electronic computer-aided design (ECAD) such as that implemented by a kernel developer / designer. When encoded on machine-readable data transmission, gate array, or storage media, design structure 2020 can be accessed and processed by one or more hardware and / or software modules within design process 2000 to simulate or otherwise functionally represent electronic components, circuits, electronic or logic modules, devices, apparatuses, or systems such as those shown herein. Thus, the design structure 2020 may include files or other data structures, including human- and / or machine-readable source code, compiled structures, and computer-executable code structures, which, when processed by a design or simulation data processing system, functionally simulate circuits or otherwise represent other levels of hardware logic design. Such data structures may include hardware description language (HDL) design entities or other data structures that conform to and / or are compatible with low-level HDL design languages ​​such as Verilog and VHDL and / or high-level design languages ​​such as C or C++.

[0106] Design process 2000 preferably employs and integrates hardware and / or software modules for synthesizing, transforming, or otherwise processing design / simulation functional equivalents of the components, circuits, devices, or logic structures shown herein to generate a netlist 2080 that may contain design structures such as design structure 2020. Netlist 2080 may include, for example, compiled or otherwise processed data structures representing lists of wires, discrete components, logic gates, control circuits, I / O devices, models, etc., describing connections to other elements and circuits in the integrated circuit design. Netlist 2080 may be synthesized using an iterative process in which it is resynthesized once or multiple times depending on the device's design specifications and parameters. As with other design structure types described herein, netlist 2080 may be recorded on a machine-readable storage medium or programmed into a programmable gate array (FPGA). This medium may be a non-volatile storage medium, such as a disk or optical disc drive, a programmable gate array, compact flash memory, or other flash memory. Alternatively, the medium may be a system or cache memory, or a buffer space.

[0107] Design process 2000 may include hardware and software modules for processing various input data structure types, including netlist 2080. Such data structure types may, for example, reside within library elements 2030 and include a set of commonly used components, circuits, and devices, including models, layouts, and symbolic representations for a given manufacturing technology (e.g., different technology nodes, 32nm, 45nm, 90nm, etc.). Data structure types may further include design specifications 2040, feature data 2050, verification data 2060, design rules 2070, and test data files 2085, which may include input test modes, output test results, and other test information. Design process 2000 may also include, for example, standard mechanical design processes, such as stress analysis, thermal analysis, mechanical event simulation, process simulation for operations such as casting, molding, and compression molding. Design process 2000 may also include modules for performing standard circuit design processes, such as timing analysis, verification, design rule checking, placement, and routing operations.

[0108] Design process 2000 employs and combines logical and physical design tools, such as HDL compilers and simulation model building tools, to process design structure 2020 and some or all of the described supporting data structures, as well as any additional mechanical design or data (if applicable), to generate a second design structure 2090. Design structure 2090 resides on a storage medium or programmable gate array in a data format used for exchanging mechanical devices and structures (e.g., information stored in IGES, DXF, Parastall XT, JT, DRG, or any other suitable format for storing or presenting such mechanical design structures). Similar to design structure 2020, design structure 2090 preferably includes one or more files, data structures, or other computer-coded data or instructions residing on a transmission or data storage medium and, when processed by an ECAD system, produces a logically or otherwise functionally equivalent form of one or more embodiments of the invention. In one embodiment, design structure 2090 may include a compiled executable HDL simulation model that functionally simulates the device shown herein.

[0109] Design structure 2090 may also employ data formats for exchanging layout data of integrated circuits and / or symbol data formats (e.g., information stored in GDSII (GDS2), GL1, OASIS, drawing files, or any other suitable format used to store such design data structures). Design structure 2090 may include information such as symbol data, drawing files, test data files, design content files, manufacturing data, layout parameters, wiring, metal grades, vias, shapes, data for routing through the production line, and any other data required by the manufacturer or other designer / developer to produce the devices or structures described above and herein. Design structure 2090 may then proceed to stage 2095, where, for example, design structure 2090: is fabricated, delivered for manufacturing, delivered to a mask plant, sent to another design plant, returned to the customer, etc.

[0110] As described above, in at least one embodiment, the data processing system serving the source host from which a logical partition is migrated may include a system architecture, system memory, a memory controller communicatively coupled to the system memory, and a link controller communicatively coupled to the system architecture and configured to communicatively couple to a destination host unrelated to the source host via a communication link. The source host further includes multiple processing units communicatively coupled to the system architecture. These processing units, for example, are configured by a hypervisor or virtual machine monitor to execute logical partitions having datasets residing in system memory and to migrate the logical partitions to the destination host via a communication link. The migration of the logical partitions includes migrating the dataset of the logical partitions executed on the source host from the source host's system memory to the destination host's system memory via a communication link. After migrating at least a portion of the dataset, the state of the logical partitions is migrated from the source host to the destination host via a communication link, such that the logical partitions are subsequently executed on the destination host. This migration technique provides reduced latency for the migration and improved jitter for the logical partitions.

[0111] In the previously described example, the source host uses a real address to specify the storage location in the system memory of the destination host to which the memory page will be copied. Those skilled in the art will understand that in other examples, the storage location in the system memory of the destination host may instead be specified using valid or virtual addresses, which undergo one or more layers of address translation to obtain a real address in the destination host's real address space.

[0112] Although various embodiments have been specifically shown and described, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the scope of the appended claims, and that all such alternative implementations fall within the scope of the appended claims.

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions mentioned in the blocks may not occur in the order indicated in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0114] Although aspects have been described with respect to computer systems that execute program code guiding the functions of the present invention, it should be understood that the present invention can alternatively be implemented as a program product including a computer-readable storage device storing program code, which can be processed by a processor of a data processing system to cause the data processing system to perform the described functions. The computer-readable storage device may include volatile or non-volatile memory, optical disks or magnetic disks, etc., but excludes non-statutory subjects such as the propagation of signals themselves, the transmission medium itself, and the form of energy itself.

[0115] As an example, the program product may include data and / or instructions that, when executed on a data processing system or otherwise processed, generate a logically, structurally, or otherwise functionally equivalent representation (including a simulation model) of the hardware components, circuits, devices, or systems disclosed herein. Such data and / or instructions may include hardware description language (HDL) design entities or other data structures that conform to and / or are compatible with low-level HDL design languages ​​such as Verilog and VHDL and / or high-level design languages ​​such as C or C++. Furthermore, the data and / or instructions may also employ data formats and / or symbolic data formats used for exchanging layout data of integrated circuits (e.g., information stored in GDSII (GDS2), GL1, OASIS, mapping files, or any other suitable format used to store such design data structures).

Claims

1. A method for performing data processing in a data processing environment, the data processing environment comprising a first data processing system and a second data processing system, wherein the first data processing system is a source host and the second data processing system is a destination host, the method comprising: One or more hardware threads of a logical partition are executed on one or more processor cores of the source host, wherein each of the one or more hardware threads has a corresponding thread state including the contents of registers within the processor core executing the corresponding hardware thread. The source host uses communication via a network connection to coordinate the migration of the logical partition from the source host to the destination host; The dataset of the logical partition is dynamically migrated from the system memory of the source host to the system memory of the destination host, which is incoherent with respect to the source host, via a link controller in the source host and a communication link coupling the source host and the destination host, while one or more hardware threads of the logical partition are running on the source host, wherein performing the dynamic migration includes disabling the use of the network connection to migrate the dataset from the source host to the destination host. After migrating at least a portion of the dataset, the state of the logical partition is migrated from the source host to the destination host via the link controller and the communication link, wherein the state of the logical partition includes the corresponding thread state of each of the one or more hardware threads of the logical partition; as well as After the state of the logical partition is migrated to the destination host, the one or more hardware threads of the logical partition are executed on the destination host.

2. The method of claim 1, wherein performing dynamic migration of the dataset comprises migrating at least some of the dataset of the logical partition from the source host to the destination host after the one or more hardware threads of the logical partition begin execution on the destination host.

3. The method of claim 1, wherein performing dynamic migration of the dataset includes the source host copying one or more memory pages of the dataset from system memory in the source host to system memory in the destination host based on a request from the destination host.

4. The method of claim 1, wherein performing the dynamic migration of the dataset includes the source host initiating a copy of one or more memory pages of the dataset from system memory in the source host to system memory in the destination host.

5. The method according to claim 1, wherein: The dataset includes multiple memory pages, each memory page having a corresponding associated page table entry that defines the virtual-to-real address translation for the associated memory page; as well as The method also includes the source host tracking in a data structure which page table entries of the plurality of memory pages have been migrated to the destination host.

6. The method according to claim 1, wherein, The source host includes a system architecture and a link controller that is communicatively coupled to the system architecture; and Performing dynamic migration of the dataset includes: The source host issues a storage request on the system architecture, and the storage request specifies an address in the system memory of the destination host. as well as The link controller in the source host receives the storage request from the system architecture and, based on the address, transmits the storage request to the memory controller in the destination host via the communication link.

7. The method of claim 1, wherein performing dynamic migration of the dataset includes the source host performing a batch flush of all memory pages in the dataset that have undergone updates by the source host from the source host's cache to system memory.

8. A data processing system serving as a source host, the data processing system comprising: System structure; System memory; The system memory has a memory controller, wherein the memory controller is communicatively coupled to the system architecture and the system memory; A link controller, communicatively coupled to the system architecture and configured to be communicatively coupled via a communication link to a destination host of a data processing system independent of the source host, wherein the destination host is incoherent relative to the source host; as well as Multiple processor cores are communicatively coupled to the system architecture, wherein each of the multiple processor cores is configured to execute one or more hardware threads of a logical partition having a dataset residing in the system memory, and to perform dynamic migration of the logical partition to the destination host via the link controller and the communication link while the one or more hardware threads of the logical partition are executed on the source host: The migration of the logical partition from the source host to the destination host is coordinated using communication via a network connection; The dataset in the logical partition is migrated from the system memory of the source host to the system memory of the destination host via the link controller and the communication link, while the one or more hardware threads of the logical partition are executed on the source host, wherein migrating the dataset includes disabling the use of the network connection to migrate the dataset; After migrating at least a portion of the dataset, the state of the logical partition is migrated from the source host to the destination host via the link controller and the communication link, such that the one or more hardware threads of the logical partition are subsequently executed on the destination host, wherein the state of the logical partition includes the corresponding thread state of each of the one or more hardware threads of the logical partition.

9. The data processing system of claim 8, wherein migrating the dataset comprises migrating at least some of the dataset of the logical partition from the source host to the destination host after the execution of the one or more hardware threads of the logical partition begins on the destination host.

10. The data processing system of claim 8, wherein migrating the dataset includes the source host copying one or more memory pages of the dataset from system memory in the source host to system memory in the destination host based on a request from the destination host.

11. The data processing system of claim 8, wherein migrating the dataset includes the source host initiating a copy of one or more memory pages of the dataset from system memory in the source host to system memory in the destination host.

12. The data processing system according to claim 8, wherein: The dataset comprises multiple memory pages, each memory page having a corresponding associated page table entry, which defines the virtual-to-real address translation for the associated memory page; and The plurality of processing units are also configured to execute: The data structure tracks which page table entries of the multiple memory pages have been migrated to the destination host.

13. The data processing system of claim 8, wherein migrating the dataset comprises: The source host issues a storage request on the system architecture of the source host, the storage request specifying an address in the system memory of the destination host; as well as The link controller of the source host receives the storage request and, based on the address, transmits the storage request to the memory controller in the destination host via the communication link.

14. The data processing system of claim 8, wherein migrating the dataset includes the source host performing a batch flush of all memory pages in the dataset that have undergone updates by the source host from the source host's cache to system memory.

15. A program product comprising: The program code, when executed by a first data processing system acting as the source host, causes the first data processing system to perform a dynamic migration of a logical partition to a second data processing system acting as the destination host, wherein the dynamic migration of the logical partition includes: One or more hardware threads of a logical partition are executed on one or more processor cores of the source host, each of the one or more hardware threads having a corresponding thread state; The migration of the logical partition from the source host to the destination host is coordinated using communication via a network connection; The dataset of the logical partition is dynamically migrated from the system memory of the source host to the system memory of the destination host, which is incoherent with respect to the source host, via a link controller in the source host and a communication link coupling the source host and the destination host, while one or more hardware threads of the logical partition are running on the source host, wherein performing the dynamic migration includes disabling the use of the network connection to migrate the dataset from the source host to the destination host. After migrating at least a portion of the dataset, the state of the logical partition is migrated from the source host to the destination host via the link controller and the communication link, wherein the state of the logical partition includes the corresponding thread state of each of the one or more hardware threads of the logical partition; and After the state of the logical partition is migrated to the destination host, the one or more hardware threads of the logical partition are executed on the destination host.

16. The program product of claim 15, wherein performing the dynamic migration of the dataset comprises migrating at least some of the dataset of the logical partition from the source host to the destination host after the one or more hardware threads of the logical partition begin execution on the destination host.

17. The program product of claim 15, wherein performing dynamic migration of the dataset includes the source host copying one or more memory pages of the dataset from system memory in the source host to system memory in the destination host based on a request from the destination host.

18. The program product of claim 15, wherein performing the dynamic migration of the dataset includes the source host initiating a copy of one or more memory pages of the dataset from system memory in the source host to system memory in the destination host.

19. The program product according to claim 15, wherein: The dataset includes multiple memory pages, each memory page having a corresponding associated page table entry that defines the virtual-to-real address translation for the associated memory page; as well as The program code also enables the source host to track in a data structure which page table entries of the plurality of memory pages have been migrated to the destination host.

20. The program product of claim 15, wherein performing dynamic migration of the dataset comprises: The source host issues a storage request on its system architecture, and the storage request specifies an address in the system memory of the destination host. as well as The link controller in the source host, which is communicatively coupled to the system architecture, receives the storage request from the system architecture and, based on the address, transmits the storage request to the memory controller in the destination host via the communication link.

21. The program product of claim 15, wherein performing the dynamic migration of the dataset includes the source host performing a batch flush of all memory pages in the dataset that have undergone updates by the source host from the source host's cache to system memory.

Images