Optimizing replication-based virtual machine migration

Abstract

Methods, apparatuses, and systems for migrating a virtual machine (VM) from a source to a target in a computing environment, and more specifically, for migration-based replication are disclosed herein. To meet user-defined or system constraints, a VM is controlled to manage the transfer of data from a source location to a target location in relation to one or more VMs. Dynamic control and adjustment of system parameters related to migration are also disclosed herein.

Description

[0001] Cross-reference to related applications

[0002] This application is a continuation of U.S. Patent Application No. 16 / 934,433, filed July 21, 2020, which is incorporated herein by reference. Background Technology

[0003] Cloud computing involves providing on-demand computing services via networks, such as wide area networks (WANs) and / or one or more local area networks (LANs) of the Internet. Cloud computing provides information technology infrastructure in the form of one or more sets of network elements. These network elements typically provide computing power and data storage to remote users over the network and are generally located in data centers. Machine virtualization is commonly used in cloud computing. A typical virtual computing system consists of multiple computing nodes interconnected by a high-speed network. The computing nodes run virtual machines (VMs) that consume physical resources, such as central processing units (CPUs), memory, persistent storage, and network resources. These physical resources may reside on one or more network elements, also known as hosts. Summary of the Invention

[0004] This disclosure provides techniques, apparatus, and systems for VM migration. As an example, the techniques include replication-based VM migration.

[0005] One aspect of this disclosure includes a method for migrating computing resources over a network, the computing resources being associated with one or more virtual machines. The method may include: determining one or more migration parameters based on one or more performance metrics; and, based on the determined one or more migration parameters, receiving performance metrics associated with at least one of the one or more virtual machines via at least one computing device connected to the network to control the migration process, thereby controlling the timing of at least one of memory snapshots, data transfers, and switch requests: the switch requests being associated with migrating the computing resources associated with the one or more virtual machines.

[0006] According to this aspect of the technology, the computing resources may include a first disk having storage areas associated with memory locations, and the control migration process includes generating at least one command that causes data located at a first memory location on the first disk to be transferred to a second memory location on a second disk, the data being associated with a first virtual machine of the one or more virtual machines. Alternatively, the control migration process may also include issuing at least one command that causes a second virtual machine to process data stored at the second memory location. Furthermore, the at least one command may be issued based on a threshold.

[0007] The control migration process may also include the at least one computing device generating at least one command that causes data located in a first memory location to be transferred to a second memory location, the data being associated with a first virtual machine of the one or more virtual machines.

[0008] This aspect of the technology also includes receiving migration requests that identify the one or more virtual machines associated with the computing resources.

[0009] For example, migration parameters may include at least one of the following: the time to perform a memory snapshot, the time to migrate computing resources associated with a given virtual machine on the storage medium, and the time to switch over the instantiation of the given virtual machine.

[0010] Additionally, this aspect of the technology may include outputting at least one of the following: estimated migration completion time, percentage of data volume migrated, non-convergent data, and threshold adjustment selection.

[0011] Performance metrics may include one or more of the following: snapshot data, compressibility of the compute resources to be migrated, available bandwidth of the network, CBT, and variance of each of the one or more virtual machines. Performance metrics may also include one or more of the following: disk size, disk rewrite rate, switch threshold, snapshot duration, and network speed.

[0012] Another aspect of this technology may include a computing system for managing the migration of computing resources associated with virtual machines over a network. The computing system may include: one or more computing devices; one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media, executable by at least one of the one or more computing devices. The program instructions cause the at least one computing device to: determine one or more migration parameters based on the one or more performance metrics; and control the migration process based on the determined one or more migration parameters to control the timing of at least one of memory snapshots, data transfers, and one or more switching requests related to the migration of the computing resources associated with the one or more virtual machines.

[0013] Another aspect of the technology includes a communication interface connected to the at least one computing device and a network, the communication interface receiving the one or more performance metrics related to the one or more virtual machines via the network.

[0014] In this aspect of the technology, the control migration process includes program instructions that cause the at least one computing device to generate at least one command that causes a second virtual machine to process data stored in the second memory location.

[0015] Another aspect of the technology may include a non-transitory computer-readable medium containing program instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform the following steps: determining one or more migration parameters based on one or more performance metrics; and controlling a migration process based on the determined one or more migration parameters to control the timing of at least one of memory snapshots, data transfers, and switching requests, the switching requests being related to the migration of computing resources associated with the one or more virtual machines. Attached Figure Description

[0016] Figure 1 It is a functional diagram of an example system based on various aspects of this disclosure.

[0017] Figure 2 Example systems based on various aspects of this disclosure are shown.

[0018] Figure 3 It is a functional diagram of an example system based on various aspects of this disclosure.

[0019] Figure 4 This is a functional block diagram of an example system based on various aspects of this disclosure.

[0020] Figure 5 This is a functional block diagram of an example system based on various aspects of this disclosure.

[0021] Figure 6 It is a flowchart based on various aspects of this disclosure.

[0022] Figure 7 It is a functional block diagram illustrating illustrative use cases of methods or systems according to various aspects of this disclosure.

[0023] Figure 8 It is a functional block diagram illustrating illustrative use cases of methods or systems according to various aspects of this disclosure.

[0024] Figure 9 It is a functional block diagram illustrating illustrative use cases of methods or systems according to various aspects of this disclosure.

[0025] Figure 10 It is a functional block diagram illustrating illustrative use cases of methods or systems according to various aspects of this disclosure.

[0026] Figure 11 It is a functional block diagram illustrating illustrative use cases of methods or systems according to various aspects of this disclosure.

[0027] Figure 12 It is a functional block diagram illustrating illustrative use cases of methods or systems according to various aspects of this disclosure. Detailed Implementation

[0028] Overview

[0029] This technology involves migrating virtual machines (VMs) between a source and a target in a computing environment, and more specifically, replication-based migration. Migration, or VM migration, generally refers to the act or operation of copying and / or moving a VM from a source location (virtual or physical) to a target location (virtual or physical) in a computing environment. Replication-based migration typically involves migrating a VM by periodically taking snapshots of the VM in the source or of data associated with the VM (potentially while the VM is running in the source) and copying or pasting these snapshots (or changes / amounts of change between snapshots) to the target. Factors or metrics typically considered when migrating a VM include: the size of the disk containing the VM's associated data, the disk rewrite rate (e.g., how quickly a given source disk of the VM is rewritten), the cut-over threshold (typically a value used to determine when a compute operation switches to the target VM), the amount of data to be transferred, and / or the available bandwidth between the source and the target, as well as other metrics discussed in detail below. Higher rewrite rates and strict cut-over thresholds often result in the need to take more snapshots. Increasing the number of snapshots that need to be taken and migrated before the switch-over operation is ready can cause the migration wave to never be ready for the switch, i.e., the migration does not converge.

[0030] In one instance, one aspect of the disclosed technology includes a process or program running on a computing device that can determine parameters associated with a migration event, monitor and determine the status of the migration event based on various metrics including those discussed above, and adjust the migration process to meet customer needs (e.g., specific VMs must complete migration around the same time) and reduce non-convergence. For example, the process may receive user input information confirming a large-scale migration event (e.g., migration of multiple VMs) and a switching threshold. The process then determines the disk size and rewrite rate of all VMs confirmed for migration. This determination may involve polling the source disks in the migration to determine their size and rewrite rate. For a given disk size and rewrite rate, the process then determines the timing at which each VM involved in the migration should be started. As part of this determination, the process considers the switching threshold. For example, the process may determine a given desired switching threshold, disk size, rewrite rate, the possibility that a switch may never occur, or that a switch will not occur within a time frame requested by the end user.

[0031] In another example, the process described above shortens the time it takes for the entire migration wave to reach the "ready to switch" state by reducing the total amount of data that needs to be transferred. For instance, to estimate / determine the timing of how the frequent snapshotting mechanism of replication-based migration might reduce the amount of data that needs to be transferred from the source to the destination, the process can dynamically calculate the timing of initiating each VM migration in the current large-scale migration wave based on different metrics and real-time tests. Disk size, rewrite rate, and available bandwidth can be fed back to the process, which can then dynamically adjust the timing of VM snapshots within a given migration wave.

[0032] In another instance, the disclosed technology is executed in a backend system running the aforementioned processes. This system acts as an interface between the end user and the VMs provided to the user through a cloud computing environment. The system receives migration requests from the user, executes the migration requests according to the user's needs, monitors the migration, provides feedback to the user during the migration, and / or adjusts migration settings to achieve the migration and ultimately switchover. For example, the system can use the user to provide the VM's identifier and a switching threshold as input. The system then provides the user with one or more of the following as outputs: estimated convergence time, percentage related to the integrity of the migration wave, probability of non-convergence, options to adjust the switching threshold, switching readiness status, and estimated switching downtime.

[0033] In another example, the system can automatically manage the entire process, including determining one or more appropriate switching thresholds for the identified VMs and initiating the switch when the thresholds are met. In this example, the user can specify the target VM and the required switch time, and the system will then determine all appropriate settings and automatically execute the switch.

[0034] In another instance, as part of the migration process, the system also dynamically adjusts parameters associated with migration waves or events. For a given set of VMs as migration targets, the system can determine or be provided with an initial switching threshold. The system can then detect performance metrics, such as disk size, disk rewrite rate, data transfer rate, data compression, etc. These detected metrics can then be used to tune or adjust one or more migration parameters, such as the timing of data transfers associated with one or more target VMs, the timing of taking one or more snapshots, or the timing of initiating a switch. The initial switching threshold can also be tuned or controlled based on the detected metrics. For example, if the metrics indicate that the initial switching threshold will not result in a timely switch, for example, to meet end-user requests or due to other events such as data center maintenance events, the initial switching threshold can be adjusted or tuned. By detecting metrics and using them to control migration, the system can monitor and control a migration event or wave when it occurs. Once the switching threshold is met, optionally, the system can control the actual switch by issuing one or more commands that trigger the actual switch.

[0035] Examples of systems and / or environments

[0036] Figure 1This is an example system 100 according to various aspects of this disclosure. System 100 includes user nodes 106, agent nodes 120, backend systems 130, and cloud computing systems 150, 160, and 170, which can communicate via a local area network (LAN) and / or a wide area network (WAN). User node 106 includes computing device 110, which is connected to agent node 120 via one or more LANs and / or WANs. User node 106 may include an enterprise utilizing cloud computing services such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and / or Software as a Service (SaaS). For example, if an enterprise is operating the system, user node 106 can use cloud systems 150, 160, or 170 as a service to provide software applications, such as accounting, word processing, inventory tracking, etc., to computing device 110. Optionally, user node 106 may include a collection of one or more computing devices 110 that access services provided through cloud computing systems 150, 160, or 170, such as web-based applications like email, banking, and video. Therefore, computing devices may include a variety of client devices, such as laptops, desktops, smartphones, or any computing device containing data processing hardware, memory, and storage. Computing devices 110 are connected to network 114 via wireless (e.g., WiFi or cellular) or wired connections (e.g., Ethernet).

[0037] User node 106 can be coupled to backend system 130 via proxy node 120. Proxy node 120 typically includes one or more proxy servers, which act as a gateway between user node 106 and backend system 130. Proxy node 120 is coupled to backend system 130 via one or more local area networks (LANs) and / or wide area networks (WANs). Proxy node 120 is responsible for handling data traffic between user node 106 and backend system 130.

[0038] Backend system 130 includes one or more processing devices 132, memory 134, and RAM 136. Memory 134 may store instructions used by processing devices 132 to control data migration within and between cloud systems 150, 160, and 170, or between user nodes 106, for example, in an enterprise operation within one or more of cloud systems 150, 160, and 170. Backend system 130 is considered a migration controller that interacts with end-user nodes 106 and cloud computing systems 150, 160, or 170 to manage and control migration events. For example, backend system 130 may receive migration requests from end-user nodes 106 belonging to specific VMs running in one or more of cloud computing systems 150, 160, or 170. System 130 may then request information from the relevant cloud systems and issue commands based on the information received from these systems to control migration-related timing, switching, etc. In some cases, backend system 130 may calculate output values ​​that optimize the data migration process, such as the start time of different data migration events. System 130 can also provide feedback on the migration status to user node 106, or provide options to influence the migration process.

[0039] Apart from Figure 1 In addition to the embodiment of backend system 130 shown, another option is for backend system 130 to send the data it collects (e.g., data layer data as VM disk data, or control layer data as metrics, rewrite rate, etc.) to a cloud system that manages the migration process, decides how to process the information from the source / backend, and how to process it. In this regard, for migration control and user-facing information, as much logic as possible is needed in the cloud system, while allowing the backend system (e.g., source worker / source agent) to do minimal work (e.g., reading source VM disks, taking source snapshots, enabling CBT on the source VM, etc.), and making the backend system as seamless and frictionless as possible with the customer. Generally speaking, the functions performed by backend system 130 can be distributed across, such as... Figure 1 The system 130 shown is connected to the components in the cloud. These functionalities can also be fully implemented within a given cloud.

[0040] Cloud computing systems 150, 160, and 170 may include one or more data centers connected via high-speed communication or computing networks. A given data center within any of systems 150, 160, and 170 may include dedicated space within a building for housing computing systems and their associated components, such as storage and communication systems. Typically, a data center will include communication equipment racks, servers / hosts, and disks. Servers / hosts and disks comprise physical computing resources used to provide virtual computing resources, such as VMs. To some extent, a given cloud computing system may contain more than one data center, which will typically be located in different geographical locations. These data centers are selected to provide timely and cost-effective service while offering redundancy and maintaining high availability. Similarly, different cloud computing systems are often provided in different geographical locations.

[0041] Figure 2 This is a network diagram of example system 200, which includes multiple computing devices 210, 220, 230, and 240 and a storage system 250 connected via network 260. Figure 2 It shows Figure 1 The example shown is a system view centered on the communication network.

[0042] like Figure 2 As shown, each of computing devices 210, 220, 230, and 240 includes one or more processors, memory, data, and instructions. For example, server computing device 210 includes one or more processors 272, memory 274, instructions 276, and data 278.

[0043] Memory 274 stores one or more pieces of information accessible to processor 272, including instructions 276 and data 278 that can be executed or used by processor 272. Memory 274 can be any type capable of storing processor-accessible information, including computer-readable media or other media for storing data, which can be read by means of electronic devices such as hard disks, memory cards, ROM, RAM, DVDs or other optical discs, and other writable and read-only memories. Systems and methods include different combinations of the above, wherein different portions of instructions and data are stored on different types of media.

[0044] Instruction 276 can be any set of instructions that can be executed directly by the processor (such as machine code) or indirectly (such as a script). For example, instructions can be stored as computer code on a computer-readable medium. In this regard, the terms "instruction" and "program" are used interchangeably. Instructions can be stored in object code format for direct processor processing, or in any other computer language, including scripts or collections of standalone source code modules that can be interpreted on demand or pre-compiled. The procedures, functions, methods, and routines of these instructions are explained in more detail below.

[0045] According to instruction 276, data 278 can be retrieved, stored, or modified by processor 272. For example, data 278 associated with memory 274 may contain data for supporting services of one or more client devices, such as 220, 230, or more. This data may include support for hosted network-based applications, file-sharing services, communication services, games, video or audio file sharing, or any other network-based services.

[0046] One or more processors 272 can be any conventional processor, such as a commercially available CPU. Optionally, one or more processors can be special-purpose devices, such as ASICs or other hardware-based processors. Although Figure 2 The processor, memory, and other elements of computing device 210 are functionally described within the same box, but those skilled in the art will understand that a processor, computing device, or memory may actually include multiple processors, computing devices, or memory that is not located in or stored in the same physical container. In one example, one or more computing devices 210 may include one or more server computing devices, which may include multiple computing devices, such as a load-balancing server cluster that exchanges information with different nodes in a network to receive, process, and transmit data sent to or from other computing devices.

[0047] Computing device 210 can function as a server, and computing device 120 includes... Figure 1 The proxy server 120 is mentioned. Computing device 210 also represents a cloud computing environment, which includes multiple computing devices acting as host devices.

[0048] Network 260 and intermediate nodes can include a variety of configurations and protocols, including short-range communication protocols such as Bluetooth™, Bluetooth Low Energy™, the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local area networks, private networks using one or more proprietary communication protocols, Ethernet, WiFi, HTTP, and various combinations thereof. Any device capable of transmitting data to or from other computing devices, such as modems and wireless interfaces, can facilitate this communication.

[0049] like Figure 2 As shown, each client computing device 220, 230 can be a personal computing device for user use, and the client computing device includes all components typically used in relation to a personal computing device, including: one or more processors (e.g., a central processing unit (CPU)), memory for storing data and instructions (e.g., RAM and internal hard disk drives), displays such as displays 224, 234, and 244 (e.g., monitors with screens, touchscreens, projectors, televisions, or other devices used to display information), speakers, and user input devices 226, 236, and 246 (e.g., mice, keyboards, touchscreens, or microphones). The client computing device may also include cameras for recording video streams, speakers, network interface devices, and all components that interconnect these elements.

[0050] Computing device 240 can also be a workstation associated with network devices, such as one or more server computing devices 210, and computing device 240 is used for management and / or maintenance, including as a backup system. In this regard, computing device 240 can be operated by a network administrator who supports, maintains, and manages security related to server computing device 210. Regarding maintenance, computing device 240 allows the network administrator to access server information, such as the current version of software running on the server, installed updates, patches, etc. Computing device 240 can also be used to access information about storage system 250, such as disk size, rewrite rate, etc. Similarly, although... Figure 2 Only a single backend system 240 is shown, but the system can actually include many such workstations, or many backend systems.

[0051] While each of client computing devices 220 and 230 may include a full-size personal computing device, optionally, they may include mobile computing devices capable of wirelessly exchanging data with server computing devices (such as server computing device 210) via a network, such as the Internet. By way of example only, client computing device 220 may be a mobile phone or device, such as a wireless-enabled PDA, tablet, wearable computing device or system, or a computer capable of accessing information via the Internet or other networks. In another example, client computing device 230 may be a wearable computing system, such as a smartwatch. For example, a user may use a keypad, buttons, a microphone, a camera utilizing visual signals, or a touchscreen to input information.

[0052] Similar to memory 274, storage system 250 can be any type of computerized memory capable of storing information accessible to server computing device 210, such as hard disk drives, disks, memory cards, ROM, RAM, DVDs, CD-ROMs, writable memory, and read-only memory. Furthermore, storage system 250 can include a distributed storage system in which data is stored on multiple different storage devices, which may be physically located in the same or different geographical locations. Storage system 250 can be connected to computing devices via network 260 as shown, and / or directly connected to or incorporated into any of computing devices 210, 220, 230, etc. Storage system 250 can store various types of information as described in detail below. To perform some or all of the functions described herein, this information is retrieved or otherwise accessed by one or more server computing devices, such as device 210, and / or one or more client computing devices, such as client computing device 220.

[0053] Figure 3 This is a functional block diagram of system 300 according to various aspects of this disclosure. System 300 includes computing hardware 310 configured to function as one or more virtual machines (VM1 to VMn) 330 communicating with disk 350. Note that... Figure 3 The diagram shows a single disk 350 for simplicity. However, system 300 may include multiple disks 350. Computing hardware 310 includes processing device 304, volatile memory 308, and non-volatile memory 312. Processing device 304 can execute programming instructions stored in memory 308, 312. Computing hardware 310 may include a server or host. Disk 350 is a partitionable storage device that provides physical or virtual storage for virtual machine 330 to provide computing services.

[0054] System 300 is an example of a virtual machine (VM), which can be implemented in cloud computing systems 150, 160, and 170 and resides in a data center on computing hardware or host 310. Typically, a data center will include a large number of computing hardware 310s and disks 350 to support the same or even more number of VMs. As mentioned above, a VM essentially comprises a computer that does not run on its own dedicated hardware, but is hosted on hardware capable of running multiple VMs. VM1 through VMn are shown as dashed lines, indicating that they are essentially virtualized on host 310.

[0055] Figure 4 This is a functional block diagram of an example system 400 that displays data streams according to various aspects of this disclosure. System 400 includes multiple virtual machines, VM1 to VMk and VMq to VMy. Figure 4Multiple storage disks or volumes (“D / V”), D / V1 to D / Vk and D / V100 to D / Vm are also shown. A disk / volume typically refers to a storage device / partition (physical or virtual) attached to a VM. Disks / volumes include persistent storage (i.e., non-volatile), but can also include cases where the storage device is volatile (e.g., RAM).

[0056] VM1 is associated with D / V1 and uses D / V1 to retrieve data for processing or storage. VM1 and D / V1, other VMs and other D / Vs, up to VMk and D / Vk, all reside in the first computing environment 410. Although Figure 4 (And other diagrams) imply a 1:1 relationship between VMs and D / Vs, but this usage example is shown in the diagrams for simplification. It is common to associate multiple D / Vs with a given VM. First, computing environment 410 can include cloud computing systems, such as any of 150, 160, etc., data centers within cloud computing systems, or physical hardware within data centers, such as computing hardware 310, which can also be considered data center hardware.

[0057] Figure 4 The diagram illustrates the operation of migrating a virtual machine (VM) or VM-related data from a first environment 410 (source) to a second environment 450 (destination). Migration can include moving / copying a VM from on-premises to the cloud, or from one cloud to another. Migration also includes movement / copying operations between projects and / or regions, or even within a data center. Migration typically requires at least the transfer of data from the VM's disks from the source to the destination. Note that a single virtual machine has more than one disk (and typically at least one). On-premises refers to the virtual machine running on hardware maintained by an entity such as an individual or organization. Figure 4 In the example, the second environment 450 can be a different cloud computing system, a different data center, or different hardware within a data center than the first environment 410. As shown, data is copied from D / V1 to, for example, D / Vm. The copied or moved data may include code or instructions used by the VM, data used by the aforementioned code or instructions as input data, or data provided by the VM as output data.

[0058] Figure 5 yes Figure 4The diagram shows the functional block diagram of the system after the migrated VM is switched to the second environment 450. As shown, data in D / V1 has been copied and / or moved to D / Vm, and data in D / Vk has been copied and / or moved to D / V100. Furthermore, the VM in the second environment 450 is now associated with D / V100 and D / Vm to perform tasks previously executed in the first environment 410. A switchover may be initiated when data is copied or moved between different computing environments. Further options include running the source VM in parallel with the target VM—whether the target VM is a "test clone" or a VM product—in which case the switchover operation means the target VM is now the "master" VM, not its respective source VM, thus creating a conceptual separation between the two, and each can modify its local data differently. Additionally, at some point, such copying or moving will end the migration of the current snapshot to complete the migration process, although another snapshot will be captured and transferred when the switchover begins, possibly in parallel with the currently migrating snapshot; therefore, the target has information about the latest state. In addition, switching may cancel the current snapshot migration and may start another different snapshot migration.

[0059] Example Method

[0060] Figure 6 The process flowchart 600 is provided according to various aspects of this disclosure. As shown, the process mainly includes three processing blocks or steps: in block 605, receiving a migration request; in block 625, determining migration and / or switching parameters; and in block 645, performing process control. As shown in block 665, process 600 may also include a step of outputting information to a user or another system. Process 600 can be implemented using backend systems 130 or 240, or as a combination of a cloud VM "controller" (which is, for example, part of the target product) and a backend system running at the source. Process 600 includes a process for controlling migration and switching, which can be performed dynamically and automatically, as will be described in detail below.

[0061] At block 605, for example, a migration request is received at system 130 or 240. This migration request may include input generated by user node 106. The migration request may also be generated automatically. For example, backend system 130 may monitor the performance of cloud computing resources on a cloud computing system or data center. Monitoring may include, for example, latency related to receiving and processing data streams, or the time end-user requests for computing resources remain in the queue, whether the cloud computing environment bandwidth is within threshold ranges such as capacity, disk size, disk rewrite rate, and CPU utilization, and other information related to performance metrics including data collected from any involved computers to provide cloud computing services.

[0062] The migration request described above can also identify one or more VMs to be migrated and a switching threshold. The switching threshold includes a threshold that determines when a switching operation can be performed. For example, this threshold could define how much of the state of a given VM should be preserved so that it is copied at most once before a switching operation is allowed. The stricter the threshold, the more VM snapshots are required and transferred from the source to the destination in order to perform the switching operation. A VM snapshot refers to the complete state of the VM, all of the VM's disks, read from a past point in time regardless of the VM's current state. A VM snapshot can include one or more disk snapshots. A disk snapshot includes the complete state of the disk, read from a past point in time regardless of the disk's current state, and will be read by the migration process as this state is transferred from the source to the destination over the network. All disk snapshots that make up a VM snapshot are from a given point in time (and preferably the same point in time). Therefore, disk snapshots provide a complete and consistent state of the VM.

[0063] For example, regarding the switching threshold, suppose a VM has a 100GB disk, the disk is rewritten at a rate of 1GB / hour (i.e., the rewrite speed), and migrating this VM would involve using a 1Gbps (i.e., 1 / 8Gbps) Ethernet bus. Migrating the first snapshot of this VM would take approximately 100GB / (1 / 8GBps) = 800 seconds = ~13.33 minutes = ~0.22 hours. During this time, the disk on the source is rewritten at a rate of 1GB / hour, so when the first snapshot of the VM is migrated, approximately 0.22GB of disk space will be rewritten (1GB / hour * ~0.22 hours), which is 0.22% of the disk space (0.22GB / 100GB). If the switching threshold is defined as 1% of the disk size, then once the first snapshot of the VM is migrated, the switching operation can proceed because the remaining work (0.22%) is less than the threshold (1%).

[0064] Furthermore, the system calculates the timing beforehand, which allows us to: issue a switch operation that shuts down the source VM, capture the final source snapshot, migrate the final source snapshot in parallel with the currently migrating [LAST-1] snapshot, and potentially, for example, utilize unused resources required for the migration. If we wait for the [LAST-1] cycle to complete before issuing the switch operation, the entire process (migrating two parallel snapshots) will finish faster than this. In other words, timing calculations can take into account, for example, switch thresholds and rewrite rates.

[0065] On the other hand, if the threshold is 0.1% of the disk size, then another snapshot of the VM needs to be captured and transferred before preparing for the switchover operation. Note that transferring the second snapshot (i.e., the amount of change between it and the first snapshot) will take ~0.22GB / (1 / 8GBps) = 1.78 seconds, and during this time, 1.78 seconds * 1GB / hour = 0.5MB of the source disk will be overwritten. Since 0.5MB << 0.1% * 100GB = 102.4MB, there is no need to capture a third snapshot of the VM in preparation for the switchover operation.

[0066] Typically, the switching threshold is not necessarily based on a percentage of the disk / VM snapshot size. For example, the switching threshold could be a constant 1GB. The switching threshold can also be the result of any algorithmic calculation, which in turn can be based on any data, such as, but not limited to, the size of the disk / VM snapshot, available network bandwidth, different disk rewrite rates, the type of guest operating system installed on the VM, and other data belonging to the VM and / or the migration environment. The threshold is not necessarily based on the amount of data remaining to be transferred, but may also be based on other factors that may be considered as determining factors. Additionally, the threshold can differ for different VMs. Note that different switching thresholds can be defined for each VM, or a single switching threshold can be defined for the entire system, or any other combination of these options is collectively referred to as the switching thresholds discussed below. Another option for combining switching thresholds could be, for example, defining a common threshold for all VMs and allocating a (potentially different) budget to each VM, for example, defining the threshold as having at most 100GB remaining for transfer during a switching operation, based on the overall size of the VM disks distributed evenly among the VMs.

[0067] Taking into account the identity of the VM and the switching threshold, the process proceeds to block 625. In block 625, migration and / or switching parameters are determined. These parameters may include: (i) the time(s) or timing of migrating compute resources in memory associated with one or more given virtual machines; (ii) the time or timing of switching one or more given virtual machines; and (iii) the time or timing at which snapshots should be taken and / or executed.

[0068] In block 625, migration and / or switching parameters are determined. This determination process relies on, but is not limited to, one or more of the following system parameters: disk size, rewrite rate, switching threshold, snapshot duration, and network bandwidth. In this regard, considering multiple WAN optimizations that may already be in place, the disk size / snapshot size can be considered as the actual size of the data to be transmitted over the migration network, in order to reduce that size. Such optimizations include, but are not limited to, compacting zero-range operations, compressing data, and utilizing data inference mechanisms.

[0069] Typically, migrations are large-scale migration events, i.e., the act of migrating multiple VMs in parallel. In this context, a migration wave is considered a large-scale migration of a well-defined set of VMs, which may only be a subset of the total number of VMs a user might want to migrate (across different migration waves). Furthermore, replication-based migrations can be viewed as the following behavior: migrating VMs by periodically taking VM snapshots in the source (possibly while it is still running in the source), copying (copying) these snapshots (or the amount of change between them) to the target, and finally performing a "switchover" operation. In this operation, the source VM is shut down, its last snapshot is taken and transferred to the target, and the target VM can then be started in the target with its latest state in the source. Typically, it will be started in the target if requested by the user.

[0070] Determining migration timing or sequence can include determining the start time for a given VM migration. For example, if a user has two VMs—VM1 and VM2—and VM1 has a 1TB disk while VM2 has only a 1GB disk, and the user wants both VMs to start on the target almost simultaneously, then if both VMs are running in the source during the migration process (which is typically the case), snapshots of both VMs are periodically taken and copied to the target. If the VMs are running in the source, they may modify the contents of their disks. Because the source VMs are constantly running and changing their data, further snapshots are needed to minimize switchover downtime. Typically, the state of the target needs to be as close as possible to the state of the source. Otherwise, the switchover operation will take longer than necessary and may result in more downtime.

[0071] Furthermore, if both VMs are allocated similar network bandwidth for their respective migrations, migrating the first snapshot of VM1 will take 1024 times longer than migrating the first snapshot of VM2 (1TB / 1GB = 1024). If the entire contents of VM2's disk are overwritten while VM2's first snapshot is being migrated, this will require creating and migrating a second snapshot of VM2 while VM1's first snapshot is still being migrated. This will add an additional 1GB of data that needs to be transferred over the network from the source to the destination. This process may repeat, for example, an additional 1022 times, while VM1's first snapshot is being migrated.

[0072] However, choosing a better timing to initiate the VM2 migration can avoid migrating redundant data. For example, the first snapshot of VM2 should only be acquired and migrated if approximately 1-2 GB of data remains available for transfer within the 1TB of the first snapshot of VM1. Alternatively, the following should be taken into consideration: available network speed (to allocate different portions to different VM snapshot migrations), time spent acquiring snapshots (potentially considering disk size / expected snapshot size, if the Change Blocks Table (CBT) or Dirty Blocks Table (DBT) functionality is available), disk rewrite rates of the different VMs being migrated together, etc. These examples provide some migration factors to consider when determining migration parameters. Typically, migration factors or metrics are computational and / or network parameters that influence the migration process, including migration timing.

[0073] By taking these factors into account to improve the timing of when to initiate snapshot collection and migration for each VM during a large-scale migration wave, the total amount of data that needs to be transferred from the source to the destination can be reduced (due to fewer snapshots being collected, and, for example, considering disk rewrite rates). This, in turn, reduces the time required to prepare for the switchover operation.

[0074] The parameters defined in Block 625, as specified in Block 645, are used to control VM migration and eventual switchover. This control may be influenced by backend systems 130 and 240, which are connected to compute clouds 150, 160, and 170 or other systems in the data center. These compute clouds or data centers may initiate snapshot requests, migration requests, or switchover requests, or monitor rewrite rates, disk sizes, data transfer rates, or other metrics. More generally, because backend systems 130 and 240 can also monitor this information or receive measurement data affecting migration rates and remaining workloads, they can dynamically control the process and provide feedback to end users. For example, as part of Block 645, if the process appears not to converge, the backend system can determine a switchover threshold that will lead to convergence.

[0075] More specifically, because the migration system monitors various metrics affecting migration rate and remaining work, it can also identify non-convergent processes. For example, if the backend system determines that the total source rewrite rate is fast enough that the effective rewrite rate (e.g., considering data compression before transmission over the WAN) is faster than the available network bandwidth between the source and destination, it may warn or notify the user that a switchover operation needs to be initiated at some point (and how long it may take) even though the switchover operation during downtime requires migrating a large amount of data.

[0076] In this scenario, besides notification, the solution is to adjust the switching threshold to, for example, require at least one target snapshot for each VM in the migration wave. Another example is a very strict definition of the switching threshold (e.g., 1% of the disk size of each VM), such that even after collecting and migrating three snapshots of some source VMs, there remains a 5% change between the latest state of the source and the state of the previous source snapshot, and this 5% change still occurs after collecting and migrating a fourth snapshot; or, more seriously, this change increases with the number of snapshots. In this case, the user can be alerted or notified of the non-convergent process, and it can be suggested that the switching threshold be adjusted accordingly, taking into account the real-time dynamic data being collected, or even that a specific threshold, strict enough to maintain a low switching operation time (within the constraints at hand), can be used.

[0077] As part of block 645, backend systems 130 and 240 can also provide users with a time estimate (ETC) of when the handover threshold will be reached or passed. This estimate is made by considering factors such as: the amount of data remaining to be transferred, the number of snapshots each VM may require (e.g., the VM's rewrite rate), the available network bandwidth for migration, and the handover threshold. This prediction can be updated based on dynamic changes in network throughput, variable rewrite rates, uncertain migration start times, etc.

[0078] Optionally, users can define their own set of constraints, and the system will provide feedback on how switching thresholds, such as ETC, affect the estimated time. For example, users can choose to limit the network bandwidth available for migration to a maximum value, thereby directly increasing the time required to pass the switching threshold (compared to the case without network bandwidth limitation).

[0079] As part of block 645, backend systems 130 and 240 can also be configured to automatically initiate failover operations. For example, the system can be configured to allow users to select an automatic failover operation once a failover threshold is met. Furthermore, users can define a set of constraints that will automatically initiate or start a failover operation when these constraints are met. For example, users can limit network bandwidth to a maximum value, define the failover threshold as at least one snapshot migration per VM, and define failover downtime as no more than 3 hours. These combined constraints can correspondingly define a new threshold (hereafter referred to as the "automatic failover threshold"), which differs from the failover threshold, and an automatic failover operation will be issued when this new automatic failover threshold is met.

[0080] For example, the above choice can be useful when the variance of the rewrite rate is large at different times of the day. This leads to a situation where a switching threshold is set, but after an hour, that threshold criterion is no longer met. For instance, there might be a one-hour window during which a switching operation has been initiated (based on possibly given constraints), but the rewrite rate becomes higher at this time – by the switching threshold's ETA > 0, and / or the switching downtime's ETA is too high compared to some constraints defined in the system. This one-hour window might be during a period when the user is not monitoring the system, and – assuming all constraints are met, the user would prefer to initiate the switching operation at that time.

[0081] When a switchover operation is initiated, the source VM of the migration wave will gracefully shut down, take a final snapshot of the VM, and migrate the snapshot to the target. The VM can then be started in the target if needed. The time required to perform this operation depends directly on the compute and / or network environment (e.g., maximum read rate from the source, maximum network speed, etc.). Furthermore, it depends on the amount of change between the final source snapshot and the previous source snapshot (if any). The greater this change, the more data needs to be transferred during the switchover operation, and therefore the more downtime the user will experience. Downtime can be minimized by making the target state as close as possible to the source state, while the impact of this feature—such as the associated costs, network utilization, etc.—on the target state after the switchover threshold is reached should also be considered in light of user preferences and needs.

[0082] As part of block 645, once the migration wave is ready to switch from the source, source snapshots of the VMs continue to be collected and migrated to the target until the switch operation is initiated, so that the target state and source state are as close as possible, and thus the switch time is kept as short as possible. Alternatively, instead of collecting snapshots between the event that the switch threshold criteria are met and the event that the switch operation is initiated, the system is configured to wait for the start of the switch operation and only collect the final snapshot at that time. This will reduce the amount of data transferred from the source to the target, which is desirable for some users, and can reduce the potential redundancy in collecting and migrating snapshots, which is undesirable for some users. Any choice between these is also possible, for example, continuing to collect additional snapshots of each VM every 24 hours until the switch operation is initiated, or implementing an exponential rollback mechanism, for example, collecting additional snapshots after 1 hour, 2 hours, 4 hours, 8 hours, etc., possibly reaching a limit such as 72 hours. Another approach, for example, is to collect additional snapshots if a VM has at least X data to migrate, and if we want to collect a source snapshot of the VM at that point in time.

[0083] On the other hand, data blocks rewritten at the source (“working set”) are also taken into account. In some cases, this is more important than the rewrite rate itself when deciding whether it is worthwhile to capture and migrate another snapshot for a given VM before initiating a switchover operation. For example, CBT / DBT can be utilized. CBT / DBT represents the data blocks that have changed between two given snapshots (disk snapshots or virtual machine snapshots). This is useful when you want to transfer only the amount of change between the two snapshots from source to destination to reduce the required traffic, so that you don’t have to calculate the amount of change on demand (but it is usually calculated dynamically by the provider of the function), or even transfer the entire updated snapshot. CBT can include the offset of each data block, the size (if it is not constant across all blocks in the CBT), and the data (although it is usually read separately to keep the CBT small and treat the CBT only as metadata).

[0084] If the CBT (Cost-Blocking Buffer) between snapshots of each VM is tracked, the amount of variation and variance between them can be examined. A variance matching score can be assigned based on the differences in the offsets and sizes of the data blocks between, for example, the CBT between snapshots 1 and 2, and the CBT between snapshots 2 and 3. The lower the variance, the less efficient it is to collect more and more snapshots before issuing a switch operation. For example, if there were no variance in the previous example, then the CBT must be transferred once a switch operation is issued, so migrating another similar snapshot before issuing a switch operation might not make sense. Note that the amount of variation between snapshots 1-2 and 2-3 includes the same block offsets, but these blocks contain different data. Therefore, based on the variance of their CBTs, collecting and migrating another snapshot of a different VM would be preferred over issuing a switch operation.

[0085] As part of Block 645, an estimate of the time required to complete the switchover operation, i.e., the Cut-Over Downtime (ECODT), can be generated. This estimate is generated by considering, for example, (1) the remaining work to be done (firstly, the aggregate size of the amount of change between the final source snapshot and its previous source snapshots); (2) the compression ratio of the source data; (3) the CBT variance per VM (as described above); and (4) the available bandwidth. The ECODT can be provided after the switchover is initiated. Users can also be provided with options to set constraints and show them how those constraints affect the ECODT. For example, a user might prefer to limit the network bandwidth available for migration to a maximum value, directly increasing the switchover downtime (compared to the case without network bandwidth limitations).

[0086] Use Case Examples

[0087] Figure 7 Use case 700 according to one aspect of this application is shown. In use case 700, the migration time of the VM is selected based on disk size.

[0088] Figure 7 The diagram illustrates a migration wave with three VMs: VM1 has two disks, each 100GB; VM2 has one 50GB disk; and VM3 has one 100GB disk. In this example, the first snapshot is initialized based on the original VM snapshot sizes of these three VMs, and the migration for each VM is initiated. The largest VM, VM1, is started first, with a total VM snapshot size of 200GB. When only 100GB of data remains to be transferred from VM1's first snapshot, VM3's first snapshot is initiated. Since its initial snapshot size is 100GB, VM3's migration also begins at that point. At this point, in this example, the same proportion of network bandwidth is allocated to VM1 and VM3 for transferring their respective data from source 720 to migration destination 740.

[0089] When both VM1's first snapshot and VM3's first snapshot have approximately 50GB remaining to transfer, VM2's first snapshot is initiated, as its initial snapshot size is 50GB. Therefore, all three VMs can be migrated in parallel—providing each VM with an equal proportion of network bandwidth for migration. This ensures timely initiation of migration for each VM. Consequently, all VMs will complete the transfer of their respective initial VM snapshots at approximately the same time, without incurring any additional costs, such as redundant migration time for some VMs' multiple snapshots while other VMs haven't even completed their initial snapshot migrations.

[0090] Note that the above system may consider whether parts of different VM disks do not contain data or have been zeroed, optimize migration so that no zeroed data is transferred, and therefore only consider non-zeroed data when determining the migration start time for each VM in the migration wave.

[0091] It's also important to note that this diagram is not scaled to the time required to transfer data. For example, in the first phase, only VM1's migration utilizes the network, meaning the network bandwidth or speed is 100% available. In the second phase, two VMs share the network, and each VM's speed is halved compared to the first phase. Similarly, in the third phase, each VM's migration speed is only one-third of that in the first phase. Therefore, although not shown, it takes three times longer to transfer three times the amount of data in the third phase compared to the transfer rate illustratively shown in the first phase.

[0092] Figure 8Use case 800 according to one aspect of this application is shown. In use case 800, VM1, VM2 and VM3 begin migration simultaneously.

[0093] In this example, for simplicity, we assume that when a snapshot of a VM (any snapshot of any VM) is transferred, the entire contents of all VM disks have been overwritten. Therefore, taking another VM snapshot at that time will create a completely different snapshot from the previous one. To keep the target synchronized with the source state, an additional amount of data needs to be transferred, roughly equal to the total size of the VM disks.

[0094] With this in mind, initiating the migration of all three VMs simultaneously will require transferring significantly more data until all three VMs have at least one snapshot on the target. This happens because, for example, while VM1's first snapshot is being transferred, VM2's first snapshot is completed, VM2's second snapshot is captured and transferred, and so on, until all four snapshots of VM2 have been transferred. This means that, at least for VM2, if we want to initiate the VM migration at a better time, we need to transfer four times the amount of data we could have previously transferred.

[0095] Notice, Figure 8 and Figure 7 The timeline shown is similar. However, because... Figure 8 Since more data needs to be transmitted through the same network, these two graphs represent different amounts of time. Therefore, Figure 8 Compare Figure 7 It takes more time.

[0096] Figure 9 Use case 900 according to one aspect of this application is illustrated. In use case 900, the disk rewrite rate is taken into account in the VM migration start timing. Specifically, in the same migration wave, the start time required for the migration of each of the three VMs is based on their disk size, a switching threshold based on disk size, the disk rewrite rate of each VM, and the network bandwidth available for migration.

[0097] In this example, the switching threshold is 0.5% of the total disk size of each VM (threshold / VM), assuming a stable 1Gbps (i.e., 1 / 8GBps) network connection between the source and destination, allocating roughly the same proportion of available network bandwidth to each VM during migration. This means that when migrating three VMs in parallel, each VM will have a network speed of 42.67MBps (=1Gbps / 3). Similarly, for two parallel VMs, the speed is 64MBps per VM, and for a single VM, it is 128MBps.

[0098] In this example, the disks of VM1 and VM2 will not be overwritten, so we will only take a single snapshot of each of them at an appropriate time. Since VM2 is the smallest of the three VMs and has never been overwritten, its migration should occur at a specific time so that it will be the last to join the migration. This means that a single snapshot of all VM2 will be migrated at a rate of 42.67 MBps, which will take (50GB * 1024MB / GB) / 42.67 MBps = 1200 seconds = 20 minutes = 1 / 3 hour.

[0099] It's also important to note that migrating VM3 requires at least one of the following speeds: [42.67, 64, 128] MBps, to migrate its first snapshot (100GB), which will take [6 / 9, 4 / 9, 2 / 9] hours respectively. During that time, the amount of data to be overwritten on VM3's source disk will be [6.66, 4.44, 2.22] GB (10% * 100GB = 10GB / hour), which is [6.66, 4.44, 2.22]% of the VM3 snapshot size. Compared to the 0.5% switching threshold, this means at least one more VM3 snapshot is needed to pass that threshold. In turn, the amount of change between the second and first snapshots of VM3 will take a maximum of 160 seconds (= 6.66GB / 42.67 MBps) to transfer. During that period, a maximum of 0.44GB (=10GB / hour * 160 seconds) of VM3's source disk was overwritten, which is less than 0.5% of VM3's total data (0.44GB / 100GB = 0.44%). This means that a third snapshot of VM3 is not required before it passes the VM3 switchover threshold. This implies that the total amount of data transferred to pass the VM3 switchover threshold is between 102.22GB and 106.66GB. Therefore, since we need to transfer 50GB of data from VM2's first snapshot to pass the VM2 switchover threshold, the VM3 migration should be initiated before the VM2 migration. Similarly, the VM1 migration should begin first because 200GB of data from VM1's first snapshot needs to be transferred to pass its switchover threshold.

[0100] In summary, in order to prepare for the switching state of the entire migration wave, the maximum amount of data that needs to be transmitted using the above method is (200 + 106.66 + 50 =) 356.66 GB.

[0101] For comparison, in Figure 10 Simultaneously initiate the migration of all three VMs. Specifically, Figure 10Use Case 1000 is illustrated, where, because VM1's first snapshot is the largest among all three VMs and has never been rewritten, only VM1's first snapshot, which is 200GB, needs to be migrated. Since only VM2's first snapshot has never been rewritten, only VM2's first snapshot, which is 50GB, needs to be migrated. Because VM3's disk is rewritten at a rate of 10GB / hour, once VM2's first snapshot migration is complete, the remaining portion of VM3's first snapshot needs to be captured and transferred from source to destination, as well as further, all of VM3's snapshots. Therefore, while migrating VM1's first snapshot, increasingly more VM3 data is being transferred. Compared to VM3, VM1 has a larger disk, and / or VM3 has a higher disk rewrite rate, requiring more data to be transferred from source to destination; therefore, and Figure 9 In comparison, it takes more time to switch thresholds for all VMs.

[0102] Figure 11 Use case 1100 according to one aspect of this application is shown. Figure 11 This illustrates how to maintain synchronization with the source after passing the switching threshold and performing the switching operation. Use Case 1100 shows a complete migration cycle for a 3-VM migration wave, where the switching threshold is defined as having at least one target snapshot for each VM in the migration wave.

[0103] like Figure 7 As shown, when initiating a migration, a migration start time is selected for each VM, thereby reducing the overall amount of data transferred from the source to the target to pass the switching threshold, while allowing all three initial VM snapshots to complete the migration in approximately the same amount of time. In this case, the estimated time to pass the switching threshold can be dynamically calculated by dividing the total disk size by the effective network upload rate from the source to the target (while also taking into account applicable compression and other WAN optimization methods).

[0104] In this example, after the switching threshold is passed, an increasing number of snapshots are periodically (e.g., continuously) and continuously captured and migrated to keep the target and source states as close as possible in a fair manner. Handling different virtual machines in a fair manner can be based on, for example, dividing network speed by the number of VMs, and / or taking into account the VM's disk size and rewrite rate. In this example, after the switching threshold is passed and before the switching operation is initiated, VM1 has three snapshots [snapshots 2, 3, and 4]. In this case, because the VM's rewrite rate is constant, these three snapshots are similar in size to each other. On the other hand, during this period, for VM3, since there are few changes to apply, a first, relatively small initial snapshot (snapshot 2) is captured, but another snapshot is immediately taken afterward. This snapshot is larger—because a large portion of VM3's disk has already been rewritten when we process VM3's snapshot 2. Furthermore, for VM2, no snapshots are taken during this period because there are no changes to apply (no disk rewrites at all). Then, at some point during a routine check of changes, the changes are detected and processed by capturing and migrating VM2's snapshot 2. During this process, the user initiates a switchover operation, which terminates the processing of snapshot 2 of VM2 and starts a different snapshot – snapshot 3 of VM2. Optionally, if a check is initiated at the time the switchover operation is started, it would be faster to complete the processing of snapshot 2 of VM2 and then only collect snapshot 3 of VM2 (this would be faster than terminating all the work done on snapshot 2 so far and then redoing it as part of snapshot 3).

[0105] When the above switching operation is initiated, the downtime period begins. The downtime ends when the source VM is shut down, the final snapshot of each VM is captured and migrated (i.e., snapshot 5 of VM1, snapshot 3 of VM2, and snapshot 4 of VM3), and these three VMs are either started or instantiated at the target location.

[0106] Note that the process shown here illustrates that the downtime described above is as short as possible because we are constantly utilizing available resources (e.g., network bandwidth) to keep the state of the target as close as possible to the source state at any given time. This ensures that the system is always ready whenever the user chooses to initiate a handover operation, providing the shortest possible downtime.

[0107] Figure 12 Use case 1200 according to one aspect of this application is shown. Specifically, the figure illustrates the non-convergent process and the adjustment of the switching threshold.

[0108] In this example, the switching threshold was initially defined as a switching downtime of up to 10 minutes. Each of the three VMs started migrating at a scheduled time, so these initial snapshots would complete migration around the same time. After these initial snapshots completed migration, an inspection was performed to determine if the disks of the three VMs had changed—this would require at least one more snapshot to fully synchronize the target state with the source state. When comparing the current snapshot state with the initial snapshot state, the system determined that VM2's disk had not changed at all, while VM1 and VM3 each had a 50GB disk change. Assuming a network upload bandwidth of 1Gbps from the source to the destination, the migration would take (50GB*2 / 1Gbps=) 13.33 minutes, which is greater than the 10-minute threshold. This means the switching threshold has not yet been met, so a second snapshot is taken for VM1 and VM3 to continue bringing the target state as close to the source state as possible and as quickly as possible.

[0109] When the second snapshot was captured and migrated, the system checked and detected that VM1 and VM3 now each had a 100GB change (e.g., the amount of change between their current state and their respective second snapshot states). This means that if a switchover operation is initiated at this time, the downtime will be (100GB*2 / 1Gbps=) 26.67 minutes – which is even greater than the previous value.

[0110] At this point, since we notice that the process may not converge, the system will issue an alert to the user about a non-convergent process and automatically adjust the switching threshold to, for example, 30 minutes.

[0111] The system continues to collect snapshots and migrate them to make the target state as close as possible to the source state.

[0112] If it is determined to continue taking snapshots that contain a significant amount of change compared to their predecessors, and the user has not initiated a handover operation, the system may decide to delay taking the next snapshot (e.g., in an exponential rollback manner) to avoid wasting expensive bandwidth and / or computing resources. During this period, the amount of change can be sampled periodically to determine its magnitude. If it is too large, for example, if it results in a handover downtime exceeding an increased handover threshold, then the handover threshold may be updated again, or the rate at which more snapshots are taken may be increased.

[0113] Another aspect of this application is the use case for determining the switching threshold. Because the switching threshold can take various forms (e.g., the minimum number of target snapshots per VM, the amount of data remaining to be transferred, etc.), the system can be configured to suggest, or even dynamically adjust, the switching threshold in such a way that it balances: (1) preparing a converged migration wave for switching; (2) reducing the time it takes for it to cross the switching threshold; and (3) the expected downtime, all three of which may take into account the switching threshold and / or other arbitrary influencing factors.

[0114] For example, if the primary goal is to minimize switchover downtime, the system can continuously (e.g., at high frequency) or periodically collect and migrate snapshots until a switchover operation is initiated. In this way, because the source is kept up-to-date as quickly and as possible, switchover downtime will be minimized. However, due to potential performance constraints or limiting switchover time to a certain point, additional constraints such as "no more than X snapshots per VM" will require finding the right balance among all relevant parameters. This can be achieved using classic linear programming, as in the example below (bold values ​​are variables, italic values ​​are user-provided numbers or numbers calculated from the runtime environment):

[0115] Switching time can be defined as:

[0116] T = Switching threshold / Network bandwidth

[0117] The network traffic consumed can be defined as:

[0118] N = Average of the magnitude of the change + Average of the increase in the magnitude of the change * Snapshot frequency

[0119] Simultaneous restrictions:

[0120] Switching threshold size > 0

[0121] The size of the switching threshold is less than the maximum value of the threshold.

[0122] Snapshot frequency > 0

[0123] Snapshot frequency < Maximum snapshot frequency

[0124] Therefore, the formula below can be defined as our "cost" function (switching time + network traffic consumed):

[0125] B = T + factor * N, such as

[0126] B = Switching threshold / Network bandwidth + Factor * (Average of change magnitude + Average of change growth * Snapshot frequency)

[0127] The possible calculations here are based on user-provided or environment-aware parameters (in italics):

[0128] Network bandwidth – perceived bandwidth or maximum permissible output

[0129] Average magnitude of change – typical rewrite rate / per snapshot period

[0130] The average growth rate of change – the typical amount of unique data in each snapshot period (e.g., data that has not been rewritten time and time again).

[0131] Maximum threshold size – User-defined limit based on the migration management window

[0132] Maximum snapshot frequency – User-defined limit

[0133] Reducing this "cost" function under given constraints will achieve an optimal balance between (A) the snapshot frequency after the switching threshold, (B) the switching time, and (C) the network traffic consumed.

[0134] As mentioned above, under the constraints described, this expression can be minimized using classical linear programming, for example, by running the simplex algorithm (or a similar algorithm) on the input.

[0135] Unless otherwise stated, the optional systems, methods, and examples described above are not mutually exclusive, but can be implemented through various combinations to achieve unique advantages. Variations and combinations of these or other features discussed above can be utilized without departing from the scope of the claims as defined, and the implementation of the above embodiments is illustrative rather than limiting of the subject matter of the claims. Furthermore, the provisions of the examples described herein, and terms such as "for example," "comprising," etc., should not be construed as limiting the subject matter of the claims to the specific examples; rather, these examples are merely for illustrating one or more possible embodiments. Additionally, reference numerals in different figures may designate the same or similar elements.

Claims

1. A replication-based migration method for migrating one or more virtual machines and associated computing resources over a network, characterized in that, The method includes: Determine one or more migration parameters based on one or more performance metrics; and Based on the determined migration parameters, the system uses at least one computing device connected to the network to receive the one or more performance metrics associated with at least one virtual machine of the one or more virtual machines to control the migration process and to control the timing of memory snapshots, data transfers, and switch requests associated with the replication-based migration. The control migration process includes: periodically creating memory snapshots and migrating the memory snapshots to a migration destination while the one or more virtual machines are running, until a switchover operation is initiated; and, The control migration process is based on user-defined constraints, which establish a switching threshold that identifies when an operation performed on the source virtual machine is switched to the migration destination.

2. The method according to claim 1, characterized in that, The computing resources include a first disk having storage areas associated with memory locations, and the control migration process includes generating at least one command that causes data located at a first memory location on the first disk to be transferred to a second memory location on a second disk, the data being associated with a first virtual machine of the one or more virtual machines.

3. The method according to claim 2, characterized in that, The control migration process includes issuing at least one command to cause the second virtual machine to process data stored in the second memory location.

4. The method according to claim 3, characterized in that, The at least one command is issued based on the switching threshold.

5. The method according to claim 1, characterized in that, The control migration process includes the at least one computing device generating at least one command to cause data located in a first memory location to be transferred to a second memory location, the data being associated with a first virtual machine of the one or more virtual machines.

6. The method according to claim 1, characterized in that, It also includes receiving migration requests, the migration requests identifying the one or more virtual machines associated with the computing resources.

7. The method according to claim 1, characterized in that, The migration parameters include at least one of the following: the time to perform a memory snapshot, the time to migrate the one or more virtual machines and computing resources associated with the given virtual machine on the storage medium, and the time to switch over the instantiation of the given virtual machine.

8. The method according to claim 1, characterized in that, It also includes outputting at least one of the following: estimated migration completion time, percentage of data volume migrated, non-convergent data, and threshold adjustment selection.

9. The method according to claim 1, characterized in that, The performance metrics include one or more of the following: snapshot data, compressibility of the computing resources to be migrated, available bandwidth of the network, modified block table (CBT), and variance of each of the one or more virtual machines.

10. The method according to claim 1, characterized in that, The performance metrics include one or more of the following: disk size, disk rewrite rate, the switching threshold, snapshot duration, and network speed.

11. A computing system for managing replication-based migration of one or more virtual machines and computing resources associated with said one or more virtual machines over a network, characterized in that, The computing system includes: One or more computing devices; One or more computer-readable storage media; and Program instructions stored on the one or more computer-readable storage media, executable by at least one of the one or more computing devices, the program instructions causing the at least one computing device to: One or more migration parameters are determined based on the one or more performance metrics; The migration process is controlled based on the determined migration parameters to control the timing of memory snapshots, data transfers, and one or more switch requests associated with the replication-based migration. The control migration process includes: periodically creating memory snapshots and migrating the memory snapshots to a migration destination while the one or more virtual machines are running, until a switchover operation is initiated; and, The control migration process is based on user-defined constraints, which establish a switching threshold that identifies when an operation performed on the source virtual machine is switched to the migration destination.

12. The computing system according to claim 11, characterized in that, The system further includes a communication interface connected to the at least one computing device and a network, the communication interface receiving the one or more performance metrics related to the one or more virtual machines via the network.

13. The computing system according to claim 12, characterized in that, The performance metrics include one or more of the following: snapshot data, compressibility of the computing resources to be migrated, available bandwidth of the network, modified block table (CBT), and variance of each of the one or more virtual machines.

14. The computing system according to claim 11, characterized in that, The control migration process also causes the at least one computing device to generate at least one command, which causes the second virtual machine to process data stored in the second memory location.

15. The computing system according to claim 11, characterized in that, The migration parameters include at least one of the following: the time to perform a memory snapshot, the time to migrate the one or more virtual machines and computing resources associated with the given virtual machine in memory, and the time to switch over the instantiation of the given virtual machine.

16. The computing system according to claim 11, characterized in that, The control migration process also causes the at least one computing device to generate at least one command that causes data located in a first memory location to be transferred to a second memory location, the data being associated with a first virtual machine of the one or more virtual machines.

17. The computing system according to claim 14, characterized in that, The at least one command is generated based on the switching threshold.

18. A non-transitory computer-readable medium comprising program instructions for a replication-based migration of one or more virtual machines and computing resources associated with said one or more virtual machines over a network, characterized in that, When one or more computing devices execute the instructions, the instructions cause the one or more computing devices to perform the following steps: One or more migration parameters are determined based on one or more performance metrics; as well as Based on the determined migration parameters, the migration process is controlled to control the timing of memory snapshots, data transfers, and switch requests associated with the copy-based migration. The control migration process includes: periodically creating memory snapshots and migrating the memory snapshots to a migration destination while the one or more virtual machines are running, until a switchover operation is initiated; and, The control migration process is based on user-defined constraints, which establish a switching threshold that identifies when an operation performed on the source virtual machine is switched to the migration destination.

Images