Dynamic migration of virtual machines in distributed computing systems

Abstract

A virtual disk and an associated method are disclosed herein for facilitating dynamic migration of virtual machines in a distributed computing system. In one embodiment, a method includes: during dynamic migration of a virtual machine, determining whether a destination virtual disk on a node contains a data block corresponding to the migrated virtual machine. The method may also include: in response to determining that the destination virtual disk does not contain the data block, retrieving the data block from the source virtual disk of the migrated virtual machine, writing the retrieved data block to one or more empty sectors on the destination virtual disk, and marking the written one or more sectors containing the data block as read-only on the destination virtual disk.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to dynamic migration of virtual machines in a distributed computing system. Background Art

[0002] Cloud computing allows multiple users to access and share a pool of configurable computing resources through a computer network. Such shared computing resources can be located in one or more data centers or other suitable distributed computing systems. In such a system, routers, switches, bridges, load balancers, or other network devices interconnect a large number of servers, network storage devices, and other computing devices. Individual servers can host one or more virtual machines, virtual switches, or other types of virtualization functions that are configured to provide computing, communication, storage, or other suitable types of computing services to users. The computing services provided are generally referred to as "cloud computing services" or "cloud services." Summary of the invention

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] In computing, a virtual machine generally refers to a simulation of a physical computing system. A virtual machine may be based on certain computer architectures (e.g., x86) and provide the functionality of a physical computer. In some distributed computing systems, the instantiation of a virtual machine involves allocating computing resources (e.g., servers or processors), network storage (e.g., permanent data storage), and other suitable types of computing resources to the virtual machine. Instantiation may also involve obtaining digital files or "images" of operating systems, system and / or user applications, and other suitable data from the image storage in the distributed computing system. One or more allocated servers may then access and execute the instructions of the obtained image to provide a suitable computing environment for the virtual machine.

[0005] In some computing systems, network storage allocated to a virtual machine may appear as a "virtual disk" or "virtual drive" on the virtual machine. Virtual disks and virtual drives are software components that emulate or "virtualize" one or more actual disk storage devices (such as optical drives, floppy disk drives, or hard disk drives). In one implementation, a virtual disk may be implemented as a base disk configured to contain a read-only parent image and one or more difference disks, the one or more difference disks being configured to contain a set of modified data blocks associated with the parent image (maintained in separate files referred to as "child images"). The difference disk may allow operations to undo changes. For example, when enabled, all changes to the parent image are stored in one or more child images. Thus, options are available to undo one or more changes to the parent image, or to permanently merge the parent image with the child image.

[0006] During operation, an instantiated virtual machine may need to be moved or "migrated" from one hosting node (e.g., a physical server) to another due to workload balancing, server hardware / software failures, system maintenance, or other suitable reasons. One migration technique (referred to herein as "live migration") involves moving a running virtual machine between different physical hosts without disconnecting the client (e.g., a user or another application) from the migrated virtual machine. During live migration, the virtual machine's memory, storage devices, and network connectivity are transferred from the source node to the destination node.

[0007] Pre-copy and post-copy migration are two technologies for dynamically migrating virtual machines. According to pre-copy migration, while the virtual machine is still running on the source node, the data of the migrated virtual machine is copied from the source node to the destination node. After completing the copying of the data, the virtual machine is started on the destination node. However, during the copying process, if certain data is changed (i.e., becomes "dirty") due to application processing or other reasons at the virtual machine, the dirty data is re-copied from the source node to the destination node. Such re-copying may be difficult when the virtual machine on the source node is running a high-performance computing workload ("HCP", for example, a computational chemistry workload) because the rate at which the virtual machine at the source node generates dirty data may exceed the rate at which data is copied from the source node to the destination node. Therefore, after many iterations, the amount of data that needs to be copied from the source node to the destination node may not decrease or may even increase. Therefore, the data corresponding to the migrated virtual machines on the host and destination nodes will not converge.

[0008] Post-copy migration may be more suitable than pre-copy migration for migrating virtual machines running such HPC workloads. Post-copy migration is initiated by suspending the virtual machine at the source node. When the virtual machine is suspended at the source node, a minimum subset of the virtual machine's operating state (e.g., CPU state, registers, non-pageable memory, etc.) may be transferred to the destination node. The virtual machine is then restarted (resume) at the destination node using the subset of operating state received from the source node. Subsequently, the source node may actively push the remaining data blocks of the migrated virtual machine to the destination node in a process known as pre-paging.

[0009] However, pre-paging may degrade the performance of applications running in the virtual machine on the destination node when the virtual machine attempts to access many data blocks that have not yet been transferred. Conversely, the source node may push other data blocks that the virtual machine does not currently need via pre-paging. By implementing a base cache disk at the destination node to pipeline the transfer of data blocks of the migrated virtual machine from the source node to the destination node during post-copy migration, several embodiments of the disclosed technology can address certain aspects of the aforementioned difficulties. Specifically, the base cache disk can be configured to perform on-demand extraction of one or more data blocks from the source node as needed instead of or in addition to pre-paging. In some embodiments, the base cache disk can be implemented as a virtual disk that can be written only once for each data sector. After the first write, the written sectors of the base cache disk become read-only.

[0010] In some implementations, before the migrated virtual machine is started on the destination node, a base cache disk ("destination disk") may be created that is the parent of the data blocks of the virtual machine on the virtual disk ("source disk") at the source node. A difference disk may also be created that is the parent of the base cache disk to isolate writes by the virtual machine on the destination node. In operation, for all reads that are not provided from the difference disk or the destination disk, the requested data may be provided from the source disk. The requested data may then be saved in the destination disk, and the corresponding sectors may be marked as read-only.

[0011] As the post-copy migration progresses, additional data blocks of the migrated virtual machine can be retrieved and written to sectors of the destination disk in response to virtual machine requests and / or via background processing at the destination node. Ultimately, the destination disk will contain a complete copy of the data blocks of the virtual machine from the source node. After completing the transfer of the complete copy of the data blocks to the destination disk at the destination node, the source disk can be removed as the parent of the destination disk. Optionally, the difference disk and the destination disk can also be merged into a dynamic virtual disk at the destination node after a specific time period or according to other suitable criteria.

[0012] Several embodiments of the disclosed technology can therefore improve the efficiency of post-copy migration by pipeline the transfer of data blocks of the migrated virtual machine from the source node to the destination node. Instead of or in addition to pre-paging, the data blocks of the migrated virtual machine can be extracted in an on-demand manner. For example, when the migrated virtual machine on the destination node requests one or more data blocks for processing, the requested data blocks can be transferred from the source disk to the destination disk. Therefore, the efficiency of dynamic migration of virtual machines can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a schematic diagram of a distributed computing system configured for rapid instantiation of virtual machines according to an embodiment of the disclosed technology.

[0014] Figure 2A and Figure 2B is a diagram illustrating an embodiment according to the disclosed technology Figure 1 Schematic diagram of hardware / software components of a distributed computing system in which a basic cache disk is utilized to facilitate dynamic migration of virtual machines.

[0015] Figures 3A-3G is a schematic diagram illustrating various operational stages of utilizing an underlying cache disk for facilitating live migration of a virtual machine according to an embodiment of the disclosed technology.

[0016] Figures 4A-4C is a schematic diagram illustrating additional operation stages for facilitating live migration of virtual machines using an underlying cache disk according to an embodiment of the disclosed technology.

[0017] Figure 5 and Figure 6 is a flow chart illustrating a process associated with efficient dynamic migration of virtual machines in a distributed computing system according to an embodiment of the disclosed technology.

[0018] Figure 7 Is suitable for Figure 1 A computing device that is part of a distributed computing system. DETAILED DESCRIPTION

[0019] Certain embodiments of computing systems, devices, components, modules, routines, and processes for efficient dynamic migration of virtual machines in distributed computing systems are described below. In the following description, specific details of the components are included to provide a thorough understanding of certain embodiments of the disclosed technology. Those skilled in the relevant art may also understand that the disclosed technology may have additional embodiments or may be used without the following references. Figure 1-7 The embodiments described are practiced without departing from the scope of the invention in several specific details.

[0020] As used herein, the term "computing cluster" generally refers to a computing system having multiple network devices that interconnect multiple servers or nodes to each other or an external network (e.g., the Internet). An example of a computing cluster is one or more racks, each rack holding multiple servers in a cloud computing data center (or part thereof) configured to provide cloud services. One or more computing clusters can be interconnected to form a "computing fabric", which forms at least part of a distributed computing system. The term "network device" generally refers to a network communication component. Example network devices include routers, switches, hubs, bridges, load balancers, security gateways, or firewalls. "Node" generally refers to a computing device configured to implement one or more virtual machines, virtual routers, virtual gateways, or other suitable virtualized computing components. In one example, a node may include a computing server having a hypervisor configured to support one or more virtual machines.

[0021] As used herein, the term "instantiation" generally refers to the generation of an instance or event of a computing object that exists during the runtime of a computer program. Instantiation binds logical requirements for resources (such as memory, processors, and network access) used by, for example, a virtual machine to a specific instance of appropriate resources in a distributed computing system. For example, instantiation of a virtual machine includes, among other things, allocating appropriate computing, storage, network, and other appropriate resources from a resource pool in a distributed computing system, obtaining an image of an operating system, user application, or other appropriate type of data, and running appropriate instructions contained in the obtained image to generate a computing environment that simulates a physical computer.

[0022] As also used herein, the term "migration" of a virtual machine generally refers to the process of moving a virtual machine from one node or storage location to another. "Dynamic migration" of a virtual machine generally refers to moving a running virtual machine between different physical nodes without disconnecting a client (e.g., a user or another user) from the migrated virtual machine. The term "pre-copy migration" generally refers to a process in which all data blocks of the migrated virtual machine are copied from a source node to a destination node while the virtual machine is still running on a source node. After such copying is completed, the virtual machine on the destination node is started. In contrast, the term "post-copy migration" generally refers to starting the migrated virtual machine on the destination node before copying at least a portion of the data blocks of the virtual machine from the source node.

[0023] Other As used herein, the term "cloud service" generally refers to one or more computing resources provided over a computer network, such as the Internet. Common examples of cloud services include Software as a Service ("SaaS"), Platform as a Service ("PaaS"), and Infrastructure as a Service ("IaaS"). SaaS is a software distribution technology in which software applications are hosted by a cloud service provider, such as in a data center, and accessed by users over a computer network. PaaS generally refers to the delivery of an operating system and associated services over a computer network without downloading or installing.

[0024] As used herein, the phrase "central mirror storage" or "mirror storage" generally refers to a network storage space containing files or file images accessible by each node of a distributed computing system. In some implementations, the central mirror storage may include a network-accessible database hosted on one or more nodes in a distributed computing system. The entries of the database may include files or file images of various suitable types. Also used herein, "file", "file image", or "image" generally refers to a collection of digital data that can be stored in a tangible computer storage device or medium. For example, a file image may include a complete copy of an operating system, user application, application data, user data, or other suitable types of data for a virtual machine. The file image may be logically divided into blocks, zones, pages, binary objects, or other suitable logical subdivisions of data. For example, the file image of data associated with a virtual machine may be logically divided into blocks of 256 kilobytes, 512 kilobytes, or other suitable sizes. Each data block may be recorded, written, or stored in one or more sectors, partitions, or other suitable divisions of a computer storage device.

[0025] Other As used herein, the term "virtual disk" or "virtual drive" generally refers to one or more software components that simulate an actual and physical disk storage device (such as an optical drive, a floppy disk drive, or a hard drive). For other programs, the virtual disk behaves like an actual physical device. The virtual disk can be implemented as a disk mirror containing the data structure of the actual storage device. According to an embodiment of the disclosed technology, a virtual disk may include one or more difference disks coupled to a base cache disk that is a parent of one or more difference disks. A "difference disk" is a virtual disk that contains data blocks representing changes to a parent virtual disk (such as a base cache disk). A "base cache disk" or "BCD" is a virtual disk whose sectors can only be written once. After the first write to the sector is completed, the sector of the base cache disk is marked as read-only. In contrast, a "base disk" is a read-only virtual disk and cannot be written to at all.

[0026] In some computing systems, post-copy migration can be implemented via pre-paging after the migrated virtual machine is started on the destination node. However, pre-paging may degrade the performance of applications running in the virtual machine on the destination node when the started virtual machine attempts to access data blocks that have not yet been transferred. In addition, once the virtual machine is started on the destination node and the application operation is restarted, the data of the migrated virtual machine may reside on both the source node and the destination node. Therefore, if the migrated virtual machine fails on the destination node, repair of the migrated virtual machine may be impossible.

[0027] Several embodiments of the disclosed technology can streamline the transfer of data blocks of a migrated virtual machine from a source node to a destination node during a post-copy migration by implementing a base cache disk at the destination node. In certain implementations, before the migrated virtual machine is started on the destination node, a base cache disk ("destination disk") can be created that is the parent of the virtual disk of the virtual machine on the source node ("source disk"). A difference disk can also be created (which is the parent of the base cache disk) to isolate writes by the virtual machine on the destination node. In operation, for all reads that do not exist or are provided from the difference disk or the destination disk, the requested data can be provided from the source disk. The requested data can then be saved in the destination disk, and the corresponding sectors can be marked as read-only. Once the transfer is completed, the destination disk can include a complete read-only copy of the data blocks from the source node. Therefore, the efficiency of post-copy dynamic migration of virtual machines can be improved, as described below with reference to Figure 1-7 Described in more detail.

[0028] Figure 1 is a schematic diagram of a distributed computing system 100 configured for efficient dynamic migration of virtual machines according to an embodiment of the disclosed technology. Figure 1 As shown, the distributed computing system 100 may include a computer network 108 that interconnects multiple users 101 via client devices 102 and computing infrastructure 104. Even though specific components of the distributed computing system 100 are shown in Figure 1 In other embodiments, the distributed computing system 100 may also include additional and / or different components. For example, the distributed computing system 100 may include additional computing architectures, network storage devices, public infrastructure, and / or other suitable components.

[0029] The client devices 102 may each comprise a computing device that facilitates a corresponding user 101 to access cloud services provided by the computing architecture 104 via the computer network 108. For example, in the illustrated embodiment, the client devices 102 solely comprise a desktop computer. In other embodiments, the client devices 102 may also include a laptop computer, a tablet computer, a smart phone, or other suitable computing device. Even though three users 101 and corresponding client devices 102 are shown in FIG. 1 for illustrative purposes, the present invention provides a plurality of computing devices, each of which is illustrated in FIG. Figure 1 In other embodiments, the distributed computing system 100 may also facilitate any suitable number of users 101 to access computing services provided by the computing architecture 104 .

[0030] like Figure 1 As shown, computer network 108 may include one or more network devices 112 that interconnect users 101 with components of computing architecture 104. Examples of network devices 112 may include routers, switches, firewalls, load balancers, or other suitable network components. Even though a specific connection scheme is shown for illustrative purposes, Figure 1 In other embodiments, the network devices 112 may also be operatively coupled in a hierarchical, flat, "mesh" or other suitable topology. In one embodiment, the computer network 108 includes the Internet. In other embodiments, the computer network 108 may also include a local area network, a wide area network, a virtual private network, or other suitable types of computer networks.

[0031] Also like Figure 1 As shown, the computing architecture 104 may include a mirrored storage 111 and a plurality of nodes 106 that are operatively coupled to each other via a network device 112. In some embodiments, the node 106 may individually include a processor, a physical server, or a blade that includes several physical servers. In other embodiments, the node 106 may also include a virtual server or several virtual servers. The nodes 106 may be organized into racks, availability zones, groups, sets, computing clusters, or other suitable divisions. For example, in the illustrated embodiment, the nodes 106 are grouped into three computing clusters 105 (individually shown as first, second, and third computing clusters 105a-105c, respectively), which are operatively coupled to corresponding network devices 112 in the computer network 108. Even though the three computing clusters 105 are shown in Figure 1 for illustrative purposes, the three computing clusters 105 are shown in Figure 11 for illustrative purposes. Figure 1 In other embodiments, the computing architecture 104 may also include one, two, eight, sixteen, or any other suitable number of computing clusters 105 having similar or different components and / or configurations.

[0032] The image storage 111 may be configured to contain a suitable Figure 2A and Figure 2B144) of one or more virtual machines or "VMs" 144. For example, individual images 113 may include files such as or In another example, the image 113 may also include one or more user applications, resource files, or other suitable types of data suitable for instantiating the virtual machine 144. Figure 1 As shown, in another embodiment, the image storage 111 may also include at least a partial copy of the data blocks corresponding to one or more virtual machines 144 or applications executed on one or more nodes in the nodes 106. Figure 1 Although shown as a single database, in other embodiments, the mirror storage 111 can also be implemented as a distributed database hosted on multiple nodes 106.

[0033] In operation, a user 101 may request the computing infrastructure 104 to instantiate one or more virtual machines 144 by, for example, submitting a request via a user port 107 using a client device 102. In response, the computing infrastructure 104 may authenticate the user request and, upon authentication, allocate appropriate computing (e.g., one or more nodes 106), storage, network, or other suitable types of computing resources. The computing infrastructure 104 may then direct the allocated nodes 106 to instantiate the requested virtual machines 144, via, for example, an infrastructure controller (not shown). As discussed in more detail below with reference to FIG. 2A-2B, the nodes 106 may be configured to implement ( Figure 2A and Figure 2B One or more base cache disks 148 (shown in ) to facilitate efficient post-copy dynamic migration of virtual machines 144 between a pair of nodes 106.

[0034] Figure 2A and Figure 2B is a diagram illustrating an embodiment according to the disclosed technology Figure 1 A schematic diagram of certain hardware / software components of the distributed computing system 100 in FIG. Figure 2A and Figure 2B In the example, some components of the distributed computing system 100 are omitted for clarity. For example, only two nodes 106, namely the source node 106a and the destination node 106b, are shown in FIG. Figure 2A and Figure 2B In other embodiments, the distributed computing system 100 may include any suitable number of nodes 106 and / or computing clusters 105 .

[0035] In addition, Figure 2A and Figure 2BIn and in other figures of this article, individual software components, objects, classes, modules and routines can be computer programs, processes or processes written as source code in C, C++, C#, Java and / or other suitable programming languages. Components can include, but are not limited to, one or more modules, objects, classes, routines, attributes, processes, threads, executables, libraries or other components. Components can be in source or binary form. Components can include aspects of source code before compilation (e.g., classes, attributes, processes, routines), compiled binary units (e.g., libraries, executables) or artifacts (e.g., objects, processes, threads) instantiated and used at runtime. In some embodiments, the various components and modules described below can be implemented using roles (actors). In other embodiments, the generation of applications and / or related services can also be implemented using monolithic applications, multi-layer applications or other suitable components.

[0036] Components within a system may take different forms within the system. As an example, a system comprising a first component, a second component, and a third component may include, but is not limited to, a system having a first component as an attribute in a source code, a second component as a binary compiled library, and a third component as a thread created at runtime. A computer program, process, or process may be compiled into a target, intermediate, or machine code and proposed for execution by one or more processors of a personal computer, a network server, a laptop, a smart phone, and / or other suitable computing device. Similarly, a component may include a hardware circuit system.

[0037] Those of ordinary skill in the art will appreciate that hardware can be thought of as fossilized software, and software can be thought of as liquefied hardware. As just one example, the software instructions in a component can be burned into a programmable logic array circuit, or can be designed as a hardware circuit with a suitable integrated circuit. Likewise, hardware can be simulated by software. Various implementations of source, intermediate and / or target code and associated data can be stored in a computer memory including a read-only memory, a random access memory, a disk storage medium, an optical storage medium, a flash memory device, and / or other suitable computer-readable storage medium that does not include a propagating signal.

[0038] like Figure 2AAs shown, user 101 can access user port 107 provided by corresponding client device 102 to send user request 160 requesting to instantiate virtual machine 144 in distributed computing system 100. User request 160 can include a name, template, a list of one or more specified features, a location, or other suitable information related to virtual machine 144. In other embodiments, user 101 can also provide display configuration, certificates, execution configuration, subscription information, or other suitable data via user port 107. In other embodiments, developers, administrators, or other suitable types of entities can provide configuration and / or other suitable information instead of or in addition to user 101.

[0039] Also like Figure 2A As shown, the provisioning controller 109 (e.g., the fabric controller or a component thereof) may be configured to receive the user request 160, identify the user request 160 as a request for a virtual machine 144 based on information included in the user request 160, and allocate, for example, the source node 106a for instantiating the requested virtual machine 144 (at Figure 2A 144a"). Subsequently, the provisioning controller 109 or other suitable component of the distributed computing system 100 may instruct the assigned source node 106a to initiate an instantiation process for providing the requested source virtual machine 144a. In response, the source node 106a may instantiate the source virtual machine 144a for use by the user 101.

[0040] Subsequently, the provisioning controller 109 or other suitable components in the distributed computing system 100 may be configured to determine that the source virtual machine 144a is to be migrated to the destination node 106b due to workload balancing, server hardware / software failure, system maintenance, or other suitable reasons. In response, the provisioning controller 109 may issue commands 162 and 162' to both the source node 106a and the destination node 106b to initiate a post-copy dynamic migration of the source virtual machine 144a from the source node 106a to the destination node 106b. In response, the source node 106a may suspend the source virtual machine 144a and transfer a minimum subset of the execution state (e.g., CPU state, registers, non-pageable memory, etc.) of the source virtual machine 144a to the destination node 106b. The destination node 106b may then instantiate the destination virtual machine 144b using the subset of the execution state received from the source node 106a. Once started, the destination virtual machine 144b can restart execution of one or more applications (not shown) for the user 101 before a complete copy of the data blocks of the source virtual machine 144a is transferred to the destination node 106b. Components of the source node 106a and the destination node 106b configured to facilitate efficient post-copy live migration of the virtual machine 144 are described in more detail below.

[0041] like Figure 2A As shown, the source node 106a and the destination node 106b can each include an interface component 133 and a control component 135 for managing the operation of the virtual disks on the source node 106a and the destination node 106b. The interface component 133 can be configured to receive the commands 162 and 162', analyze the content of the commands 162 and 162', and forward the results of the analysis to the control component 135. For example, the interface component 133 can indicate to the control component 135 the type (e.g., difference, base, base cache, etc.), size (e.g., fixed or dynamic) or other suitable information related to the virtual disks 145 associated with the source virtual machine 144a and the destination virtual machine 144b. Then, the control component 135 can configure and facilitate the operation of the virtual disk 145, such as those described below with reference to 3A-4C, as indicated by the interface component 133. In the illustrated example, one virtual machine 144 with a corresponding virtual disk 145 is shown on a single node 106 for illustrative purposes. In other examples, each node 106 may support one, three, four, or any suitable number of virtual machines 144 with corresponding virtual disks 145 .

[0042] like Figure 2A As shown, source virtual machine 144a can be associated with source virtual disk 147. In some embodiments, source virtual disk 147 can include a base disk with one or more difference disks (not shown). In other embodiments, source virtual disk 147 can include one or more other suitable types of virtual disks with fixed or dynamic sizes. Figure 2A As shown, the source virtual disk 147 may include ( Figure 3A ) data block 115, which represents the data block corresponding to the source virtual machine 144a and other suitable types of data. In other embodiments, at least a portion of the data block 115 may be stored in, for example, the central mirror storage 111 ( Figure 1 ) (such as Figure 2B as shown), or stored in other suitable locations in the distributed computing system 100.

[0043] Also like Figure 2AAs shown, the destination virtual machine 144b can be associated with a virtual disk 145, which includes a difference disk 146 coupled to a base cache disk 148. The base cache disk 148 is therefore the parent of the difference disk 146, as indicated by arrow 130. According to an embodiment of the disclosed technology, the control component 135 can also configure the data blocks 115 in the source virtual disk 147 as the parent of the base cache disk 148, as indicated by arrow 130'. Therefore, any data block 115 contained in the source virtual disk 147 can serve as a base disk for the base cache disk 148 on the destination node 106b. The base cache disk 148 can be configured to retrieve the data blocks 115 of the virtual machine 144a from the source virtual disk 147 in an on-demand manner, and thus streamline the post-copy live migration process. Details of the operation of the source virtual disk 147, the base cache disk 148, and the difference disk 146 are described below with reference to Figures 3A-3G Describe in more detail.

[0044] even though Figure 2A The source virtual machine 144a is illustrated as including a complete copy of the data block 115 of the source virtual machine 144a on the source node 106a. In other embodiments, at least a portion of the data block 115 may also be located at, for example, the mirror storage 111. Figure 2B As shown. The virtual disk 145 on the source node 106a can be implemented as a source difference disk 149 that is a child of the data block 115 in the mirror storage 111. Therefore, the data block 115 is the parent of the source difference disk 149, as indicated by arrow 130", and the source difference disk 149 is the parent of the base cache disk 148 on the destination node 106b, as indicated by arrow 130'. Details of the operation of the source difference disk 149 and the base cache disk 148 are described below with reference to Figures 4A-4C Describe in more detail.

[0045] Figures 3A-3G is an illustration of utilizing the underlying cache disk 148 to facilitate the source virtual machine 144a ( Figure 2A )from Figure 2A FIG. 1 is a schematic diagram of various operation stages of a post-copy dynamic migration from a source node 106a to a destination node 106b in a distributed computing system 100. Figure 3A As shown, the source disk 147 of the source node 106a may include a copy of the data blocks 115 of the source virtual machine 144a. In the illustrated example, nine data blocks 115 identified as blocks AI are shown for illustrative purposes. In other embodiments, the data blocks 115 may include any suitable number of blocks, or may be divided into data blocks or other suitable divisions.

[0046] Also like Figure 3AAs shown, the base cache disk 148 may be initially empty because the data blocks 115 have not yet been copied from the source node 106a, even though the destination virtual machine 144b has been instantiated and started. However, the difference disk 146 on the destination node 106b may contain certain data blocks 117 that are unique to the destination virtual machine 144b, such as data blocks written to the difference disk 146 by the destination virtual machine 144b after launch. Such data blocks 117 may represent differences between the corresponding data on the base cache disk 148 and the destination virtual machine 144b.

[0047] like Figure 3A As shown, during initial operation, destination virtual machine 144b on destination node 106b may determine that certain data blocks 115 are now needed to execute instructions for an application (not shown) or other suitable software component executing in destination virtual machine 144b. In the illustrated example, blocks A, D, and H are indicated as needed. Therefore, destination virtual machine 144b requests control component 135 ( Figure 2A ) provides data from blocks A, D, and H. Control component 135 may then determine whether difference disk 146 contains such data. In response to determining that difference disk 146 does not contain such data, control component 135 may determine whether base cache disk 148 on destination node 106b contains such data. Figure 3A As shown, the base cache disk 148 also does not contain the requested data. In response, the control component 135 can access the parent of the base cache disk 148, that is, the data block 115 on the source disk 147 for retrieving the blocks A, D and H without downloading the entire data block 115.

[0048] like Figure 3B As shown, upon receiving the data from blocks A, D, and H, the control component 135 may cause the received data to be written to the corresponding sectors in the base cache disk 148, and after the writing is completed, mark the sectors corresponding to blocks A, D, and H as read-only. The control component 135 may then provide a copy of the data received from the base cache disk 148 to the destination virtual machine 144b via the difference disk 146. Subsequently, the destination virtual machine 144b may determine that the data from block B is now needed for execution. Figure 3C As shown, control component 135 can determine whether difference disk 146 contains such data. Figure 3D As shown, in response to determining that the difference disk 146 contains such data, the control component 135 provides the data of block B from the difference disk 146 to the destination virtual machine 144b.

[0049] As destination virtual machine 144b requests more and more additional data from data blocks 115 of virtual machine 144a or via background processing at destination node 106b, remaining data blocks 115 may be written to corresponding sectors of underlying cache disk 148 and marked as read-only. Figure 3E As shown, in some embodiments, once a complete copy of the data block 115 has been written to the base cache disk 148, the control component 135 can remove the data block 115 as the parent of the base cache disk 148 at the source disk 147. Subsequent requests for the data block 115 (e.g., for the destination virtual machine 144b) can be provided from the base cache disk 148 without accessing the source disk 147, as shown in FIG. Figure 3F As shown. Thus, a complete copy of data block 115 of source virtual machine 144a can be maintained on destination node 106b during live migration of source virtual machine 144a. At the same time, changes to data block 115 are captured by difference disk 146. Thus, when destination virtual machine 144b experiences an error or other suitable type of failure, destination node 106b (e.g., via a hypervisor) can easily repair such an error by instantiating and starting another destination virtual machine 144b based on a complete copy of data block 115 in base cache disk 148 and data block 117 in difference disk 146.

[0050] Optionally, in some embodiments, base cache disk 148 and difference disk 146 may be merged into destination disk 150, such as Figure 3G Such merging may be based on elapsed time or other suitable criteria. In other embodiments, the merging of base cache disk 148 with difference disk 146 may be omitted.

[0051] Figures 4A-4C is an illustration of utilizing the underlying cache disk 148 to facilitate Figure 2B 1 is a schematic diagram of various operational stages of dynamic migration of virtual machines 144 in distributed computing system 100. Even though only source node 106a and destination node 106b are shown in FIG. Figures 4A-4C However, in other embodiments, similar operations may be implemented for any suitable number of nodes 106 ( Figure 1 ).

[0052] like Figure 4A As shown, the destination virtual machine 144b on the destination node 106b may request certain data blocks 115 during the initial phase of the live migration process. Figure 4A As shown, destination virtual machine 144b requests blocks A, E, and I for illustration purposes. Figure 3A and Figure 3BIn response to determining that the difference disk 146 does not contain the requested data, the control component ( Figure 2B ) can determine whether the base cache disk 148 at the destination node 106b contains such data.

[0053] In response to determining that the base cache disk 148b does not contain such data, in some embodiments, the control component 135 may determine whether the source difference disk 149, which is the parent of the base cache disk 148, contains such data. In response to determining that the source difference disk 149 contains such data, for example, data block B, the control component 135 retrieves a copy of the required data from the source difference disk 149 and stores the retrieved data in the base cache disk 148. In response to determining that the source difference disk 149 does not contain such data, the control component 135 may request a copy of the required data from the data block 115 in the mirror storage 111.

[0054] like Figure 4B As shown, the mirror storage 111 can provide a copy of the requested data block 115 to be stored in the corresponding sector to the base cache disk 148. The written sector is then marked as read-only. The base cache disk 148 can then make a copy of the requested data block 115 available to the destination virtual machine 144b. For example, Figure 4B As shown, base cache disk 148 may retrieve or pull blocks A, D, and H from mirror storage 111. Base cache disk 148 may store the received database 115 in corresponding sectors and mark those sectors as read-only before providing the data to destination virtual machine 144b for processing.

[0055] like Figure 4C As shown, as more data blocks 115 are requested from the source difference disk 149 and / or the mirror storage 111 or via background processing, the base cache disk 148 may eventually contain a complete copy of the data block 115 and / or the data block 117 in the source difference disk 149. After determining that the base cache disk 148 contains a complete copy of the data block 115, the source difference disk 149 may be removed as a parent from the base cache disk 148. Subsequent data requests from the destination virtual machine 144b may be provided from the base cache disk 148 and / or the difference disk 146 without accessing the source difference disk 149 or the mirror storage 111.

[0056] Figure 5 and Figure 6 is a flow chart illustrating a process associated with dynamic migration of virtual machines in a distributed computing system according to an embodiment of the disclosed technology. Figure 5 and Figure 6 The process shown in the figure is about Figure 1-4CThe process is described with reference to the distributed computing system 100 of FIG. 1 , but in other embodiments, the process may be implemented in a computing system having additional and / or different components.

[0057] like Figure 5 As shown, a process 200 for dynamic migration of a virtual machine may include receiving a request for one or more data blocks of a source virtual machine at stage 202, the source virtual machine to be migrated from a source node to a destination node as a destination virtual machine. The process 200 may then include a decision stage 204 to determine whether the requested data blocks exist on a difference disk associated with the destination virtual machine. Such a determination may be based on a file mapping or other suitable metadata of the difference disk. In response to determining that the difference disk contains the data blocks, the process 200 may include providing the data blocks from the difference disk to the destination virtual machine.

[0058] In response to determining that the difference disk does not contain the data block, the process 200 may include another decision stage 208 to determine whether the base cache disk as the parent of the difference disk contains such a data block. In response to determining that the base cache disk contains the data block, the process 200 may include providing the data block from the base cache disk at stage 206. In response to determining that the base cache disk does not contain the data block, the process 200 may retrieve the data block from the source disk at stage 210, and the source disk contains a copy of the requested data block of the source virtual machine on the source node. Such retrieval may include querying a database, requesting a copy of the data block, receiving the requested data block, and / or other suitable operations. The process 200 may then include storing the received data block in the base cache disk in the corresponding sector before returning to providing the data block at stage 206 and thereafter marking such a sector as read-only.

[0059] Figure 6 The diagram shows how to manage basic cache disks such as Figures 3A-3G 220). For example, Figure 6As shown, process 220 may include determining the completeness of the copy of the data blocks of the source virtual machine on the base cache disk at stage 222. The base cache disk is a child of the source disk utilized by the source virtual machine, which is to be dynamically migrated from the source node to the destination node as the destination virtual machine corresponding to the base cache disk. Process 220 may then include a decision phase for determining whether the copy of the data blocks of the source virtual machine on the base cache disk is complete. In response to determining that the copy of the data blocks of the source virtual machine on the base cache disk is complete, process 220 may include removing the source disk of the source virtual machine as the parent of the base cache disk at stage 226. Optionally, process 220 may also include merging the base cache disk with the difference disk associated with the destination virtual machine at stage 230. Otherwise, process 220 may include retrieving additional data blocks from the source disk on demand or via background processing at stage 228. Process 220 may then store the retrieved data blocks in the underlying cache disk in corresponding sectors and thereafter mark such sectors as read-only at stage 232 before returning to determine the integrity of the data blocks on the underlying cache disk at stage 222 .

[0060] Figure 7 Is suitable for Figure 1 The computing device 300 may be suitable for some components of the distributed computing system 100 in the embodiment of the present invention. For example, the computing device 300 may be suitable for Figure 1 Node 106 or Figure 2A and 2B The preset controller 109. In a very basic configuration 302, the computing device 300 may include one or more processors 304 and a system memory 306. A memory bus 308 may be used for communication between the processor 304 and the system memory 306.

[0061] Depending on the desired configuration, the processor 304 can be of any type, including but not limited to a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), or any combination thereof. The processor 304 can include a multi-level cache, such as a level 1 cache 310 and a level 2 cache 312, a processor core 314, and registers 316. An example processor core 314 can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. An example memory controller 318 can also be used with the processor 304, or in some implementations, the memory controller 318 can be an internal part of the processor 304.

[0062] Depending on the desired configuration, system memory 306 may be of any type, including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 306 may include an operating system 320, one or more applications 322, and program data 324. The depicted basic configuration 302 is illustrated in FIG4 by those components within the inner dashed line.

[0063] The computing device 300 may have additional features or functions, as well as additional interfaces, to facilitate communication between the basic configuration 302 and any other devices and interfaces. For example, a bus / interface controller 330 may be used to facilitate communication between the basic configuration 302 and one or more data storage devices 332 via a storage interface bus 334. The data storage device 332 may be a removable storage device 336, a non-removable storage device 338, or a combination thereof. Examples of removable storage devices and non-removable storage devices include magnetic disk devices such as flexible disk drives and hard disk drives (HDDs), optical disk drives, such as compact disk (CD) drives or digital versatile disk (DVD) drives, solid state drives (SSDs), and tape drives, to name a few. Example computer storage media may include volatile and non-volatile media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. The term "computer-readable storage medium" or "computer-readable storage device" does not include propagating signals or other types of signals and communication media.

[0064] System memory 306, removable storage device 336, and non-removable storage device 338 are examples of computer-readable storage media. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 300. Any such computer-readable storage media may be part of computing device 300. The term "computer-readable storage media" does not include propagating signals and communication media.

[0065] The computing device 300 may also include an interface bus 340 for facilitating communication from various interface devices (e.g., output devices 342, peripheral interfaces 344, and communication devices 346) to the basic configuration 302 via a bus / interface controller 330. Example output devices 342 include a graphics processing unit 348 and an audio processing unit 350, which may be configured to communicate to various external devices, such as a display or speakers, via one or more A / V ports 352. Example peripheral interfaces 344 include a serial interface controller 354 or a parallel interface controller 356, which may be configured to communicate with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices, etc.) or other peripheral devices (e.g., printers, scanners, etc.) via one or more I / O ports 358. Example communication devices 346 include a network controller 360, which may be arranged to facilitate communication with one or more other computing devices 362 over a network communication link via one or more communication ports 364.

[0066] A network communication link can be an example of a communication medium. Communication media can generally be embodied by computer-readable media, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission medium and include any information delivery medium. A "modulated data signal" can be a signal having one or more of its characteristics set or changed in a manner that encodes information in the signal. By way of example and not limitation, communication media can include wired media such as a wired network or a direct wired connection and wireless media such as acoustic, radio frequency (RF), microwave, infrared (IR) and other wireless media. The term computer-readable media as used herein can include both storage media and communication media.

[0067] The computing device 300 may be implemented as part of a small form factor portable (or mobile) electronic device such as a cellular phone, a personal digital assistant (PDA), a personal media player device, a wireless network watch device, a personal head mounted device, a dedicated device, or a hybrid device including any of the above functionality. The computing device 300 may also be implemented as a personal computer including both laptop and non-laptop configurations.

[0068] From the foregoing, it will be appreciated that the specific embodiments of the present disclosure have been described herein for illustrative purposes, but various modifications may be made without departing from the present disclosure. In addition, many elements in the elements of an embodiment may be combined with other embodiments as additions to or replacements for elements of other embodiments. Therefore, the present technology is limited only by the appended claims.

Claims

1. A method for dynamically migrating virtual machines on nodes interconnected via a computer network in a distributed computing system, the method comprising: Suspending a source virtual machine on the first node, wherein after being suspended, the source virtual machine has a plurality of data blocks contained in a source virtual disk; Instantiating and starting a destination virtual machine on the second node, wherein the destination virtual machine includes a destination virtual disk as a parent and a difference disk as a child of the destination virtual disk; receiving, at the second node, a request from the started destination virtual machine, the request being for a data block among the plurality of data blocks of the suspended source virtual machine; as well as In response to receiving the request from the started destination virtual machine, determining whether the requested data block is available at the difference disk; In response to determining that the requested data block is not available at the difference disk, determining whether the destination virtual disk on the second node contains the requested data block; as well as In response to determining that the destination virtual disk does not contain the requested data block, retrieving, via the computer network, the requested data block from the source virtual disk, the source virtual disk being a parent of the destination virtual disk of the started destination virtual machine; writing the retrieved data blocks to one or more empty sectors of the destination virtual disk; After completing writing of the retrieved data block, marking one or more sectors containing the written data block as read-only on the destination virtual disk; as well as A copy of the retrieved data block is provided from the destination virtual disk to the destination virtual machine, thereby enabling operation of the destination virtual machine without waiting for retrieval of other data blocks of the source virtual machine.

2. The method according to claim 1, wherein the method further comprises: In response to receiving the request for the data block, determining whether the requested data block is available at the difference disk; as well as In response to determining that the requested data block is available at the differencing disk, the data block is provided from the differencing disk in response to the received request.

3. The method according to claim 1, wherein the method further comprises: In response to receiving the request for the data block, determining whether the requested data block is available at the difference disk; In response to determining that the requested data block is not available at the difference disk, determining whether the destination virtual disk on the second node contains the requested data block; as well as In response to determining that the destination virtual disk contains the requested data block, providing the data block from the destination virtual disk in response to the received request without retrieving the data block from the source virtual disk of the first node.

4. The method of claim 1, wherein the destination virtual disk is a child of the source virtual disk, and wherein the method further comprises: Repeating, via background processing on the second node, the fetching, writing, and marking operations for the additional plurality of data blocks; determining whether the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine; as well as In response to determining that the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine, removing the source virtual disk on the first node as a parent of the destination virtual disk on the second node.

5. The method of claim 1 , wherein the destination virtual disk is a child of the source virtual disk, and wherein the method further comprises: determining whether the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine; In response to determining that the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine, removing the source virtual disk on the first node, the source virtual disk serving as a parent of the destination virtual disk on the second node; as well as The plurality of data blocks are provided from the destination virtual disk in response to an append request for the plurality of data blocks.

6. The method of claim 1, wherein the destination virtual disk is a child of the source virtual disk, and wherein the method further comprises: determining whether the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine; as well as In response to determining that the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine, removing the source virtual disk on the first node, the source virtual disk serving as a parent of the destination virtual disk on the second node; as well as The plurality of data blocks contained in the destination virtual disk and the data contained in the difference disk are merged into the destination virtual disk.

7. The method of claim 1 , retrieving the requested data block comprising: Determining whether the source difference disk contains the requested data block; as well as In response to determining that the source differencing disk contains the requested data block, the requested data block is provided from the source differencing disk to the destination virtual disk via the computer network.

8. The method of claim 1 , retrieving the requested data block comprising: Determining whether the source difference disk contains the requested data block; as well as In response to determining that the source difference disk does not contain the requested data block, Retrieving the requested data block from mirror storage; as well as The retrieved data blocks are provided from the mirror storage to the destination virtual disk via the source differencing disk.

9. A computing device interconnected with additional computing devices in a distributed computing system via a computer network, the computing device comprising: processor; as well as a memory containing instructions executable by the processor to cause the computing device to: instantiating and starting a destination virtual machine on the computing device for live migration of a source virtual machine from another computing device, the computing device having a destination virtual disk that is a child of a source virtual disk and a differencing disk that is a child of the destination virtual disk, the source virtual disk containing a plurality of data blocks of the source virtual machine after suspension; receiving a request from the destination virtual machine, the request being for a data block of the source virtual machine; as well as In response to receiving the request for the data block, determining whether the requested data block is available at the difference disk; as well as In response to determining that the requested data block is not available at the difference disk, determining whether the destination virtual disk contains the requested data block; as well as In response to determining that the destination virtual disk does not contain the requested data block, Retrieving the requested data block from the source virtual disk via the computer network; writing the retrieved data blocks to one or more empty sectors of the destination virtual disk; After completing writing of the retrieved data block, marking one or more sectors containing the written data block as read-only; as well as A copy of the retrieved data block is provided from the destination virtual disk to the destination virtual machine.

10. The computing device of claim 9, wherein the memory includes additional instructions executable by the processor to cause the computing device to: In response to receiving the request for the data block, determining whether the requested data block is available at the difference disk; and In response to determining that the requested data block is available at the differencing disk, the data block is provided from the differencing disk in response to the received request.

11. The computing device of claim 9, wherein the memory includes additional instructions executable by the processor to cause the computing device to: In response to receiving the request for the data block, determining whether the requested data block is available at the difference disk; In response to determining that the requested data block is not available at the differencing disk, determining whether the destination virtual disk contains the requested data block; and In response to determining that the destination virtual disk contains the requested data block, the data block is provided from the destination virtual disk in response to the received request without retrieving the data block from the source virtual disk.

12. The computing device of claim 9, wherein the memory includes additional instructions executable by the processor to cause the computing device to: Repeating the fetching, writing and marking operations for the additional plurality of data blocks via background processing; determining whether the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine; and In response to determining that the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine, removing the source virtual disk on the first node as a parent of the destination virtual disk on the second node.

13. The computing device of claim 9, wherein the memory includes additional instructions executable by the processor to cause the computing device to: determining whether the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine; and In response to determining that the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine, removing the source virtual disk, the source virtual disk being a parent of the destination virtual disk; and The plurality of data blocks contained in the destination virtual disk and the data contained in the difference disk are merged into the destination virtual disk.

14. A method for dynamically migrating virtual machines on nodes interconnected via a computer network in a distributed computing system, the method comprising: During a post-copy live migration of a source virtual machine on a first node to a destination virtual machine on a second node, receiving, at the second node, a request from the destination virtual machine, the request being for a data block among a plurality of data blocks of the suspended source virtual machine, wherein the destination virtual machine includes a destination virtual disk as a parent and a difference disk as a child of the destination virtual disk; as well as In response to receiving the request from the destination virtual machine, determining whether a requested data block of the plurality of data blocks is available at the difference disk; In response to determining that the requested data block is not available at the difference disk, determining whether the destination virtual disk on the second node contains the requested data block among the requested multiple data blocks of the source virtual machine after the suspension of the source virtual machine on the first node, after which sectors of the destination virtual disk can be written only once and are read-only; and In response to determining that the destination virtual disk does not contain the requested data block from the plurality of data blocks, Retrieving a copy of the data block from a source virtual disk at the first node via the computer network; writing the retrieved data block to one or more sectors on the destination virtual disk at the second node; The one or more sectors containing the written data blocks are marked as read-only on the destination virtual disk.

15. The method according to claim 14, further comprising: obtaining a copy of the data block from the one or more read-only sectors of the destination virtual disk; as well as The obtained copy of the data block from the destination virtual disk is provided to the destination virtual machine to continue the live migration.

16. The method of claim 14, wherein the destination virtual machine includes a differencing disk as a child of the destination virtual disk, and wherein the method further comprises: determining whether the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine; as well as In response to determining that the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine, removing the source virtual disk on the first node as a parent of the destination virtual disk on the second node.

17. The method of claim 14, wherein the destination virtual machine includes a differencing disk as a child of the destination virtual disk, and wherein the method further comprises: determining whether the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine; as well as In response to determining that the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine, removing the source virtual disk on the first node, the source virtual disk serving as a parent of the destination virtual disk on the second node; as well as After detecting a failure of the destination virtual machine, another destination virtual machine is instantiated and started on the second node using the complete copy of the plurality of data blocks in the destination virtual disk and the data contained in the difference disk.

18. The method of claim 14, wherein the destination virtual machine includes a differencing disk as a child of the destination virtual disk, and wherein the method further comprises: determining whether the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine; as well as In response to determining that the destination virtual disk contains a complete copy of the plurality of data blocks corresponding to the source virtual machine, the plurality of data blocks contained in the destination virtual disk and the data contained in the difference disk are merged into the destination virtual disk.

Images