Storage device and method for assisting virtual machine migration

Abstract

A storage device and a method of assisting virtual machine migration are disclosed. The storage device can include at least one controller for a virtual machine (VM) on a source host, a first storage device that can store data of the VM, a second storage device that can store a storage state of the VM, a storage device controller that can process at least one read request for the VM received from the controller using the first storage device and process at least one write request for the VM received from the controller using the first storage device, and a VM migration state monitoring and capturing module that can assist the VM to migrate from the source host to a destination host.

Description

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 040,516, filed June 17, 2020, which is incorporated herein by reference for all purposes. Technical Field

[0002] The inventive concept generally relates to storage devices, and more specifically, to storage devices that support the live migration of virtual machines. Background Technology

[0003] The volume of data generated, analyzed, and consumed is constantly growing. One of the technologies adopted to efficiently utilize data centers is virtualization. Virtualization technology enables physical servers to run multiple virtual machines (VMs) for user applications on a single physical server. This technology allows data centers to support a large number of user applications using existing resources and reduces the underutilization of physical resources.

[0004] Data centers can move (i.e., migrate) virtual machines from one server to another. Migrating VMs can utilize the resources of both the source and destination machines and may involve temporarily halting the VM being migrated. Therefore, VM migration is expected to be as efficient as possible.

[0005] Improvements are still needed in migrating VMs from one server to another. Summary of the Invention

[0006] According to an embodiment of the inventive concept, a storage device is provided, the storage device comprising: at least one controller for a virtual machine (VM), the VM being on a source host; a first storage device for storing data of the VM; a second storage device for storing the state of the VM; a storage device controller for using the first storage device to process at least one read request for the VM received from the controller, and for using the first storage device to process at least one write request for the VM received from the controller; and a VM migration state monitoring and capture module for assisting the VM in migrating from the source host to a destination host.

[0007] According to an embodiment of the inventive concept, a method is provided, the method comprising: receiving, on a storage device on a source host, a command for assisting in migrating a virtual machine (VM) from the source host to a destination host; tracking a first change in the storage state of the VM in the storage device; tracking a second change in the storage data of the VM in the storage device; sending the first change in the storage state of the VM from the storage device; and sending the second change in the storage data of the VM from the storage device.

[0008] According to an embodiment of the inventive concept, a method is provided, the method comprising: receiving, on a storage device on a destination host, a command for assisting in migrating a virtual machine (VM) from a source host to a destination host; receiving, on the storage device, a change in the storage state of the VM; updating the storage device to match the change in the storage state of the VM; receiving, on the storage device, a change in the storage data of the VM; and updating the storage data of the VM on the storage device to match the change in the storage data of the VM. Attached Figure Description

[0009] Figure 1 The machine shown is an embodiment of the invention, including a storage device equipped to assist in real-time migration.

[0010] Figure 2 Examples of embodiments according to the inventive concept are shown. Figure 1 Additional details about the machine.

[0011] Figure 3 The illustration shows an embodiment of a virtual machine (VM) supported according to the inventive concept. Figure 1 Machines and storage devices.

[0012] Figure 4 Examples of embodiments according to the inventive concept are shown. Figure 1 Details of the storage device.

[0013] Figure 5 The invention illustrates an embodiment of a virtual machine being migrated from one machine to another.

[0014] Figure 6 The embodiments shown are accessible according to the inventive concept. Figure 1 The storage device holds details of the state information.

[0015] Figure 7 This illustrates an embodiment of the inventive concept that determines whether a threshold number of logical block addresses (LBAs) have been written. Figure 4 VM migration status monitoring and capture module.

[0016] Figure 8 Examples of embodiments according to the inventive concept are shown. Figure 4 The VM migration status monitoring and capture module is Figure 1 The status of the storage device generates a log page.

[0017] Figures 9A to 9B Examples of embodiments according to the inventive concept are shown. Figure 1 Storage devices support VMs from Figure 1 A flowchart illustrating an example process of real-time migration of a machine.

[0018] Figure 10 Examples of embodiments according to the inventive concept are shown. Figure 4 The flowchart shows an example process by which the VM migration status monitoring and capture module determines whether a threshold number of LBAs have been written.

[0019] Figure 11 Examples of embodiments according to the inventive concept are shown. Figure 4 The VM migration status monitoring and capture module generates and transmits VM migration status. Figure 4 A flowchart of an example process for reporting the storage status.

[0020] Figures 12A to 12B Examples of embodiments according to the inventive concept are shown. Figure 1 Storage devices support VM to Figure 1 A flowchart illustrating an example process of real-time migration of a machine. Detailed Implementation

[0021] Referring now to embodiments of the inventive concept, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the inventive concept. However, it should be understood that those skilled in the art can practice the inventive concept without these specific details. In other instances, well-known methods, processes, components, circuits, and networks are not described in detail to avoid unnecessarily obscuring aspects of the embodiments.

[0022] It will be understood that although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of the inventive concept, a first module may be referred to as a second module, and similarly, a second module may be referred to as a first module.

[0023] The terminology used herein in the description of the inventive concept is for the purpose of describing particular embodiments only and is not intended to limit the inventive concept. As used in the description of the inventive concept and the appended claims, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein means and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Components and features in the drawings are not necessarily drawn to scale.

[0024] As the volume of data generated, analyzed, and consumed grows, data center sizes are expanding to keep pace. New business models for cloud-based IT resources are emerging. Cloud computing enables users to have more economical IT infrastructure that allows them to use resources on demand, pay-as-you-go, and use what they need, providing significant cost benefits to end users. Because of these factors, data centers are becoming increasingly complex and large, involving both capital expenditures (capex) and operating expenditures (opex). Therefore, the efficient utilization of data center resources, such as servers, has become crucial.

[0025] One of the many technologies developed and adopted to efficiently utilize data centers is virtualization. Virtualization technology enables physical servers to run multiple virtual machines (VMs) for user applications on a single physical server. This technology allows data centers to support a large number of user applications with existing resources and reduces the underutilization of physical resources.

[0026] A virtualization operating system (OS) (also known as a hypervisor) runs on a physical server and facilitates the operation and management of virtual machines (VMs). The hypervisor handles VM startup, resource allocation, isolation and fairness among multiple VMs, and VM migration from one server to another. Each VM requires four main resources—1) compute, 2) system memory, 3) network connectivity, and 4) storage—for smooth and proper operation. The hypervisor provides, manages, and allocates these resources to the VMs running on that server. In a typical data center, thousands of physical servers can run a very large number of VMs. Therefore, load balancing of VMs across a given set of servers or server cluster can also be critical.

[0027] Sometimes, maintenance and upgrades are required for these servers. Sometimes, VM requirements may need to be upgraded due to resource demands or user requests. It's also possible that some servers may be running too few VMs, and consolidating these VMs on another server may make sense to save costs. All these reasons necessitate VM migration capabilities in servers and virtual machine monitors.

[0028] VM migration should be as non-intrusive and non-destructive as possible. Ideally, applications running on the VM should remain unaware that the underlying VM is being migrated to a different server. This process is also known as live migration. Therefore, the goal is to reduce and minimize the time spent moving VMs from the source server to the destination server. When a VM migrates, the VM monitor essentially copies the VM's state from the source server to the destination server. The VM's state includes processor state, memory state, network state, and storage state.

[0029] As further described below, VM migration is typically completed in three phases: 1) Preparation, 2) Data Replication (brownout), and 3) Incremental Replication (blackout). In the preparation phase, various resources required by the candidate VM are allocated or provisioned on the destination server. This negotiation occurs between the source and destination servers. Once resources are reserved and allocated for the VM on the destination server, the second phase begins. In the data replication phase (also known as the brownout phase), the source VM monitor begins moving VM data to the destination server. During this phase, both the VM and the VM monitor actively access the VM data on the source solid-state drive (SSD). It is possible that during this phase, the VM may modify some data being moved to the destination. Once the entire data replication is complete, the live migration enters the incremental replication phase (also known as blackout). In this phase, the source VM monitor suspends or pauses the VM being migrated. Once the VM is frozen, it cannot change any data or state of its resources. At this point, the VM monitor performs replication on incremental changes, compared to the replication operations performed by the VM monitor during the second phase. Once the incremental changes have been replicated, the destination VM monitor starts the VM on the destination server, completing the live migration process.

[0030] The VM's storage resources can be provided from directly attached SSDs or remotely from discrete storage devices. During the aforementioned data replication operation, the source server hosting the VM can allocate compute resources, memory resources, and in some cases, storage resources to track incremental changes in the third phase. This state can be captured as a bitmap of individual LBAs or a series of LBAs. Similarly, the destination server can allocate similar resources for the transfer. These resources can add additional costs to the solution and, among other limitations, unintentionally impose latency constraints on existing visitors to these servers.

[0031] Embodiments of the inventive concept may include systems and methods for performing fast and efficient real-time VM migrations from a source server to a destination server using SSD assistance. VM migrations are performed for various reasons, such as server consolidation and load balancing. As part of the VM migration process, data and state of various VM resources (such as CPU, memory, storage devices, and networking) can be copied on the destination server. The VM can be paused during state transfer and restart at the destination. Minimal VM pauses are desired so that applications running on the VM are unaffected. Embodiments of the inventive concept may use an SSD controller to efficiently assist in moving stored data to the destination server.

[0032] When a virtual machine monitor makes a decision to migrate a VM, the source virtual machine monitor (i.e., the virtual machine monitor on the source server) can alert the SSD device (on the source server) to the decision. The SSD controller can then begin tracking storage state changes and data changes. The SSD controller can create dirty state and LBA-range logs to minimize pause times during VM transfer of storage data and state. The SSD controller can also adjust data prefetching and write-to-data cache policies during the VM migration activation period, enabling fast and efficient access to data belonging to that VM. The SSD controller can also track changes to the last snapshot, and the source virtual machine monitor can subsequently use the increments in a streaming manner. Embodiments of the inventive concept described herein can also be used in other storage device use cases (such as snapshot, archiving, and backup features) to achieve efficient execution.

[0033] System Architecture

[0034] A source server can host multiple virtual machines (VMs). Each VM can be assigned to a physical function and a non-volatile memory fast (NVMe) controller in an attached SSD. (While NVMe controllers have been specifically discussed, embodiments of the inventive concept may use other protocols for communication and should not be considered limited to NVMe controllers.) The server may have multiple SSDs attached to it, and each SSD may contain one or more NVMe controllers. From a pool of NVMe controllers (also referred to as storage controllers), one or more controllers can be assigned to each VM hosted by a virtual machine monitor virtualization operating system (OS) running on the server. Each VM may have its own guest OS, system software stack, and applications. Each VM has access to server resources such as central processing unit (CPU) bandwidth, system memory, network bandwidth, and storage bandwidth and capacity. These resources can be managed, allocated, and assigned to each VM hosted on the server by the virtual machine monitor.

[0035] The destination server can have similar settings and configurations to the source server. One or more VMs from the source server can be migrated to the destination server for various reasons such as load balancing and server consolidation.

[0036] VM storage status

[0037] Each VM running on a virtual machine monitor can have various state vectors at any given time. VM state can have multiple components, including processor state, application execution state, memory state, network state, and storage state. Regarding storage state, VM state indicates VM activity related to persistent storage of data. Each VM can have storage devices attached to the VM via a passthrough mechanism through the virtual machine monitor.

[0038] The storage device state can then have three main components: 1) PCIe transfer connection state, 2) NVMe controller state, and 3) Flash Translation Layer (FTL) state.

[0039] The PCIe interface between the source server and the SSD storage device may have certain configurations and parameters set by the source virtual machine monitor or VMPCIe driver. Some examples of PCIe configuration states include: various PCIe capability settings (such as Fast Peripheral Component Interconnect Fast (PCIe), Message Signaling Interruption (MSI and MSI-X), Advanced Error Reporting (AER), power management, etc.) and configuration space settings (such as interrupts, maximum load size, etc.).

[0040] During the lifetime of a VM, the VM NVMe driver can perform various NVMe configurations and settings. NVMe state can consist of NVMe registers exposed through the PCIe BAR window. NVMe state can also consist of various feature settings and configurations performed through the NVMe driver. Some examples of NVMe-level storage device states include: memory page size, doorbell stride, timeout value, commit queue (SQ) arbitration settings, etc.; feature settings (such as power management, queue count, asynchronous event notification, interrupt merging, atomicity, etc.); created namespaces, security settings, etc.; and log pages created so far, etc.

[0041] FTL states can correspond to the state of logical-to-physical address mapping, flash media block usage, write / read cache states, and the associated NVMe controller and consequently, the settings of the associated VM. In some embodiments of the inventive concept, VM migration can reproduce the FTL states on the destination SSD as close as possible to the source SSD. In other embodiments of the inventive concept, copying the FTL states may not be performed.

[0042] The SSD can also track input / output (I / O) activity. That is, the SSD controller can record VM data writes according to LBA ranges. In some embodiments of the inventive concept, the SSD controller can record write LBA ranges below a certain threshold based on the number of unique LBAs written. Therefore, the SSD controller can track the range of LBAs written by the VM and a flag indicating whether the number of written LBAs exceeds a certain threshold (which may vary depending on the VM).

[0043] SSD architecture optimized for VM migration

[0044] SSDs use flash memory for persistent data storage. An SSD controller typically consists of the following main modules: 1) a host interface, 2) a flash translation layer (FTL), and 3) flash channels. The SSD controller architecture allows the host to access the performance of the flash memory.

[0045] The host interface essentially provides a logical view of persistent storage to the attached host. Flash storage capacity can be advertised in units of logical blocks or sectors, or it can be advertised as object storage. In the case of block storage, the host can issue I / O commands using logical block addresses (LBAs), or the host can also use object storage protocols (such as key-value (KV)). The host interface can implement PCIe as a transport protocol to carry host storage protocols (such as NVMe). The host can issue storage I / O commands and management commands using command submission queues (SQs). The host interface can perform SQ arbitration and select the command to be executed. The host interface can perform direct memory access (DMA) operations to or from host memory as needed. After command execution, the host interface can send a completion entry (CE) to the host. The host interface can also send interrupts (such as MSI or MSI-X) to the host to indicate that one or more commands have been executed.

[0046] Once the host interface selects the command to execute, it retrieves and interprets the command issued by the host. The host interface extracts appropriate fields (such as Logical Block Address (LBA), data transfer length, and source / destination addresses). The host interface then provides these parameters to the Flash Translation Layer (FTL) for further command execution. The FTL module essentially translates logical addresses into actual physical addresses where data can reside in the flash media. The FTL layer maintains a logical-to-physical address translation table and performs lookups to obtain the physical addresses. In addition to address translation, the FTL performs other functions required for flash media management.

[0047] Flash media may possess certain read / write and persistence characteristics that should be managed. An SSD controller manages flash media, enabling its use to provide reliable, high-performance, and cost-effective host data storage. A multitude of internal house-keeping operations exist that an SSD controller can perform for the normal and healthy operation of the SSD. Some examples of these internal or background operations are: recirculation, garbage collection, FTL data structure updates, various statistics and log updates, cache unloading, and others.

[0048] When commands are written to a flash channel, the FTL layer provides the physical address and data. The flash channel can be used to perform all actual data read or write operations on the flash media. The flash channel implements the specific bus protocol required to access the media, and it also enables other important functions such as error detection and correction.

[0049] Embodiments of the inventive concept can implement a number of features to facilitate real-time VM migration processing. When a source server host decides to move a VM to another destination server, the source server can notify the PCIe Physical Function (PF) / NVMe controller associated with the VM of this decision. The virtual machine monitor can set flags in the appropriate SSD and NVMe controller. The host can use NVMe setting feature commands to set the "VM_MIGRATION_ACTIVE" flag in the host interface. The host can also use custom vendor-defined NVMe commands to set the VM_MIGRATION_ACTIVE flag in the SSD. These commands are merely examples, and the host can use any other such mechanism to set such a flag in the SSD.

[0050] While the examples above focus on the NVMe protocol, similar flags, operations, commands, and / or functions can be obtained using other protocols: the specific descriptions of commands, operations, functions, and / or features relating to NVMe herein should not be considered as limited to NVMe, and similar commands, operations, functions, and / or features of other protocols may be used interchangeably.

[0051] In some embodiments of the inventive concept, the VM to be migrated can still be active (live), and the SSD can be fully owned by the VM (pass-through). In such embodiments of the inventive concept, if the active path is not available to the virtual machine monitor, NVMe commands can be sent through the management interface.

[0052] Once the VM_MIGRATION_ACTIVE flag is set in the SSD, the SSD can begin monitoring all I / O activity and status of the PF / NVMe controller. The VM migration status monitoring and capture module can create logs or reports of the VM storage status described earlier. If the host changes any status parameters during the migration phase, the SSD can update the logs / reports appropriately. At some point, the source virtual machine monitor can read the status logs or reports from the source SSD using the NVMe Get Log command. The source virtual machine monitor can also use custom vendor-defined NVMe commands to read the storage status of the VM being migrated. The above commands are just examples, and other mechanisms are also possible.

[0053] The source virtual machine monitor reads the storage state log page and then sends the log or report to the destination server. At the destination server, the destination virtual machine monitor uses the NVMe Set Feature command to provide the storage state log or report to the appropriate SSD and NVMe controllers for the VM being migrated. The SSD controller can use the storage state log received from the host and can insert settings and configurations into the appropriate modules (such as PCIe, NVMe controller, and FTL). Using this method, the storage state of the destination SSD / NVMe controller can be ready or very close to being ready for VM migration. This feature helps reduce the time required to stop the VM on the source server and restart it on the destination server. Therefore, downtime (also known as power outage time) during live migration is reduced.

[0054] As described above, once the VM_MIGRATION_ACTIVE flag is set, the SSD controller can track I / O operations performed by the VM. The SSD controller can create and maintain a dirty log of LBA ranges modified by the VM. After setting the VM_MIGRATION_ACTIVE flag, the source virtual machine monitor can move all flash data blocks (LBAs, namespaces, etc.) to the destination server. The source virtual machine monitor can perform a sequential read of all data belonging to the VM and send that data to the destination virtual machine monitor. The destination virtual machine monitor can then write that data to the associated SSD and NVMe controllers on the destination server. In another embodiment of the inventive concept, the destination virtual machine monitor may have already prepared the storage at the destination server using a last known backup image or the original VM image and created checksums for the LBAs by a single count or range. In this case, the destination virtual machine monitor can compare the old checksum with the new checksum and copy only the mismatched LBAs.

[0055] In one embodiment of the inventive concept, the SSD can track the actual LBA range written by the VM during its lifetime as part of the storage state of each VM. This information can be a portion of a storage state log or report page. Using this information, the source virtual machine monitor can avoid reading the entire flash memory capacity allocated to the VM. Instead, the source virtual machine monitor can only read the actual LBA range written by the VM, thus reducing the amount of data moved.

[0056] In some embodiments of the inventive concept, a threshold may exist, exceeding which the source virtual machine monitor may move only the entire capacity as a sequential read workload. For example, consider a scenario where approximately 100GB of storage capacity is allocated to a VM. If the VM has only written approximately 4GB of LBAs before the source virtual machine monitor decides to migrate the VM, the source virtual machine monitor may only move the actual LBAs to the destination virtual machine monitor, which may be less than the entire allocated capacity (approximately 100GB) that must be moved. However, by using a threshold, the burden on the source virtual machine monitor is reduced. If the VM has already written, for example, approximately 98GB of LBAs before the source virtual machine monitor decides to migrate the VM, the LBA write information is less helpful because almost the entire allocated capacity needs to be moved to the destination server or destination SSD. Therefore, if the source virtual machine monitor detects that a threshold flag is set (indicating that the written LBAs are greater than a certain threshold), the source virtual machine monitor may copy the entire data capacity to the destination server.

[0057] As mentioned earlier, once the source virtual machine monitor sets the VM_MIGRATION_ACTIVE flag in the SSD controller, the SSD controller can create a dirty log for LBAs written by the VM from that point onwards. The dirty log can be used to replicate modified data after the initial full data move. While the source virtual machine monitor is moving data to the destination server, the VM may remain active during what is sometimes called the power-down phase. This is the phase where both the source virtual machine monitor and the VM have active access to the SSD. Any LBAs modified during this phase can be replicated to the destination server later, before the VM is started on the destination server. At the end of the power-down phase, the source virtual machine monitor can stop or pause the VM on the source server, replicate any incremental changes to its state and data to the destination server, and finally start the VM on the destination server. This final phase is sometimes also referred to as the power-down phase.

[0058] The objective is to minimize the length or time spent in the power-down phase. Power-down time is the period during which the VM is interrupted. Embodiments of the inventive concept can use assistance from an SSD to help expedite the power-down phase. After the source VM monitor stops the VM, no further state changes or LBA data changes occur on the source SSD. At this point, the source VM monitor can read the dirty log from the SSD to identify what LBAs and state parameters were modified during the power-down phase. The source VM monitor can then copy the incremental state changes and / or modified (or incremental or dirty) LBAs to the destination. Once the incremental state and LBA data are copied to the destination, the VM can be started on the destination server. At this point, the live migration of the VM is successfully completed.

[0059] In another embodiment of the inventive concept, the SSD controller may take additional steps to assist the live VM migration process. Once the VM_MIGRATION_ACTIVE flag is set, the SSD controller understands that a live VM migration is in progress and that the source virtual machine monitor is about to copy the VM data to the destination. The SSD controller may begin prefetching sequential data of interest or some of the LBA range from the flash media. In some embodiments of the inventive concept, the data stored in the flash media may be encrypted: data decryption may take additional time. Prefetching such encrypted data can result in data decryption before the migration data is requested. Such a data prefetching strategy can result in faster completion of virtual machine monitor data read commands. This reduction in the latency of the source virtual machine monitor read commands can help reduce the overall power-down phase duration (where fewer data are modified during the power-down phase, the power-down phase may require less time to copy the reduced modified data).

[0060] In another embodiment of the inventive concept, once the VM_MIGRATION_ACTIVE flag is set, the SSD controller can provide higher priority to associated VM data for read and write caching. That is, the SSD controller can preferentially cache data accessed by that VM. Such caching can accelerate command completion for both the migrating VM and the source virtual machine monitor.

[0061] In another embodiment of the inventive concept, the source SSD can track all changes to the last known image. The destination server SSD can be synchronized with this last known image. Here, both SSDs understand the image state definition. Once the source server is ready to transfer the VM, it can instruct the source and destination SSDs to be ready for streaming. In streaming mode, the SSDs can prepare the data to be transferred, and once started, the host server can facilitate the streaming of data from the source SSD to the destination SSD. The source and destination servers can transfer the data as is without decoding it.

[0062] Migration Assist NVMe Commands

[0063] The source virtual machine monitor can configure live migration assistance features in the SSD controller using the NVMe command extensions described below. These NVMe commands use standard formats and protocols, with some fields reused or assigned new syntax. Newly customized vendor-defined NVMe commands can also be used for this purpose. The source virtual machine monitor and the SSD controller can also use completely different communication methods to achieve the same functionality. The following sections describe some examples of possible NVMe command extensions that can be used to facilitate live VM migration assistance from SSDs.

[0064] Sources can use the NVMe Set Features – VM MIGRATION ACTIVE command to set certain NVMe controller features in an SSD. Some examples of existing features that can be set via a source include: command submission queue arbitration parameters, power management settings, temperature thresholds, queue count, and interrupt merging configuration.

[0065] The source virtual machine monitor can also set the following parameters: VM_MIGRATION_ACTIVE, LBA_WRITTEN_THRESHOLD, and LBA_WRITTEN_THRESHOLD_EXCEEDED.

[0066] VM_MIGRATION_ACTIVE can be a flag set by the source virtual machine monitor. This flag instructs the virtual machine monitor during the migration process of VMs associated with this PF / NVMe controller. When this flag is set, the SSD controller can prepare certain storage state logs and reports associated with this PF / NVMe controller as described in the preceding sections. The SSD controller can also begin creating dirty logs for VM I / O activity. When this flag is set, the SSD controller can also employ different data prefetching settings and data caching strategies.

[0067] The LBA_WRITTEN_THRESHOLD flag can be used by the SSD controller to track the actual LBA ranges written by the VM during its lifetime. If the threshold is set to 0, all LBA ranges written by the VM are logged. For non-zero threshold values, LBA ranges written by the host are only logged up to the threshold size set by the host. If the threshold size is exceeded, the SSD controller can set the LBA_WRITTEN_THRESHOLD_EXCEEDED flag for source reads. During VM migration, if the source VM monitor detects that the LBA_WRITTEN_THRESHOLD_EXCEEDED flag is set, the source VM monitor can move only the entire capacity allocated to that VM to the destination server. If the LBA_WRITTEN_THRESHOLD_EXCEEDED flag is not set, the source VM monitor can move only the logged LBA ranges, thus reducing the amount of data transferred to the destination server.

[0068] Once the VM_MIGRATION_ACTIVE flag is set via the source virtual machine monitor, the SSD controller can log the LBA range written to LBA_DIRTY_LOG. The source can use this log to detect and replicate incremental data changes made by the VM during power-down phases. In other words, the source virtual machine monitor can initially replicate all VM data while the VM is still active, and then only replicate the differences after the VM is paused for migration. Using this data replication process, the source virtual machine monitor can ensure data integrity on the destination server when the VM is restarted.

[0069] The source virtual machine monitor can use the NVMe Get Features – VM MIGRATION ACTIVE command to read the current values ​​of the parameters of the VMMIGRATION ACTIVE feature. The source can obtain parameter values ​​such as: VM_MIGRATION_ACTIVE, LBA_WRITTEN_THRESHOLD, and LBA_WRITTEN_THRESHOLD_EXCEEDED.

[0070] The destination virtual machine monitor can use the NVMe Set Features – VM STORAGE STATE command extension to provide VM storage state values ​​to the SSD controller on the destination SSD. The SSD controller on the destination SSD can then insert these values ​​into the appropriate functional layers (such as PCIe transport, NVMe host layer, and FTL). By doing so, the destination SSD controller becomes ready or nearly ready to store I / O commands from the migrated VM. The source VM storage state can be obtained from the source SSD by the source virtual machine monitor using the Get Log Page command, as described elsewhere. The source virtual machine monitor can then transfer these state values ​​to the destination server.

[0071] The source virtual machine monitor can use the NVMe Get Log Page – VM STORAGE STATE command to read the VM storage state from the attached SSD. Various possible parameters and fields of the VM storage state are described elsewhere. After reading the VM storage state log, the source virtual machine monitor can send the log to the destination virtual machine monitor. The destination virtual machine monitor can then use the new Set Features – VM STORAGE STATE command to provide the relevant storage parameters and values ​​to the SSD attached to the destination server.

[0072] After setting the VM_MIGRATION_ACTIVE flag in the source virtual machine monitor, the source virtual machine monitor can use the NVMe Get Log Page – VM DIRTY STATE command to read incremental changes that occur in the VM storage state. The source virtual machine monitor can use this command after the VM on the source server has been paused or frozen. The source virtual machine monitor can initially copy the entire storage state while the VM is running, and then only copy the storage state differences after the VM has stopped. The incremental storage state can be sent to the destination virtual machine monitor, which can then use the Set Features – VMSTORAGESTATE command to reflect the final VM storage state on the destination server's SSD.

[0073] The source virtual machine monitor can use the NVMe Get Log Page –VM DIRTY LBA RANGES command to read LBAs written or modified by the VM during a power-down phase. During a power-down phase, the source virtual machine monitor can replicate all LBAs allocated by the VM and / or written data to the destination server while the VM is running concurrently. Because the VM is active during this phase, it can write to and modify some storage LBAs. Therefore, during the power-down phase (after the VM stops on the source server), the source virtual machine monitor can replicate modified or dirty LBAs to the destination server. After the virtual machine monitor sets the VM_MIGRATION_ACTIVE flag, the SSD can track and create a dirty LBA range log. The SSD controller can use LBA_WRITTEN_THRESHOLD to log the range of LBAs updated by the VM. If the number of LBAs written by the VM exceeds the threshold set by the source virtual machine monitor, the SSD can set the LBA_WRITTEN_THRESHOLD_EXCEEDED flag. After reading the log, the source virtual machine monitor can use the LBA range to read the data and send it to the destination server for persistent storage on the SSD attached to the destination server for its lifetime. If the source virtual machine monitor finds that the LBA_WRITTEN_THRESHOLD_EXCEEDED flag is set in the log page, the source virtual machine monitor can choose to ignore the dirty LBA range and only copy the entire LBA space.

[0074] SSD controllers can use the Identify–VM MIGRATION ASSIST SUPPORT data structure to indicate VM live migration assistance support for both the source and destination virtual machine monitors. Unused fields of the Identify data structure can be used for this purpose.

[0075] All the commands described above are examples of some embodiments of the inventive concept. Other embodiments of the inventive concept may use different commands or create new, custom, vendor-defined commands to achieve the same functionality.

[0076] Source Virtual Machine Monitor Operation Procedure

[0077] At a high level, live VM migration from the source server to the destination server is completed in three phases. In the first initial phase, following the migration decision, migration planning and resource provisioning take place. In the second phase (also known as the power-down phase), the source VM monitor replicates the current state and values ​​of VM resources (such as storage, network, and storage devices) to the destination server. In the final phase (also known as the power-down phase), the source VM monitor stops or pauses the VM and replicates incremental differences from the earlier copy to the destination server, after which the VM can be restarted on the destination server.

[0078] To replicate storage state and storage data from the source server to the destination server, the source virtual machine monitor performs a series of steps. First, during a power-down phase, while the VM is still running, the source virtual machine monitor sets the VM_MIGRATION_ACTIVE flag on the SSD. The source virtual machine monitor then initiates the first replication of the storage state and stored data. The source virtual machine monitor can use the Get Log Page command to read the current storage state. The read storage state can then be passed to the destination server for restoration. Afterward, the source virtual machine monitor can decide whether to replicate certain LBA ranges to the destination or move the entire capacity allocated to the VM. The source virtual machine monitor can use the Get Features command to retrieve the LBA ranges written by the VM during its lifetime. If the LBA_WRITTEN_THRESHOLD_EXCEEDED flag is set, the virtual machine monitor reads the entire capacity and sends that data to the destination server. If the LBA_WRITTEN_THRESHOLD_EXCEEDED flag is not set, the source virtual machine monitor selectively reads data from the LBA ranges written by the VM and sends that data to the destination server. After all VM resource state and data have been moved to the destination server, the source VM monitor can stop the VM. The source VM monitor can then use the Get Log Page command to read incremental modifications to the storage state and LBA data. The source VM monitor can then move the state and data differences to the destination server. At this point, the entire state and data values ​​of the VM being migrated can be restored and prepared on the destination server. The VM can then be restarted on the destination server.

[0079] Destination Virtual Machine Monitor Operation Procedure

[0080] The destination virtual machine monitor can restore the VM resource state on the destination server. For a successful VM migration, the VM resource state and data can be recreated on the destination server. By making the VM resource state and data on the destination server identical or very close to those on the source server, applications running on the migrated VM will be unaffected. In other words, the purpose of live VM migration is to make the migration process transparent to the actual application.

[0081] The destination virtual machine monitor receives the state of VM resources on the source server. For storage state, the destination virtual machine monitor can use the Set Features command to provide the VM STORAGE STATE to the attached SSD. The destination virtual machine monitor receives all VM data from the source server. The destination virtual machine monitor retains all data received from the source server. The destination virtual machine monitor then receives incremental differences in VM state and data. The destination virtual machine monitor then retains the modified data and performs any reconfiguration of resources based on incremental state changes on the source server. At this point, the destination server can replicate the state and VM data of the VM resources. In other words, at this point, the destination virtual machine monitor is ready to launch the migrated VM on its server.

[0082] In another embodiment of the inventive concept, the VM storage state and data migration mechanism can also be used to capture snapshot images and serve as an archiving, backup, and recovery mechanism.

[0083] Figure 1 An embodiment of the invention is shown, comprising a machine equipped with a storage device to assist in real-time migration. Figure 1 In the diagram, machine 105 (also referred to as a host) is shown. Machine 105 may include processor 110. Processor 110 may be any type of processor. (For ease of illustration, processor 110 and other components discussed below are shown externally to machine 105; embodiments of the inventive concept may include these components within machine 105.) Although Figure 1 A single processor 110 in machine 105 is shown, but machine 105 may include any number of processors, each of which may be a single-core or multi-core processor, each of which may implement a reduced instruction set computer (RISC) architecture or a complex instruction set computer (CISC) architecture (and other possibilities), and may be mixed in any desired combination.

[0084] Machine 105 may also include memory 115. Memory 115 may be any type of memory (such as flash memory, dynamic random access memory (DRAM), static random access memory (SRAM), persistent random access memory, ferroelectric random access memory (FRAM), or non-volatile random access memory (NVRAM) (such as magnetoresistive random access memory (MRAM)). Memory 115 may also be any desired combination of different memory types. Machine 105 may also include a memory controller 120, which can be used to manage access to memory 115.

[0085] Machine 105 may also include storage device 125. Storage device 125 can be used to store data. Processor 110 can run device driver 130, which can support access to storage device 125. Embodiments of the inventive concept may include any desired storage device that can operate using any desired storage principle. Thus, the storage device may be a solid-state drive (SSD), hard disk drive, or any other desired storage device, and data may be stored using sector-based storage, block-based storage, or key-based storage, and other possibilities. Although Figure 1 Only one storage device 125 is shown, but embodiments of the inventive concept can support any number of installed storage devices in machine 105.

[0086] although Figure 1 Machine 105 is described as a server (which may be a standalone server or a rack server), but embodiments of the inventive concept may include any desired type of machine 105 without limitation. For example, machine 105 may be replaced by a desktop computer or laptop computer or any other machine that may benefit from embodiments of the inventive concept. Machine 105 may also include dedicated portable computing machines, tablet computers, smartphones, and other computing machines. Additionally, applications that can access data from storage device 125 may reside in another machine, which is separate from machine 105 and accesses machine 105 via a network connection traversing one or more networks of any type (wired, wireless, global, etc.).

[0087] Figure 2 Show Figure 1 Additional details about Machine 105. Figure 2Typically, machine 105 includes one or more processors 110, which may include a memory controller 120 and a clock 205. Processor 110 can be used to coordinate the operation of components of machine 105. Processor 110 may also be integrated with memory 115, which, for example, may include random access memory (RAM), read-only memory (ROM), or other state-saving media. Processor 110 may also be integrated with storage device 125 and network connector 210, which may be, for example, an Ethernet connector or a wireless connector. Processor 110 may also be connected to bus 215, and user interface 220 and input / output interface ports that can be managed using input / output engine 225, as well as other components, may be attached to bus 215.

[0088] Figure 3 Showing support for virtual machines (VMs) Figure 1 Machine 105 and Figure 1 Storage device 125. When machine 105 supports VMs, each VM operates as if it were a physically separate machine and is unaware that other VMs may be running on machine 105. (This does not mean that virtual machines are unaffected by the operations of other VMs: while one VM is executing a command, another VM can be stopped, waiting for the resources of machine 105 to become available.) Therefore, although Figure 3 Machine 105 is shown to support only VM 305, but embodiments of the inventive concept may include any number (zero or more) of VMs on machine 105.

[0089] Each VM can expect to access various resources of machine 105 (such as, Figure 1 Processor 110 Figure 1 The memory 115 and Figure 1 The VM 305 can therefore be considered to include an application 310, which can run on a system software stack 315, which may reside on an operating system 320 (the operating system native to the VM 305 and not necessarily shared across multiple VMs). The VM 305 may also include a memory 325, which may be a storage device 125. Figure 1 The portion of memory 115 (or, in Figure 1 At least a portion of memory 115 may be used in some embodiments of the inventive concept of more than one VM, and memory 325 may be the entirety of memory 115. VM 305 may also include a network driver 330 for accessing a network (which may be a local area network (LAN), wide area network (WAN), metropolitan area network (MAN), global network (such as the Internet) or a combination thereof, and may be wired, wireless or a combination thereof) and a storage driver 335 for accessing a storage device (such as storage device 125).

[0090] To support VM 305, machine 105 may include virtual machine monitor 340. Virtual machine monitor 340 can be considered as an operating system that manages VMs. Each VM communicates with virtual machine monitor 340, which in turn can communicate with network or storage device 125.

[0091] Storage device 125 may disclose one or more functions (such as physical function (PF) 345). The disclosed functions may be PF, virtual function (VF), or a combination thereof. Additionally, as described in U.S. Patent Application No. 16 / 846,271, filed April 10, 2020, now pending (claiming the benefit of U.S. Provisional Patent Application No. 62 / 865,962, filed June 24, 2019, and U.S. Provisional Patent Application No. 62 / 964,114, filed January 21, 2020, all of which are incorporated herein by reference for all purposes), storage device 125 may be associated with a lightweight bridge that may present the PF to machine 105 on behalf of storage device 125. Therefore, the PF and / or VF may be disclosed for storage device 125 by storage device 125 itself or by another component representing storage device 125.

[0092] For each function (such as PF 345), the storage device may have an associated non-volatile memory fast (NVMe) controller (such as NVMe controller 350). NVMe controller 350 manages operations within storage device 125 in response to requests sent via function 345. In some embodiments of the inventive concept, each disclosed function may have its own NVMe controller; in other embodiments of the inventive concept, a single NVMe controller may handle requests on behalf of two or more disclosed functions.

[0093] In some embodiments of the inventive concept, each VM uses a unique function for its request to storage device 125. In other words, when a VM (such as VM 305) sends a request intended for storage device 125, virtual machine monitor 340 can receive the request and direct it to the function associated with that VM. (Therefore, virtual machine monitor 340 can store a mapping between the various functions exposed by storage device 125 and the VMs running on machine 105 to facilitate proper communication between VMs and storage device 125.) Because each VM can use different functions, this option provides storage device 125 with a simple way to know which VM is making the request. Thus, for example, if the storage device provides 16 exposed functions, 16 different VMs can each use one of the exposed functions. However, if each VM uses a unique exposed function, the total number of VMs that machine 105 can support can be limited to the total number of functions exposed by all installed storage devices. Therefore, for example, if machine 105 has a total of three storage devices installed, two of which expose eight functions and the third storage device exposes 16 functions, then the total number of VMs that machine 105 can support can be limited to 32 (eight plus eight plus sixteen).

[0094] In other embodiments of the inventive concept, multiple VMs may share the functionality requested by storage device 125. In other words, multiple VMs may send requests to virtual machine monitor 340, and some (or all) of these requests may be sent to shared functionality exposed by storage device 125. Embodiments of the inventive concept may include all functionality exposed by storage device 125 being shared by at least two VMs, each functionality exposed by storage device 125 being used by a single VM, or any desired combination of both.

[0095] Note that in embodiments of the inventive concept where a single function (such as function 345) is shared by two or more VMs, storage device 125 may not automatically have sufficient information to know what data is actually stored for a given VM. That is, for any given write request sent to storage device 125 via function 345, storage device 125 may not know which VM issued the write request. Therefore, for example, if storage device 125 has already stored, for example, approximately 1GB of data received via function 345, storage device 125 may not know how much data is written by the VM being migrated (compared to the data written by the VMs not being migrated).

[0096] There are ways to notify storage device 125 which VM issued a specific write request across functions shared by multiple VMs. For example, virtual machine monitor 340 can add tags (as metadata) to identify which VM issued the write request. (Including such tags can also be useful to virtual machine monitor 340 to support the return of data read from storage device 125 to the appropriate VM, rather than having virtual machine monitor 340 identify read requests sent by storage device 125, for example, by checking the LBA (or some other metadata) associated with the data. In this way, even when two or more VMs share the exposed functions of storage device 125, storage device 125 can use tags to track which data belongs to which VM.)

[0097] An example of such a label that can be used to identify data is the assignment of write requests to streams. For example, virtual machine monitor 340 can assign write requests from different VMs to different streams. Storage device 125 can then use stream identifiers to track data written through different VMs. For example, (when storage device 125 is an SSD) storage device 125 can use data streaming to write all data associated with a specific stream ID to a set of blocks that do not contain data from other streams. Data streaming can be beneficial for garbage collection in the SSD: if a VM is to be migrated, all data associated with that VM can be moved from the SSD to another storage device, and all blocks occupied by that data can be invalidated and subjected to garbage collection without leaving any data in those blocks that needs to be programmed into other blocks in the SSD. Note that the term "stream ID," as used herein, can identify a stream within the SSD, or it can identify a stream within machine 105 (and allow SSD management to map the management of stream IDs within machine 105 to streams within the SSD).

[0098] Examples of how to manage streaming can be found in the following patent applications: U.S. Patent Application No. 15 / 089,237, filed April 1, 2016 (now U.S. Patent No. 9,959,046, granted May 1, 2018), which claims the benefit of U.S. Provisional Patent Application No. 62 / 273,323, filed December 30, 2015; and U.S. Patent Application No. 15 / 146,708, filed May 4, 2016 (now U.S. Patent No. 9,898,202, granted February 20, 2018), which claims the benefit of U.S. Provisional Patent Application No. 62 / 273,323, filed December 30, 2015; and U.S. Patent Application No. 15 / 146,708, filed May 4, 2016 (now U.S. Patent No. 9,898,202, granted February 20, 2018). The benefit of U.S. Provisional Patent Application No. 62 / 302,162, filed March 1, 2016, and a continuation-in-part of U.S. Patent Application No. 15 / 046,439, filed February 17, 2016 (now U.S. Patent No. 9,880,780, granted January 30, 2018), which claims the benefit of U.S. Provisional Patent Application No. 62 / 261,303, filed November 30, 2015; and U.S. Patent Application No. 15 / 690,270, filed August 29, 2017, which is currently pending. The benefits of U.S. Provisional Patent Application No. 62 / 501,087, filed May 3, 2017, and U.S. Provisional Patent Application No. 62 / 511,957, filed May 26, 2017; U.S. Patent Application No. 15 / 821,708, filed November 22, 2017 (now permitted), which claims the benefit of U.S. Provisional Patent Application No. 62 / 561,674, filed September 21, 2017, and is a continuation-in-part of U.S. Patent Application No. 15 / 620,814, filed June 12, 2017. The application claims the benefit of U.S. Provisional Patent Application No. 62 / 490,027, filed April 25, 2017; and U.S. Patent Application No. 16 / 775,262, filed January 28, 2020, which is a divisional application of U.S. Patent Application No. 15 / 167,974, filed May 27, 2016 (now U.S. Patent No. 10,592,171, granted March 17, 2020), which claims the benefit of U.S. Provisional Patent Application No. 62 / 309,446, filed March 16, 2016.U.S. Patent Application No. 16 / 856,020, filed April 22, 2020, is a currently pending application to U.S. Patent Application No. 15 / 499,877, filed April 27, 2017 (now U.S. Patent No. 10,656,838, granted May 19, 2020). This U.S. Patent Application claims the benefits of U.S. Provisional Patent Application No. 62 / 458,566, filed February 13, 2017, and U.S. Provisional Patent Application No. 62 / 471,350, filed March 14, 2017. U.S. Patent Application No. 15 / 499,877, filed April 27, 2017, is also a pending application to U.S. Patent Application No. 15 / 499,877, filed April 27, 2017. A continuation-in-part of U.S. Patent Application No. 15 / 344,422, filed November 4, 2016 (now U.S. Patent No. 10,282,324, granted May 7, 2019), which claims the benefit of U.S. Provisional Patent Application No. 62 / 383,302, filed September 2, 2016, and is a continuation-in-part of U.S. Patent Application No. 15 / 144,588, filed May 2, 2016, which is currently pending. This U.S. Patent Application No. 62 / 245,100, filed October 22, 2015, and U.S. Provisional Patent Application No. 62 / 192,044, filed July 13, 2015, also claims the benefit of U.S. Provisional Patent Application No. 62 / 245,100, filed October 22, 2015, and U.S. Provisional Patent Application No. 62 / 192,044, filed July 13, 2015. The benefit of U.S. Provisional Patent Application No. 5; a continuation-in-part application of U.S. Patent Application No. 15 / 499,877, filed April 27, 2017, which is also U.S. Patent Application No. 15 / 090,799, filed April 5, 2016 (now U.S. Patent No. 10,509,770, granted December 17, 2019), which claims the benefit of U.S. Provisional Patent Application No. 62 / 245,100, filed October 22, 2015, and U.S. Provisional Patent Application No. 62 / 192,045, filed July 13, 2015; and U.S. Patent Application No. 16 / 866,545, filed May 4, 2020, which is currently pending. This U.S. patent application is a continuation of U.S. Patent Application No. 16 / 219,936, filed December 13, 2018 (now granted), which is a continuation of U.S. Patent Application No. 15 / 458,968, filed March 14, 2017 (now granted February 26, 2019), and claims the benefit of U.S. Provisional Patent Application No. 62 / 448,958, filed January 20, 2017, and U.S. Provisional Patent Application No. 62 / 413,177, filed October 26, 2016, all of which are incorporated herein by reference.

[0099] NVMe controller 350 can communicate with host interface 355. Each function 345 can have its own NVMe controller 350. Host interface 355 can act as a general interface for requests received from machine 105. Host interface 355 can communicate with flash translation layer (FTL) 360, which can handle the translation of logical identifiers (or logical addresses) received from machine 105 into physical addresses on storage device 125 that actually stores data. These logical identifiers do not actually need to be addresses: for example, while these logical identifiers could be logical block addresses (LBAs) for block-based storage devices, they could also be keys for key-value storage devices. This information can be passed to flash channel 365, which can be used to access data from flash chip 370. (The use of flash channel 365 and flash chip 370 indicates...) Figure 3 The storage device 125 can be an SSD, and the flash channel 365 and flash chip 370 can be replaced with other similar components for other types of storage devices (such as hard drives).

[0100] Although Figure 3 An abstract diagram of storage device 125, which is part of machine 105 supporting VM, is shown, but Figure 4 Show Figure 1 Another illustration of the storage device 125. Figure 4 In this configuration, SSD 125 may include host interface 355 (also referred to as “host interface logic” or “HIL”), SSD controller 405, and various flash memory chips 370-1 to 370-8 (also referred to as “flash storage”), which can be organized into various channels 365-1 to 365-4. Host interface logic 355 manages the interaction between SSD 125 and other components (such as…). Figure 1 Communication between processor 110 and processor 110. In some embodiments of the inventive concept, (with via as Figure 1 (In contrast to the network interface of machine 105) SSD 125 can also support its own network connectivity: In this embodiment of the inventive concept, host interface logic 355 can also manage devices remote from SSD 125 (i.e., not considered as such). Figure 1The communication between the machine 105 and the device communicating with the SSD 125. These communications may include read requests to read data from the SSD 125, write requests to write data to the SSD 125, and delete requests to delete data from the SSD 125. The host interface logic 355 may manage an interface spanning only a single port, or it may manage an interface spanning multiple ports. Optionally, the SSD 125 may include multiple ports, each with a separate host interface logic 355 to manage the interface across that port. Embodiments of the inventive concept may also combine this possibility (e.g., an SSD with three ports may have a first host interface logic for managing one port and a second host interface logic for managing the other two ports). To support direct communication with remote devices over a network, the HIL 355 may include an Ethernet component ( Figure 4 (not shown in the image) or some equivalent network communication components.

[0101] SSD controller 405 can use flash controller ( Figure 4 (Not shown) to manage read and write operations, garbage collection, and other operations on flash memory chips 370-1 to 370-8. The SSD controller 405 may include a flash translation layer 360, a storage state 410, and a VM migration state monitoring and capture module 415. (Refer to the above...) Figure 3 The discussed flash memory conversion layer 360 is manageable by Figure 1 The machine 105 uses a conversion between logical data identifiers and the physical locations in flash memory chips 370-1 to 370-8 that actually store the data represented by the logical data identifiers.

[0102] Figure 3 Both the NVMe controller 350 and the SSD controller 405 are described as being used in embodiments of the inventive concept: there may be confusion regarding the presence of two controllers. The SSD controller 405 manages reading data from and writing data to flash memory chips 370-1 to 370-8; on the other hand, Figure 3 The NVMe controller 350 can act as part of the interface for VMs. Their operations are related but not entirely overlapping.

[0103] When the SSD controller 405 is typically implemented using some form of hardware (e.g., as a custom controller for the SSD 125, such as an application-specific integrated circuit (ASIC); however, the SSD controller 405 may be implemented using some kind of general-purpose processor with appropriate software (such as a central processing unit, graphics processing unit (GPU), or general-purpose GPU (GPGPU)) or using a properly configured field-programmable gate array (FPGA), and other possibilities), Figure 3The implementation of the NVMe controller 350 can be more diverse. In some embodiments of the inventive concept, Figure 3 The NVMe controller 350 can be implemented using hardware similar to the SSD controller 405 (but separate from the SSD controller 405). Therefore, Figure 3 The NVMe controller 350 can be implemented using hardware such as an ASIC or a suitably configured FPGA. In other embodiments of the inventive concept, Figure 3 The NVMe controller 350 can be implemented using software (which can run on suitable available hardware). In other embodiments of the inventive concept, Figure 3 The NVMe controller 350 may include both hardware and software components (such as certain general-purpose processors running appropriate software).

[0104] In some embodiments of the inventive concept, Figure 3 The NVMe controller 350 may be implemented separately from the SSD controller 405 (but communicatively integrated with the SSD controller 405) (possibly as part of the host interface 355, or possibly separate from the host interface 355); in other embodiments of the inventive concept, Figure 3 The NVMe controller 350 can be implemented as part of the SSD controller 405. Figure 3 The NVMe controller 350 can also be implemented externally to the storage device 125. Figure 3 The idea that the NVMe controller 350 can be implemented outside the storage device 125 may seem to support specific functions exposed by the storage device 125. Figure 3 The description of the NVMe controller 350 is the opposite. However, U.S. Patent Application No. 16 / 846,271, filed April 10, 2020 and now pending (claiming the benefit of U.S. Provisional Patent Application No. 62 / 865,962, filed June 24, 2019 and U.S. Provisional Patent Application No. 62 / 964,114, filed January 21, 2020, all of which are incorporated herein by reference for all purposes) describes how the functions of the storage device 125 can be represented using a lightweight bridge that can be external to the storage device itself. Therefore, Figure 3 The NVMe controller 350 can be implemented as part of the lightweight bridge, or as part of a component that may exist somewhere within one of the publicly disclosed functions of the lightweight bridge and storage device 125. Figure 3 In embodiments of the inventive concept, the NVMe controller 350 may be located external to the storage device 125. Figure 3The NVMe controller 350 can communicate with multiple different storage devices 125 (e.g., data from a single VM can span multiple storage devices, or data from one VM can be written to one storage device while data from another VM can be written to another storage device). In this embodiment of the inventive concept, Figure 3 The NVMe controller 350 can store information associated with which storage device can store a specific data segment (similar to a flash translation table, but mapping LBAs or other logical identifiers to storage devices rather than addresses within the storage device).

[0105] Storage status 410 includes information about the status of SSD 125. See the following reference... Figure 6 As further discussed, storage state 410 may include information about the use of Figure 1 Information regarding the configuration of one or more transport protocols for communication between machine 105 and SSD 125, and information about... Figure 1 The communication between machine 105 and SSD 125 spans the interface (i.e., Figure 3 Information regarding the configuration of the PF 345, information about input / output (I / O) activity associated with each VM, and information about the flash translation layer of each VM. Note that storage state 410 may represent the state of all SSDs 125, or storage state 410 may represent the state of SSD 125 associated with a specific VM (since different VMs may have different configurations). In this case, multiple storage states 410 may exist stored in SSD 125 (or a single storage state 410, but with information described by the VM). Storage state 410 may be stored in any desired form of storage device (such as certain forms of memory (e.g., RAM or equivalent) or non-volatile storage device), and may be stored in the same storage device (e.g., flash chips 370-1 to 370-8) or a different medium, which may be the same as the storage used to store data.

[0106] The VM migration status monitoring and capture module 415 can manage and capture information about VM migration status from... Figure 1 The VM migration status monitoring and capture module 415 provides information on the storage status 410 of each VM being migrated from machine 105 to another VM. The VM migration status monitoring and capture module 415 can also assist in transferring data to the destination storage device. Additionally, the VM migration status monitoring and capture module 415 can assist in configuring storage for VMs being migrated to... Figure 1 The machine 105 contains the VM's SSD 125. When assisting in receiving the migrated VM, the VM migration status monitoring and capture module 415 can receive data from the source storage device (directly from the source storage device or indirectly from...). Figure 1The virtual machine monitor 340 on the source or destination machine 105 receives status information and configures the SSD 125 accordingly. The VM migration status monitoring and capture module 415 can be connected to the host interface logic 355, the flash translation layer 360, the cache 420 (and the cache policy 425 and / or prefetch policy 425), and the flash channels 365-1 to 365-4 interfaces.

[0107] The SSD controller 405 may also include a cache 420 and a cache policy 425 and / or a prefetch policy 425. The cache 420 can be used to store data in a storage device (e.g., some form of memory such as RAM or equivalent) that is faster than the flash memory chips 370-1 to 370-8. For example, the cache 420 can be used to buffer data that will be written to the flash memory chips 370-1 to 370-8, or it can store data recently written to the flash memory chips 370-1 to 370-8. Figure 1 The data written to machine 105 is expected to be needed again soon. Alternatively, cache 420 can be used to prefetch data from flash memory chips 370-1 to 370-8 that SSD 125 is expected to request soon. Note that the two uses of cache 420 are compatible: cache 420 can support data from... Figure 1 The caching strategy for the data received by machine 105 and the expected data to be cached by Figure 1 The prefetching strategy for data requested by machine 105 is the same for both. For example, the caching strategy keeps cache lines in cache 420 until the cache line is sent to the virtual machine monitor on the source machine (e.g., ...). Figure 5 The virtual machine monitor 340-1 in the virtual machine and the storage device on the destination machine (e.g., Figure 5 At least one of the destination SSDs 125-2). Furthermore, in some embodiments of the inventive concept, the VM may have some control over caching and / or prefetching policies, in which case the SSD 125 may include multiple caching and / or prefetching policies requested by different VMs storing data on the SSD 125.

[0108] When a VM is being migrated, the caching policy 425 and / or prefetch policy 425 can be modified to better support VM migration. For example, since the SSD 125 is expected to send the requested data to the destination storage device soon, the caching policy can be adjusted accordingly. Figure 1Data received by machine 105 may be held in cache 420 for a longer period than normal. Similarly, the prefetch strategy may be modified to request more data for VM storage while anticipating that the request to send the data to the destination storage device will be made soon. Prefetching VM data may also trigger certain data processing functions (such as error correction and decryption, if necessary) so that such data processing functions are completed or at least begin earlier than expected in response to the data read request.

[0109] Although Figure 4 The SSD controller 405 is shown as including a flash translation layer 360, storage state 410, a VM migration state monitoring and capture module 415, a cache 420, and cache policies 425 and / or prefetch policies 425; however, embodiments of the inventive concept may position these modules in any desired location. Embodiments of the inventive concept may also position these modules within different portions of the SSD 125 (e.g., none of these modules may be located within the SSD controller 405).

[0110] Although Figure 4 The SSD 125 is shown as comprising eight flash memory chips 370-1 to 370-8 organized into four channels 365-1 to 365-4; however, embodiments of the inventive concept can support any number of flash memory chips organized into any number of channels. Similarly, although Figure 4 The structure of an SSD is shown, but other storage devices (e.g., hard disk drives) can be implemented using different structures, but with similar potential benefits.

[0111] Figure 5 This shows a virtual machine being migrated from one machine to another. Figure 5 In this process, VM 305 operates on machine 105-1 and will be migrated to machine 105-2. VM 305 has associated data (including status information) stored on the source SSD 125-1, which can be migrated along with VM 305 for use on the destination SSD 125-2. Once the migration is complete, VM 305 will operate on machine 105-2, ideally with no evidence that VM 305 previously operated on machine 105-1.

[0112] So far, VM migration has been briefly discussed without delving into the details of how to perform VM migrations or how to select specific VMs for migration. While the process of selecting specific VMs for migration is beyond the scope of this document, there are many reasons why VMs can be migrated. When new servers are built in a data center, VMs can be migrated to those new servers to utilize their resources (and alleviate the burden on older servers). A particular server may require offline maintenance, in which case VMs running on that server can be migrated so that they can continue to run. A particular server may be old and need to be decommissioned, in which case VMs running on that server should be migrated to other servers. The load on different servers may vary, resulting in one server being heavily loaded while another is lightly loaded: load balancing can justify migrating VMs from heavily loaded servers to lightly loaded servers. A VM may change its requirements, in which case the server on which the VM is running may not be sufficient to meet the new requirements, and a new server should be used. And there may be other reasons why VMs can be selected for migration.

[0113] VM migration is a complex process that can require significant processing and resource utilization. Therefore, performing a VM migration can impact the performance of the VM itself and other VMs installed on the machine. Consequently, it is desirable to perform VM migrations in a manner that minimizes this load, thus reducing the impact on user applications.

[0114] VM migration should also be transparent to the VM itself. That is, the VM should not know which server it is installed on, or the fact that the VM is being (or has already been) migrated. Therefore, performance latency during VM migration should not be expected.

[0115] The process of migrating a VM typically unfolds in three phases. In the first phase, known as the preparation phase, destination machine 105-2 prepares to receive VM 305. During the preparation phase, resources can be allocated on destination machine 105-2. These resources may include… Figure 1 The cores allocated in the processor 110 Figure 1 The portion of memory 115 used by VM 305, the portion of storage space on source storage device 125-1, and the portion to be used by VM 305. Figure 3 The publicly disclosed features include 345, etc. Note that at the start of the preparation phase, VM 305 has been selected for migration, as has the destination machine 105-2 to which VM 305 will be migrated.

[0116] In the second phase, known as the power-limiting phase, the virtual machine monitor 340-1 begins to... Figure 4The storage state 410 and data of VM 305 are migrated to the destination machine 105-2. During the power-down phase, VM 305 continues to run and has access to its data and resources. Therefore, some information related to VM 305 may change during the power-down phase. These changes may include, for example, data stored on source storage device 125-1, changes to the configuration of source storage device 125-1, etc. Figure 3 The data in memory 325, etc. Therefore, although most of the data in VM 305 can be transferred to destination machine 105-2 during the power-limited phase, it is reasonable to expect that some data will have been changed but not yet migrated.

[0117] To address changes made to VM 305 during the power-down phase, VM migration includes a third phase known as the power-down phase. During the power-down phase, VM 305 is stopped (other terms used to describe what happens to VM 305 during the power-down phase include suspended, paused, stopped, frozen, etc.). By stopping VM 305, any further changes to its data are prevented. During the power-down phase, VM monitor 340-1 can migrate any changes or increments made to VM 305 relative to the replication performed during the power-down phase. Once all changes or increments have been replicated to destination machine 105-2, VM 305 stored on both source machine 105-1 and destination machine 105-2 should be identical. VM 305 can then be started on destination machine 105-2, and replication of VM 305 on source machine 105-1 can be terminated, releasing the resources used by VM 305 on source machine 105-1.

[0118] As mentioned above, VM migration is expensive in terms of processing and resources. Since virtual machine monitor 340-1 handles most of the VM migration (there are corresponding operations performed by virtual machine monitor 340-2 on the destination machine 105-2), virtual machine monitor 340-1 has fewer resources available for other VMs that are not being migrated. By offloading some of the work related to VM migration to the source storage device 125-1, the load on virtual machine monitor 340-1 can be reduced to benefit all VMs on the source machine 105-1.

[0119] Because source storage device 125-1 cannot access all content associated with VM 305 (e.g., information about what is running on VM 305). Figure 3 Application 310 information, or Figure 3The source storage device 125-1 does not necessarily take over VM migration entirely, as it is capable of storing only data on the source storage device 125-1. VM migration is also not necessarily limited to transferring only data stored on the source storage device 125-1. However, given the storage device's capacity to store trillions of bytes of data, the source storage device 125-1 can manage the sharing of most of the data related to the VM. Involving the source storage device 125-1 in migration information significantly reduces the load on the virtual machine monitor 340-1. Furthermore, since the source storage device 125-1 can track what requests are received on each public function, it can easily determine what data is stored for the VM being migrated. The virtual machine monitor 340-1 may not necessarily know what data is stored on the source storage device 125-1 for VM 305 (which is being migrated), and may have to request such information from the source storage device 125-1 (via...). Figure 3 Function 345). Similarly, source storage device 125-1 can access information that may be hidden or unknown to virtual machine monitor 340-1 at any time (such as the number of logical identifiers used by VM 305, or information about the translation tables of those logical identifiers).

[0120] Therefore, upon a request from the virtual machine monitor 340-1, the source storage device 125-1 can begin collecting information for the auxiliary virtual machine monitor 340-1 to perform VM migration. This request can set a flag within the source storage device 125-1. Figure 3 With its publicly disclosed function 345, source storage device 125-1 can determine which VM is being migrated. Source storage device 125-1 can then begin collecting information that can be used during VM migration. For example, source storage device 125-1 can collect the storage state 410 of VM 305, and source storage device 125-1 can begin identifying all logical identifiers (such as LBAs or keys) used by VM 305 to write data. Source storage device 125-1 can also modify caching policies and / or prefetching policies to benefit VM migration. That is, source storage device 125-1 can modify caching policies to retain data in... Figure 4 Instead of flushing data to the storage medium, the data is stored in the cache 420, and / or the source storage device 125-1 can modify the prefetch strategy to facilitate the reading of data written for VM 305. Figure 4 The data is stored in cache 420. Any or both of these policy modifications can accelerate the process of transferring data from VM 305 to destination storage device 125-2.

[0121] At the request of virtual machine monitor 340-1, source storage device 125-1 can generate including Figure 4The source storage device 125-1 can send the log page or other report of the storage status 410 to the virtual machine monitor 340-1, which can then send the information to the virtual machine monitor 340-2 on the destination machine 105-2, which can then use the data. Figure 4 The destination storage device 125-2 is configured using storage state 410. In this way, the destination storage device 125-2 can ultimately be configured to mirror the configuration of the source storage device 125-1.

[0122] Optionally, the source storage device 125-1 can Figure 4 The storage status 410 (again, as a log page or other report) is sent directly to the destination storage device 125-2. The destination storage device 125-2 can then... Figure 4 The storage state 410 is used to configure itself. For the source storage device 125-1 to... Figure 4 The storage status 410 is sent directly to the destination storage device 125-2, and the source storage device 125-1 can use its own network connection (such as an Ethernet connection) or the network connection of the source machine 105-1.

[0123] The source storage device 125-1 can also determine whether a threshold number of LBAs (or other logical identifiers) have been written to the storage device. This determination can speed up the process of transferring data from the source storage device 125-1 to the destination storage device 125-2.

[0124] To illustrate how this determination might be helpful, consider a source storage device 125-1 allocated approximately 100 gigabytes of data storage capacity to VM 305. If VM 305 has already written, for example, only about 4 gigabytes of data to source storage device 125-1, virtual machine monitor 340-1 could more efficiently identify those LBAs that have been written and transfer them directly to destination storage device 125-2 (e.g., by requesting those LBAs from source storage device 125-1 and then sending read requests for those LBAs to destination storage device 125-2), or send those LBAs to virtual machine monitor 340-2 on destination machine 105-2. However, if approximately 98 gigabytes of data have already been written to source storage device 125-1 for VM 305, it would be more efficient to simply transfer the entire allocated space of VM 305 from source storage device 125-1 to destination storage device 125-2.

[0125] To set the entire process in the environment, during the VM migration preparation phase, the virtual machine monitor 340-1 of the source machine 105-1 can first send a request to the source storage device 125-1, informing it that VM 305 will be migrated to the destination machine 105-2. The virtual machine monitor 340-1 of the source machine 105-1 can use this request to set a flag, which can be the marked VM_MIGRATION_ACTIVE flag. Once this flag is set, the source storage device 125-1 can begin preparing to migrate VM 305 from the source machine 105-1 to the destination machine 105-2. This preparation may include, for example, identification... Figure 4 The storage state 410 and / or modification of the cache policy and / or prefetch policy to facilitate Figure 4 The data of VM 305 is stored in the cache 420. The virtual machine monitor 340-2 of the destination machine 105-2 can also notify the destination storage device 125-2 that the VM migration is in progress, so that the destination storage device 125-2 can be configured for VM 305 and receive data from VM 305.

[0126] Ultimately, the virtual machine monitor 340-1 on the source machine 105-1 can monitor... Figure 4 The request for storage status 410 is sent to the source storage device 125-1. This request can specify... Figure 4 The storage status 410 is prepared in the log page or some other report. This request can also specify... Figure 4 Storage status 410 can be used by virtual machine monitor 340-1 of source machine 105-1 (by sending a message back to virtual machine monitor 340-1 of source machine 105-1, or by...) Figure 4 The storage status 410 is stored in a location accessible to the virtual machine monitor 340-1 of the source machine 105-1, or sent to the destination storage device 125-2 (via a network). This request may include a GET_LOG_PAGE request for VM_STORAGE_STATE data. The destination storage device 125-2 can then, according to... Figure 4 The storage state 410 is used to configure itself (whether the destination storage device 125-2 performs this configuration on itself or under the guidance of the virtual machine monitor 340-2 of the destination machine 105-2 may vary depending on the specific implementation). The request may include a SET_LOG_PAGE request for VM_STORAGE_STATE data.

[0127] Finally, the virtual machine monitor 340-1 of the source machine 105-1 can enter the power-limited phase of the VM migration. The source storage device 125-1 can compare the number of logical identifiers used for VM 305 with a threshold that can be marked as LBA_WRITTEN_THRESHOLD. The virtual machine monitor 340-1 of the source machine 105-1 can set the value of LBA_WRITTEN_THRESHOLD. Based on this comparison, the source storage device 125-1 can set a flag that can be marked as LBA_WRITTEN_THRESHOLD_EXCEEDED, which indicates whether the number of logical identifiers is large enough to justify copying all the data of VM 305, or whether the number of logical identifiers is small enough to make copying only the logical identifiers that have actually been written more efficient.

[0128] The virtual machine monitor 340-1 of source machine 105-1 can access the LBA_WRITTEN_THRESHOLD_EXCEEDED flag to determine the appropriate method for copying data from source storage device 125-1 to destination storage device 125-2. The virtual machine monitor 340-1 of source machine 105-1 can then read data from VM 305 (each logical identifier or simply request all data of VM 305) from source storage device 125-1 and send that data to the destination virtual machine monitor 340-2 of destination machine 105-2 for writing to destination storage device 125-2. Alternatively, source storage device 125-1 can send data directly to destination storage device 125-2, which can be done in any desired manner: for example, source storage device 125-1 can send data from VM 305 as a stream to destination storage device 125-2. Destination storage device 125-2 may then store data of VM 305 (whether destination storage device 125-2 stores data of VM 305 on its own or stores data of VM 305 under the guidance of virtual machine monitor 340-2 of destination machine 105-2 may depend on the specific implementation).

[0129] The virtual machine monitor 340-1 of source machine 105-1 can select a snapshot of the data of VM 305 on the destination storage device 125-2, instead of reading data from the source storage device 125-1. That is, at some point (before or after the virtual machine monitor 340-1 of source machine 105-1 begins migrating VM 305 from source machine 105-1 to destination machine 105-2, and potentially even before VM 305 is selected for migration), a snapshot of the data on the source storage device 125-1 can be taken. This snapshot may include all data stored on the source storage device 125-1, or only a snapshot of the data of VM 305 on the source storage device 125-1. Besides the fact that the data of VM 305 can be stored somewhere instead of being sent to the destination storage device 125-2 to support the migration of VM 305 from source machine 105-1 to destination machine 105-2, this snapshot can be... Figure 4 VM migration status monitoring and capture module 415 to be compatible with Figure 4 The VM migration status monitoring and capture module 415 can be used to select VM 305 data generated in the same way as migrating from source storage device 125-1 to destination storage device 125-2.

[0130] During the power-limiting phase, source storage device 125-1 can track any changes made to source storage device 125-1 by VM 305. These changes may include data being written to source storage device 125-1, data being deleted from source storage device 125-1, and changes to the configuration of source storage device 125-1. These changes or increments represent differences in the information provided to destination storage device 125-2 during the preparation or power-limiting phase.

[0131] During the power outage phase, the virtual machine monitor 340-1 on the source machine 105-1 can pause VM 305. By pausing VM 305, any further changes to its configuration or data are prevented (but VM 305 cannot perform any actions on behalf of the user).

[0132] Virtual machine monitor 340-1 on source machine 105-1 can then be requested. Figure 4 The storage state 410 increment. This request can also specify... Figure 4 The incremental storage state 410 can be used by the virtual machine monitor 340-1 of the source machine 105-1 (by sending a message back to the virtual machine monitor 340-1 of the source machine 105-1, or by...) Figure 4The storage status 410 is stored in a location accessible to the virtual machine monitor 340-1 of the source machine 105-1, or it is sent to the destination storage device 125-2 (via a network). The request may include GET_LOG_PAGE for VM_DIRTY_STATE data. The destination storage device 125-2 can then, based on the data... Figure 4 The storage state 410 is configured incrementally to itself (whether the destination storage device 125-2 performs this configuration on itself or under the guidance of the virtual machine monitor 340-2 of the destination machine 105-2 may depend on the specific implementation). The request may include a SET_LOG_PAGE request for VM_STORAGE_STATE data.

[0133] The virtual machine monitor 340-1 of source machine 105-1 can then request an increment of data on the source storage device 125-1 of VM 305. This request can also specify that the increment of data on the source storage device 125-1 of VM 305 is available to the virtual machine monitor 340-1 of source machine 105-1 (either by sending a message back to the virtual machine monitor 340-1 of source machine 105-1, or by...). Figure 4 The storage state 410 is stored in a location accessible to the virtual machine monitor 340-1 of the source machine 105-1, or it is sent to the destination storage device 125-2 (via a network). The request may include a GET_LOG_PAGE for VM_DIRTY_LBA_RANGES data. The destination storage device 125-2 may then store incremental data for VM 305 (whether the destination storage device 125-2 stores incremental data for VM 305 on itself or under the guidance of the virtual machine monitor 340-2 of the destination machine 105-2 may depend on the specific implementation).

[0134] once Figure 4 Storage state 410 and all incremental data on the source storage device 125-1 of VM 305 have been transferred to the destination storage device 125-2, which is now ready for VM 305 to be activated on the destination machine 105-2. The virtual machine monitor 340-2 on the destination machine 105-2 can then notify the destination storage device 125-2 that the VM migration is complete and can start VM 305 on the destination machine 105-2. The virtual machine monitor 340-1 on the source machine 105-1 can also notify the source storage device 125-1 that the VM migration is complete and can terminate VM 305 on the source machine 105-1 (and delete all information related to VM 305 from the source storage device 125-1), at which point the VM migration is complete.

[0135] Storage devices 125-1 and / or 125-2 may store data in compressed, encoded, and / or encrypted formats. For clarity, compression is intended to indicate that the data has undergone an algorithm that typically reduces the amount of space required to store the data; encoding is intended to indicate that the data is stored in a different representation than that provided to storage devices 125-1 and / or 125-2; and encryption is intended to indicate that the data has undergone an algorithm that typically renders the original content unrecognizable and unrecoverable without an encryption cipher or encryption key (the term "encryption key" is intended to cover both possibilities and any other terms that may represent similar concepts, whether or not they are listed herein). Therefore, for example, data can be compressed using algorithms such as dictionary compression, prefix encoding, run-length encoding (RLE), cluster encoding, sparse encoding, indirect encoding, bit packing, or Lempel-Ziv (LZ) compression; data can be encoded in formats such as binary, hexadecimal, decimal, base64, Distinguished Encoding Rule (DER), or Basic Encoding Rule (BER); and data can be encrypted using algorithms such as private-key cryptosystems (e.g., Data Encryption Standard (DES), Triple DES (3DES), Advanced Encryption Standard (AES)) or public-key cryptosystems (e.g., Diffie-Hellman key exchange, Rivest-Shamir-Adleman (RSA), and Good Privacy (PGP)). The above examples are not intended to be limiting, and embodiments of the inventive concept may also use other compression algorithms, encoding schemes, and encryption algorithms.

[0136] As an example of encoding, consider Binary Coded Data (BCD). BCD encodes each digit 0-9 as a separate number, which may require 4 bits of data. Therefore, the number 1024, composed of four digits 1, 0, 2, and 4, can be represented in BCD as 0001000000100100. On the other hand, if viewed as a number, 1024 can be represented in binary as 00000100000000000 (assuming a 16-bit representation of an integer). Both BCD and binary are efficient ways to represent and store the number 1024; they are simply different encodings.

[0137] Compression, encoding, and encryption are theoretically orthogonal operations: applying one operation does not prevent any other operation from being performed on the data. (In practice, combining compression and encoding does not provide more benefit than either alone, and encryption can sufficiently alter the data that compression and / or encoding may not offer significant benefit. However, this fact is irrelevant to the operation of embodiments of the inventive concept.) Therefore, data stored on source storage device 125-1 can be compressed, encoded, and / or encrypted. Source storage device 125-1 can apply compression, encoding, and / or encryption in any desired order, and these methods can be used in any desired combination. For example, one storage device can compress data, then encode it, then encrypt it, while another storage device can encrypt data, then compress it, but effortlessly encode it.

[0138] Storage devices 125-1 and / or 125-2 may also provide error correction services. For example, storage devices 125-1 and / or 125-2 may use error correction codes (such as single error correction double detection (SECDED) codes) to help prevent the possibility of data errors in read or write requests. Storage devices 125-1 and / or 125-2 may use error correction codes regardless of whether the data is compressed, encrypted, and / or encoded.

[0139] In some embodiments of the inventive concept, source storage device 125-1 may provide data to VM 305 precisely as it is stored. Therefore, if the data is stored in a compressed, encoded, and / or encrypted manner, the data of VM 305 ultimately received by destination storage device 125-2 may also be compressed, encoded, and / or encrypted. Similarly, if the data includes error correction codes, the data of VM 305 ultimately received by destination storage device 125-2 may include error correction codes. This may be acceptable if destination storage device 125-2 uses the same compression, encoding, encryption, and / or error correction code scheme (although for encrypted data, if the encryption algorithm is applied by source storage device 125-1 rather than VM 305, source storage device 125-1 may also need to provide destination storage device 125-2 with an encryption key for the encrypted data in order to eventually decrypt the data of VM 305 for use). However, if the source storage device 125-1 and the destination storage device 125-2 use different compression, encoding, and / or encryption algorithms or different error correction schemes, the data of VM 305 can be processed to remove any processing that the destination storage device 125-2 may not be equipped with for processing.

[0140] In other embodiments of the inventive concept, source storage device 125-1 may remove any compression, encoding, encryption, and / or error correction codes from the data of VM 305 before it is sent for final delivery to destination storage device 125-2. Therefore, as part of migrating VM 305 from source machine 105-1 to destination machine 105-2, source storage device 125-1 may decompress, decode, decrypt, and / or remove error correction codes from the data of VM 305 before sending it. Note that the order in which these processes are performed may be relevant: if the data is encrypted first and then compressed, attempting to decrypt the data before decompressing it may result in the final data being completely unusable by VM 305 after migration to destination machine 105-2. Furthermore, various embodiments of the inventive concept may revoke some, but not all, of these methods. For example, source storage device 105-1 may first compress and encrypt the data of VM 305 before storage. As part of migrating VM 305 from source machine 105-1 to destination machine 105-2, source storage device 105-1 may decrypt but not decompress the data of VM 305 before sending the data to VM 305.

[0141] Furthermore, while the above discussion addresses compression, encoding, encryption, and error correction codes that may be used to manipulate data on VM 305, source storage device 125-1 may also perform other manipulations on the data of VM 305. These other manipulations may be reversed or not reversed as needed and suitable for storage devices 125-1 and 125-2, before the data on VM 305 is transmitted.

[0142] As described above, storage devices 125-1 and 125-2 can each act as a source and / or destination, depending on which VMs are being migrated and between which machines. Therefore, storage devices 125-1 and 125-2 can each perform the operations attributed to the source storage device 125-1 and destination storage device 125-2, respectively. That is, storage devices 125-1 and 125-2 can each notify virtual machine monitor 340-1 or virtual machine monitor 340-2 that a VM migration is about to occur, and can collect or set... Figure 4 The storage status 410 (and appropriately generates log pages or reports) can send or receive data from the VM305, and can provide information about increments. Figure 4 The data in storage state 410 or VM 305. Therefore, operations belonging to source storage device 125-1 or destination storage device 125-2 can be performed by either source storage device 125-1 or destination storage device 125-2.

[0143] The above discussion focuses only on operations related to the migration of information about VM 305 stored on source storage device 125-1. VM migration may also involve transferring and Figure 3 Application 310 Figure 3 The status of the system software stack 315, Figure 3 The status of the operating system 320 Figure 3 The status of memory 325, Figure 3 The status of network drive 330 and Figure 3 Information relating to the status of storage drive 335. This information may not be available to source storage device 125-1, and therefore, in addition to the processing described above, it may also be transmitted from virtual machine monitor 340-1 of source machine 105-1 to virtual machine monitor 340-2 of destination machine 105-2.

[0144] While the above processing has been described with reference to VM migration, embodiments of the inventive concept support using the VM migration operation of storage device 125 for other purposes. For example, as described above, the virtual machine monitor 340-1 of source machine 105-1 can use snapshots to transfer data of VM 305 to destination storage device 125-2. The virtual machine monitor 340-1 of source machine 105-1 (or in fact, any processing on source machine 105-1, even if source machine 105-1 does not support VMs) can request a snapshot of the data on source storage device 125-1. The virtual machine monitor 340-1 of source machine 105-1 or any processing on source machine 105-1 can then request source storage device 125-1 in a state where the VM may be migrated. Source storage device 125-1 can then track changes to the data stored on storage device 125-1. Any processing on the virtual machine monitor 340-1 of the source machine 105-1 or on the source machine 105-1 can subsequently request changes relative to the snapshot, which is very similar to the virtual machine monitor 340-1 of the source machine 105-1 requesting increments of data on VM 305 on the source storage device 125-1. For this purpose, Figure 4 The VM migration status monitoring and capture module 415 can also be referred to as an incremental module (although the functionality of an incremental module may be less than that of a standard module). Figure 4 The VM migration status monitoring and capture module 415 is functional because the source storage device 125-1 may not capture or track it. Figure 4 The change of storage state 410, or may not be used to configure the source storage device 125-1 in a specific state.

[0145] The above discussion includes references to various commands that can be sent to storage devices 125-1 and 125-2. Additional details regarding these commands are provided below.

[0146] NVMe commands include commands called Set Feature and Get Feature. These commands enable machines 105-1 and 105-2 to set (using Set Feature) or read (using Get Feature) within storage devices 125-1 and 125-2. Figure 3 The values ​​of certain characteristics of the NVMe controller 350. Examples of such characteristics may include command submission queue arbitration parameters, power management settings, temperature thresholds, queue count, and interrupt merging configuration. New characteristics can be added for these characteristics and can be tagged as VM_MIGRATION, VM_STORAGE_STATE, VM_DIRTY_STATE, and VM_DIRTY_LBA_RANGES.

[0147] Using the VM_MIGRATION feature, Machine 105-1 and Machine 105-2 can set or read certain parameters (such as those marked above as VM_MIGRATION_ACTIVE, LBA_WRITTEN_THRESHOLD, and LBA_WRITTEN_THRESHOLD_EXCEEDED).

[0148] VM_MIGRATION_ACTIVE can be used to... Figure 3 The NVMe controller 350 (therefore informing storage devices 125-1 and 125-2) that VM 305 is migrating from source machine 105-1 to destination machine 105-2. VM_MIGRATION_ACTIVE can also be read to determine... Figure 3 Is the NVMe controller 350 currently actively assisting in the migration of VM 305?

[0149] LBA_WRITTEN_THRESHOLD can be used to... Figure 3 The NVMe controller 350 notifies the user how many LBA ranges to log (when data is written to source storage device 125-1). If more LBAs than the value of LBA_WRITTEN_THRESHOLD are written to source storage device 125-1, storage device 125-1 can set the LBA_WRITTEN_THRESHOLD_EXCEEDED flag. The virtual machine monitor 340-1 of source machine 105-1 can read the LBA_WRITTEN_THRESHOLD_EXCEEDED flag. If this flag is set, the virtual machine monitor 340-1 of source machine 105-1 can move the full capacity of VM 305 allocated on source storage device 125-1 to destination storage device 125-2. Otherwise, the virtual machine monitor 340-1 of source machine 105-1 can access the data including the LBA ranges written to source storage device 125-2. Figure 1 The VM 305 writes the log pages of LBAs or other reports, and then only reads those LBAs from the source storage device 125-1 to write the data to the destination storage device 125-2.

[0150] Regarding LBA_WRITTEN_THRESHOLD and LBA_WRITTEN_THRESHOLD_EXCEEDED, two things are worth noting. First, although the parameters represent "LBA," they can be used to track data written using other methods. For example, in key-value stores, these parameters can be used to track how many keys have been written to the source storage device 125-1 of the VM 305 instead of LBA. (If needed, the parameters can be renamed to KEY_WRITTEN_THRESHOLD and KEY_WRITTEN_THRESHOLD_EXCEEDED or some other comparable label: for the purposes of this discussion, references to LBA should be understood to include references to other identifiers of data written to source storage device 125-1.)

[0151] Secondly, since tracking how much data has been written to source storage device 125-1 throughout the lifetime of the VM, rather than just since the VM migration began, is meaningful, the virtual machine monitor 340-1 of source machine 105-1 can set the value of the LBA_WRITTEN_THRESHOLD parameter when VM 305 is first instantiated on source machine 105-1. Therefore, source storage device 125-1 can track the number of logical identifiers written to source storage device 125-1 throughout the lifetime of VM 305 (or at least as long as VM 305 is instantiated on source machine 105-1), not just since the VM migration began.

[0152] Using the VM_STORAGE_STATE feature, machines 105-1 and 105-2 can set or read information about... Figure 4 The information regarding the storage status 410. Refer to the following... Figure 6 Further discussion Figure 4 Storage status 410.

[0153] Using the VM_DIRTY_STATE feature, source machine 105-1 can read the data stored in source storage device 125-1 during the power-limited phase. Figure 4 The storage state 410 has undergone what increments (changes). These increments can be transferred to the destination storage device 125-2 during the power-down phase. To enable VM_DIRT Y_STATE to track this information, the source machine 105-1 can be configured to indicate the changes made to the storage state 410. Figure 4Any further changes to the storage state 410 should be flagged by the source storage device 125-1. Optionally, the source storage device 125-1 may assume that the most recent read from the source machine 105-1... Figure 4 After the storage status 410, Figure 4 Any changes made to the storage state 410 should be tracked by VM_DIRTY_STATE. Note that VM_DIRTY_STATE does not necessarily need to be the counterpart used to set the state of the destination storage device 125-2: the VM_STORAGE_STATE feature can be used to track changes made to the source storage device 125-1. Figure 4 The change in storage state 410 is written to the destination storage device 125-2. However, embodiments of the inventive concept may include a special command for setting the VM_DIRTY_STATE characteristic to write such a change to the destination storage device 125. Figure 4 Storage status 410.

[0154] Using the VM_DIRTY_LBA_RANGES feature, machine 105-1 can read a list of LBAs (or other logical identifiers) that have been changed (written or deleted) since the virtual machine monitor 305-1 of source machine 105-1 transferred the data of VM 305 from source storage device 125-1 to destination storage device 125-2 during a power-down phase. These increments can be transferred to destination storage device 125-1 during a power-down phase. To enable VM_DIRTY_LBA_RANGES to track this information, source machine 105-1 can set a flag indicating that any further changes to the data of VM 305 should be tracked by source storage device 125-1. Optionally, source storage device 125-1 can assume that any changes made to the data of VM 305 after source machine 105-1 has transferred the data of VM 305 to destination storage device 125-2 should be tracked by VM_DIRTY_LBA_RANGES. Note that VM_DIRTY_LBA_RANGES does not necessarily require a counterpart for storing data from VM 305 in destination storage device 125-2, as data can be written to destination storage device 125-2 using regular write requests. However, embodiments of the inventive concept may include special commands for writing changes to data from VM 305 to destination storage device 125-2.

[0155] The NVMe specification includes the Identify data structure. This Identify data structure can be used to indicate VM live migration assistance that supports Virtual Machine Monitor 340-1 and Virtual Machine Monitor 340-2. Unused fields of the Identify structure can be used for this purpose.

[0156] While the above describes how existing NVMe commands can be enhanced to support VM migration, embodiments of the inventive concept may also use other commands (including custom vendor-defined commands) to achieve support for VM migration.

[0157] Figure 6 It shows that it can be accessed Figure 1 The storage device 125 holds details of the state information. Figure 6 The image shows storage state 410. Storage state 410 may include various information (such as transport configuration information 605, interface configuration information 610, I / O activity 615, and translation table 620). Transport configuration information 605 may include information about how to configure the transport protocol. For example, if... Figure 1 The storage device 125 is connected to the peripheral component interconnect (PCIe) via a rapid connection. Figure 1 If machine 105 communicates, then the configuration information 605 transmitted may include information about Figure 1 Information regarding the PCIe configuration of the storage device 125. Examples of information that can be considered as transmission configuration information 605 may include various PCIe capability settings (such as message signaling interrupts (MSI and MSI-X), advanced error reporting (AER), power management, etc.) and configuration space settings (such as interrupt information, maximum load size, etc.).

[0158] Interface configuration information 610 may include information about Figure 1 Information regarding the configuration of the interfaces in the storage device 125. For example, if... Figure 1 Storage device 125 uses NVMe and Figure 1 If the machine 105 communicates, then the interface configuration information 610 may include information about... Figure 1 Information about the NVMe configuration of storage device 125. Examples of information that can be considered interface configuration information 610 may include memory page size, doorbell step size, timeout value, commit queue arbitration settings, power management, number of queues, asynchronous event notification, interrupt merging, atomicity, created namespaces, security settings, log pages created to date, etc.

[0159] I / O activity 615 may include information about Figure 1 Information on the I / O activity of storage device 125. This information may include VM data writes in terms of LBA range and whether the number of VM data writes exceeds thresholds (such as VM_DIRTY_LBA_RANGES and LBA_WRITTEN_THRESHOLD as described above).

[0160] Transformation table 620 may include information on the conversion of logical identifiers to... Figure 1The translation table 620 contains information about one or more translation tables for physical addresses in the storage device 125. The translation table 620 may also include other information such as flash memory block usage, write / read cache status, and information used for... Figure 3 The setup of the NVMe controller 350 (and subsequently the VM 305).

[0161] Figure 5 The destination storage device 125-2 can use some or all of the information in storage state 410 to reproduce Figure 5 The state of the source storage device 125-1. For example, the transmission configuration information 605 can be used to configure... Figure 5 The transmission protocol of the destination storage device 125-2 is matched. Figure 5 The transmission protocol of the source storage device 125-1, and the interface configuration information 610 can be used to configure the interface. Figure 5 The interface of the destination storage device 125-2 is matched. Figure 5 The interface of the source storage device 125-1, I / O activity 615 can be used to... Figure 5 The destination storage device 125-2 provides, for example, Figure 5 The source storage device 125-1 is known about Figure 3 The information on the I / O activity of the VM305, conversion table 620 can be used to convert... Figure 5 The translation table and cache of the destination storage device 125-2 are arranged to match as closely as possible. Figure 5 Configuration of the source storage device 125-1.

[0162] Figure 7 This shows the logical block address (LBA) used to determine whether a threshold number of entries have been written. Figure 4 The VM migration status monitoring and capture module 415. As described above, Figure 1 Storage device 125 can track data from... Figure 3 The VM migration status monitoring and capture module 415 can compare the number 705 with a threshold 710: if the number 705 exceeds the threshold 710, the VM migration status monitoring and capture module 415 can set an indication that it has been... Figure 3 The VM 305 will write enough data. Figure 1 The storage device 125's flag 715 (such as LBA_WRITTEN_THRESHOLD_EXCEED) is assigned to VM 305. Figure 5 The entire capacity of the source storage device 125-1 was transferred to Figure 5 The destination storage device 125-2. Otherwise, Figure 1 The storage device 125 is traceable and has been targeted Figure 3 VM 305 written to Figure 1 The various LBA addresses (or ranges thereof) of the storage device 125.

[0163] Although Figure 7 Show Figure 1 The storage device 125 tracks the data from the storage device 125. Figure 3 The VM 305 writes 705 LBAs, but embodiments of the inventive concept can track other conditions. For example, embodiments of the inventive concept can track... Figure 1 The space occupied on storage device 125, the number of logical identifiers (which may differ from LBAs: for example, the keys used in key-value stores), or the ... Figure 3 The range of LBAs written by the VM 305. If Figure 1 If the storage device 125 can track other information, the VM migration status monitoring and capture module 415 can use other thresholds for comparison.

[0164] Figure 8 Show Figure 4 VM migration status monitoring and capture module 415 is designed for Figure 5 The status log page of source storage device 125-1 is generated. Figure 8 In this process, the VM migration state monitoring and capture module 415 can access storage state 410 (or a portion thereof) and generate log pages or reports 805 from storage state 410. The log pages or reports 805 may take any desired form and / or format and should be suitable for use by [the system / entity]. Figure 5 destination storage device 125-2 Figure 5 The virtual machine monitor 340-2 is used to configure the destination storage device 125-2 accordingly.

[0165] Figures 9A to 9B An embodiment of the invention is shown for use in accordance with the inventive concept. Figure 1 Storage devices support VMs from Figure 1 A flowchart illustrating an example process of real-time migration of machine 105. Figure 9A In the middle, in box 905, Figure 5 The source storage device 125-1 can be obtained from Figure 5 The source machine 105-1 Figure 5 Virtual Machine Monitor 340-1 receives Figure 5 VM 305 will be migrated (from Figure 5 Source machine 105-1 to Figure 5 The destination machine 105-2, despite Figure 5 The source storage device 125-1 may not know Figure 5 The request to the destination machine (105-2). In box 910, Figure 5 The source storage device 125-1 is modifiable Figure 4The caching strategy 425 and / or prefetch strategy 425 are designed to facilitate the caching of cached data. Figure 5 Data retention and / or migration of VM 305 Figure 4 The cache is 420.

[0166] In box 915, Figure 5 The source storage device 125-1 can be obtained from Figure 5 The source machine 105-1 Figure 5 The virtual machine monitor 340-1 receives the input from the virtual machine monitor 340-1. Figure 5 VM 305 Figure 4 The request for storage status 410. In response, in box 920, Figure 5 The source storage device 125-1 can Figure 4 Storage status 410 sent to Figure 5 The source machine 105-1 Figure 5 Virtual Machine Monitor 340-1 Figure 5 The destination machine 105-2 Figure 5 Virtual Machine Monitor 340-2, or Figure 5 Any one of the destination storage devices 125-2.

[0167] In box 925, Figure 5 The source storage device 125-1 can be obtained from Figure 5 The source machine 105-1 Figure 5 Virtual Machine Monitor 340-1 receives requests Figure 5 The request for data from VM 305. In box 930, Figure 5 The source storage device 125-1 can access data from... Figure 5 The VM 305 data is decompressed, decoded, decrypted, and / or error-correcting code removed. Note that if... Figure 5 The data from the VM 305 will be compressed, encoded, encrypted, and transmitted using error-correcting codes, as shown by dashed line 935; box 930 can be omitted. Additionally, in box 930, Figure 1 The source storage device 125-1 can perform some operations but not others (such as, to...). Figure 5 The data on the VM 305 is decrypted but not decompressed, depending on... Figure 5 Is the destination storage device 125-2 capable of handling... Figure 1 (Data manipulated by VM 305).

[0168] In box 940 ( Figure 9B ), Figure 5 The source storage device 125-1 can send Figure 5 The data from VM 305. As mentioned above, Figure 5 The source storage device 125-1 can Figure 5Data sent from VM 305 Figure 5 The source machine 105-1 Figure 5 Virtual Machine Monitor 340-1 Figure 5 destination storage device 105-2 Figure 5 Virtual Machine Monitor 340-2, or Figure 5 Any one of the destination storage devices 125-2.

[0169] In box 945, Figure 5 The source storage device 125-1 is traceable Figure 5 Any increment (i.e., change) to the storage state 410. In box 950, Figure 5 The source storage device 125-1 is traceable Figure 5 Any incremental (i.e., change) to the data on VM 305. Note that boxes 945 and 950 can be executed in parallel rather than sequentially.

[0170] In box 955, Figure 5 The source storage device 125-1 can be obtained from Figure 5 The source machine 105-1 Figure 5 The virtual machine monitor 340-1 receives the input from the virtual machine monitor 340-1. Figure 4 The storage state 410 is requested incrementally. In box 960, Figure 5 The source storage device 125-1 can send Figure 4 The increment of storage state 410. As described above, Figure 5 The source storage device 125-1 can Figure 4 The incremental storage status 410 is sent to Figure 5 The source machine 105-1 Figure 5 Virtual Machine Monitor 340-1 Figure 5 destination storage device 105-2 Figure 5 Virtual Machine Monitor 340-2, or Figure 5 Any one of the destination storage devices 125-2.

[0171] In box 965, Figure 5 The source storage device 125-1 can be obtained from Figure 5 The source machine 105-1 Figure 5 The virtual machine monitor 340-1 receives the input from the virtual machine monitor 340-1. Figure 5 The request is for incremental data from VM 305. In box 970, Figure 5 The source storage device 125-1 can send Figure 5 The incremental data of VM 305. As mentioned above, Figure 5 The source storage device 125-1 can Figure 5 Incremental data sent to VM 305 Figure 5The source machine 105-1 Figure 5 Virtual Machine Monitor 340-1 Figure 5 destination storage device 105-2 Figure 5 Virtual Machine Monitor 340-2, or Figure 5 None of the destination storage devices 125-2. Additionally, although not in Figure 9B It is clearly stated in the document, but Figure 5 The source storage device 125-1 can be sent Figure 5 Before incrementally updating the data on VM 305, adjust as needed. Figure 5 The VM 305 incrementally decompresses, decodes, decrypts, and / or removes error correction codes from the data.

[0172] Finally, in box 975, Figure 5 The source storage device 125-1 can receive instructions Figure 5 VM 305 from Figure 5 Source machine 105-1 to Figure 5 The message indicating that the migration of destination machine 105-2 is complete is received at this time. Figure 5 The source storage device 125-1 can stop the auxiliary VM migration. As shown by dashed line 980, box 975 can be omitted because once Figure 5 The VM 305 has been removed from Figure 5 The source machine 105-1 was migrated to Figure 5 The destination machine 105-2 can then be accessed from... Figure 5 Source storage device 125-1 removal Figure 5 Any data from the VM 305 will be efficiently terminated. Figure 5 Storage device 125-1 auxiliary Figure 5 VM 305 migration away Figure 5 The source storage device 125-1.

[0173] Figure 10 An embodiment of the invention is shown for use in accordance with the inventive concept. Figure 4 The flowchart illustrates an example process by which the VM migration status monitoring and capture module determines whether a threshold number of LBAs have been written. Figure 10 In the middle, in box 1005, Figure 5 The source storage device 125-1 can track targets Figure 5 The number of LBAs written to the VM 305 is 705. As mentioned above, this trace can be obtained by the total storage space occupied, the number of LBAs (or other logical identifiers) written, Figure 5 The VM 305 writes the LBA range or other conditions. In box 1010, Figure 4 The VM migration status monitoring and capture module 415 can target Figure 5The number of LBAs written by VM 305 is 705, compared to a threshold of 710. If targeting... Figure 5 If the number of LBAs written by VM 305 exceeds the threshold of 710, then in box 1015, Figure 4 The VM migration status monitoring and capture module 415 can be configured to indicate that it has been targeted Figure 5 The VM 305 is assigned a flag indicating that enough data has been written. Figure 5 VM 305 Figure 5 The full capacity of the source storage device 125-1, not Figure 5 The selected set of data from the VM305 can be transmitted to Figure 5 Destination storage device 125-2.

[0174] Figure 11 An embodiment of the invention is shown for use in accordance with the inventive concept. Figure 4 The VM migration status monitoring and capture module generates and transmits VM migration status. Figure 4 A flowchart illustrating an example process for reporting storage status 410. Figure 11 In the middle, in box 1105, Figure 4 The VM migration status monitoring and capture module 415 can (in Figure 5 The source machine 105-1 Figure 5 (Request from Virtual Machine Monitor 340-1) Generate information about Figure 5 VM 305 Figure 4 The storage status 410 log page or other reports. This log page or other reports may include, for example... Figure 6 Transmission configuration information 605 Figure 6 Interface configuration information 610 Figure 6 I / O activities 615 and Figure 6 The transformation table 620, or the log page or report, may only include a subset of this data. In box 1110, Figure 4 The VM migration status monitoring and capture module 415 can store log pages or other reports. Figure 5 In the memory somewhere in the source storage device 125-1, such that (in box 1115) Figure 5 The source machine 105-1 Figure 5 The virtual machine monitor 340-1 can be accessed from Figure 5 The source storage device 125-1 reads log pages or other reports. Optionally, in box 1120, Figure 4 The VM migration status monitoring and capture module 415 can send log pages or other reports in a message. Figure 5 The source machine 105-1 Figure 5 The virtual machine monitor 340-1 (or sent to some other destination, such as, Figure 5The destination machine 105-2 Figure 5 Virtual Machine Monitor 340-2 or Figure 5 Destination storage device 125-2).

[0175] Figures 12A to 12B An embodiment of the invention is shown for use in accordance with the inventive concept. Figure 1 Storage device 125 supports VM to Figure 1 A flowchart illustrating an example process of real-time migration of machine 105. Figure 12A In the middle, in box 1205, Figure 5 The destination storage device 125-2 can be from Figure 5 The destination machine 105-2 Figure 5 The virtual machine monitor 340-2 receives requests to assist in... Figure 5 VM305 from Figure 5 The source machine 105-1 was migrated to Figure 5 The destination machine is 105-2. In box 1210, Figure 5 The destination storage device 125-2 can receive Figure 4 Storage status 410. Figure 5 The destination storage device 125-2 can be from Figure 5 The source machine 105-1 Figure 5 Virtual Machine Monitor 340-1 Figure 5 The destination machine 105-2 Figure 5 Virtual Machine Monitor 340-2, or Figure 5 Any one of the source storage devices 125-1 receives Figure 4 Storage status 410. In box 1215, Figure 5 The destination storage device 125-2 can be based on Figure 4 The information received in the storage state 410 is used to target Figure 5 The VM 305 is configured appropriately.

[0176] In box 1220, Figure 5 The destination storage device 125-2 can receive Figure 5 Data from VM 305. Figure 5 The destination storage device 125-2 can be from Figure 5 The source machine 105-1 Figure 5 Virtual Machine Monitor 340-1 Figure 5 The destination machine 105-2 Figure 5 Virtual Machine Monitor 340-2, or Figure 5 Any one of the source storage devices 125-1 receives Figure 5 Data from VM 305. In box 1225, Figure 5The destination storage device 125-2 can be used for Figure 5 The data on VM 305 is appropriately compressed, encoded, encrypted, and / or error-correcting codes are introduced. As shown by dashed line 1230, box 1225 can be omitted. Then, in box 1235, Figure 5 The destination storage device 125-2 can store Figure 5 Data from VM 305.

[0177] In box 1240 ( Figure 12B ), Figure 5 The destination storage device 125-2 can receive Figure 4 The increment (change) of storage state 410. Figure 5 The destination storage device 125-2 can be from Figure 5 The source machine 105-1 Figure 5 Virtual Machine Monitor 340-1 Figure 5 The destination machine 105-2 Figure 5 Virtual Machine Monitor 340-2, or Figure 5 Any one of the source storage devices 125-1 receives Figure 4 The incremental storage state 410. In box 1245, Figure 5 The destination storage device 125-2 can update its storage status to match the current status of the destination storage device 125-2. Figure 4 The incremental storage state 410 is consistent.

[0178] In box 1250, Figure 5 The destination storage device 125-2 can receive Figure 5 The incremental (change) of data on VM 305. Figure 5 The destination storage device 125-2 can be from Figure 5 The source machine 105-1 Figure 5 Virtual Machine Monitor 340-1 Figure 5 The destination machine 105-2 Figure 5 Virtual Machine Monitor 340-2, or Figure 5 Any one of the source storage devices 125-1 receives Figure 5 The incremental data for VM 305. In box 1255, Figure 5 Destination storage device 125-2 is updatable Figure 5 The data for VM 305. Additionally, although not in Figure 12B It is clearly stated in the document, but in storage Figure 5 Before the data increment of VM 305, Figure 5 The destination storage device 125-2 can, when appropriate, [access / resources / resources]. Figure 5 The incremental data of the VM 305 is compressed, encoded, encrypted, and / or error-correcting codes are introduced.

[0179] Finally, in box 1260, Figure 5 The destination storage device 125-2 can be from Figure 5 The destination machine 105-2 Figure 5 Virtual Machine Monitor 340-2 receives Figure 5 VM 305 from Figure 5 Source machine 105-1 to Figure 5 The message indicating that the migration of destination machine 105-2 is complete is received at this time. Figure 5 The destination storage device 125-2 can stop auxiliary VM migration. (With) Figure 5 The source storage device 125-1 (may not receive) Figure 5 VM 305 from Figure 5 The source storage device 125-1 to Figure 5 The message indicating the migration completion of the destination storage device 125-2 is different, due to... Figure 5 The destination storage device 125-2 can support Figure 5 The forward operation of the VM 305, therefore Figure 5 The destination storage device 125-2 should receive such a message.

[0180] exist Figures 9A to 12B Some embodiments of the inventive concept are shown in the figure. However, those skilled in the art will recognize that other embodiments of the inventive concept are possible by changing the order of the boxes, by omitting boxes, or by including links not shown in the figures, regardless of any elements that may be specifically omitted. All such variations of the flowchart, whether explicitly described or not, are considered embodiments of the inventive concept.

[0181] Embodiments of the inventive concept include technical advantages over conventional implementations. By using a storage device to facilitate VM migration, the load on the virtual machine monitor managing VM migration can be reduced. By reducing the load on the virtual machine monitor, it can be freed up to handle other tasks (such as managing the execution of other VMs, selecting other VMs for migration to other machines, etc.). Embodiments of the inventive concept also result in storage devices that are aware of VM migration and can transfer resources to benefit VM migration. For example, since accessing data from a cache is faster than accessing data from the underlying storage medium of the storage device, the cache in the storage device can be used to store VM data.

[0182] The following discussion is intended to provide a brief, general description of one or more suitable machines in which certain aspects of the inventive concept can be implemented. One or more machines can be controlled at least in part by input from conventional input devices (such as keyboards, mice, etc.) and by instructions received from another machine, interaction with a virtual reality (VR) environment, biometric feedback, or other input signals. As used herein, the term "machine" is intended to broadly include a single machine, a virtual machine, or a system of machines, virtual machines, or devices that operate communicatively together. Exemplary machines include computing devices (such as personal computers, workstations, servers, portable computers, handheld devices, telephones, tablet computers, etc.) and transportation devices (such as private or public transportation vehicles (e.g., cars, trains, taxis, etc.)).

[0183] One or more machines may include embedded controllers (such as programmable or non-programmable logic devices or arrays, application-specific integrated circuits (ASICs), embedded computers, smart cards, etc.). One or more machines may utilize one or more connections to one or more remote machines (such as via network interfaces, modems, or other communication links). Machines may be interconnected via physical and / or logical networks (such as intranets, the Internet, local area networks, wide area networks, etc.). Those skilled in the art will understand that network communications may utilize various wired and / or wireless short-range or long-range carriers and protocols, including: radio frequency (RF), satellite, microwave, Institute of Electrical and Electronics Engineers (IEEE) 802.11, etc. Optics, infrared, cables, lasers, etc.

[0184] Embodiments of this invention can be described by reference to or in conjunction with associated data, including functions, programs, data structures, application programs, etc., which, when accessed by a machine, cause the machine to perform tasks or define abstract data types or low-level hardware contexts. The associated data may be stored, for example, in volatile and / or non-volatile memory (e.g., RAM, ROM, etc.), or in other storage devices and their associated storage media (including hard disk drives, floppy disks, optical storage devices, magnetic tapes, flash memory, memory sticks, digital video disks, bio-memory disks, etc.). The associated data may be transmitted over a transmission environment (including physical and / or logical networks) in the form of packets, serial data, parallel data, propagated signals, etc., and may be used in compressed or encrypted formats. The associated data may be used in a distributed environment and stored locally and / or remotely for machine access.

[0185] Embodiments of the inventive concept may include a tangible, non-transitory machine-readable medium, which may include instructions executable by one or more processors, including instructions for performing elements of the inventive concept as described herein.

[0186] Having described and illustrated the principles of the inventive concept with reference to the illustrated embodiments, it will be appreciated that the illustrated embodiments may be modified in arrangement and detail without departing from such principles, and may be combined in any desired manner. Furthermore, although the foregoing discussion has focused on particular embodiments, other configurations (constructions) are contemplated. Specifically, although expressions such as "according to an embodiment of the inventive concept" are used herein, these phrases are intended to refer generally to the possibilities of embodiments and not to limit the inventive concept to a particular embodiment configuration. As used herein, these terms may refer to the same or different embodiments that can be combined into other embodiments.

[0187] The foregoing illustrative embodiments should not be construed as limiting the inventive concept. Although some embodiments have been described, those skilled in the art will readily understand that many modifications are possible with respect to those embodiments without substantially departing from the novel teachings and advantages of this disclosure. Therefore, all such modifications are intended to be included within the scope of the inventive concept as defined in the claims.

[0188] Embodiments of the inventive concept can be extended to the following statements without limitation:

[0189] Statement 1: Embodiments of the inventive concept include a storage device, the storage device comprising:

[0190] At least one controller for virtual machines (VMs) on the source host;

[0191] The first storage device is used to store data for the VM;

[0192] The second storage device is used for storing the VM's state;

[0193] A storage device controller, configured to use a first storage device to process at least one read request for a virtual machine (VM) received from the controller, and to use the first storage device to process at least one write request for a VM received from the controller; and

[0194] The VM migration status monitoring and capture module is used to assist VMs in migrating from the source host to the destination host.

[0195] Statement 2: Embodiments of the inventive concept include the storage device described in Statement 1, wherein the storage device includes a solid-state drive (SSD).

[0196] Statement 3: Embodiments of the inventive concept include the storage device according to Statement 1, wherein the storage state includes at least one of the following: transport configuration information for a controller of a VM, interface configuration information for a controller of a VM, input / output (I / O) activity of the VM, and a table that translates logical identifiers used by the VM into physical addresses used by the storage device.

[0197] Statement 4. An embodiment of the inventive concept includes the storage device according to Statement 1, wherein the VM migration state monitoring and capture module tracks a first change in the storage state of a VM in the storage device and a second change in the storage data of the VM in the storage device.

[0198] Statement 5. Embodiments of the inventive concept include the storage device described in Statement 4, wherein the VM migration state monitoring and capture module sends a first change and a second change to at least one of a virtual machine monitor on the source host and a second storage device on the destination host.

[0199] Statement 6. Embodiments of the inventive concept include the storage device according to Statement 4, wherein the VM migration status monitoring and capture module sends the storage status and storage data of the VM to at least one of a virtual machine monitor on the source host and a second storage device on the destination host.

[0200] Statement 7. Embodiments of the inventive concept include the storage device described in Statement 6, wherein the stored data is subjected to at least one of compression, encoding, and encryption.

[0201] Statement 8. Embodiments of the inventive concept include the storage device according to Statement 7, wherein, after at least one of decompressing, decoding, and decrypting the stored data, the VM migration status monitoring and capture module sends the storage status and storage data of the VM to at least one of a virtual machine monitor on the source host and a second storage device on the destination host.

[0202] Statement 9. Embodiments of the inventive concept include the storage device described in Statement 4, wherein the VM migration state monitoring and capture module sets a flag based at least in part on the amount of data written by the VM to the storage device exceeding a threshold.

[0203] Statement 10: Embodiments of the inventive concept include the storage device according to Statement 9, wherein a virtual machine monitor on a source host uses the flag to select between all storage data of a VM being transferred from the storage device or a subset of storage data of a VM being transferred from the storage device.

[0204] Statement 11: Embodiments of the inventive concept include the storage device described in Statement 4, wherein the VM migration status monitoring and capture module generates a report including the storage status.

[0205] Statement 12. Embodiments of the inventive concept include the storage device described in Statement 1, the storage device further comprising: a cache and a caching strategy.

[0206] Statement 13. Embodiments of the inventive concept include the storage device described in Statement 12, wherein a caching strategy retains cache lines in the cache until the cache lines are sent to at least one of a virtual machine monitor on the source host and a second storage device on the destination host.

[0207] Statement 14. Embodiments of the inventive concept include the storage device described in Statement 1, the storage device further comprising: a cache and a prefetching strategy.

[0208] Statement 15. Embodiments of the inventive concept include the storage device according to Statement 14, wherein a prefetch strategy transfers VM storage data from a first storage device to a cache to accelerate the transmission of VM storage data from the storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host.

[0209] Statement 16. Embodiments of the inventive concept include the storage device according to Statement 15, wherein the storage device performs at least one of decompression, decoding, and decryption on stored data transferred from a first storage device to a cache according to a prefetch strategy.

[0210] Statement 17. Embodiments of the inventive concept include the storage device according to Statement 1, wherein the at least one controller includes a non-volatile memory fast (NVMe) controller.

[0211] Statement 18. Embodiments of the inventive concept include the storage device according to Statement 1, wherein the at least one controller is associated with one of the physical function (PF) and virtual function (VF) disclosed for the storage device.

[0212] Statement 19. Embodiments of the inventive concept include the storage device described in Statement 18, wherein the at least one controller is associated with only one of PF and VF.

[0213] Statement 20: Embodiments of the inventive concept include the storage device according to Statement 1, wherein the at least one controller includes at least one of a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose GPU (GPGPU), and software.

[0214] Statement 21: Embodiments of the inventive concept include the storage device according to Statement 1, wherein the storage device controller includes the at least one controller.

[0215] Statement 22: Embodiments of the inventive concept include a storage device, the storage device comprising:

[0216] The first storage device is used to store data;

[0217] A storage device controller, configured to use a first storage device to process at least one read request received from a host, and to use the first storage device to process at least one write request received from a host; and

[0218] The incremental module is used to identify changes in the stored data in the first storage device relative to the snapshot.

[0219] Statement 23: Embodiments of the inventive concept include the storage device described in Statement 22, wherein the incremental module sends changes to the stored data to the host.

[0220] Statement 24. An embodiment of the inventive concept includes a method comprising:

[0221] The storage device on the source host receives commands that assist in migrating virtual machines (VMs) from the source host to the destination host;

[0222] Track the first change in the storage state of the VM in the storage device;

[0223] The second change in the stored data of the VM in the storage device is tracked;

[0224] Send the first change to the storage state of the VM from the storage device; and

[0225] The second change is sending the VM's stored data from the storage device.

[0226] Statement 25. Embodiments of the inventive concept include the method according to Statement 24, wherein the storage device includes a solid-state drive (SSD).

[0227] Statement 26. Embodiments of the inventive concept include the method according to Statement 24, wherein:

[0228] The step of sending a first change in storage state from a storage device includes: sending the first change in storage state from the storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host; and

[0229] The step of sending a second change to the storage data of a VM from a storage device includes: sending the second change to the storage data of the VM from the storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host.

[0230] Statement 27. Embodiments of the inventive concept include the method according to Statement 24, the method further comprising:

[0231] Send the storage state of the VM from the storage device; and

[0232] Send the VM's storage data from the storage device.

[0233] Statement 28. Embodiments of the inventive concept include the method according to Statement 27, wherein:

[0234] The step of sending the storage state of a VM from a storage device includes: sending the storage state of the VM from the storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host; and

[0235] The steps of sending storage data of a VM from a storage device include: sending the storage data of the VM from the storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host.

[0236] Statement 29. Embodiments of the inventive concept include the method according to Statement 28, wherein the stored data is subjected to at least one of compression, encoding, and encryption.

[0237] Statement 30, embodiments of the inventive concept include the method according to Statement 29, wherein the step of sending storage data of a VM from a storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host includes:

[0238] Decompressing, decoding, and decrypting stored data to produce non-encoded stored data; and

[0239] Send non-encoded storage data of a VM from a storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host.

[0240] Statement 31: Embodiments of the inventive concept include the method according to Statement 27, wherein:

[0241] The method further includes: receiving a request for the storage state of the VM on the storage device; and

[0242] The step of sending the storage status of a VM from a storage device includes: in response to a request for the storage status of a VM, sending the storage status of the VM from the storage device.

[0243] Statement 32. Embodiments of the inventive concept include the method according to Statement 27, wherein:

[0244] The method further includes: receiving a request for storage data for the VM on the storage device; and

[0245] The steps of sending storage data of a VM from a storage device include: in response to a request for storage data of a VM, sending storage data of the VM from the storage device.

[0246] Statement 33. Embodiments of the inventive concept include the method according to Statement 27, the method further comprising:

[0247] Track the amount of data written from the VM to the storage device; and

[0248] At least in part, a flag is set in the storage device based on the fact that the amount of data written by the VM to the storage device exceeds a threshold.

[0249] The source host can use the flag to select between transferring all storage data of the VM from the storage device or transferring a subset of storage data of the VM from the storage device.

[0250] Statement 34. Embodiments of the inventive concept include the method according to Statement 27, wherein the step of sending the storage state of the VM from the storage device includes:

[0251] Generate reports including the storage status of VMs; and

[0252] The report is stored on a storage device.

[0253] The virtual machine monitor on the source host can read the report from the storage device.

[0254] Statement 35. Embodiments of the inventive concept include the method according to Statement 24, wherein the storage state includes at least one of the following: transport configuration information for the VM controller, interface configuration information for the VM controller, VM input / output (I / O) activity, and a table that translates logical identifiers used by the VM into physical addresses used by the storage device.

[0255] Statement 36. Embodiments of the inventive concept include the method according to Statement 24, the method further comprising: adjusting at least one of a caching policy and a prefetching policy to facilitate the migration of a VM from a source host to a destination host.

[0256] Statement 37. Embodiments of the inventive concept include the method according to Statement 36, wherein the caching strategy retains cache lines in the cache until the cache lines are sent to at least one of a virtual machine monitor on the source host and a second storage device on the destination host; and

[0257] The prefetch strategy moves VM storage data from the storage device to the cache to accelerate the sending of VM storage data from the storage device to at least one of the virtual machine monitor on the source host and a second storage device on the destination host.

[0258] Statement 38. Embodiments of the inventive concept include the method according to Statement 37, the method further comprising at least one of decompressing, decoding and decrypting the stored data of the VM in the cache.

[0259] Statement 39. Embodiments of the inventive concept include the method according to Statement 24, the method further comprising: receiving, on a storage device on a source host, a command indicating that the migration of a virtual machine (VM) from the source host to the destination host is complete.

[0260] Statement 40: An embodiment of the inventive concept includes a method comprising:

[0261] The storage device on the destination host receives commands that assist in migrating virtual machines (VMs) from the source host to the destination host;

[0262] The storage device receives changes in the storage state of the VM;

[0263] Update the storage device to match the changes in the VM's storage state;

[0264] The storage device receives changes to the VM's stored data; and

[0265] Update the storage data of the VM on the storage device to match the changes to the VM's storage data.

[0266] Statement 41. Embodiments of the inventive concept include the method according to Statement 40, the method further comprising: receiving a second command from a storage device that the VM is active.

[0267] Statement 42. Embodiments of the inventive concept include the method according to Statement 40, wherein the storage device includes a solid-state drive (SSD).

[0268] Statement 43. Embodiments of the inventive concept include the method according to Statement 40, wherein:

[0269] The step of receiving a change in the storage state of a VM on a storage device includes: receiving the change in the storage state of a VM from at least one of a virtual machine monitor on a destination host and a second storage device on a source host;

[0270] The step of receiving changes to the storage data of a VM on a storage device includes: receiving changes to the storage data of a VM from at least one of a virtual machine monitor on a destination host and a second storage device on a source host.

[0271] Statement 44. Embodiments of the inventive concept include the method according to Statement 40, the method further comprising:

[0272] The storage device receives the storage status of the VM;

[0273] Configure the storage device to match the storage state of the VM;

[0274] Receives the VM's stored data from the storage device; and

[0275] Store the VM's storage data on a storage device.

[0276] Statement 45. Embodiments of the inventive concept include the method according to Statement 44, wherein:

[0277] The step of receiving the storage state of a VM from a storage device includes: receiving the storage state of a VM from at least one of a virtual machine monitor on a destination host and a second storage device on a source host;

[0278] The step of receiving storage data of a VM on a storage device includes: receiving storage data of the VM from at least one of a virtual machine monitor on a destination host and a second storage device on a source host.

[0279] Statement 46. Embodiments of the inventive concept include the method according to Statement 45, wherein the stored data is subjected to at least one of compression, encoding, and encryption.

[0280] Statement 47. Embodiments of the inventive concept include the method according to Statement 46, wherein the step of receiving storage data of a VM from at least one of a virtual machine monitor on a destination host and a second storage device on a source host includes:

[0281] The storage device receives non-encoded data of the VM from at least one of the virtual machine monitor on the source host and a second storage device on the destination host; and

[0282] Compressing, encoding, and encrypting at least one of the following to produce stored data:

[0283] Statement 48. Embodiments of the inventive concept include the method according to Statement 40, wherein the storage state includes at least one of the following: transport configuration information for the VM controller, interface configuration information for the VM controller, VM input / output (I / O) activity, and a table that translates logical identifiers used by the VM into physical addresses used by the storage device.

[0284] Statement 49. Embodiments of the inventive concept include an article comprising a non-transitory storage medium having instructions stored thereon, which, when executed by a machine, cause:

[0285] The storage device on the source host receives commands that assist in migrating virtual machines (VMs) from the source host to the destination host;

[0286] Track the first change in the storage state of the VM in the storage device;

[0287] The second change in the stored data of the VM in the storage device is tracked;

[0288] Send the first change to the storage state of the VM from the storage device; and

[0289] The second change is sending the VM's stored data from the storage device.

[0290] Statement 50: Embodiments of the inventive concept include the articles described in Statement 49, wherein the storage device includes a solid-state drive (SSD).

[0291] Statement 51: Embodiments of the inventive concept include the article described in Statement 49, wherein:

[0292] The step of sending a first change in storage state from a storage device includes: sending the first change in storage state from the storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host; and

[0293] The step of sending a second change to the storage data of a VM from a storage device includes: sending the second change to the storage data of the VM from the storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host.

[0294] Statement 52: Embodiments of the inventive concept include the article according to Statement 49, wherein a non-transitory storage medium has further instructions stored thereon, which, when executed by a machine, cause:

[0295] Send the storage state of the VM from the storage device; and

[0296] Send the VM's storage data from the storage device.

[0297] Statement 53: Embodiments of the inventive concept include the articles described in Statement 52, wherein:

[0298] The step of sending the storage state of a VM from a storage device includes: sending the storage state of the VM from the storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host; and

[0299] The steps of sending storage data of a VM from a storage device include: sending the storage data of the VM from the storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host.

[0300] Statement 54. Embodiments of the inventive concept include the article described in Statement 53, wherein the stored data is subjected to at least one of compression, encoding, and encryption.

[0301] Statement 55, embodiments of the inventive concept include the article according to Statement 54, wherein the step of sending storage data of a VM from a storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host includes:

[0302] Decompressing, decoding, and decrypting stored data to produce non-encoded stored data; and

[0303] Send non-encoded storage data of a VM from a storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host.

[0304] Statement 56. Embodiments of the inventive concept include the articles described in Statement 52, wherein:

[0305] The non-transitory storage medium has additional instructions stored thereon, which, when executed by the machine, cause: the storage device to receive a request for the storage state of the VM; and

[0306] The step of sending the storage status of a VM from a storage device includes: in response to a request for the storage status of a VM, sending the storage status of the VM from the storage device.

[0307] Statement 57. Embodiments of the inventive concept include the articles described in Statement 52, wherein:

[0308] The non-transitory storage medium has additional instructions stored thereon, which, when executed by the machine, cause: the storage device to receive a request for storage data for the VM; and

[0309] The steps of sending storage data of a VM from a storage device include: in response to a request for storage data of a VM, sending storage data of the VM from the storage device.

[0310] Statement 58. Embodiments of the inventive concept include the article according to Statement 52, wherein the non-transitory storage medium has additional instructions stored thereon, which, when executed by a machine, cause:

[0311] Track the amount of data written from the VM to the storage device; and

[0312] At least in part, a flag is set in the storage device based on the fact that the amount of data written by the VM to the storage device exceeds a threshold.

[0313] The source host can use the flag to select between transferring all storage data of the VM from the storage device or transferring a subset of storage data of the VM from the storage device.

[0314] Statement 59. Embodiments of the inventive concept include the article according to Statement 52, wherein the step of sending the storage state of the VM from the storage device includes:

[0315] Generate reports including the storage status of VMs; and

[0316] The report is stored on a storage device.

[0317] The virtual machine monitor on the source host can read the report from the storage device.

[0318] Statement 60: Embodiments of the inventive concept include the article according to Statement 49, wherein the storage state includes at least one of the following: transport configuration information for the controller of the VM, interface configuration information for the controller of the VM, input / output (I / O) activity of the VM, and a table that translates logical identifiers used by the VM into physical addresses used by the storage device.

[0319] Statement 61: Embodiments of the inventive concept include the article according to Statement 49, a non-transitory storage medium having additional instructions stored thereon, which, when executed by a machine, cause at least one of a caching strategy and a prefetching strategy to assist in migrating a VM from a source host to a destination host.

[0320] Statement 62. Embodiments of the inventive concept include the article according to Statement 61, wherein the caching strategy retains cache lines in the cache until the cache lines are sent to at least one of a virtual machine monitor on the source host and a second storage device on the destination host; and

[0321] The prefetch strategy moves VM storage data from the storage device to the cache to accelerate the sending of VM storage data from the storage device to at least one of the virtual machine monitor on the source host and a second storage device on the destination host.

[0322] Statement 63. Embodiments of the inventive concept include the article according to Statement 62, wherein a non-transitory storage medium has further instructions stored thereon, which, when executed by a machine, cause at least one of: decompressing, decoding, and decrypting stored data of a VM in a cache.

[0323] Statement 64. Embodiments of the inventive concept include the article according to Statement 49, a non-transitory storage medium having further instructions stored thereon, which, when executed by a machine, cause: a storage device on a source host to receive a command indicating that the migration of a virtual machine (VM) from the source host to the destination host is complete.

[0324] Statement 65. Embodiments of the inventive concept include an article comprising a non-transitory storage medium having instructions stored thereon, which, when executed by a machine, cause:

[0325] The storage device on the destination host receives commands that assist in migrating virtual machines (VMs) from the source host to the destination host;

[0326] The storage device receives changes in the storage state of the VM;

[0327] Update the storage device to match the changes in the VM's storage state;

[0328] The storage device receives changes to the VM's stored data; and

[0329] Update the storage data of the VM on the storage device to match the changes to the VM's storage data.

[0330] Statement 66. Embodiments of the inventive concept include the article according to Statement 65, a non-transitory storage medium having further instructions stored thereon, which, when executed by a machine, cause: a second command to be received from the storage device to make the VM active.

[0331] Statement 67. Embodiments of the inventive concept include the articles described in Statement 65, wherein the storage device includes a solid-state drive (SSD).

[0332] Statement 68. Embodiments of the inventive concept include the article described in statement 65, wherein:

[0333] The step of receiving a change in the storage state of a VM on a storage device includes: receiving the change in the storage state of a VM from at least one of a virtual machine monitor on a destination host and a second storage device on a source host;

[0334] The step of receiving changes to the storage data of a VM on a storage device includes: receiving changes to the storage data of a VM from at least one of a virtual machine monitor on a destination host and a second storage device on a source host.

[0335] Statement 69. Embodiments of the inventive concept include the article according to Statement 65, wherein a non-transitory storage medium has further instructions stored thereon, which, when executed by a machine, cause:

[0336] The storage device receives the storage status of the VM;

[0337] Configure the storage device to match the storage state of the VM;

[0338] Receives the VM's stored data from the storage device; and

[0339] Store the VM's storage data on a storage device.

[0340] Statement 70: Embodiments of the inventive concept include the article described in statement 69, wherein:

[0341] The step of receiving the storage state of a VM from a storage device includes: receiving the storage state of a VM from at least one of a virtual machine monitor on a destination host and a second storage device on a source host;

[0342] The step of receiving storage data of a VM on a storage device includes: receiving storage data of the VM from at least one of a virtual machine monitor on a destination host and a second storage device on a source host.

[0343] Statement 71: Embodiments of the inventive concept include the article described in Statement 70, wherein the stored data is subjected to at least one of compression, encoding, and encryption.

[0344] Statement 72, embodiments of the inventive concept include the article according to Statement 71, wherein the step of receiving storage data of a VM from at least one of a virtual machine monitor on a destination host and a second storage device on a source host includes:

[0345] The storage device receives non-encoded data of the VM from at least one of the virtual machine monitor on the source host and a second storage device on the destination host; and

[0346] Compressing, encoding, and encrypting at least one of the following to produce stored data:

[0347] Statement 73: Embodiments of the inventive concept include the article according to Statement 65, wherein the storage state includes at least one of the following: transport configuration information for the controller of the VM, interface configuration information for the controller of the VM, input / output (I / O) activity of the VM, and a table that translates logical identifiers used by the VM into physical addresses used by the storage device.

[0348] Therefore, given the extensive variety of substitutions to the embodiments described herein, this detailed description and the accompanying materials are intended to be illustrative only and should not be considered as limiting the scope of the inventive concept. Thus, the inventive concept claims all such modifications that fall within the scope and spirit of the claims and their equivalents.

Claims

1. A storage device, comprising: At least one controller is used for virtual machines, which reside on the source host. The first storage device is used to store data for virtual machines; The second storage device is used to store the virtual machine's state; A storage device controller is configured to use a first storage device to process at least one read request for a virtual machine received from the controller, and to use the first storage device to process at least one write request for a virtual machine received from the controller. and The virtual machine migration status monitoring and capture module is used to assist virtual machines in migrating from the source host to the destination host. The virtual machine migration state monitoring and capture module tracks the first change in the storage state of the virtual machines in the storage device, and tracks the second change in the storage data of the virtual machines in the storage device. The virtual machine migration status monitoring and capture module sends the first and second changes to the second storage device on the destination host.

2. The storage device according to claim 1, wherein, The storage state includes at least one of the following: transport configuration information for the virtual machine controller, interface configuration information for the virtual machine controller, input / output activities of the virtual machine, and a table that translates logical addresses used by the virtual machine into physical addresses used by the storage device.

3. The storage device according to claim 1, wherein, The virtual machine migration status monitoring and capture module sends the virtual machine's storage status and storage data to at least one of the virtual machine monitor on the source host and a second storage device on the destination host.

4. The storage device according to claim 1, wherein, The virtual machine migration status monitoring and capture module sets a flag based, at least in part, on the amount of data written by the virtual machine to the storage device exceeding a threshold.

5. The storage device according to claim 1, wherein, The virtual machine migration status monitoring and capture module generates reports that include storage status.

6. The storage device according to any one of claims 1 to 5, further comprising: A caching and prefetching strategy, wherein the prefetching strategy transfers virtual machine storage data from a first storage device to a cache to accelerate the transmission of virtual machine storage data from the storage device to at least one of a virtual machine monitor on a source host and a second storage device on a destination host.

7. The storage device according to any one of claims 1 to 5, wherein, The at least one controller is associated with one of the physical and virtual functions disclosed for the storage device.

8. The storage device according to claim 1, wherein, The storage device controller includes the at least one controller.

9. A method for assisting virtual machine migration, comprising: The storage device on the source host receives commands that assist in migrating the virtual machine from the source host to the destination host; Track the first change in the storage state of the virtual machine in the storage device; The second change in the stored data of the virtual machine in the storage device is tracked; Send the first change to the storage state of the virtual machine from the storage device; and The second change is sending the virtual machine's stored data from the storage device. The step of sending a first change in storage state from the storage device includes: sending the first change in storage state from the storage device to a second storage device on the destination host, and The step of sending the second change of the virtual machine's storage data from the storage device includes: sending the second change of the virtual machine's storage data from the storage device to a second storage device on the destination host.

10. The method according to claim 9, wherein: The step of sending the first change in storage state from the storage device further includes: sending the first change in storage state from the storage device to a virtual machine monitor on the source host; and The step of sending the second change of the virtual machine's storage data from the storage device further includes: sending the second change of the virtual machine's storage data from the storage device to the virtual machine monitor on the source host.

11. The method of claim 9, further comprising: Send the storage state of the virtual machine from the storage device to at least one of the virtual machine monitor on the source host and the second storage device on the destination host; and The storage data of the virtual machine is sent from the storage device to at least one of the virtual machine monitor on the source host and the second storage device on the destination host.

12. The method according to claim 11, wherein: The method further includes: receiving a request for the storage state of a virtual machine from the storage device; and The steps of sending the storage state of a virtual machine from a storage device include: in response to a request for the storage state of a virtual machine, sending the storage state of the virtual machine from the storage device.

13. The method of claim 11, further comprising: Track the amount of data written to storage devices by virtual machines; and At least in part, a flag is set in the storage device based on the fact that the amount of data written to the storage device by the virtual machine exceeds a threshold. The source host uses the flag to select between transferring all storage data of the virtual machine from the storage device or transferring a subset of storage data of the virtual machine from the storage device.

14. The method according to claim 11, wherein, The steps for sending the storage state of a virtual machine from a storage device include: Generate reports including the storage status of virtual machines; and The report is stored on a storage device. The virtual machine monitor on the source host reads the report from the storage device.

15. The method according to claim 9, wherein, The storage state includes at least one of the following: transport configuration information for the virtual machine controller, interface configuration information for the virtual machine controller, virtual machine input / output activities, and a table that translates logical addresses used by the virtual machine into physical addresses used by the storage device.

16. The method according to any one of claims 9 to 15, further comprising: Adjust at least one of the caching and prefetching policies to assist the virtual machine in migrating from the source host to the destination host.

17. A method for assisting virtual machine migration, comprising: The storage device on the destination host receives commands that assist in migrating the virtual machine from the source host to the destination host; The storage device receives changes in the storage state of the virtual machine; Update the storage device to match the changes in the virtual machine's storage state; The storage device receives changes to the virtual machine's stored data; and Update the virtual machine's storage data on the storage device to match the changes to the virtual machine's storage data. The second storage device on the source host includes a virtual machine migration state monitoring and capture module that tracks changes in the storage state and storage data of virtual machines. The virtual machine migration status monitoring and capture module sends changes in the virtual machine's storage status and storage data to the storage device on the destination host.

18. The method of claim 17, further comprising: The storage device receives the storage status of the virtual machine; Configure the storage device to match the storage state of the virtual machine; The storage device receives the virtual machine's stored data; and The virtual machine's storage data is stored on a storage device.

19. The method according to claim 17 or claim 18, wherein, The storage state includes at least one of the following: transport configuration information for the virtual machine controller, interface configuration information for the virtual machine controller, virtual machine input / output activities, and a table that translates logical addresses used by the virtual machine into physical addresses used by the storage device.

Images