Method for transmitting information by storage device and storage device for transmitting information

Abstract

A method for transmitting information by a storage device and a storage device for transmitting information are provided, the method comprising: determining that a first area of ​​a source storage device includes a storage location, the storage location includes data to be copied from the source storage device to a target storage device, the first area having a size equal to a first granularity size; determining that a second area within the first area includes the storage location, the second area having a size equal to a second granularity size, the second granularity size being smaller than the first granularity size; in response to determining that the second area is associated with a threshold, exiting a query mode including the first granularity size and the second granularity size; and sending a result data structure indicating that the second area includes the storage location.

Description

[0001] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 415,558, filed on October 12, 2022, entitled “Efficient Passing of Mapped and Dirty Bitmaps Tracked Through Namespaces,” and U.S. Patent Application No. 18 / 160,414, filed on January 27, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] One or more aspects according to one or more embodiments of the present disclosure relate to systems and methods for data storage. Background Art

[0003] This Background section is intended only to provide background, and the disclosure of any embodiment or concept in this section does not constitute an admission that such embodiment or concept is prior art.

[0004] In the field of data storage, a live migration (or "LM") operation (or live migration process) may refer to an operation or process that copies data from a source storage device to a target storage device (e.g., a migration process) while the source storage device can potentially receive a user data read or user data write from a server (e.g., a source server or host) (e.g., the source storage device is active). As used herein, a "live migration" component (e.g., a "live migration server" or "live migration storage device," etc.) refers to a component that can be involved in transferring data from a source storage device to a target storage device and that can have higher / additional rights (e.g., higher / additional rights for accessing data within the system) than other components of the system. Some processes for live migration may create metadata that tracks the location of data within the source storage device to be copied to the target storage device.

[0005] Therefore, there may be methods, apparatus, and systems suitable for improving the delivery of metadata for performing live migrations. Summary of the Invention

[0006] Aspects of embodiments of the present disclosure relate to computer storage systems and provide improvements to the delivery of metadata (eg, bitmaps and / or LBA lists) used to identify locations in a source storage device having data to be copied to a target storage device.

[0007] According to one or more embodiments of the present disclosure, a method for transmitting information by a storage device is provided, the method comprising: determining that a first area of ​​a source storage device includes a storage location, the storage location includes data to be copied from the source storage device to a target storage device, the first area having a size equal to a first granularity size; determining that a second area within the first area includes the storage location, the second area having a size equal to a second granularity size, the second granularity size being smaller than the first granularity size; determining that the second area is associated with a threshold; exiting a query mode including the first granularity size and the second granularity size; and sending a result data structure indicating that the second area includes the storage location.

[0008] The method may also include: receiving a request for "information indicating that one or more storage locations include data to be copied from a source storage device to a target storage device", the request being associated with a first granularity size, wherein the resulting data structure includes: a bitmap indicating that the first area or the second area includes one or more storage locations, the one or more storage locations including data to be copied from the source storage device to the target storage device; or a list of one or more logical block addresses (LBAs), the one or more logical block addresses including the data to be copied.

[0009] The method may also include determining that the number of storage locations within the first area or the second area and including data to be copied from the source storage device to the target storage device is equal to or less than a threshold number of storage locations as the threshold, wherein the resulting data structure includes a list of the one or more LBAs.

[0010] The method may further include determining that the number of storage locations corresponding to the second bitmap and including data to be copied from the source storage device to the target storage device is equal to or less than a threshold number of storage locations for each bitmap as the threshold, wherein the first region is associated with the first bitmap, wherein the second region is associated with the second bitmap, wherein the second bitmap indicates that one or more storage locations within the second region include data to be copied from the source storage device to the target storage device, and wherein the resulting data structure includes a list of the one or more LBAs.

[0011] The method may further include determining that a second granule size is equal to or less than a threshold granule size as the threshold, wherein the resulting data structure includes the list of the one or more LBAs.

[0012] The method may further include copying data corresponding to all storage locations within the second region, wherein the threshold may include a threshold granularity size or a number of remaining contiguous LBAs.

[0013] The method may further include determining a first granularity size or a second granularity size based on information stored on a storage device, wherein the request for information is directed to a namespace including the one or more storage locations, and wherein the first region and the second region correspond to the namespace.

[0014] The query mode may include a recursive query mode, the second granularity size may be greater than a third granularity size corresponding to the third region, the third region is within the second region and includes the storage location, the first region may be associated with a first data structure, the first data structure indicates that one or more storage locations within the first region include data to be copied from the source storage device to the target storage device, the second region may be associated with a second data structure, the second data structure indicates that one or more storage locations within the second region include data to be copied from the source storage device to the target storage device, and the result data structure may include a self-describing return data structure, the self-describing return data structure includes at least one of a bitmap header and an LBA list header, the first data structure, the second data structure, and a global header.

[0015] According to one or more other embodiments of the present disclosure, a storage device for transmitting information is provided, wherein the storage device is configured to: determine that a first area of ​​a source storage device includes a storage location, the storage location includes data to be copied from the source storage device to a target storage device, and the first area has a size equal to a first granularity size; determine that a second area within the first area includes the storage location, the second area has a size equal to a second granularity size, and the second granularity size is smaller than the first granularity size; determine that the second area is associated with a threshold; exit a query mode including the first granularity size and the second granularity size; and send a result data structure indicating that the second area includes the storage location.

[0016] The storage device may be configured to: receive a request for "information indicating that one or more storage locations include data to be copied from a source storage device to a target storage device", the request being associated with a first granularity size, wherein the resulting data structure comprises: a bitmap indicating that the first area or the second area includes one or more storage locations, the one or more storage locations including data to be copied from the source storage device to the target storage device; or a list of one or more logical block addresses (LBAs), the one or more logical block addresses including the data to be copied.

[0017] The storage device may be configured to determine that the number of storage locations within the first area or the second area and comprising data to be copied from the source storage device to the target storage device is equal to or less than a threshold number of storage locations serving as the threshold, wherein the resulting data structure comprises a list of the one or more LBAs.

[0018] The storage device may be configured to: determine that the number of storage locations corresponding to the second bitmap and including data to be copied from the source storage device to the target storage device is equal to or less than a threshold number of storage locations for each bitmap as the threshold, wherein the first area is associated with the first bitmap, wherein the second area is associated with the second bitmap, wherein the second bitmap indicates that one or more storage locations within the second area include data to be copied from the source storage device to the target storage device, and wherein the resulting data structure includes a list of the one or more LBAs.

[0019] The storage device may be configured to determine that the second granularity size is equal to or less than a threshold granularity size as the threshold, wherein the resulting data structure includes the list of the one or more LBAs.

[0020] The method may further include copying data corresponding to all storage locations within the second region, wherein the threshold may be a threshold granularity size or a number of remaining contiguous LBAs.

[0021] The storage device may be configured to determine the first granularity size or the second granularity size based on information stored on the storage device, wherein the request for information is directed to a namespace including the one or more storage locations, and wherein the first region and the second region correspond to the namespace.

[0022] The query mode may include a recursive query mode, the second granularity size may be greater than a third granularity size corresponding to the third region, the third region is within the second region and includes the storage location, the first region may be associated with a first data structure, the first data structure indicates that one or more storage locations within the first region include data to be copied from the source storage device to the target storage device, the second region may be associated with a second data structure, the second data structure indicates that one or more storage locations within the second region include data to be copied from the source storage device to the target storage device, and the result data structure may include a self-describing return data structure, the self-describing return data structure includes at least one of a bitmap header and an LBA list header, the first data structure, the second data structure, and a global header.

[0023] According to one or more other embodiments of the present disclosure, a method for transmitting information by a storage device is provided, the method comprising: receiving an initial request for "information indicating that one or more storage locations include data to be copied from a source storage device to a target storage device", the initial request being associated with a first granularity size; determining that a first area of ​​the source storage device includes storage locations, the storage locations include data to be copied from the source storage device to the target storage device, the first area having a size equal to the first granularity size; and based on the initial request, automatically determining that a second area within the first area includes the storage locations, the second area having a size equal to a second granularity size, the second granularity size being smaller than the first granularity size.

[0024] The method may also include determining that the second region is associated with the threshold; and transmitting a resulting data structure.

[0025] The method may also include determining that a second region is associated with the threshold; and exiting a recursive query mode including the first granularity size and the second granularity size.

[0026] The first region may be associated with a first data structure indicating that one or more storage locations within the first region include data to be copied from a source storage device to a target storage device, the second region is associated with a second data structure indicating that one or more storage locations within the second region include data to be copied from the source storage device to the target storage device, and the result data structure includes a self-describing return data structure including at least one of a bitmap header and an LBA list header, the first data structure, the second data structure, and a global header. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Non-limiting and non-exhaustive embodiments of the present disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0028] Figure 1 is a system diagram illustrating a system for communicating information for performing live migration according to one or more embodiments of the present disclosure.

[0029] Figure 2 is a conceptual diagram depicting a method of delivering information for performing live migration with respect to a simple example according to one or more embodiments of the present disclosure.

[0030] Figure 3 is a diagram depicting the generation of a cumulative results data structure conveying information used to perform live migration, according to one or more embodiments of the present disclosure.

[0031] Figure 4is a diagram depicting a self-describing return structure that conveys information for performing live migration, according to one or more embodiments of the present disclosure.

[0032] Figure 5 is a flow diagram depicting example operations of a method of communicating information for performing live migration in accordance with some embodiments of the present disclosure.

[0033] Throughout the several views of the drawings, corresponding reference numerals indicate corresponding components. Those skilled in the art will understand that the elements in the drawings are shown for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements and regions in the drawings may be exaggerated relative to other elements and regions to help improve the clarity and understanding of the various embodiments. In addition, common but well-known elements and components not relevant to the description of the embodiments may not be shown to facilitate a less obstructed view of the various embodiments and to clarify the description. DETAILED DESCRIPTION

[0034] By referring to the specific implementation methods and drawings of one or more embodiments, the aspects of the present disclosure and the methods for implementing the aspects of the present disclosure can be more easily understood. Hereinafter, the embodiments will be described in more detail with reference to the drawings. However, the described embodiments can be implemented in various different forms and should not be interpreted as being limited to the embodiments shown herein. On the contrary, these embodiments are provided as examples so that the present disclosure will be thorough and complete and will fully convey the aspects of the present disclosure to those skilled in the art. Therefore, descriptions of processing, elements and techniques that are unnecessary for those of ordinary skill in the art to fully understand the aspects and features of the present disclosure may be omitted.

[0035] Unless otherwise specified, throughout the drawings and written description, the same reference numerals, characters, or combinations thereof denote the same elements, and therefore, their description will not be repeated. In addition, for clarity of description, parts not related to the description of the embodiment may not be shown. In the drawings, the relative sizes of elements and regions may be exaggerated for clarity.

[0036] In the detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments. However, it is apparent that the various embodiments can be practiced without these specific details or with one or more equivalent arrangements.

[0037] It will be understood that although the terms "zeroth," "first," "second," "third," etc. may be used herein to describe various elements, components, regions, and / or parts, these elements, components, regions, and / or parts should not be limited by these terms. These terms are used to distinguish one element, component, region, or part from another element, component, region, or part. Thus, a first element, first component, first region, or first part described below may be referred to as a second element, second component, second region, or second part without departing from the spirit and scope of the present disclosure.

[0038] It will be understood that when an element or component is referred to as being “on,” “connected to,” or “coupled to” another element or component, it can be directly on, connected to, or coupled to the other element or component, or one or more intervening elements or components may be present. However, “directly connected / directly coupled” means that one component is directly connected or coupled to another component without intervening components. Meanwhile, other expressions describing the relationship between components (such as, “between,” “directly between,” or “adjacent to,” and “directly adjacent to”) may be similarly interpreted. Furthermore, it will be understood that when an element or layer is referred to as being “between” two elements or components, it can be the only element or component between the two elements or components, or one or more intervening elements or components may also be present.

[0039] The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to also include the plural form. It will also be understood that when the terms "comprise", "have" and "include" are used in this specification, the features, wholes, steps, operations, elements and / or components stated are described, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their groups are not excluded. As used herein, each of the terms "or" and "and / or" includes any combination and all combinations of one or more of the associated listed items.

[0040] For the purposes of this disclosure, expressions such as “at least one of” when following a list of elements modify the entire list of elements and do not modify the individual elements in the list. For example, “at least one of X, Y, and Z” and “at least one selected from the group consisting of X, Y, and Z” may be interpreted as X only, Y only, Z only, or any combination of two or more of X, Y, and Z (such as, for example, XYZ, XYY, YZ, and ZZ).

[0041] As used herein, the terms "substantially," "about," "approximately," and similar terms are used as terms of approximation rather than terms of degree, and are intended to take into account the inherent deviations in measured or calculated values ​​that one of ordinary skill in the art would recognize. As used herein, "about" or "approximately" is inclusive of the stated value and means within an acceptable range of deviation for the particular value as determined by one of ordinary skill in the art, taking into account the measurement in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system). For example, "about" may mean within one or more standard deviations, or within ±30%, ±20%, ±10%, or ±5% of the stated value. Additionally, the use of "may" when describing embodiments of the present disclosure means "one or more embodiments of the present disclosure."

[0042] When one or more embodiments can be implemented differently, a specific processing order can be performed differently from the order described. For example, two consecutively described processes can be performed substantially simultaneously or in an order opposite to the order described.

[0043] Any component or any combination of components described (e.g., in any system diagram included herein) may be used to perform one or more of the operations in any flowchart included herein. Furthermore, (i) the operations are examples only and may involve various additional operations not explicitly covered, and (ii) the temporal order of the operations may be changed.

[0044] The electronic or electrical devices and / or any other related devices or components according to the embodiments of the present disclosure described herein can be implemented using any suitable hardware, firmware (e.g., an application specific integrated circuit), software, or a combination of software, firmware, and hardware. For example, the various components of these devices can be formed on an integrated circuit (IC) chip or on separate IC chips. In addition, the various components of these devices can be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or formed on a substrate.

[0045] In addition, the various components of these devices can be processes or threads that execute computer program instructions and interact with other system components to perform the various functions described herein, running on one or more processors in one or more computing devices. The computer program instructions are stored in a memory, which can be implemented in a computing device using a standard memory device (such as, for example, a random access memory (RAM)). The computer program instructions can also be stored in other non-transitory computer-readable media (such as, for example, a CD-ROM, a flash drive, etc.). In addition, those skilled in the art will recognize that, without departing from the spirit and scope of the embodiments of the present disclosure, the functions of the various computing devices can be combined or integrated into a single computing device, or the functions of a particular computing device can be distributed on one or more other computing devices.

[0046] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention pertains. It will also be understood that, unless expressly defined otherwise herein, terms (such as those defined in common dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and / or in this specification, and should not be interpreted in an idealized or overly formal manner.

[0047] As mentioned above, in the field of data storage, a live migration operation (or live migration process) may refer to a process or process in which data is copied from a source storage device to a target storage device while the source storage device may potentially receive user data reads or writes from a source host (e.g., a host that may be assigned lower supervisory privileges than a hypervisor (or live migration server)). For example, a data center owner, operator, and / or software platform operator that sells computing resources may implement a live migration strategy in which data corresponding to a source storage device may be moved from the source storage device to a new (or target) storage device while the data at the source storage device may be changed (e.g., the data may be updated based on receiving user data writes). The live migration strategy may involve moving a namespace (e.g., a group of objects in a storage device) from a source controller (or source storage device) to a target controller (or target storage device).

[0048] As will be discussed in further detail below with reference to the accompanying drawings, the source live migration controller can assist the source live migration server in managing the live migration operation by creating tracking metadata (e.g., metadata for tracking valid pages or dirty pages). For example, the step of generating tracking metadata may include: creating a bitmap ("BMP") of "mapping" pages (e.g., pages corresponding to valid data to be copied from the source storage device to the target storage device) at the beginning of the live migration process, and creating a bitmap ("BDP") of one or more subsequent "dirty" pages (e.g., pages corresponding to mapping pages that receive user data writes during the copying of valid data from the source storage device to the target storage device) during the live migration process. For example, the source live migration controller may construct a BMP on demand and use the BMP for the first iteration of data copying of the live migration process. Then, following a previous BMP or previous BDP query from the source live migration server, the source live migration controller may track subsequent user data writes in the BDP during a second or later iteration of the data migration.

[0049] As used herein, a valid page refers to a page (or logical block address (LBA)) that corresponds to a particular namespace and has data (e.g., valid data) to be copied from a source storage device to a target storage device. Valid data may exist at a source storage location that has received a write or copy of user data (e.g., from a source virtual machine (VM)). Valid data may not exist at a source storage location that has not received a write since the last format operation or the last erase operation. Valid data may not exist at a source storage location that has not received a write since the last write was released. Released writes represent unmapped (or "trimmed") writes. In some embodiments, data may be written and then released, but not regrouped as invalid. In such embodiments, valid data may include data at a storage location corresponding to the released write. In some embodiments, Write Zero and / or Write Uncorrectable may be considered valid data, depending on the preferred use case definition.

[0050] As used herein, a write zero (or write multiple zeros) (e.g., a write zero command) represents a write command that does not require the transfer of data. The write zero command may cause the storage device to write the data value "zero" (e.g., all data values ​​"zero") to the LBA range provided in the write zero command. The storage device may program these "zeros" to the storage medium for robust reading of the data later, and the storage device may also mark a sentinel value in a logical-to-physical (L2P) lookup table. For example, the sentinel value may be 0xFFFFFFFF. When the live migration server later reads the LBA corresponding to the sentinel value, the storage device may check the L2P lookup table to find the sentinel value. For example, the sentinel value may be outside the range of possible physical locations, which allows the storage device to determine that the sentinel value is a sentinel value. Looking up the meaning of the sentinel value may cause the storage device to return zero (e.g., all zeros) for the data of the LBA. If the write zero command corresponds to a range of LBAs, the sentinel value may be placed in the L2P table for each LBA in the range.

[0051] As used herein, write uncorrectable refers to a write command similar to a write zero command; however, instead of programming zeros to the storage medium, the storage device may program data and ECC metadata that may not be correctly decoded by an error correction code (ECC) decoding engine when read in the future. In some embodiments, a sentinel value different from the sentinel value used for the write zero command (e.g., 0xFFFFFFFE) may be placed in the L2P lookup table. When the live migration server reads an LBA that was previously written with an uncorrectable value, the read may not be decoded. The storage device may return an appropriate error or completion status to the read.

[0052] In some embodiments, written zeros and / or written uncorrectables may not be considered valid data, depending on a preferred use case definition. In some embodiments, valid data may be considered mapped data because even if data is written to zeros (or similar), the storage device continues to point to the location at the source storage device where the data can be found.

[0053] As used herein, a dirty page refers to a page (or LBA) on a source storage device that has changed data (e.g., dirty data) that may be copied to a target storage device because the data at the LBA in the source storage device does not match the data at the LBA in the target storage device. Dirty data may exist at a source storage location that receives a write or copy of user data (e.g., from a source VM). In some embodiments, a page that has received a write or copy and has therefore been marked as a dirty page may be released (or "unmarked"). In some embodiments, the released page may be considered to include dirty data, according to a preferred use case definition. In some embodiments, the released page may be considered to not include dirty data, according to a preferred use case definition. In some embodiments, write zeros and / or write uncorrectables may be considered to be dirty data, according to a preferred use case definition. In some embodiments, write zeros and / or write uncorrectables may not be considered to be dirty data, according to a preferred use case definition.

[0054] To provide an overview of live migration, one or more of the following operations may be part of the live migration process. A host (e.g., a source live migration server) may use BMP at the start of the live migration process so that the host may selectively copy (e.g., only copy) mapping data from the source storage device to the target storage device. Using BMP may reduce the number of requests that pass through two drivers (e.g., a source live migration storage device and a target live migration storage device) and across a network in the data center (e.g., across a "live migration server to target" link 10 (see below). Figure 1 and Figure 2 ))The amount of data moved can speed up the migration process.

[0055] The host can query to receive the BDP following the initial BMP and any subsequent BDPs. Using the BDP allows the host to selectively copy (e.g., copy only) data that has changed since the last query. Because the copy process can be performed more quickly (e.g., less data can be copied in each subsequent copy iteration of the live migration process), the amount of data queried can be expected to "tend to ε (i.e., infinitesimal)." Ultimately, the copied data can be determined to be at an acceptably small size, and once the entire final migration state and final data described by the last remaining BDP are moved from the source storage device and source VM to the target storage device and target VM (e.g., target server), the host can suspend operations of both the source VM (e.g., source server) and the storage device (e.g., source storage device and target storage device). As a final operation of the live migration process, the target LM server can restart the target storage device and target VM.

[0056] Tracking metadata can help provide information to the source live migration server. Tracking metadata can indicate where data at the source storage device has been updated or changed after the source live migration server copies the data from the source storage device to the target storage device.

[0057] Due to the relatively large consumption of computing resources (e.g., resources corresponding to memory usage and bandwidth), the performance of the live migration system may be overloaded during the live migration. For example, in the case of creating an initial BMP, the source live migration storage controller may create or maintain a BMP having a size corresponding to the entire namespace. The size of subsequent BDPs may also be huge. Therefore, creating and transmitting the initial BMP information and subsequent BDP information may consume a large amount of computing resources at the source live migration storage device. In addition, transmitting the BMP and each subsequent BDP from the source live migration storage device to the source live migration server and parsing the BMP and subsequent BDP by the source live migration server may consume a large amount of bandwidth and may cause significant delays.

[0058] For example, in one or more embodiments, the live migration system may use a query pattern (e.g., a recursive large-to-small granularity query pattern) to avoid sending a large portion of the BMP that lacks any valid pages and / or to avoid sending a relatively large portion of the BDP that does not have dirty pages. In one or more embodiments, the recursive large-to-small granularity query pattern may include one or more of the following operations described below. Although the present disclosure discusses the recursive query pattern in detail, it should be understood that the present disclosure is not limited to this. For example, in some embodiments, the live migration system may use an iterative query pattern.

[0059] The source live migration server may send a first request (e.g., a first query or a first management command) for information from the source live migration storage device, the information indicating whether a range of LBAs having a large granularity size includes valid pages or dirty pages. For example, the information may indicate the location of data to be copied from the source storage device to the target storage device. The source live migration storage device may divide the namespace into a plurality of large areas having a large granularity size. For example, the first request may be directed to the namespace. The source live migration storage device may determine which large areas include LBAs having valid pages or dirty pages. The source live migration storage device may send a first response to the first request of the source live migration server, the first response having a first data structure (e.g., a first bitmap) indicating which large areas include valid pages or dirty pages.

[0060] The source live migration server may send a second request (e.g., a second query or a second management command) to the source live migration storage device for information indicating whether valid pages or dirty pages are included in an LBA range having a size smaller than the granularity size of the previous request and within the LBA range from the previous request. The source live migration storage device may determine which smaller areas include LBAs having valid pages or dirty pages. The source live migration storage device may send a second response to the second request from the source live migration server, the second response having a second data structure (e.g., a second bitmap) indicating which large areas include valid pages or dirty pages.

[0061] In one or more embodiments, this recursive large-to-small granularity query pattern may be repeated until the granularity size associated with the request is equal to the size of one LBA. In one or more embodiments, the recursive large-to-small granularity query pattern may be repeated until the granularity size associated with the request is small enough or equal to or less than a threshold value (e.g., eight LBAs) such that additional or redundant copying of LBAs that do not include valid data or dirty data is deemed acceptable.

[0062] To improve bandwidth and latency, it may be desirable to reduce the number of management commands from the source live migration server to the source live migration storage device before the source live migration storage device sends a response indicating the precise LBAs that include valid or dirty pages. To improve bandwidth and latency, it may also be desirable to reduce the number of management commands from the source live migration server to the source live migration storage device before the source live migration server determines that a given granularity size is small enough so that every LBA within the region corresponding to the given granularity size can be copied by the source live migration server, even though some LBAs within the region may not include valid or dirty pages. For example, because some LBAs within the region do not include valid or dirty pages, some additional or redundant copying may be performed. However, at a sufficiently small granularity size, any such additional or redundant copying may be acceptable as long as all changes to the data are captured.

[0063] A system configured to deliver information for performing live migration using recursive large-to-small granularity queries can have improved bandwidth and latency characteristics by, for example, (i) reducing the number of management commands based on an exit threshold, (ii) reducing the number of management commands based on automation of recursive query and response patterns, and / or (iii) using a self-describing return structure that includes an accumulated result data structure (e.g., sending aggregated results with a header).

[0064] For example, the source live migration server may send a first request (e.g., a first query or a first management command) for information from the source live migration storage device indicating whether a range of LBAs having a large granularity size includes valid pages or dirty pages. In some embodiments, the source live migration storage device may enter a recursive large-to-small granularity query mode. In one or more embodiments, before sending a response including a bitmap indicating relevant areas including valid pages or dirty pages, the source live migration controller may determine whether a threshold has been met (e.g., whether a threshold associated with the number of LBAs having valid pages or dirty pages within one or more relevant areas has been exceeded). The threshold (e.g., an exit threshold or an early exit threshold) may correspond to the relevant area, and the threshold being met may indicate that the number of LBAs having valid pages or dirty pages within the relevant area is equal to or less than the threshold. For example, the relevant area may be LBA128 to LBA135, and the threshold may be less than two LBAs within the relevant area having valid data or dirty data. In response to determining that a threshold has been met (e.g., only one of LBA128 to LBA135 has valid data or dirty data), the source live migration controller may exit the query mode early (e.g., before the minimum granularity size of the source storage device is queried) and may send a resulting data structure including a bitmap or LBA list to the source live migration server to indicate the precise storage location including the valid page or dirty page.

[0065] In one or more embodiments, the granularity and / or exit threshold of the recursive large-to-small granularity query pattern may be stored on the source live migration storage device, such that the recursive large-to-small granularity query pattern may be automated within the source live migration storage device based on a single command from the source live migration server. For example, a single command may be sent to the source live migration storage device to trigger a recursive query and response pattern associated with one or more large-to-small granularity query cycles (e.g., one or more iterations).

[0066] In one or more embodiments, the source live migration storage device may accumulate bitmaps corresponding to each iteration of the recursive query mode. Otherwise, these bitmaps may have been sent back to the source live migration server by the source live migration storage device after each iteration. In some embodiments, when an exit threshold is reached, the source live migration storage device may exit the recursive mode and may send a result data structure including the accumulated bitmaps and / or LBA lists to the source live migration server using a self-describing return structure.

[0067] Figure 1 is a system diagram illustrating a system for communicating information for performing live migration according to one or more embodiments of the present disclosure.

[0068] Reference Figure 1 , system 1 may include a source host system 100S and a target host system 100T. Source host system 100S may be connected to a source storage device 200. Target host system 100T may be connected to a target storage device 300. Source host system 100S and target host system 100T may be associated with or may include a central processing unit (CPU). Source storage device 200 and target storage device 300 may include logical block addresses (LBAs) (e.g., LBA0 to LBA255, some of which are depicted in FIG. 1 for simplified example). Figure 1 The LBA may be associated with a physical location in the source storage device 200 and the target storage device 300 for storing data (eg, user data).

[0069] The target host system 100T may include a target server 100c. The target server 100c may be a host or a target VM. The target server 100c may be connected to the target storage device 250c via a communication link 20. The communication link 20 may include various technologies (e.g., Peripheral Component Interconnect Express (PCIe), Non-Volatile Memory Express (NVMe) over PCIe, NVMe over a Network (NVMe-oF)), etc.) or may be implemented by various technologies (e.g., Peripheral Component Interconnect Express (PCIe), Non-Volatile Memory Express (NVMe) over PCIe, NVMe over a Network (NVMe-oF)), etc.).

[0070] The target storage device 250c may include a target storage device controller 210c and a target controller memory 220c. The target controller memory 220c may include or may be RAM (e.g., static random access memory (SRAM) or dynamic random access memory (DRAM)). The target storage device controller 210c may include embedded logic or may be implemented as embedded logic. The embedded logic may be embedded within the target storage device 250c. The embedded logic may enable the target storage device controller 210c to process requests from a server (e.g., target server 100c) to copy / write data to the target storage device 300.

[0071] The target storage device 250c may include the target storage device 300 (or a portion of the target storage device 300). For example, the target storage device 300 may include target memory 330. The target memory 330 may include or may be long-term storage. For example, the target memory 330 may include non-volatile memory and / or a memory layer (including volatile memory and non-volatile memory) and may correspond to the long-term storage device of one or more target storage devices 250c. For example, the target storage device 300 may represent long-term storage composed of one or more target storage devices 250c of a distributed storage system (e.g., a virtualized distributed storage system). The target storage device 250c may be a solid-state drive (SSD) including one or more underlying target storage devices 250c, which may be virtual or physical. The target storage device 250c may be connected to the target storage device 300 via a storage interface 22. The storage interface 22 may include or may be a flash memory interface (e.g., a NAND flash memory interface, a toggle mode interface, or an open NAND flash interface (ONFI)) or a remote interface (e.g., an interface implemented via Ethernet).

[0072] In one or more embodiments, the communication link 20 and the storage interface 22 may be shared connections. For example, the source storage device 200 may be all identical NAND, and the source storage device controller 210a and the source live migration controller 210b may be the same controller (e.g., the same ASIC controller). In such an embodiment, the storage interface 22 may be the same NAND channel running between the two entities (e.g., the source storage device 250a and the source storage device 200). Similarly, the communication link 20 may connect the source storage device controller 210a and the source live migration controller 210b to the source server 100a and the source live migration server 100b, respectively, over the same link (e.g., the same PCIe link).

[0073] The source host system 100S may include a source server 100a. The source server 100a may be a host or a source VM. The source server 100a may service read and write requests from user applications. For example, a user may use an application (e.g., a word processing application) to send a user data write (UDW) (e.g., a user application data write request) or a user data read (UDR) (e.g., a user application data read request) to the source storage device 200. The read / write request may be sent to the source storage device 200 via the source storage device 250a.

[0074] The source server 100a can be connected to the source storage device 250a via a communication link 20. The communication link 20 can include various technologies (e.g., PCIe, NVMe over PCIe, NVMe-oF, etc.) or be implemented by various technologies (e.g., PCIe, NVMe over PCIe, NVMe-oF, etc.).

[0075] The source storage device 250a may include a source storage device controller 210a and a source controller memory 220a. The source controller memory 220a may include or may be RAM (e.g., SRAM or DRAM). The source storage device controller 210a may include, or may be implemented as, embedded logic (e.g., software or firmware embedded in the source storage device 250a), the embedded logic being configured to process requests from a server (e.g., source server 100a) to copy / write data to the source storage device 200. The source storage device 250a may include the source storage device 200 or a portion thereof.

[0076] For example, the source storage device 200 may include source memory 230. The source memory 230 may include or may be long-term memory. For example, the source memory 230 may include non-volatile memory and / or a memory layer (including volatile memory and non-volatile memory). The source memory 230 may correspond to a long-term memory device of one or more source storage devices 250a. For example, the source storage device 200 may represent a long-term memory composed of one or more source storage devices 250a of a distributed storage system (such as a virtualized distributed storage system). The source storage device 250a may be a solid-state drive (SSD) including one or more underlying source storage devices 250a. The underlying source storage device may be virtual or physical. The source storage device 250a may be connected to the source storage device 200 via a storage interface 22. The storage interface 22 may include or may be a flash memory interface (e.g., a NAND flash memory interface, switch mode, or ONFI), or a remote interface that may be implemented via Ethernet.

[0077] The source host system 100S may include a source live migration server 100b. The source live migration server 100b may be a server assigned higher / additional permissions compared to the source server 100a or the target server 100c. The permissions may correspond to data access within the system 1. The source live migration server 100b may be assigned to manage live migration operations (e.g., live migration processing) for copying data from the source storage device 200 to the target storage device 300. The source live migration server 100b may include a source live migration server memory 120b.

[0078] The source live migration server memory 120b may include or may be RAM (e.g., SRAM or DRAM). The source live migration server memory 120b may be used to store metadata used to manage (e.g., monitor and / or direct) live migration operations. The source live migration server 100b may be connected to the source live migration storage device 250b via a communication link 20. The communication link 20 may include various technologies (e.g., PCIe, NVMe over PCIe, NVMe-oF, etc.) or may be implemented using various technologies (e.g., PCIe, NVMe over PCIe, NVMe-oF, etc.).

[0079] The source live migration storage device 250b may be a storage device assigned higher / additional permissions than the source storage device 250a or the target storage device 250c. The assigned permissions may correspond to access to data within the system 1. The source live migration storage device 250b may be assigned to facilitate live migration operations by reading data from the source storage device 200.

[0080] The source live migration storage device 250b may include a source live migration controller memory 220b and a source live migration controller 210b.

[0081] The source live migration controller memory 220b may include or may be RAM (eg, SRAM or DRAM).The source live migration controller memory 220b may be used to store metadata for managing (eg, monitoring and / or directing) live migration operations.

[0082] The source live migration controller 210b may include or be implemented as embedded logic to help the source live migration server 100b copy data from the source storage device 200 to the target storage device 300. The embedded logic may be part of the source live migration storage device 250b.

[0083] The source live migration storage device 250b may be connected to the source storage device 200 via the storage interface 22. The storage interface 22 may include or may be a flash memory interface (eg, a NAND flash memory interface) or a remote interface (eg, implemented by Ethernet).

[0084] The source live migration server 100b may be connected or linked to the target host system 100T via a “live migration server to target” link 10. The “live migration server to target” link 10 may be implemented by various techniques depending on whether the system 1 is implemented via hardware (HW), software (SW), or a combination of HW and SW.

[0085] In addition to the target server 100c discussed above, the target host system 100T may include a target live migration server 100d. Target live migration server 100d may be a server assigned higher / additional permissions compared to source server 100a or target server 100c. The permissions may correspond to data access within system 1. Target live migration server 100d may be assigned to assist source live migration server 100b in managing live migration operations. Target live migration server 100d may include target live migration server storage 120d.

[0086] The target live migration server memory 120d may include or may be RAM (e.g., SRAM or DRAM). The target live migration server memory 120d may be used to store metadata used to manage (e.g., monitor and / or direct) live migration operations. The target live migration server 100d may be connected to the target live migration storage device 250d via a communication link 20. The communication link 20 may include various technologies (e.g., PCIe, NVMe over PCIe, NVMe-oF, etc.) or may be implemented by various technologies (e.g., PCIe, NVMe over PCIe, NVMe-oF, etc.).

[0087] The target live migration storage device 250d may be a storage device assigned higher / additional permissions than the source storage device 250a or the target storage device 250c. The assigned permissions may correspond to access to data within the system 1. The target live migration storage device 250d may be assigned to facilitate live migration operations by writing data (e.g., data from the source storage device 200) to the target storage device 300.

[0088] The target live migration storage device 250d may include a target live migration controller memory 220d. The target live migration controller memory 220d may include or may be RAM (e.g., SRAM or DRAM). The target live migration controller memory 220d may be used to store metadata used to manage (e.g., monitor and / or direct) live migration operations.

[0089] The target live migration storage device 250d may further include a target live migration controller 210d. The target live migration controller 210d may include embedded logic or be implemented as embedded logic to assist the target live migration server 100d in writing data to the target storage device 300. The embedded logic may be part of the target live migration storage device 250d. The target live migration storage device 250d may be connected to the target storage device 300 via a storage interface 22. The storage interface 22 may include or may be a flash memory interface (e.g., a NAND flash memory interface) or a remote interface (e.g., implemented via Ethernet).

[0090] The servers and storage devices of system 1 may be implemented via hardware, software, or a combination of hardware and software. For example, in some embodiments, the "live migration server to target" link 10 may be a physical network connection (e.g., Transmission Control Protocol / Internet Protocol (TCP-IP), Ethernet, Remote Direct Memory Access (RDMA), InfiniBand, Fibre Channel, etc.). In some embodiments, one or more of the servers (e.g., source server 100a, source live migration server 100b, target live migration server 100d, or target server 100c) may be software entities. For example, one or more of the servers may be VMs managed by a hypervisor associated with one or more CPUs.

[0091] Likewise, one or more of the storage devices 250a, 250b, 250c, the source storage device 200, and the target storage device 300 may be virtualized and implemented by means of HW and / or SW technologies. For example, one or more of the storage devices 250a, 250b, 250c, the source storage device 200, and the target storage device 300 may be provided by any combination of physical storage devices. In one or more embodiments, and as Figure 1As shown by the dashed lines in , source storage device 250a, source live migration storage device 250b, and source storage device 200 may be components of a source SSD. Similarly, target storage device 250c, target live migration storage device 250d, and target storage device 300 may be components of a target SSD. In one or more other embodiments, the migration may be from a first virtual machine to a second virtual machine, noting that both virtual machines may be supported by a single physical storage device. It should be understood that various other combinations of physical and virtual devices may be implemented without departing from the spirit and scope of this disclosure. For example, in one or more embodiments, the migration may be a local migration involving a server system, and there may not be a "live migration server to target" link 10 between the two live migration servers. In one or more embodiments, the migration may involve a single live migration storage device. For example, one or more live migration storage controllers within the same SSD may manage the migration of data from source storage device 200 to target storage device 300 on the same SSD, utilizing potential new namespace and LBA changes. Alternatively, one or more live migration storage controllers within the same SSD may manage the migration through namespace sharing and NVMe access changes.

[0092] In some embodiments, source live migration server 100b may be linked (e.g., connected) to target storage device 250c via communication link 20. In some embodiments, source server 100a may be linked to source live migration server 100b via a "live migration server to source" link. For example, the "live migration server to source" link may be implemented using various techniques, which may be implemented depending on whether system 1 is implemented via HW, SW, or a combination of HW and SW.

[0093] The source live migration server 100b and the source live migration controller 210b can coordinate the management of the live migration process by transmitting metadata indicating the status of the data corresponding to the LBAs in the source storage device 200. For example, the source live migration server 100b can send metadata associated with the namespace to the source live migration controller 210b. The source live migration controller 210b can work to map the namespace to the relevant data (e.g., valid data) within the source storage device 200 and to be copied to the target storage device 300. The source live migration controller 210b can create a bitmap BM of the metadata when mapping the namespace to the source storage device.

[0094] For example, during the mapping phase of the live migration process, the bitmap BM may be referred to as a bitmap of mapped pages ("BMP"), with each LBA being referred to as a "page" or "NAND page." During the data copying phase of the live migration process, the bitmap BM may be referred to as a bitmap of dirty pages ("BDP"). During the mapping phase, the source live migration server 100b refers to the BMP to determine which storage locations of the source storage device 200 include data to be copied to the target storage device 300. When the source live migration server 100b copies the data from the source storage device 200 to the target storage device 300 according to the BMP, the source live migration storage device 250b may use the BDP to track user data writes UDW occurring at the source storage device 200. After the source live migration server 100b has completed the process of copying the data based on the BMP, the source live migration controller 210b may send information indicating the location of the dirty pages to the source live migration server 100b based on the BDP to copy the data from the source storage device 200 to the target storage device 300.

[0095] The BMP and BDP may include a bit that indicates whether a given LBA includes valid data or dirty data to be copied to the target storage device 300. Each LBA may include 4 kilobytes (KB) of data. Each LBA may be represented by one bit in the bitmap. For example, a bit "0" may indicate that a given LBA does not have valid data or dirty data to be copied to the target storage device 300 (e.g., see Figure 1 denoted by C in the reference numeral. In addition, a bit "1" may indicate that a given LBA has valid data or dirty data C to be copied to the target storage device 300. Thus, an entire drive having a large storage capacity (e.g., 8 terabytes (TB) of data in the source storage device 200) may be represented by way of a bitmap having, for example, 256 megabytes (MB) of metadata.

[0096] Still refer to Figure 1 , the bitmap BM corresponds to the BDP of the simple example. Figure 1 The bitmap BM in FIG2 is depicted as having tracked user data writes (UDWs) to a drive in a namespace having 256 LBAs. The bitmap BM has tracked user data writes UDWs to LBA128, LBA130, LBA131, LBA133, LBA134, LBA162, and LBA193. Therefore, the bitmap BM having a number of LBAs ranging from 0 to 255 indicates that LBA128, LBA130, LBA131, LBA133, LBA134, LBA162, and LBA193 have data C to be copied from the source storage device 200 to the target storage device 300 through the shaded boxes indicating bits “1” in the bitmap BM.

[0097] As discussed above, the source live migration storage device 250b may receive a request from the source live migration server 100b to send information to the source live migration server 100b. The information may indicate whether one or more storage locations (e.g., LBAs) within one or more large-granularity areas of the source storage device 200 include data C to be copied from the source storage device 200 to the target storage device 300. The source live migration storage device 250b may enter a recursive large-to-small granularity query mode. The source live migration storage device 250b may generate a result data structure RDS based on one or more query and response iterations of the recursive large-to-small granularity query mode.

[0098] The request from the source live migration server 100b may include a pointer P that informs the source live migration storage device 250b where to send the result data structure RDS in the source live migration server memory 120b. For example, the pointer P may correspond to a return buffer RB for storing the result data structure RDS.

[0099] In addition to improving bandwidth and latency, one or more aspects of one or more embodiments of the present disclosure can reduce the amount of memory consumed by storing the result data structure RDS. For example, all LBAs including data to be copied from a hypothetical source storage device 200 having a drive capacity of 1 petabyte (PB) can be indicated within a return buffer RB having a size of only 16 kilobytes (KB).

[0100] Figure 2 is a conceptual diagram depicting a method of delivering information for performing live migration with respect to a simple example according to one or more embodiments of the present disclosure.

[0101] Reference Figure 2 , provide the above (about Figure 1 ) to discuss further details. In this simple example, the source live migration server 100b and the source live migration storage device 250b (see Figure 1 ) may be limited to a bitmap size of 4 bits. Therefore, if the bitmap BM were to query four LBAs at a time from LBA0 to LBA255, a lot of time / resources / overhead would be wasted on the bitmap BM corresponding to the namespace having 256 LBAs and where the data C to be copied from the source storage device 200 to the target storage device 300 is only at LBA128, LBA130, LBA131, LBA133, LBA134, LBA162, and LBA193. Therefore, in order to provide a more efficient transfer of the mapping bitmap and the dirty bitmap that are tracked through the namespace corresponding to 256 LBAs, the source live migration storage device 250b may be configured to respond to a recursive large-to-small granularity query pattern.

[0102] The source live migration storage device 250b may receive a request for information indicating that one or more LBAs corresponding to the bitmap BM include data C to be copied from the source storage device 200 to the target storage device 300. The recursive large-to-small granularity query pattern may be associated with multiple assumptions. For example, as mentioned above, the bitmap size may be limited to 4 bits. In addition, the granularity setting may include a first granularity size G1 of 64LBA, a second granularity size G2 of 16LBA, a third granularity size G3 of 4LBA, and a fourth granularity size G4 of 2LBA. The recursive pattern may also be associated with a threshold TV (e.g., an exit threshold). For example, as will be discussed in further detail below, the threshold TV may be a threshold number TN of storage locations. The threshold TV may be a threshold number TNB of storage locations for each bitmap (e.g., a threshold number of storage locations corresponding to each bitmap). The threshold TV may be a threshold granularity TG, or may be a threshold of storage locations remaining in a given area and having a sufficiently small granularity.

[0103] Based on the aforementioned recursive large-to-small granularity query pattern assumption, in one or more embodiments, after receiving a request from the source live migration server 100b, the source live migration storage device 250b may proceed as follows. The source live migration storage device 250b may query the drive at LBA0 in a first iteration IT1 with a first granularity G1 of 64 LBAs. Therefore, the source live migration storage device 250b may divide the namespace into four regions R1, R2, R3, and R4. Each of the four regions may have a size of 64 LBAs. R1 may correspond to LBA0 to LBA63, R2 may correspond to LBA64 to LBA127, R3 may correspond to LBA128 to LBA191, and R4 may correspond to LBA192 to LBA255. Because regions R1 and R2 do not include any dirty pages, and because regions R3 and R4 include dirty pages, the source live migration storage device 250b may generate a result data structure bitmap RDSB including a bitmap BM1 including four bits (0, 0, 1, 1) indicating the presence of dirty LBAs in LBA ranges 128 to 191 and 192 to 255. In one or more embodiments, the source live migration storage device 250b may send the bitmap BM1 to the source live migration server 100b.

[0104] The source live migration storage device 250b may query the drive at LBA128 at a second granularity G2 of 16 LBAs in the second iteration IT2. Therefore, the source live migration storage device 250b may divide the region R3 into four regions R31, R32, R33, and R34. Each of the four regions may have a size of 16 LBA. R31 may correspond to LBA128 to LBA143, R32 may correspond to LBA144 to LBA159, R33 may correspond to LBA160 to LBA175, and R34 may correspond to LBA176 to LBA191. Figure 2 The single asterisk (*) depicted next to R33, R41, and TV in FIG is intended to link R33 and R41 to an example case discussed below, in which R33 and R41 meet (e.g., the number of storage locations with data C in R33 and R41 is equal to or less than) a threshold number of storage locations TN or a threshold number of storage locations per bitmap TNB). Because regions R31 and R33 include dirty pages, and because regions R32 and R34 do not include any dirty pages, the source live migration storage device 250b may generate a result data structure bitmap RDSB including a bitmap BM2 including four bits (1, 0, 1, 0) indicating the presence of dirty LBAs in LBA ranges 128 to 143 and 160 to 175. In one or more embodiments, the source live migration storage device 250b may send the bitmap BM2 to the source live migration server 100b.

[0105] The source live migration storage device 250b may also query the drive at LBA 192 in a second iteration IT2 at a second granularity G2 of 16 LBAs. Therefore, the source live migration storage device 250b may divide region R4 into four regions R41, R42, R43, and R44. Each of the four regions may have a size of 16 LBAs. R41 may correspond to LBAs 192 to 207, R42 may correspond to LBAs 208 to 223, R43 may correspond to LBAs 224 to 239, and R44 may correspond to LBAs 240 to 255. Because region R41 includes dirty pages, and because regions R42, R43, and R44 do not include any dirty pages, the source live migration storage device 250b may generate a result data structure bitmap RDSB including a bitmap BM3, which includes four bits (1, 0, 0, 0), indicating that dirty LBAs exist in the LBA range 192 to 207. In one or more embodiments, the source live migration storage device 250b may send the bitmap BM3 to the source live migration server 100b.

[0106] The source live migration storage device 250b may query the drive at LBA128 at a third granularity G3 of 4 LBA in the third iteration IT3. Therefore, the source live migration storage device 250b may divide the region R31 into four regions. The four regions may include: Figure 2 R311 shown in, and Figure 2 Unlabeled R312, R313 and R314. Figure 2 131 and 132. The double asterisks (**) depicted next to R311 and TV in FIG are intended to link R311 to the example case discussed below, in which R311 satisfies the threshold granularity TG. Each of the four regions may have a size of 4 LBAs. R311 may correspond to LBAs 128 to 131, R312 may correspond to LBAs 132 to 135, R313 may correspond to LBAs 136 to 139, and R314 may correspond to LBAs 140 to 143. Because regions R311 and R312 include dirty pages, and because regions R313 and R314 do not include any dirty pages, the source live migration storage device 250b may generate a result data structure bitmap RDSB including a bitmap BM4 including four bits (1, 1, 0, 0), indicating that dirty LBAs exist in LBA ranges 128 to 131 and 132 to 135. In one or more embodiments, the source live migration storage device 250b may send the bitmap BM4 to the source live migration server 100b.

[0107] In one or more embodiments, the source live migration storage device 250b may query the drive at LBA 128 and LBA 132 at a fourth granularity G4 of 2 LBAs in a fourth iteration IT4. Alternatively, in one or more other embodiments, the source live migration storage device 250b may determine that the recursive large-to-small granularity query mode is associated with a threshold TV, which is the size of the threshold granularity TG of 2 LBAs, meaning that 8 consecutive LBAs that may include dirty pages may remain. Therefore, in such embodiments, the source live migration storage device 250b may exit the recursive query mode and may return the LBA list or cause the source live migration server 100b to copy all 8 remaining LBAs (e.g., LBA ranges 128 to 135 corresponding to regions R311 and R312) (because the number of consecutive LBAs remaining in the relevant regions is small enough, it is assumed that all 8 LBAs include dirty pages). For example, any resulting copying of potentially clean LBAs (e.g., non-dirty LBAs) will not introduce significant delay.

[0108] The source live migration storage device 250b may query the drive at LBA160 at a third granularity G3 of 4 LBA in the third iteration IT3. Therefore, the source live migration storage device 250b may divide the region R33 into four regions. The four regions may include: Figure 2 R331 shown in, and Figure 2 Each of the four regions may have a size of 4 LBAs. R331 may correspond to LBAs 160 to 163. Region R331 includes dirty pages, and regions R332, R333, and R334 do not include any dirty pages.

[0109] In one or more embodiments, the source live migration storage device 250b may determine that the recursive large-to-small granularity query mode is associated with a threshold value TV, where the threshold value TV is: (i) a threshold number TN of storage locations of 1 LBA; or (ii) a threshold number TNB of storage locations per bitmap of 1 LBA. Therefore, in such embodiments, because region R331 includes only one dirty LBA, or because region R331 corresponds to bitmap BM5 that includes only one dirty LBA, the source live migration storage device 250b may exit the recursive query mode and may generate a result data structure list RDSL that includes an LBA list including LBA162, instead of sending bitmap BM5.

[0110] Similarly, in one or more embodiments, the source live migration storage device 250b may query the drive at LBA 192 at a third granularity G3 of 4 LBAs in a third iteration IT3 and may determine that the recursive large-to-small granularity query mode is associated with a threshold TV, where the threshold TV is: (i) a threshold number TN of storage locations of 1 LBA; or (ii) a threshold number TNB of storage locations per bitmap of 1 LBA. Therefore, because region R411 includes only one dirty LBA or because region R411 corresponds to bitmap BM6 that includes only one dirty LBA, the source live migration storage device 250b may exit the recursive query mode and may generate a result data structure list RDSL including an LBA list including LBA 193 instead of sending bitmap BM6.

[0111] It should be understood that Figure 2 The example is a simplified example, in part because in practice the resulting bitmaps can be kilobytes (KB) long. For example, each resulting bitmap can be 4KB in size instead of 4 bits.

[0112] Figure 3 is a diagram depicting the generation of a cumulative results data structure conveying information used to perform live migration, according to one or more embodiments of the present disclosure.

[0113] Reference Figure 3 In one or more embodiments, the source live migration server 100b (see Figure 1) and the source live migration storage device 250b may be configured to reference one or more parameters of an agreed-upon recursive large-to-small granularity query pattern. In one or more embodiments, the parameters may be stored on the source live migration storage device 250b and / or stored on the source live migration server 100b. The parameters may include agreed-upon transfer settings and granularity settings. For example, the source live migration storage device 250b may be configured to send the available granularity settings to the source live migration server 100b. Optionally, the source live migration storage device 250b may be configured to receive granularity settings from the source live migration server 100b. Therefore, in one or more embodiments, the recursive large-to-small granularity query pattern may be executed by the source live migration storage device 250b based on receiving only one management command from the source live migration server 100b. For example, the source live migration storage device 250b may perform the above-mentioned instructions in a more programmatic (or automated) manner than processing and responding to one query from the source live migration server 100b at a time. Figure 2 Simple example for discussion. For example, the source live migration storage device 250b may automatically perform each iteration of the recursive query mode based on a management command from the source live migration server 100b.

[0114] In one or more embodiments, the source live migration storage device 250b may perform a recursive large-to-small granularity query pattern and may generate or create a result data structure RDS, which includes a result data structure bitmap RDSB (e.g., a result bitmap composed of one or more bitmaps) and / or includes a result data structure list RDSL (e.g., an LBA list including one or more LBAs). The result data structure RDS may include all of the same information conveyed by processing and responding to a query from the source live migration server 100b at one time, but includes such information in a cumulative format. For example, the cumulative result data structure RDS may include two different pieces of information (e.g., data structures) configured to be sent from the source live migration storage device 250b to the source live migration server 100b: (i) a cumulative result data structure bitmap RDSB; and (ii) a cumulative result data structure list RDSL. For example, as described above with respect to Figure 2 As discussed, the source live migration storage device 250b may generate four result data structure bitmaps RDSB (e.g., BM1, BM2, BM3, and BM4). However, instead of sending each individual bitmap individually to the source live migration server 100b, the source live migration storage device 250b may store each result data structure bitmap RDSB at the source live migration storage device 250b and wait until the recursive large-to-small granularity query mode is exited before sending the accumulated result data structure RDS.

[0115] Figure 4is a diagram depicting a self-describing return structure for conveying information for performing live migration, according to one or more embodiments of the present disclosure.

[0116] Reference Figure 4 , the source live migration storage device 250b (see Figure 1 ) can generate a self-describing return data structure 400 to store the accumulated result data structure RDS (see Figure 3 ) is passed to the source live migration server 100b. For example, the source live migration storage device 250b may construct a header of the self-describing return data structure 400 to pass the accumulated result data structure RDS to the source live migration server 100b. In one or more embodiments, the self-describing return data structure 400 may include a global header 402. The self-describing return data structure 400 may also include one or more bitmap headers 422 and / or LBA list headers 432. For example, the self-describing return data structure 400 may include at least one of the bitmap header 422 and the LBA list header 432. Each bitmap header 422 may correspond to corresponding bitmap data 424. The LBA list header 432 may correspond to the LBA list 434. The global header 402 may indicate what information will be provided next in the self-describing return data structure 400. The bitmap header 422 may indicate the size of their respective bitmaps.

[0117] In one or more embodiments, the global header 402 may include one or more of the following entries: a source controller identifier (ID); a source namespace ID; mapped return data versus dirty return data; a timestamp start (e.g., when dirty page tracking begins); a timestamp stop (e.g., when dirty page tracking ends); the number of bitmaps (e.g., indicating that two bitmap headers are included in the self-describing return data structure 400); the size of all bitmaps (if all bitmaps are the same size); the presence of an LBA list (e.g., indicating whether an LBA list header is included in the self-describing return data structure 400); and more data pending (part 1 of 2). The "more data pending" entry may indicate whether the self-describing return data structure 400 is overflowed due to a portion of the information exceeding the return buffer RB on the source live migration server 100b (see Figure 1 ) size (eg, cannot fit in the return buffer RB on the source live migration server 100b (see Figure 1 )). For example, the self-describing return data structure 400 may be copied to a return buffer RB allocated by the source live migration server 100b for copying the result data structure RDS. The return buffer RB may correspond to a pointer P, which may have been included in the request from the source live migration server 100b. In one or more embodiments, the "timestamp start" and "timestamp stop" entries may be replaced by sequence number and time increment entries.

[0118] In one or more embodiments, the bitmap header 422 may include one or more of the following entries: the granularity of the bitmap; Starting_LBA (e.g., for a particular bitmap corresponding to a particular bitmap header 422); Ending_LBA (e.g., for a particular bitmap) and / or the size of the bitmap (e.g., used to describe a particular bitmap if not all bitmaps are the same size); and bitmap data (e.g., 0s and 1s indicating whether a given LBA includes data to be copied from the source storage device 200 to the target storage device 300).

[0119] In one or more embodiments, LBA list header 432 may include one or more of a number of entries and a list of LBAs.

[0120] Figure 5 is a flow diagram depicting example operations of a method of communicating information for performing live migration in accordance with some embodiments of the present disclosure.

[0121] Reference Figure 5 , the method 5000 may include one or more of the following operations. The source live migration storage device 250b (see Figure 1 ) may receive a request for “information indicating that one or more LBAs of the source storage device 200 include data C to be copied from the source storage device 200 to the target storage device 300” from the source live migration server 100b (operation 5001). The source live migration storage device 250b may enter a recursive query mode (e.g., a large-to-small granularity query mode) (operation 5002). The source live migration storage device 250b may determine that a first region of the source storage device 200 includes an LBA containing data C to be copied from the source storage device 200 to the target storage device 300, and that the first region has a size equal to a first granularity size (operation 5003). In response to the determination of operation 5003, the source live migration storage device 250b may determine that a second region within the first region includes the LBA (the second region may have a size equal to a second granularity size, the second granularity size being smaller (or less) than the first granularity size and being larger (or more) than a third granularity size, the third granularity size corresponding to a third region within the second region and including the LBA) (operation 5004). The source live migration storage device 250b may exit the query mode based on determining that the second region is associated with the threshold TV (e.g., determining that the second granularity size is equal to or less than the threshold TV) (operation 5005). In response to the second region being associated with the threshold TV, the source live migration storage device 250b may send a result data structure RDS to the source live migration server 100b, the result data structure RDS including an aggregated result with a header and indicating that the second region includes a storage location corresponding to the LBA (operation 5006).

[0122] As used herein, when a "data structure" is described as "indicating" the location of data, it may mean that elements of the data structure (e.g., bits of a bitmap if the data structure is or includes a bitmap) point to or indicate the location of the data.

[0123] Thus, embodiments of the present disclosure provide improvements and advantages in using recursive large-to-small granularity queries to pass tracking metadata (e.g., metadata for tracking valid or dirty pages) from a source live migration controller to a source live migration server. By allowing automation of recursive query mode and / or early exit from recursive query mode, bandwidth can be saved because fewer commands are sent from the source live migration server to the source live migration controller. In addition, latency can be reduced by reducing the time it takes for the source live migration controller to create a bitmap and by reducing the time it takes for the source live migration server to parse each bitmap. Furthermore, tracking information about a large namespace can be represented by a relatively small amount of metadata.

[0124] The disclosed example embodiments extend to the following statements, but are not limited thereto:

[0125] Statement 1. An example method includes: determining, by a storage device, that a first area of ​​a source storage device includes a storage location, the storage location including data to be copied from the source storage device to a target storage device, the first area having a size equal to a first granularity size; determining, by the storage device, that a second area within the first area includes the storage location, the second area having a size equal to a second granularity size, the second granularity size being smaller than the first granularity size; determining that the second area is associated with a threshold; exiting a query mode including the first granularity size and the second granularity size; and sending a result data structure indicating that the second area includes the storage location.

[0126] Statement 2. The example method includes the method described in Statement 1, and further includes: receiving a request at a storage device for "information indicating that one or more storage locations include data to be copied from a source storage device to a target storage device", the request being associated with a first granularity size, wherein the resulting data structure includes: a bitmap indicating that the first area or the second area includes one or more storage locations, the one or more storage locations including data to be copied from the source storage device to the target storage device; or a list of one or more logical block addresses (LBAs), the one or more logical block addresses (LBAs) including the data to be copied.

[0127] Statement 3. The example method includes the method described in Statement 2, and further includes: determining that the number of storage locations within the first area or the second area and including data to be copied from the source storage device to the target storage device is equal to or less than a threshold number of storage locations as the threshold, wherein the resulting data structure includes a list of the one or more LBAs.

[0128] Statement 4. The example method includes the method described in statement 2, and further includes: determining that the number of storage locations corresponding to the second bitmap and including data to be copied from the source storage device to the target storage device is equal to or less than a threshold number of storage locations for each bitmap as the threshold, wherein the first area is associated with the first bitmap, wherein the second area is associated with the second bitmap, wherein the second bitmap indicates that one or more storage locations within the second area include data to be copied from the source storage device to the target storage device, and wherein the resulting data structure includes a list of the one or more LBAs.

[0129] Statement 5. The example method comprises the method of statement 2, and further comprising: determining that the second granule size is equal to or less than the threshold granule size as the threshold, wherein the resultant data structure comprises the list of the one or more LBAs.

[0130] Statement 6. The example method comprises the method of statement 2, and further comprising: copying data corresponding to all storage locations within the second region, wherein the threshold may include a threshold granularity size or a number of contiguous LBAs remaining.

[0131] Statement 7. The example method includes the method described in Statement 2, and further includes: determining a first granularity size or a second granularity size based on information stored on a storage device, wherein the request for information is directed to a namespace including the one or more storage locations, and wherein the first region and the second region correspond to the namespace.

[0132] Statement 8. An example method includes the method of any one of Statements 1 to 7, wherein the query mode includes a recursive query mode, the second granularity size is greater than a third granularity size corresponding to the third region, the third region is within the second region and includes the storage location, the first region is associated with a first data structure, the first data structure indicates that one or more storage locations within the first region include data to be copied from a source storage device to a target storage device, the second region is associated with a second data structure, the second data structure indicates that one or more storage locations within the second region include data to be copied from a source storage device to a target storage device, and the result data structure includes a self-describing return data structure, the self-describing return data structure includes the first data structure, the second data structure, the global header, and the bitmap header or the LBA list header.

[0133] Statement 9. An example method includes: receiving an initial request at a storage device for "information indicating that one or more storage locations include data to be copied from a source storage device to a target storage device", the initial request being associated with a first granularity size; determining by the storage device that a first area of ​​the source storage device includes storage locations, the storage locations including data to be copied from the source storage device to the target storage device, the first area having a size equal to the first granularity size; and automatically determining, based on the initial request, that a second area within the first area includes the storage locations, the second area having a size equal to a second granularity size, the second granularity size being smaller than the first granularity size.

[0134] Statement 10. Example method comprising the method of statement 9, and further comprising: determining that the second region is associated with the threshold; and transmitting, by the storage device, the resulting data structure.

[0135] Statement 11. Example method comprising the method of statement 10, and further comprising: determining that the second region is associated with the threshold; and exiting the recursive query mode including the first granularity size and the second granularity size.

[0136] Statement 12. An example method includes the method of any one of Statements 10 and 11, wherein the first region is associated with a first data structure indicating that one or more storage locations within the first region include data to be copied from a source storage device to a target storage device, the second region is associated with a second data structure indicating that one or more storage locations within the second region include data to be copied from a source storage device to a target storage device, and the result data structure includes a self-describing return data structure including at least one of a bitmap header and an LBA list header, the first data structure, the second data structure, and a global header.

[0137] Statement 13. An example apparatus for performing the method of any one of statements 1 to 12 comprises: a controller and a memory.

[0138] While embodiments of the present disclosure have been particularly shown and described with reference to the embodiments described herein, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the disclosure as set forth in the appended claims and their equivalents.

Claims

1. A method for transmitting information, the method comprising: determining, by the storage device, that a first region of the source storage device includes storage locations, the storage locations including data to be copied from the source storage device to the target storage device, the first region having a size equal to a first granularity size; determining, by the storage device, a second region within the first region that includes the storage location, the second region having a size equal to a second granularity size, the second granularity size being smaller than the first granularity size; determining that the second region is associated with a threshold; exiting a query mode including the first granularity size and the second granularity size; as well as A resulting data structure is sent indicating that the second region includes the memory location.

2. The method according to claim 1, further comprising: receiving, at a storage device, a request for information indicating that one or more storage locations include data to be copied from a source storage device to a target storage device, the request being associated with a first granularity size, The result data structure includes: a bitmap indicating that the first region or the second region includes one or more storage locations including data to be copied from a source storage device to a target storage device; or A list of one or more logical block addresses including the data to be copied.

3. The method according to claim 2, further comprising: determining that the number of storage locations within the first area or the second area and including data to be copied from the source storage device to the target storage device is equal to or less than a threshold number of storage locations as the threshold, The result data structure includes a list of the one or more logical block addresses.

4. The method according to claim 2, further comprising: determining that the number of storage locations corresponding to the second bitmap and including data to be copied from the source storage device to the target storage device is equal to or less than a threshold number of storage locations of each bitmap as the threshold, The first region is associated with the first bitmap. wherein the second region is associated with the second bitmap, wherein the second bitmap indicates that one or more storage locations within the second region include data to be copied from the source storage device to the target storage device, and The result data structure includes a list of the one or more logical block addresses.

5. The method according to claim 2, further comprising: determining that the second granule size is equal to or smaller than a threshold granule size serving as the threshold, The result data structure includes a list of the one or more logical block addresses.

6. The method according to claim 2, further comprising: copying the data corresponding to all storage locations within the second area, The threshold value includes a threshold granularity size or the number of remaining continuous logical block addresses.

7. The method according to claim 2, further comprising: determining a first granule size or a second granule size based on information stored on a storage device, wherein the request for information is directed to a namespace comprising the one or more storage locations, and The first area and the second area correspond to namespaces.

8. The method according to claim 2, wherein: Query modes include recursive query mode; the second granularity size being greater than a third granularity size corresponding to a third region that is within the second region and includes the storage location; The first region is associated with a first data structure, the first data structure indicating that one or more storage locations within the first region include data to be copied from a source storage device to a target storage device; the second region being associated with a second data structure indicating that one or more storage locations within the second region include data to be copied from the source storage device to the target storage device; and The result data structure includes a self-describing return data structure, which includes: The first data structure, The second data structure, global headers, and Bitmap header or logical block address list header.

9. A storage device for transmitting information, the storage device being configured to: determining that a first region of the source storage device includes storage locations, the storage locations including data to be copied from the source storage device to the target storage device, the first region having a size equal to a first granularity size; determining a second region within the first region that includes the storage location, the second region having a size equal to a second granularity size, the second granularity size being smaller than the first granularity size; determining that the second region is associated with a threshold; exiting a query mode including the first granularity and the second granularity; and A resulting data structure is sent indicating that the second region includes the memory location.

10. The storage apparatus of claim 9, configured to: receive a request for information indicating that one or more storage locations include data to be copied from a source storage device to a target storage device, the request being associated with a first granularity size; in, The resulting data structure includes: a bitmap indicating that the first region or the second region includes one or more storage locations including data to be copied from a source storage device to a target storage device; or A list of one or more logical block addresses including the data to be copied.

11. The storage apparatus according to claim 10 , further comprising: determining that the number of storage locations within the first area or the second area and including data to be copied from the source storage device to the target storage device is equal to or smaller than a threshold number of storage locations as the threshold; in, The resulting data structure includes a list of the one or more logical block addresses.

12. The storage apparatus according to claim 10 , configured to: determine that the number of storage locations corresponding to the second bitmap and including data to be copied from the source storage device to the target storage device is equal to or less than a threshold number of storage locations of each bitmap as the threshold, in, The first region is associated with the first bitmap, wherein the second region is associated with the second bitmap, wherein the second bitmap indicates that one or more storage locations within the second region include data to be copied from the source storage device to the target storage device, and The result data structure includes a list of the one or more logical block addresses.

13. The storage device according to claim 10, configured to: determine that the second granularity is equal to or smaller than a threshold granularity as the threshold, in, The resulting data structure includes a list of the one or more logical block addresses.

14. The storage device according to claim 10, further comprising: copying the data corresponding to all storage locations within the second area, The threshold is a threshold granularity size or the number of remaining continuous logical block addresses.

15. The storage device according to claim 10, configured to: determine the first granularity or the second granularity based on information stored on the storage device; in, The request for information is directed to a namespace comprising the one or more storage locations, and The first area and the second area correspond to namespaces.

16. The storage device according to claim 10, wherein: Query modes include recursive query mode; the second granularity size being greater than a third granularity size corresponding to a third region that is within the second region and includes the storage location; The first region is associated with a first data structure, the first data structure indicating that one or more storage locations within the first region include data to be copied from a source storage device to a target storage device; the second region being associated with a second data structure indicating that one or more storage locations within the second region include data to be copied from the source storage device to the target storage device; and The result data structure includes a self-describing return data structure, which includes: The first data structure, The second data structure, global headers, and Bitmap header or logical block address list header.

17. A method for transmitting information, the method comprising: receiving, at a storage device, an initial request for information indicating that one or more storage locations include data to be copied from a source storage device to a target storage device, the initial request being associated with a first granularity size; determining, by the storage device, that a first region of the source storage device includes storage locations, the storage locations including data to be copied from the source storage device to the target storage device, the first region having a size equal to a first granularity size; as well as Based on the initial request, a second region within the first region is automatically determined to include the storage location, the second region having a size equal to a second granularity size, the second granularity size being smaller than the first granularity size.

18. The method according to claim 17, further comprising: determining that the second region is associated with the threshold, and The resulting data structure is sent by the storage device.

19. The method according to claim 18, further comprising: determining that the second region is associated with the threshold, and Exiting a recursive query mode including the first granule size and the second granule size.

20. The method according to claim 19, wherein The first region is associated with a first data structure, the first data structure indicating that one or more storage locations within the first region include data to be copied from a source storage device to a target storage device; the second region being associated with a second data structure indicating that one or more storage locations within the second region include data to be copied from the source storage device to the target storage device; and The result data structure includes a self-describing return data structure, which includes: The first data structure, The second data structure, global headers, and Bitmap header or logical block address list header.

Images