Fully allocated volume to de-revolume migration in storage system

Abstract

A method, computer program product and computer system for fully allocated volume to de-revolume migration in a storage system. The method includes moving a physical allocation of stored data associated with a fully allocated volume into a virtual address range in a deduplication domain and establishing deduplication metadata as a transfer. The method then performs a background deduplication process on the virtual address range after the virtual address range is populated by the physical allocation, where the hash interface is queried using the driver to perform hash calculations on the physical driver storing the data.

Description

Background Art

[0001] The present invention relates to storage systems, and more particularly, to migration of fully allocated volumes to deduplicated volumes.

[0002] A common feature found in storage controllers and devices today is data deduplication. There is an ever-increasing demand to store more user data in the same physical capacity, which reduces the overall cost of owning a storage system. Data deduplication works by identifying repeated data patterns, and it will create a reference to the duplicate data stored elsewhere in the system instead of storing the user data. The existing duplicate content can be within the same volume, within another volume in the same pool or another pool within the storage system, or within a volume used by another host.

[0003] A fully allocated volume has all of its capacity pre-allocated and the storage system has minimal interaction with the data outside of caching and RAID operations. Customers can have fully allocated and deduplicated volumes in the same storage pool.

[0004] It is common for customers to have a mix of both deduplicated and fully allocated volumes in a pool. Deduplication is a newer technology, so a migration path from fully allocated to deduplication is both valuable and necessary.

[0005] Existing techniques involve making multiple copies of the data in order to perform the migration, or reading all the data and writing it to a new location within the pool. Summary of the invention

[0006] According to an embodiment of the present invention, a computer-implemented method for migrating from a fully allocated volume to a deduplicated volume in a storage system is provided. The method is performed by one or more processors of a computer system and includes moving a physical allocation of stored data associated with the fully allocated volume to a virtual address range in a deduplication domain, and establishing deduplication metadata as a passthrough. When the virtual address range is filled with physical allocations, a background deduplication process is performed on the virtual address range, wherein a drive query hash interface is used to perform a hash calculation on a physical drive storing the data. The virtual address range becomes a deduplicated volume of data.

[0007] This provides an improvement in using idle computing power within the drive layer to calculate the deduplication hash. This allows volume migration to have low computing requirements because no data needs to be read outside the physical drive.

[0008] The method may include avoiding moving data stored on a physical drive during the method. This provides an improvement in that data does not need to be written to a new location during volume migration.

[0009] The method may include suspending input / output operations while moving a physical allocation of stored data associated with a fully allocated volume into a virtual address range in a deduplication domain. This provides an improvement in avoiding any conflicts with underlying data during the movement of the physical allocation.

[0010] The method may include retaining, by the deduplication layer, in the deduplication domain, the virtual address range to which the physical allocation associated with the fully allocated volume is moved.

[0011] Establishing the deduplication metadata as a pass-through may include creating a pass-through forward lookup structure for absorbing fully allocated volumes into the deduplicated volume, and creating a stub backward lookup structure that is updated with the deduplication metadata as the deduplicated volume is deduplicated.

[0012] Moving the physical allocation of stored data associated with the fully allocated volume into a virtual address range in the deduplication domain may be performed by the virtualization layer and may include marking the volume as a deduplication volume and activating the deduplication layer and transferring metadata.

[0013] Performing the background deduplication process may include iterating, by the deduplication layer, over grains of the deduplicated volume, wherein the grains are locked to input / output operations during grain deduplication.

[0014] Iterating over the granules may include: when a granule is a deduplication hit, creating referrer metadata on a forward lookup structure to point to the source metadata, and marking the granule as invalid. When a granule is a deduplication miss, a hash of the granule may be stored in a deduplication fingerprint database, thereby creating source metadata on a forward lookup structure to point to the granule on the data disk. For all granules, a backward lookup structure may be updated with an invalidation or virtual address.

[0015] The method may include performing a hash calculation at a flash core module (FCM) on a driver layer without reading data from outside the FCM.

[0016] According to another embodiment of the present invention, a system for migrating fully allocated volumes to deduplicated volumes in a storage system is provided, including a processor and a memory, the memory being configured to provide computer program instructions to the processor to perform the functions of the component. The system includes a physical allocation moving component in a virtualization layer, for moving the physical allocation of stored data associated with the fully allocated volume to a virtual address range in a deduplication domain. A deduplication metadata establishment component for establishing deduplication metadata as a transfer is provided. A deduplication component of the deduplication layer is provided, for performing a background deduplication process on a virtual address range after the virtual address range is filled with physical allocations, wherein a drive query hash interface is used to perform hash calculations on a physical drive storing data. The virtual address range becomes a deduplicated volume of data.

[0017] This provides an improved system that uses idle computing power within the driver layer to calculate deduplication hashes. This allows volume migration to have low computing requirements.

[0018] The physical allocation of stored data may be a managed disk of data extents associated with physical data extents stored across multiple physical drives having additional computing power at which hash calculations for the deduplication process are performed. The multiple physical drives may be flash core modules (FCMs) having hash calculation capabilities.

[0019] The system may include a pause component for pausing input / output operations while moving a physical allocation of stored data associated with a fully allocated volume into a virtual address range in a deduplication domain.

[0020] The system may include a reservation component in the deduplication layer to reserve in the deduplication domain a virtual address range to which a physical allocation associated with the fully allocated volume is moved.

[0021] The deduplication metadata establishment component may include a forward lookup component for creating a pass-through forward lookup structure for absorbing a fully allocated volume into a deduplicated volume. A backward lookup component may be included for creating a stub backward lookup structure to be updated with the deduplication metadata when the deduplicated volume is deduplicated.

[0022] Physically allocating the move component may include marking the volume as a deduplicated volume and activating the deduplication layer and transferring metadata.

[0023] The deduplication component may include a granule iteration component for iterating, by the deduplication layer, over granules of the deduplication volume and locking granules to input / output operations during granule deduplication.

[0024] The granule iteration component may include a hit component for creating a referrer metadata on a forward lookup structure to point to the source metadata when a granule is a deduplication hit, and marking the granule as invalid. A miss component may be included for storing a hash of the granule in a deduplication fingerprint database when a granule is a deduplication miss, creating source metadata on a forward lookup structure to point to the granule on a data disk. A backward lookup update component may be included for updating a backward lookup structure with an invalidation or virtual address.

[0025] According to another embodiment of the present invention, a computer program product for fully allocated volume to deduplication volume migration in a storage system is provided. The computer program product includes a computer-readable storage medium having program instructions. The program instructions are executable by a processor to cause the processor to move the physical allocation of stored data associated with the fully allocated volume to a virtual address range in a deduplication domain, and establish deduplication metadata as a transfer. When the virtual address range is filled with physical allocations, a background deduplication process is performed on the virtual address range, wherein a drive query hash interface is used to perform a hash calculation on a physical drive storing the data. The virtual address range becomes a deduplicated volume of data.

[0026] The computer-readable storage medium may be a non-transitory computer-readable storage medium, and the computer-readable program code may be executed by a processing circuit. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Exemplary embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings:

[0028] Figure 1 is a flow chart illustrating a method according to an exemplary embodiment of the present invention;

[0029] Figure 2 is a block diagram illustrating a storage system according to an exemplary embodiment of the present invention;

[0030] Figure 3A and Figure 3B is a schematic diagram illustrating an exemplary embodiment of a method according to an exemplary embodiment of the present invention;

[0031] Figure 4A and Figure 4B is a flow chart illustrating a method according to an exemplary embodiment of the present invention;

[0032] Figure 5 is a block diagram illustrating a system according to an exemplary embodiment of the present invention;

[0033] Figure 6 is a block diagram illustrating a computer system or cloud server in which exemplary embodiments of the present invention may be implemented;

[0034] Figure 7 is a schematic diagram illustrating a cloud computing environment in which exemplary embodiments of the present invention may be implemented; and

[0035] Figure 8 is a diagram illustrating abstract model layers of a cloud computing environment in which exemplary embodiments of the present invention may be implemented.

[0036] It should be understood that for simplicity and clarity of illustration, the elements shown in the figures are not necessarily drawn to scale. For example, for clarity, the size of some elements may be enlarged relative to other elements. In addition, where deemed appropriate, reference numerals may be repeated in the drawings to indicate corresponding or similar features. DETAILED DESCRIPTION

[0037] Exemplary embodiments of the present invention are provided, including methods, systems, and computer program products for volume migration from fully allocated provisioning to deduplicated volumes within a storage pool. However, it should be understood that the scope of the inventive concept is not limited in this regard. The disclosed exemplary embodiments are merely illustrative of the claimed systems, methods, and computer program products. The inventive concept may be implemented in many different forms and should not be construed as being limited to the exemplary embodiments set forth herein. Rather, these included exemplary embodiments are provided for completeness of disclosure and to facilitate understanding by those skilled in the art. In the detailed description, discussion of well-known features and techniques may be omitted to avoid unnecessarily obscuring the exemplary embodiments presented.

[0038] References in the specification to "one embodiment," "an embodiment," "an exemplary embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment must include the feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it is considered to be within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in conjunction with other embodiments, whether or not explicitly described.

[0039] In order not to obscure the presentation of the exemplary embodiments of the inventive concept, in the following detailed description, some processing steps or operations known in the art may be combined for presentation and illustration purposes, and in some instances, may not be described in detail. In addition, some processing steps or operations known in the art may not be described at all. The following detailed description focuses on the distinguishing features or elements of the inventive concept according to various exemplary embodiments.

[0040] A deduplication migration coordinator component may be provided that is responsible for intra-pool volume migration from fully allocated volumes to deduplication volumes. The migration is optimized and uses spare computing power within the driver layer to compute deduplication hashes for logical address ranges on the driver layer itself. Computing the hash calculations for deduplication migrations on the driver layer allows the migration to have low system compute requirements and eliminates the need to read data on disk from outside the driver itself.

[0041] The described migration method eliminates the requirement to write data to a new location during migration. No additional capacity or "swing space" is required for the migration process, and existing fully allocated capacity is quickly assimilated into the deduplication domain, which simplifies storage management during the migration process.

[0042] The move to deduplicated volumes is an improvement in the field of computer storage system technology and provides a more efficient storage solution.

[0043] Physical storage drivers increasingly include additional computing power to perform advanced functions at the driver level. For example, a flash core module (FCM) has the ability to encrypt or compress data. FCM drivers typically do not utilize their full computing power to provide their daily functions. Among other things, FCMs utilize field programmable gate arrays (FPGAs), which are extremely flexible computing units that can be utilized to efficiently perform hash calculations (e.g., SHA-1 calculations). The described methods and systems provide for utilizing this spare computing power in a storage environment.

[0044] Deduplication is a technique that determines if a particle of data already exists on the storage controller, and instead of writing the data out again, updates the metadata structure to point to the owner of the existing storage segment and increments the reference count for that entry. References are typically implemented by taking the incoming IO, hashing it with an algorithm such as SHA-1, and comparing the hashed fragment to a database of existing IO. In the case of a deduplication hit, the storage has effectively written a small amount of metadata instead of the data itself, thus providing capacity savings to the user.

[0045] It is common for storage users to have a mix of deduplicated volumes and fully allocated volumes in a pool. Deduplication is a newer technology, and therefore a migration path from fully allocated to deduplication is both valuable and necessary.

[0046] refer to Figure 1 , flow chart 100 illustrates an example embodiment of the method for migrating a fully allocated volume to a deduplicated volume in a storage pool.

[0047] An initial phase may be performed by the deduplication layer to reserve a data disk virtual address range for the allocated physical extents currently associated with the fully allocated volume to be migrated (step 101). The reserved data disk virtual address range may be an address range of a virtual area of ​​a data disk that is not currently associated with any physical disk extents and / or is not currently in use at this stage. The migration process may move the allocated physical disk extents associated with the fully allocated volume to this virtual area. Since the reserved virtual area is in the deduplication domain, this will become a deduplication volume.

[0048] The initial phase of the method may establish a pass-through forward lookup structure to establish deduplication metadata as a pass-through for absorbing fully allocated volumes (step 102). A desired property of the forward lookup structure is that if a read is attempted on the structure and no entry is found, the original fully allocated volume should be read. This is desired for a "pass-through" with an empty structure that results in a read for the fully allocated volume. The pass-through forward lookup structure may map each virtual address in the fully allocated volume to a virtual address in a reserved virtual address range in the deduplication domain. A stub backward lookup structure may also be created for updating deduplication metadata during the migration process (e.g., "on the fly").

[0049] The physical allocation of the stored data associated with the fully allocated volume can be moved to the reserved data disk virtual address range in the deduplication domain while suspending IO to the volume (step 103). This can be performed in the virtualization layer, and IO to the volume can be suspended while moving the physical allocation.

[0050] Once the physically allocated extents of the volume are mapped to the reserved virtual area in the deduplication domain, the ownership of the data can be passed to the deduplication control, where the data disk is the owner of the data.

[0051] As part of the move, the method may mark the volume as a deduplicated volume, with the deduplication layer and pass-forward lookup structure activated for new IO operations, step 104. IO may be suspended so that there are no conflicts with the underlying data during the move of the physical allocation and while the deduplication metadata is being established.

[0052] The method may perform a background deduplication process on the newly populated virtual address range granules (step 105 ).

[0053] The deduplication process may use the drive query hash interface to request hash calculations to be performed by the physical drives where the data is stored (step 106). Background deduplication may iterate down to the driver layer at the granularity of the volume query for hash calculations.

[0054] The method may update the backward lookup structure (eg, on the fly) with the invalidation or virtual address and provide deduplication metadata (step 107).

[0055] In this manner, the method may use a combination of data disk virtual reservation, virtual map manipulation, and / or driver layer hash calculation offload.Initial absorption of a fully allocated volume may be via a pass-through forward lookup structure, while performing runtime backward lookup structure updates.

[0056] This approach can allow the allocation of physical storage to be migrated from a fully allocated volume to a deduplicated volume without requiring the underlying data to be read or written to a new location on the physical storage in the pool. Data that is still needed stays on the physical drives where they are already.

[0057] refer to Figure 2 , schematic block diagram 200 shows an example embodiment of a storage system 210 in which the described methods and systems may be implemented. To illustrate the described methods, storage system 210 shows a virtualization layer 220, a deduplication layer 250, and a driver layer 280. The arrangement of the components of the system is illustrative and should not be considered as limiting of possible embodiments.

[0058] Storage system 210 may provide storage to host applications at one or more host servers having a storage interface through which input / output (IO) operations may be processed to write data to and read data from storage system 210 .

[0059] The storage system 210 may include a virtualized storage controller that provides a virtualization layer 220. The virtualization layer 220 may receive host IO operations for logical volumes of data disks having logical addresses of data blocks from a host application. The virtualization layer 220 may present the logical addresses to the host and maintain logical address metadata 221 that provides a logical to physical address mapping that provides virtualization of the storage system 210. The logical addresses may be provided as volumes presented to attached hosts.

[0060] The described method may include a deduplication migration coordination component 230 at the virtualization layer 220 moving a managed disk physical allocation 241 of backend data extents associated with a fully allocated volume 222 to a reserved data disk virtual address range of a new deduplication volume 224 .

[0061] The virtualization layer 220 may maintain a log structure array (LSA) structure that is used to describe the logical to physical layout of block devices in the storage system. LSA structures may be used in storage systems because they provide a simple way to implement many different data reduction techniques and are agnostic to the type of storage backend. LSA storage systems may reference physical addresses at the storage backend using logical block addressing (LBAs) in the virtual domain. The host application may only need to provide the LBAs without knowing anything about the physical backend.

[0062] The storage system 210 may include a backend storage controller 240. The backend storage controller 240 may include another deduplication migration coordination component 260 that provides a deduplication layer 250 for deduplication of data stored in the physical drive layer 280. Deduplication can be used to eliminate duplication from storage chunks of data by using deduplication metadata 251 to point to the storage chunks. The deduplication process can begin by applying a hash function to create a unique digital fingerprint or signature for a chunk of given data. The fingerprint value can be saved in an indexed fingerprint database 252 and compared to a fingerprint value created for a new incoming data chunk. By comparing the fingerprint values, it can be determined whether the data chunk is unique or whether it is a duplicate of a data chunk that has already been stored.

[0063] Deduplication metadata 251 may provide a mapping to addresses of stored data blocks. Deduplication metadata 251 may include a forward lookup structure describing the virtual to physical mapping, such as using a B-tree. A source chunk may be a chunk of data storing an original copy of the data, and a reference chunk may be a symbolic link to the data of the source chunk. A source chunk may contain a count of the number of referees that reference back to the source. A source chunk may know how many chunks reference it, but it may not know which chunks.

[0064] The backend storage system controller 240 may also include various functions, such as a garbage collection component for storage extension block garbage collection in the physical drive layer 280 .

[0065] The depicted deduplication layer 250 may include another deduplication migration coordination component 260, which may include an address range reservation component 261 that may reserve a range of logical addresses for the new deduplication volume 224. A pass-through forward lookup structure 262 may be used to absorb the fully allocated volume 222 by creating source metadata pointing to grains on physical data disks and deduplication referer metadata to point to the source metadata. A backward lookup 264 may be used to record the failure of a migrated volume and / or the update of a virtual address on the fly.

[0066] A physically allocated managed disk 241 may be maintained in the backend storage controller 240 as a logical unit of physical storage that is invisible to the host. A managed disk physical allocation 241 may be allocated to a storage pool and provide an extended block that a volume may use. A managed disk physical allocation 241 may be presented as a single logical disk that may provide an available block of physical storage without a one-to-one requirement to correspond to a physical drive.

[0067] The driver layer 280 may include storage drivers 291-293 that may provide non-volatile storage of a physical storage pool. The storage drivers 291-293 may be in the form of modules, and each may have a processor 297-299, respectively. The storage drivers 291-293 may include deduplication hash components 294-296, respectively, to which deduplication hash calculations for data on the corresponding drive may be offloaded.

[0068] The storage drivers 291-293 may have the capability to have additional calculations and may require the driver to calculate hash calculations for data areas on the drive. In an exemplary embodiment, the storage drivers 291-293 may be flash core modules (FCMs). The FCMs may be compression drivers and as part of the mechanism, they may calculate hashes. The FCMs may already have the capability to calculate hashes and depending on the scheme, range, and size of the hash, they may already calculate and store the hashes.

[0069] The described method may provide a fast and efficient mechanism for migrating fully allocated volumes to a pool of deduplicated volumes on a physical flash drive array. The deduplication hashing workload may be offloaded to the drives where the data resides. The deduplication hashing may be performed by performing SHA-1 calculations on the CPU itself or on a PCI-e connected compute board such as Lewisburg.

[0070] Migration can occur without unnecessary reads and writes of volume data outside the driver layer. Therefore, data may not be duplicated in the pool as a result of the migration, and no additional storage may be required during the migration. Unique source data may not be moved or altered. Customers can quickly see their pool capacity consumption drop due to volumes with high deduplication levels.

[0071] refer to Figure 3A and Figure 3B , schematic diagrams 300 , 310 show illustrations of example embodiments of the described methods.

[0072] exist Figure 3A , an initial stage is shown where a fully allocated volume 322 is presented to the host by the virtualization layer. The fully allocated volume 322 may have an associated physical allocation provided by a managed disk physical allocation 341 in the backend storage. The managed disk physical allocation 341 may present a logical unit of physical storage that provides an available block of physical storage in the allocated storage pool 390 in the storage drives 391-393.

[0073] The method can reserve a virtual address range 323 in the deduplication domain 320, to which the data of the fully allocated volume 322 will be migrated and then deduplicated to become a deduplicated volume 324 ( Figure 3B). Figure 3A At the stage shown, the volume may still be fully allocated and IO may not interact with the deduplication domain.

[0074] Figure 3B The situation is shown after the virtualization layer has moved the physical allocation of the volume provided by the managed disk physical allocation 341 to the reserved virtual address range 323 in the deduplication domain 320. During the movement of the physical allocation, IO to the volume may be suspended.

[0075] A forward lookup structure 362 may be provided in the deduplication domain 320 for deduplication metadata of the new deduplication volume 324 address pointed to from the fully allocated volume 322 address. A backward lookup structure 364 may be updated in the deduplication domain 320 to point to the deduplication volume 324 from the managed disk physical allocation 341.

[0076] A background deduplication process may then be performed on the new deduplicated volume 324 utilizing hash calculations offloaded to the processing power of the storage drives 391-393 storing the physical data. During migration to the deduplicated volume 324, the physical data may not be moved.

[0077] Figure 4A and Figure 4B are flow diagrams 400, 420 illustrating example embodiments of the described method of migrating a fully allocated volume to a deduplicated volume.

[0078] refer to Figure 4A The method may receive a user migration request for a fully allocated volume to a deduplicated volume (step 401).

[0079] The method at the deduplication layer may reserve a data disk LBA range for managed disk (MDISK) extents currently associated with a fully allocated volume (step 402).

[0080] The method at the deduplication layer may create a pass-forward lookup structure (e.g., in the form of a pass-forward lookup tree or table) where each LBA points to a relative LBA on a reserved LBA range of the data disk (step 403). The method may also create a stub backward lookup structure (e.g., in the form of a backward lookup tree or table) (step 404). At this stage, the volume is still fully allocated and IO may not interact with the deduplication layer.

[0081] The method may suspend IO to the volume (step 405). The method at the virtualization layer may move the extents of the volume to the reserved data disk area (step 406).

[0082] The method may mark 407 the volume as a deduplicated volume, with the deduplication layer and pass-forward lookup activated for new IO. The method may unpause IO to the volume (step 408).

[0083] refer to Figure 4B , the method at the deduplication layer may iterate on the volume particle by particle to perform deduplication (step 421). For each particle, the method may query down to the driver layer for hash calculation, which may be performed by a modified read IO (step 422). The method at the deduplication layer may lock the particle to the IO during particle migration and avoid conflicting host IO (step 423).

[0084] The method may send a hash calculation request to the driver layer, and the driver layer may process the deduplication hash calculation for the particle and return the hash result (step 424). The method may pass the hash algorithm and particle size to the module on the driver layer in advance. Each driver has the ability to calculate the hash of the data stored on itself.

[0085] For each particle and / or the next particle (step 425), it is determined whether the hash of the particle exists in the deduplication fingerprint database (decision 426). If the hash exists, the method can create a referrer metadata on the forward lookup structure to point to the source metadata of the hash (step 427). The method can mark 428 the particle in the physical drive as invalid and as a candidate for garbage collection. The particle can now be overwritten with new data.

[0086] If the hash does not exist in the deduplication fingerprint database, the granule may be marked as a deduplication miss. The method may store the hash in the deduplication fingerprint database (step 429). The method may create source metadata on a forward lookup structure to point to the granule on the data disk. The data may be retained 431 on the drive (step 430).

[0087] As each granule is processed, the method may update the backward lookup structure with the failure and / or virtual address (step 432). A determination is made as to whether there is a next granule (decision 433). If so, the method may loop to process the next granule in step 422. Once all granules have been analyzed, the volume migration may be completed (step 434).

[0088] Thus, migration can occur without unnecessary reads and writes of volume data outside the driver layer. As a result of the migration, data may not be duplicated within the pool, and no additional storage may be required during the migration. Unique source data may not be moved and / or changed. Customers will quickly see their pool capacity consumption drop due to volumes with high deduplication levels. For source granularity, tiering and heat knowledge may be preserved.

[0089] refer to Figure 5, the block diagram illustrates an example embodiment of a computing system 500. The computing system 500 may include a deduplication migration coordination component 520 and an example driver module 560. The driver module 560 may have additional computing capabilities 561 and include a deduplication migration interface 562 and a hash calculation component 563 for performing hash calculations for deduplication migration on the driver module 560.

[0090] The computing system 500 may include at least one processor 501, a hardware module or circuit for performing the functions of the described components, which may be a software unit executed on at least one processor. Multiple processors running parallel processing threads may be provided to enable parallel processing of some or all functions of the components. The memory 502 may be configured to provide computer instructions 503 to at least one processor 501 to perform the functions of the components.

[0091] The description is shown as a single component on computing system 500. However, deduplication migration component 520 may be provided across multiple computing devices, such as at a virtualization layer and a deduplication layer of a storage system.

[0092] The depicted deduplication migration component 520 may include a deduplication component 530 that includes a reservation component 535 for reserving in a deduplication domain a virtual address range to which a physical allocation associated with a fully allocated volume is moved.

[0093] The described deduplication migration component 520 may include a physical allocation move component 521 in the virtualization layer for moving the physical allocation of stored data associated with the fully allocated volume to a virtual address range in the deduplication domain. The physical allocation move component 521 may include a pause component 522, which may pause input / output operations while moving the physical allocation of stored data associated with the fully allocated volume to a virtual address range in the deduplication domain.

[0094] The described deduplication migration component 520 may include a deduplication metadata establishment component 523, which may establish the deduplication metadata as a pass-through. The deduplication metadata establishment component 523 may include a forward lookup component 524 for creating a pass-through forward lookup structure for absorbing a fully allocated volume into a deduplication volume, and a backward lookup component 525 for creating a stub backward lookup structure that is updated with the deduplication metadata when the deduplication volume is deduplicated.

[0095] The deduplication component 530 of the deduplication layer can perform a background deduplication process on the virtual address range after the virtual address range is filled with physical allocations, using a drive query hash interface 540 to perform hash calculations on the physical drive storing the data, where the virtual address range becomes a deduplicated volume of data.

[0096] The deduplication component 530 may include a particle iteration component 531 for iterating over the particles of the deduplication volume by the deduplication layer during particle deduplication, locking the particles to input / output operations. The particle iteration component 531 may include a hit component 532 for creating a referencer metadata pointing to the source metadata on a forward lookup when the particle is a deduplication hit, and marking the particle as invalid; a miss component 533 for storing a hash of the particle in a deduplication fingerprint database when the particle is a deduplication miss, creating source metadata pointing to the particle on the data disk on a forward lookup when the particle is a deduplication miss; and a backward lookup update component 534 for updating the backward lookup with invalidations and / or virtual addresses.

[0097] Figure 6 A block diagram of components of a computing system used in a virtualization layer, a deduplication layer, and / or a physical layer according to an exemplary embodiment of the present invention is depicted. It should be understood that Figure 6 This merely provides an illustration of one implementation and does not imply any limitation with respect to the environments in which different embodiments may be implemented.Many modifications may be made to the depicted computing system.

[0098] The computing system may include one or more processors 602, one or more computer-readable RAMs 604, one or more computer-readable ROMs 606, one or more computer-readable storage media 608, device drivers 612, read / write drivers or interfaces 614, and a network adapter or interface 616, all interconnected by a communications fabric 618. Communications fabric 618 may be implemented with any architecture designed to pass data and / or control information between processors (e.g., microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within the system.

[0099] One or more operating systems 610 and application programs 611 such as deduplication migration coordination components 230, 260 are stored on one or more of the computer-readable storage media 608 for execution by one or more of the processors 602 via one or more of the corresponding RAMs 604 (typically including cache memory). In the illustrated embodiment, each of the computer-readable storage media 608 may be a disk storage device of an internal hard disk drive, a CD-ROM, a DVD, a memory stick, a tape, a disk, an optical disk, a semiconductor storage device (such as RAM, ROM, EPROM, flash memory), or any other computer-readable storage medium that can store computer programs and digital information in accordance with embodiments of the present invention.

[0100] The computing system may also include a R / W drive or interface 614 to read from and write to one or more portable computer-readable storage media 626. Application programs 611 on the computing system may be stored on one or more of the portable computer-readable storage media 626, read via a corresponding R / W drive or interface 614, and loaded into a corresponding computer-readable storage medium 608.

[0101] The computing system may also include a network adapter or interface 616, such as a TCP / IP adapter card or a wireless communication adapter. Application programs 611 on the computing system may be downloaded to the computing device from an external computer or external storage device via a network (e.g., the Internet, a local area network or other wide area network or wireless network) and a network adapter or interface 616. The program may be loaded from the network adapter or interface 616 into a computer-readable storage medium 608. The network may include copper wires, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0102] The computing system may also include a display screen 620, a keyboard or keypad 622, and a computer mouse or touchpad 624. The device driver 612 interfaces with the display screen 620 for imaging, the keyboard or keypad 622, the computer mouse or touchpad 624, and / or the display screen 620 for pressure sensing of alphanumeric character input and user selection. The device driver 612, the R / W driver or interface 614, and the network adapter or interface 616 may include hardware and software stored in the computer-readable storage medium 608 and / or ROM 606.

[0103] The present invention may be a system, method and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, and the computer-readable program instructions are used to cause a processor to perform various aspects of the present invention.

[0104] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0105] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0106] The computer-readable program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, in order to perform various aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit.

[0107] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams and the combination of blocks in the flowchart illustrations and / or block diagrams can be implemented by computer-readable program instructions.

[0108] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device and / or other equipment to work in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0109] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other apparatus, so that a series of operating steps are performed on the computer, other programmable device, or other apparatus to produce a computer-implemented process, so that the instructions executed on the computer, other programmable device, or other apparatus implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0110] Flowchart and block diagram in the figure show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each frame in the flow chart or block diagram can represent the module, segment or part of instruction, which includes one or more executable instructions for realizing the specified logical function.In some alternative embodiments, the function noted in the frame can not occur in the order noted in the figure.For example, the two frames shown continuously can actually be realized as a step, and are performed simultaneously, substantially simultaneously, in a mode of partial or full time overlap, or these frames can sometimes be performed in reverse order, depending on the function involved.It will also be noted that the combination of the frames in each frame of the block diagram and / or flow chart illustration and the block diagram and / or flow chart illustration can be realized by a dedicated hardware-based system that performs a specified function or action or performs a combination of special hardware and computer instructions.

[0111] cloud computing

[0112] It should be understood that although the present disclosure includes detailed descriptions about cloud computing, the implementation of the teachings recorded herein is not limited to cloud computing environments. Instead, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0113] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal management effort or interaction with the provider of the service. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0114] Features are as follows:

[0115] On-demand self-service: Cloud consumers can unilaterally and automatically provision computing capacity (such as server time and network storage) as needed without manual interaction with the service provider.

[0116] Broad Network Access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0117] Resource pooling: The computing resources of a provider are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. There is a sense of location independence, as consumers generally do not control or know the exact location of the provided resources, but are able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0118] Rapid elasticity: In some cases, the capacity to scale out quickly and the capacity to scale in quickly can be provisioned quickly and elastically. To the consumer, the capacity available for provisioning often appears to be unlimited and can be purchased in any quantity at any time.

[0119] Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the utilized services.

[0120] The service model is as follows:

[0121] Software as a Service (SaaS): The capability provided to the consumer is to use the provider's applications running on the cloud infrastructure. The applications can be accessed from a variety of client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0122] Platform as a Service (PaaS): The capability provided to consumers is to deploy applications created or acquired by consumers using programming languages ​​and tools supported by the provider onto the cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but have control over deployed applications and possible configuration of the application hosting environment.

[0123] Infrastructure as a Service (IaaS): The capability provided to consumers is the provisioning of processing, storage, networking, and other basic computing resources on which consumers can deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0124] The deployment model is as follows:

[0125] Private cloud: The cloud infrastructure is operated only for the organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0126] Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0127] Public cloud: Cloud infrastructure is available to the general public or large industrial groups and is owned by an organization that sells cloud services.

[0128] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0129] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure consisting of a network of interconnected nodes.

[0130] Reference now Figure 7 , depicts an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which a local computing device used by a cloud consumer can communicate, such as a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an automobile computer system 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service for which the cloud consumer does not need to maintain resources on a local computing device. It should be understood that Figure 6 The types of computing devices 54A-N shown in are intended to be illustrative only, and computing node 10 and cloud computing environment 50 may communicate with any type of computerized device over any type of network and / or network addressable connection (eg, using a web browser).

[0131] Reference now Figure 8 , showing the cloud computing environment 50 ( Figure 7 ) provides a set of functional abstraction layers. It should be understood in advance that Figure 8 The components, layers, and functions shown in are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0132] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: host 61; server 62 based on RISC (Reduced Instruction Set Computer) architecture; server 63; blade server 64; storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0133] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71 ; virtual storage 72 ; virtual networks 73 , including virtual private networks; virtual applications and operating systems 74 ; and virtual clients 75 .

[0134] In one example, the management layer 80 may provide the functionality described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources for performing tasks within a cloud computing environment. Metering and pricing 82 provides cost tracking when resources are utilized within a cloud computing environment, as well as billing or invoicing for consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management so that the required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-scheduling and procurement of cloud computing resources, where future demand is anticipated based on the SLA.

[0135] The workload layer 90 provides examples of functions that can take advantage of the cloud computing environment. Examples of workloads and functions that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis processing 94; transaction processing 95; and storage deduplication processing 96.

[0136] The computer program product of the present invention comprises one or more computer-readable hardware storage devices, in which computer-readable program codes are stored. The program codes can be executed by one or more processors to implement the method of the present invention.

[0137] The computer system of the present invention includes one or more processors, one or more memories, and one or more computer-readable hardware storage devices, wherein the one or more hardware storage devices contain program codes that can be executed by the one or more processors via the one or more memories to implement the method of the present invention.

[0138] Descriptions of various embodiments of the present invention have been given for purposes of illustration, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements over existing technologies on the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0139] Modifications may be made to the foregoing without departing from the scope of the present invention.

Claims

1. A method for migrating a fully allocated volume to a deduplicated volume, the method comprising: Moving a physical allocation of stored data associated with the fully allocated volume into a virtual address range in the deduplication domain and establishing deduplication metadata as a pass-through; as well as When the virtual address range is filled with the physical allocation, a background deduplication process is performed on the virtual address range, wherein a drive query hash interface is used to perform a hash calculation on a physical drive storing the data, wherein the virtual address range becomes a deduplication volume for the data.

2. The method according to claim 1, further comprising: During the method, movement of data stored on the physical drive is avoided.

3. A method according to claim 1 or 2, wherein the physical allocation of stored data is a managed disk of data extension blocks associated with physical data extension blocks stored across multiple physical drives, and the multiple physical drives have additional computing power at which hash calculations for the deduplication process are performed.

4. The method according to any one of the preceding claims, further comprising: Input / output operations are suspended while physical allocations of stored data associated with the fully allocated volume are moved into virtual address ranges in the deduplication domain.

5. The method according to any one of the preceding claims, further comprising: The virtual address range to which the physical allocation associated with the fully allocated volume is moved is retained in the deduplication domain by a deduplication layer.

6. The method of any one of the preceding claims, wherein establishing the deduplication metadata as a transmission comprises: creating a pass-forward lookup structure for absorbing the fully allocated volume into the deduplicated volume; as well as A stub backward lookup structure is created to be updated with deduplication metadata when the deduplication volume is deduplicated.

7. A method according to any of the preceding claims, wherein moving the physical allocation of stored data associated with a fully allocated volume to a virtual address range in a deduplication domain is performed by a virtualization layer, and includes marking the volume as a deduplication volume and activating the deduplication layer and the transfer metadata.

8. The method of any one of the preceding claims, wherein performing a background deduplication process comprises iterating, by a deduplication layer, over granules of the deduplicated volume, wherein granules are locked to input / output operations during granule deduplication.

9. The method of claim 8, wherein iterating over particles comprises: When a granule is a deduplication hit, creating referrer metadata on the forward lookup structure to point to the source metadata, and marking the granule as invalid; When a particle is a deduplication miss, a hash of the particle is stored in a deduplication fingerprint database, and source metadata is created on the pass-forward lookup structure to point to the particle on the data disk; as well as Update the backward lookup structure with the invalidation or virtual address for all granules.

10. The method of any preceding claim, wherein the hash calculation is performed at a Flash Core Module (FCM) on a driver layer without reading data from outside the FCM.

11. A system for fully allocated volume to deduplicated volume migration in a storage system, the system comprising a processor and a memory, the memory being configured to provide computer program instructions to the processor to perform functions of a component, the system comprising: a physical allocation move component in the virtualization layer for moving a physical allocation of stored data associated with the fully allocated volume into a virtual address range in the deduplication domain; a deduplication metadata establishing component, used to establish the deduplication metadata as a transmission; as well as A deduplication component of the deduplication layer is used to perform a background deduplication process on the virtual address range after the virtual address range is filled with the physical allocation, wherein a drive query hash interface is used to perform hash calculations on a physical drive storing the data, wherein the virtual address range becomes a deduplication volume for the data.

12. A system according to claim 11, wherein the physical allocation of stored data is a managed disk of data extension blocks associated with physical data extension blocks stored across multiple physical drives, and the multiple physical drives have additional computing power at which hash calculations for the deduplication process are performed.

13. The system of claim 11 or 12, comprising a pause component for pausing input / output operations while moving a physical allocation of stored data associated with a fully allocated volume into a virtual address range in a deduplication domain.

14. The system of any one of claims 11 to 13, comprising a reservation component in the deduplication layer for reserving in the deduplication domain the virtual address range to which the physical allocation associated with the fully allocated volume is moved.

15. The system according to any one of claims 11 to 14, wherein the deduplication metadata establishment component comprises: a forward lookup component for creating a transfer forward lookup structure for absorbing the fully allocated volume into the deduplicated volume; as well as A backward lookup component is used to create a stub backward lookup structure to be updated with the deduplication metadata when the deduplication volume is deduplicated.

16. The system of any one of claims 11 to 15, wherein the physical allocation move component marks the volume as a deduplication volume and activates the deduplication layer and the transfer metadata.

17. The system of any one of claims 11 to 16, wherein the deduplication component comprises a granule iteration component for iterating, by the deduplication layer, over granules of the deduplication volume, wherein granules are locked to input / output operations during granule deduplication.

18. The system of claim 17, wherein the granule iteration component comprises: A hit component, for creating a referrer metadata on a forward lookup structure to point to the source metadata when a granule is a deduplication hit, and marking the granule as invalid; a miss component, for storing a hash of a particle in a deduplication fingerprint database when the particle is a deduplication miss, and creating source metadata on the pass-forward lookup structure to point to the particle on the data disk; as well as A backward lookup update component is used to update the backward lookup structure with invalid or virtual addresses.

19. The system of any one of claims 11 to 18, wherein the plurality of physical drives comprises a flash core module (FCM).

20. A computer program product for fully allocated volume to deduplicated volume migration in a storage system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to: Moving a physical allocation of stored data associated with the fully allocated volume into a virtual address range in the deduplication domain and establishing deduplication metadata as a pass-through; and When the virtual address range is filled with the physical allocation, a background deduplication process is performed on the virtual address range, wherein a drive query hash interface is used to perform a hash calculation on a physical drive storing the data, wherein the virtual address range becomes a deduplication volume for the data.

21. A computer program comprising program code means adapted to perform the method of any one of claims 1 to 10 when said program is run on a computer.

Images