System and method for balanced provisioning of storage pools in a cloud storage network

Abstract

A network of non-transitory computer readable storage media nodes containing data arranged into a cloud storage network wherein at least one storage pool contains at least one storage volume, wherein said at least one storage pool and at least one storage volume and the data therein contained are managed by a system comprising: an orchestration service; an auto-provisioning engine for the automatic allocation of data resources; a media catalog for the administration and indexing of all metadata; a snapshot management module for the management of point -in-time data copies, wherein the auto¬ provisioning engine allocates storage media with a range of capacity and performance parameters to each of the at least one storage volumes within each of the at least one storage pools in a uniform manner, and wherein the snapshot management module distributes snapshots for each of the at least one volumes across all of the at least one volumes in an accordingly uniform manner.

Description

[0001] SYSTEM AND METHOD FOR BALANCED PROVISIONING

[0002] OF STORAGE POOLS IN A CLOUD STORAGE NETWORK

[0003] FIELD OF INVENTION

[0004] This invention relates to cloud storage in general, and to high fidelity storage pools in particular.

[0005] BACKGROUND OF THE INVENTION

[0006] As more and more digital systems migrate to cloud storage, a fundamental challenge is becoming prevalent. In order to avoid cases of failure in data retrieval, system managers typically provision a storage capacity significantly above the demands of their actual system, a practice known as “overprovisioning” , producing an accumulative increase in demand without an increase in data efficiency. Increasing demand for cloud storage over a limited supply of storage capacity continues to reliably produce an increase in user costs, whose data management systems typically relate only to their own databases, not the cloud infrastructure on which those databases are hosted. For these users, avoiding partial or even total catastrophic database failure has become a function of cost, i.e. the cost of overprovisioning their database capacity and diversity to ensure fidelity, effectively through the brute force said “overprovisioning” provides.

[0007] A cloud storage system may be distributed over many physical drives, logically arranged in “storage volumes ” and units of cloud storage that can themselves be grouped together into “storage pools ” . Management of said storage pools conventionally fall victim to the practice of overprovisioning described above, and in doing so will be vulnerable to a range of data inefficiencies. Such inefficiencies further increase the demand for data storage, and in doing so further increase the risk of data retrieval failure, in turn increasing the demand for overprovisioning. A causal series produces a positive feedback loop: (a) more storage is purchased; (b) more storage is filled ineffectively; (c) less storage becomes available for useful data storage; (d) data retrieval fidelity is decreased; (e) the risk of failure increases; and (f) ultimately the demand for overprovisioning data capacity increases. Without an effective means of guaranteeing high fidelity data retrieval on cloud storage infrastructure with a fixed capacity, the risks of storage failure and the data capacity demanded itself rise together, increasing costs without long term increases to the stability or fidelity of the cloud storage system. Fundamentally, this feedback loop continues to increase costs for users because the architecture of conventional cloud storage systems cannot provide guarantees to those users that their databases will remain fidelitous without employing an overprovisioning approach.

[0008] Cloud storage architectures will typically include volumes and pools distributed over a series of physical media, as discussed, as well as an orchestration service in a manual control plane, and an autoprovisioning system to allocate and configure resources across storage pools. The inefficiencies of conventional systems in data allocation are produced not just by the limitations of functionality afforded by conventional orchestration services and auto-provisioning systems (which typically overprovision as a matter of policy), but also by the limitations of adaptability of the storage architecture they manage.

[0009] Snapshots are essential features of the functioning of cloud storage networks, buttheir high demand for storage capacity can reduce the operability of other functions within those cloud storage networks to the point of near obsolescence. For example, when data resources are characterized by a broad range of performance and capacity parameters, a characterization that is often highly dynamic, the allocation of snapshot storage to single storage volumes hosted on a single type of storage resource limits access of other processes to that resource. If that resource, now highly limited in its capacity, enters into a period of high performance relative to other resources, other data management processes are precluded from exploiting that high performance. Similarly, if the storage resource responsible for the storage of all snapshots from a volume enters into a period of low performance, then data processes operated on and between snapshots on said volume are reduced in functionality, often below the threshold for usability. These challenges can multiply to render entire networks as non-functional, and apply to a range of storage applications outside of just snapshot storage.

[0010] SUMMARY OF THE INVENTION:

[0011] The following embodiments and aspects thereof are described and illustrated in conjunction with systems, devices and methods which are meant to be exemplary and illustrative and not limiting in scope. In various embodiments, one or more of the above -described problems have been reduced or eliminated, while other embodiments are directed to other advantages or improvements.

[0012] According to a first aspect of the invention, a network of non-transitory computer readable storage media nodes contains data arranged into a cloud storage network wherein at least one storage pool contains at least one storage volume, wherein said at least one storage pool and at least one storage volume and the data therein contained are managed by a system comprising: (a) an orchestration service for the management of the cloud storage network; (b) an auto-provisioning engine for the automatic allocation of data resources according to requests provided by the orchestration service; (c) a media catalog for the administration and indexing of all metadata across all storage media nodes in the cloud storage network; (d) a snapshot management module for the management of point-in-time data copies; wherein the auto-provisioning engine allocates storage media with a range of capacity and performance parameters across each of the at least one storage volumes within each of the at least one storage pools in an uniform manner according to said parameters, and wherein the snapshot management module distributes snapshots for each of the at least one volumes across all of the at least one volumes in an accordingly uniform manner. By arranging a cloud storage network in this way, the present invention teaches a means for the automatic utilization of the data resources of cloud storage nodes (i.e. the hardware itself) for a dynamic data storage demand, inherently including a snapshot functionality. This configuration achieves a number of different objectives, including but not limited to the misallocation of snapshots to storage media nodes unable to cope with the use associated therewith. This effect is produced by the immediate and autonomous allocation of resources, said immediacy being required to prevent delays in allocation that can lead to accumulated failures or other inefficiencies in how data resources are allocated.

[0013] According to another aspect of the invention, the orchestration service requests the auto-provisioning engineer to determine available resources from the media catalog, and wherein the auto-provisioning engine autonomously provisions at least one storage pool containing at least one storage volume on said resources. According to another aspect of the invention, prior to automatic allocation, the auto-provisioning engine installs management software on at least one media node defined in the media catalog.

[0014] According to another aspect of the invention, the auto-provisioning engine creates at least one thinly provisioned storage pool containing at least one storage volume on top of at least one media node defined in the media catalog.

[0015] According to another aspect of the invention, data resources used for deleted snapshots are immediately made available for reallocation by the auto-provisioning enginer for any data request made by the orchestration service relating to any of the at least one storage volumes in the storage pool in which said deleted snapshots were generated within.

[0016] It as an effect of the present invention that monopolization of a well-performing data resource by a single volume is prevented by the auto-provisioning engine allocating to the storage pool data resources with similar capacity and performance parameters.

[0017] According to another aspect of the invention, the capacity and performance parameters are determined periodically by a designated module situated on the control plane and managed by the orchestration service.

[0018] According to another aspect of the invention, the media catalog is automatically updated with the capacity and performance parameters determined by said designated module.

[0019] According to another aspect of the invention, the capacity and performance parameters include: 10 / sec; bandwidth; resiliency; redundancy; capacity reservation; and total provisioned capacity.

[0020] According to another aspect of the invention, the auto-provisioning engine divides every available new data resource into parts and allocates parts from each resource to a new volume.

[0021] According to another aspect of the invention, the orchestration service receives a request for a storage pool with specific parameters, and according thereto the auto-provisioning engine selects data resources that are suitable for the application. According to another aspect of the invention, a method for the provisioning of a cloud storage network on a network of non-transitory computer readable media, wherein at least one storage pool contains at least one storage volume, comprises the steps: (a) receiving at an orchestration service a request from a user to create a storage pool; (b) receiving at an auto-provisioning engine a request from the orchestration service to determine the available resources as presented in the media catalog; (c) allocating with the auto-provisioning engine resources for each media node referenced in the media catalog, whereby a media network comprised of at least one storage pool containing at least one storage volume is thinly provisioned and evenly distributed across allocated media nodes in a cloud storage network.

[0022] According to another aspect of the invention, a method for the provisioning of a cloud storage network on a network of non-transitory computer readable media further comprises the step: installing with the auto-provisioning engine management software on top of each allocated media node.

[0023] According to another aspect of the invention, a method for the provisioning of a cloud storage network on a network of non-transitory computer readable media further comprises the step: thinly provisioning with the auto-provisioning engine at least one storage pool on top of each allocated media node .

[0024] According to another aspect of the invention, a method for the provisioning of a cloud storage network on a network of non-transitory computer readable media further comprises the step: thinly provisioning with the auto-provisioning engine at least one storage volume contained within said at least one storage pool on top of each allocated media node.

[0025] According to another aspect of the invention, the snapshot storage for each storage volume is distributed evenly over all media nodes allocated to the storage pool in which said storage volume is contained, and wherein the distribution and management is executed by a dedicated snapshot management module.

[0026] According to another aspect of the invention, the resources allocated for a deleted snapshot is immediately reallocated to the storage pool containing the storage volume to which the snapshot related. According to another aspect of the invention, the auto-provisioning engine allocates media resources with similar capacity and performance parameters, as periodically determined and presented in the media catalog.

[0027] According to another aspect of the invention, said capacity and performance parameters include: 10 / sec; bandwidth; resiliency; redundancy; capacity reservation; and total provisioned capacity.

[0028] BRIEF DESCRIPTION OF THE FIGURES:

[0029] Some embodiments of the invention are described herein with reference to the accompanying figures. The description, together with the figures, makes apparent to a person having ordinary skill in the art how some embodiments may be practiced. The figures are for the purpose of illustrative description and no attempt is made to show structural details of an embodiment in more detail than is necessary for a fundamental understanding of the invention.

[0030] In the Figures:

[0031] FIG. 1 constitutes a sequence diagram of a cloud storage network provisioning a storage pool, according to some embodiments of the invention.

[0032] FIG. 2 constitutes a sequence diagram of a cloud storage network provisioning a storage volume in a storage pool, according to some embodiments of the invention.

[0033] FIG. 3 constitutes a system diagram of a storage pool containing four volumes distributed over four media resources, according to some embodiments of the invention.

[0034] DETAILED DESCRIPTION OF SOME EMBODIMENTS:

[0035] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components, modules, units and / or circuits have not been described in detail so as not to obscure the invention. Some features or elements described with respect to one embodiment may be combined with features or elements described with respect to other embodiments. For the sake of clarity, discussion of same or similar features or elements may not be repeated.

[0036] Unless explicitly stated, the method embodiments described herein are not constrained to a particular order or sequence. Additionally, some of the described method embodiments or elements thereof can occur or be performed simultaneously, at the same point in time, or concurrently.

[0037] Reference is made to FIG. 1, which constitutes a sequence diagram of a cloud storage network provisioning a storage pool, according to some embodiments of the invention. A network 100 comprises a pool allocation request 110, an orchestration service 120, at least one media node 130, an autoprovisioning engine / service 140, a media catalog 150, and a cloud service provider / infrastructure 160, according to some embodiments. Utilizing the network 100, a user 101 making a pool allocation request 110 at an orchestration service 120 produces a communication from said orchestration service and an auto-provisioning engine 140, which then determines the information relating to available media 130 from the media catalog 150, and thereby receives a list of available media, according to some embodiments of the invention. With the list of available media provided, the auto-provisioning engine 140 requests a resource allocation from the provider of the cloud storage infrastructure 160, returning with a list of allocated media nodes, which the auto -provisioning engine 140 forwards back to the orchestration service 120. With the list of available media nodes, the orchestration service 120 installs a connector, assigns storage to media, and thinly provisions a pool on said nodes, according to some embodiments. According to other embodiments, the allocation and thin provisioning of the media nodes 130 is executed by the auto-provisioning engine 140. The pool thinly provisioned on the media nodes 130, whether thin provision is executed by the orchestration service 120 or the auto-provisioning engine 140, are thinly provisioned with the respect to specific operating parameters defined in the pool allocation request made by the user 101.

[0038] Reference is made to FIG. 2, which constitutes a sequence diagram of a cloud storage network provisioning a storage volume in a storage pool, according to some embodiments of the invention. A network 200 comprises a volume allocation request 210, an orchestration service 220, at least one media node 230, an auto-provisioning engine / service 240, a media catalog 250, and a cloud service provider / infrastructure 260, according to some embodiments. Utilizing the network 200, a user 201 making a volume allocation 210 request at an orchestration service 220 produces a communication from said orchestration service 220 and an auto-provisioning engine 240 in the context of a request to allocate a pool in which said volume is contained, such that the auto -provisioning engine 240 determines the information relating to available media from the media catalog 250, and thereby receives a list of available media, according to some embodiments of the invention. With the list of available media provided, the auto-provisioning engine 240 requests a resource allocation from the provider of the cloud storage infrastructure 260, returning with a list of allocated media nodes, which the auto-provisioning engine 240 forwards back to the orchestration service 220. With the list of available media nodes, the orchestration service 220 installs a connector, assigns storage to media, and thinly provisions a pool on said nodes, according to some embodiments. With the pool thinly provisioned, the orchestration service can then thinly provision the requested volume on a media node 230, according to some embodiments. According to other embodiments, the allocation and thin provisioning of the media nodes 230 is executed by the auto-provisioning engine 240. The pool and volumes contained therein thinly provisioned on the media nodes 230, whether thin provision is executed by the orchestration service 220 or the auto-provisioning engine 240, are thinly provisioned with the respect to specific operating parameters defined in the pool allocation request made by the user.

[0039] Reference is made to FIG. 3, which constitutes a system diagram of a storage pool containing four volumes distributed over four media resources, according to some embodiments of the invention. The orchestration service and auto-provisioning engine provision the volumes in a pool evenly across four different media: 351, 352, 353, and 354, such that each media is thinly provisioned with a pool 350 containing parts of each of the four volumes: 310, 320, 330, and 340. By distributing the four volumes 310, 320, 330, and 340 over the media 351, 352, 353, and 354, no single storage volume can monopolize the operation of a single medium or media node. A significant portion of each of the volumes 310, 320, 330, and 340 on each of the media 351, 352, 353, and 354 is occupied by snapshots of that volume, and here too the invention distributes snapshots evenly across the different media to prevent any one medium or media node being monopolized by the snapshot storage for a particular volume. Although the present invention has been described with reference to specific embodiments, this description is not meant to be construed in a limited sense. Various modifications of the disclosed embodiments, as well as alternative embodiments of the invention will become apparent to persons skilled in the art upon reference to the description of the invention. It is, therefore, contemplated that the appended claims will cover such modifications that fall within the scope of the invention.

Claims

CLAIMS1. A network of non-transitory computer readable storage media nodes containing data arranged into a cloud storage network wherein at least one storage pool contains at least one storage volume, wherein said at least one storage pool and at least one storage volume and the data therein contained are managed by a system comprising: a. an orchestration service for the management of the cloud storage network; b. an auto-provisioning engine for the automatic allocation of data resources according to requests provided by the orchestration service; c. a media catalog for the administration and indexing of all metadata across all storage media nodes in the cloud storage network; d. a snapshot management module for the management of point-in-time data copies; wherein the auto-provisioning engine allocates storage media with a range of capacity and performance parameters to each of the at least one storage volumes within each of the at least one storage pools in a uniform manner, and wherein the snapshot management module distributes snapshots for each of the at least one volumes across all of the at least one volumes in an accordingly uniform manner.

2. The network of claim 1, wherein the orchestration service requests the auto-provisioning engine to determine available resources from the media catalog, and wherein the auto-provisioning engine autonomously provisions at least one storage pool containing at least one storage volume on said resources.

3. The network of claim 1 , wherein prior to automatic allocation, the auto-provisioning engine installs management software on at least one media node defined in the media catalog.

4. The network of claim 1, wherein the auto-provisioning engine creates at least one thinly provisioned storage pool containing at least one storage volume on top of at least one media node defined in the media catalog.

5. The network of claim 1, wherein data resources used for deleted snapshots are immediately made available for reallocation by the auto -provisioning engine for any data request made by the orchestration service relating to any of the at least one storage volumes in the storage pool in which said deleted snapshots were generated within.

6. The network of claim 1, wherein the capacity and performance parameters are determined periodically by a designated module situated on the control plane and managed by the orchestration service.

7. The network of claim 6, wherein the media catalog is automatically updated with the capacity and performance parameters determined by said designated module.

8. The network of claim 6, wherein the capacity and performance parameters include: 10 / sec; bandwidth; resiliency; redundancy; capacity reservation; and total provisioned capacity.

9. The network of claim 6, wherein the auto-provisioning engine divides every available new data resource into parts and allocates parts from each resource to a new volume.

10. The network of claim 6, wherein the orchestration service receives a request for a storage pool with specific parameters, and according thereto the auto-provisioning engine selects data resources that are suitable for the application.

11. A method for the provisioning of a cloud storage network on a network of non -transitory computer readable media nodes, wherein at least one storage pool contains at least one storage volume, comprising the steps: a. receiving at an orchestration service a request from a user to create a storage pool; b. receiving at an auto-provisioning engine a request from the orchestration service to determine the available resources as presented in the media catalog;c. allocating with the auto-provisioning engine resources for each media node referenced in the media catalog; whereby a media network comprised of at least one storage pool containing at least one storage volume is thinly provisioned and evenly distributed across allocated media nodes in a cloud storage network.

12. The method of claim 10, further comprising the step: installing with the auto-provisioning engine management software on top of each allocated media node.

13. The method of claim 10, further comprising the step: thinly provisioning with the auto-provisioning engine at least one storage pool on top of each allocated media node .

14. The method of claim 10, further comprising the step: thinly provisioning with the auto-provisioning engine at least one storage volume contained within said at least one storage pool on top of each allocated media node.

15. The method of claim 10, wherein the snapshot storage for each storage volume is distributed evenly over all media nodes allocated to the storage pool in which said storage volume is contained, and wherein the distribution and management is executed by a dedicated snapshot management module.

16. The method of claim 12, wherein the resources allocated for a deleted snapshot is immediately reallocated to the storage pool containing the storage volume to which the snapshot related.

17. The method of claim 10, wherein the auto-provisioning engine allocates media resources with similar capacity and performance parameters, as periodically determined and presented in the media catalog.

18. The method of claim 14, wherein said capacity and performance parameters include: 10 / sec; bandwidth; resiliency; redundancy; capacity reservation; and total provisioned capacity.

Images