Preset deduplicated data store

Abstract

Systems, methods, and non-transitory machine-readable media for provisioning a deduplicated data store are provided. Example embodiments relate to provisioning a deduplicated data store. Examples include receiving input indicative of an expected data volume for a deduplicated data store and at least one retention period; performing a comparison of the at least one retention period to a stored data structure; determining at least one storage multiplier based on the comparison of the at least one retention period to the stored data structure; and determining a storage volume to provision for the deduplicated data store based on the expected data volume and the at least one storage multiplier.

Description

Background Technology

[0001] Data reduction techniques can be applied to reduce the amount of data stored in a storage system. Example data reduction techniques include deduplication. Deduplication identifies duplicate data units and aims to reduce or eliminate the number of instances of duplicate data units stored in the storage system. Attached Figure Description

[0002] Some implementation methods are described with reference to the following figures.

[0003] Figure 1 This is a schematic diagram of an example system based on some implementation methods.

[0004] Figure 2 This is an illustration of example resource allocation based on some implementation methods.

[0005] Figure 3 This is a diagram of an example data structure based on some implementation methods.

[0006] Figure 4 This is a diagram illustrating example input parameters according to some implementation methods.

[0007] Figure 5 This is a diagram illustrating example constant values ​​based on some implementation methods.

[0008] Figure 6 This is a diagram illustrating an example process based on some implementation methods.

[0009] Figure 7 This is a schematic diagram of an example computing device according to some implementation methods.

[0010] Figure 8 This is a diagram illustrating an example process based on some implementation methods.

[0011] Figure 9 This is a diagram of an example machine-readable medium storing instructions according to some implementation methods.

[0012] Throughout the accompanying drawings, the same reference numerals denote similar but not necessarily identical elements. The drawings are not necessarily drawn to scale, and the dimensions of some parts may be enlarged to illustrate the examples more clearly. Furthermore, the drawings provide examples and / or embodiments consistent with the description; however, the description is not limited to the examples and / or embodiments provided in the drawings. Detailed Implementation

[0013] In this disclosure, unless the context clearly indicates otherwise, the terms “a,” “an,” or “the” are intended to include the plural form as well. Similarly, when used in this disclosure, the terms “includes / including / comprises / comprising” or “have / having” indicate the presence of the said element but do not exclude the presence or addition of other elements.

[0014] In some examples, storage systems can perform deduplication on data to reduce the amount of space required to store the data. The storage system can perform the deduplication process by breaking down the data stream into discrete data units, or "chunks." Further, the storage system can determine the identifier or "fingerprint" of incoming data units and identify which incoming data units are copies of previously stored data units. In cases where a data unit is a copy, the storage system can store a reference to the previous data unit instead of storing the duplicate incoming data unit.

[0015] As used herein, a "fingerprint" refers to the value obtained by applying a function to the contents of a data unit (where "contents" can include all or a subset of the contents of the data unit). Examples of functions that can be applied include hash functions that generate a hash value based on the input data unit. Examples of hash functions include cryptographic hash functions such as Secure Hash Algorithm 2 (SHA-2) hash functions (e.g., SHA-224, SHA-256, SHA-384, etc.). In other examples, other types of hash functions or other types of fingerprint functions may be used.

[0016] A “storage system” may include one or more storage devices. A storage system may also include multiple storage controllers that manage access to the storage devices(s), such as storage arrays. A “data unit” may refer to any portion of data that can be individually identified within the storage system. In some cases, a data unit may refer to a block, a collection of blocks, or any other portion of data. In some examples, the storage system may store data units in persistent storage devices. Persistent storage devices may be implemented using one or more persistent (e.g., non-volatile) storage devices (such as multiple disk-based storage devices (e.g., multiple hard disk drives (HDDs)), multiple solid-state drives (SSDs) (such as multiple flash storage devices), or combinations thereof).

[0017] "Controller" can refer to hardware processing circuitry, which may include any or a combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or other hardware processing circuitry. Alternatively, "controller" can refer to a combination of hardware processing circuitry and machine-readable instructions (software and / or firmware) that can be executed on the hardware processing circuitry.

[0018] In some examples, deduplication tasks can be performed by a separate storage system, referred to herein as a “deduplication data store.” Each deduplication data store can be implemented as an abstraction of the physical storage system and includes a virtualized system or device that represents the hardware components of the physical storage system. In some examples, a single physical storage system can host multiple virtual deduplication data stores that operate independently of each other. For example, a first virtual deduplication data store might protect (i.e., process and store) backup data for a business website, and a second virtual deduplication data store might protect backup data for a company’s accounting system.

[0019] In some examples, the host system's computing resources can be allocated among one or more deduplication data stores. For instance, specific portions of the host processor, host memory, local host storage, etc., can be assigned to each deduplication data store. However, if insufficient computing resources are allocated to a particular deduplication data store, that store may not be able to efficiently process backup data. In such cases, the backup process may execute relatively slowly and may not complete before the next backup process is scheduled to run. Furthermore, if excessive computing resources are allocated to a deduplication data store, those resources may be wasted. Therefore, allocating excessive resources can lead to relatively high costs without providing any additional benefits.

[0020] According to some embodiments of this disclosure, a storage management system can determine the amount of computing resources to be pre-allocated to a specific deduplication data storage area. In some embodiments, the storage management system can receive inputs including: daily data volume (i.e., the expected amount of data to be protected each day), maximum local retention period (i.e., the maximum amount of time the data will be stored in local storage after protection is initiated (e.g., from the creation of a backup of the data), and maximum remote retention period (i.e., the maximum amount of time the data will be stored in remote storage after protection is initiated (e.g., from the creation of a backup of the data)). The storage management system can determine a local storage multiplier based on a comparison of the maximum local retention period and the stored data structure, and then determine the local storage volume based on the local storage multiplier and the daily data volume. The storage management system can also determine a remote storage multiplier based on a comparison of the maximum remote retention period and the stored data structure, and then determine the remote storage volume based on the remote storage multiplier and the daily data volume. The storage management system can then use the remote storage volume to determine the amount of remote metadata to be stored in the local storage device. Furthermore, as described below, the storage management system can determine the amount of other resources to provision for the deduplication data storage area, including memory, processor, and storage input / output (I / O) performance (e.g., operations per second (IOPS)). Additionally, in some embodiments, the deduplication data storage area can be automatically provisioned based on the determined resource amounts. Therefore, the disclosed technology can provide a deduplication data storage area with sufficient computing resources to meet the desired performance level without incurring additional costs due to excessive computing resources. In this way, the disclosed technology can provide an efficient technique for determining appropriate resource amounts for various workloads, which can improve the performance and efficiency of the deduplication storage system.

[0021] Figure 1 -Example Storage System

[0022] Figure 1An example of a storage system 100 is shown, comprising a management system 160, a host system 105, and a remote storage device 140. The host system 105 may include various computing resources, such as processor(s) 110, memory 120, and local storage device 130 (e.g., hard disk, flash storage, optical disk, etc.). Processor(s) 110 may include a microprocessor, microcontroller, processor module or subsystem, programmable integrated circuit, programmable gate array, multiple processors, a microprocessor including multiple processing cores, or other control or computing devices. Memory 120 may include dynamic random access memory (DRAM), static random access memory (SRAM), etc. Local storage device 130 and remote storage device 140 may include hard disks, flash storage devices, optical disks, etc. Remote storage device 140 may also be referred to herein as "cloud-based storage."

[0023] In some implementations, host system 105 may include virtualization manager 155 and any number of deduplication data stores 150A-150N (also referred to herein as "deduplication data store 150"). Virtualization manager 155 may be a software application executable by processor(s) 110 to initialize and manage deduplication data store 150. For example, deduplication data store 150 may be implemented as a virtual machine for performing deduplication, and virtualization manager 155 may be a hypervisor for managing deduplication data store 150. In some implementations, virtualization manager 155 may manage the allocation of host resources 110, 120, 130 to deduplication data store 150. Reference is made below. Figure 2 An example implementation of this resource allocation is described.

[0024] In some implementations, the management system 160 may include functionality for determining or recommending resource allocation for the deduplication data storage area 150. The management system 160 may (e.g., via a web page or other user interface) receive input indicating (e.g., on host system 105 or another system) desired characteristics of the deduplication data storage area 150 to be provisioned. For example, the received input may include daily data volume, maximum local retention period, and maximum remote retention period. The management system 160 may compare the local and remote retention periods with a stored data structure 170 (e.g., a lookup table) that includes a storage multiplier, and may thereby determine the local and remote storage multipliers. The management system 160 may determine the amount of local storage (i.e., a portion of local storage device 130) based on the daily data volume and the local storage multiplier. The management system 160 may also determine the amount of remote storage (i.e., a portion of remote storage device 140) based on the daily data volume and the remote storage multiplier, and may then use the remote storage amount to determine the amount of remote metadata to be stored in local storage device 130. Furthermore, in some embodiments, the management system 160 may use a constant 175 to determine the amount of other resources to provision for the deduplication data storage area, including memory, processor, and storage input / output (I / O) performance. In some embodiments, the virtualization manager 155 may receive such resource information from the management system 160 and may automatically provision the deduplication data storage area 150 based on the received resource information. (Refer to below) Figures 2 to 9 This function of management system 160 is described in further detail.

[0025] In some implementations, the management system 160 may include processor(s), memory(s), and storage(s)(s). Figure 1 A computing device (not shown). In such an embodiment, the multiplier data structure 170 and the constant 175 can be stored in one or more storage devices of the management system 160. In other embodiments, the management system 160 can be implemented as a network service or function, a webpage, an application running on the host system 105, etc.

[0026] Figure 2 -Example resource allocation

[0027] Referring now to 2, an illustration of example resource allocations 200A-200N (also referred to herein as "resource allocation 200") is shown. As illustrated, deduplication data storage area 150A can be executed using the corresponding resource allocation 200A, and deduplication data storage area 150N can be executed using the corresponding resource allocation 200N. Each resource allocation 200 can specify a specific amount or proportion of resources including processor 110, memory 120, and local storage device 130.

[0028] In some implementations, the virtualization manager 155 (e.g. Figure 1 The system (as shown) can receive or otherwise access resource allocations 200A-200N and can automatically provision deduplication data storage areas 150A-150N based on resource allocations 200A-200N. However, in other embodiments, resource allocations 200A-200N can be provided to users (e.g., human system managers) as suggestions for selecting system components to provision (e.g., by purchasing computing and storage devices, by leasing cloud services, etc.).

[0029] Figure 3 -Example Data Structure

[0030] Now for reference Figure 3 The illustration shows an example storage multiplier data structure 170 according to some implementations. As shown, the data structure 170 may include multiple rows or records, wherein each row includes different combinations of maximum retention period (e.g., from 1 day to 7 days, from 8 days to 90 days, etc.) and storage multipliers.

[0031] In some implementations, data structure 170 may be a table for looking up the storage multiplier corresponding to a specified maximum retention period. For example, management system 160 may receive an input indicating a maximum local retention period of five days, compare that input with data structure 170, and thereby determine a local storage multiplier of 0.9 (i.e., as specified in the first row of data structure 170). In another example, management system 160 may receive another input indicating a maximum remote retention period of 120 days, compare that input with data structure 170, and thereby determine a remote storage multiplier of 1.5 (i.e., as specified in the third row of data structure 170).

[0032] In some implementations, data structure 170 may be populated with a storage multiplier indicating the average storage requirements associated with different maximum retention periods. In some examples, the average storage requirements associated with different maximum retention periods may not vary in a simple or linear manner as a function of the maximum retention period (i.e., there may be a non-linear relationship between the storage multiplier and the maximum retention period). Instead, the storage requirements may vary depending on various retention schemes that are typically used for different retention periods and the amount of storage required is not a function of the retention period. For example, assuming a maximum retention period of four weeks, the most common retention scheme might include retaining data once a day in weeks 1-2 and once a week in weeks 3-4. In another example, for a maximum retention period of six months, the most common retention scheme might include retaining data once a day in month 1, once a week in months 2-3, and twice a month in months 4-6.

[0033] Therefore, in some implementations, data structure 170 can be populated with a storage multiplier based on expert information and / or empirical information regarding storage requirements associated with different maximum retention periods. For example, expert information may include knowledge of the most commonly used retention schemes for different maximum retention periods. Further, empirical information (e.g., system test results, performance measurements generated during actual system use, historical user observations, etc.) can indicate the storage requirements associated with different portions of a given retention scheme. For example, empirical information may indicate that two consecutive daily backups typically differ by X percent, two consecutive weekly backups typically differ by Y percent, two consecutive monthly backups typically differ by Z percent, etc. Further, the level of data compressibility (i.e., using deduplication) may be inversely proportional to the typical amount of variation between two consecutive backups. Thus, in this way, expert information (e.g., the most commonly used retention schemes for a given maximum retention period) and empirical information (e.g., storage requirements associated with different portions of the most commonly used retention scheme) can be used to determine the relative storage requirements associated with the corresponding maximum retention period, and can thereby be used to populate the storage multiplier in data structure 170. Furthermore, data structure 170 allows storage requirements to be estimated based on user input regarding the expected maximum retention period, without requiring additional user input to specify detailed characteristics of the deduplication data storage area (e.g., retention scheme, data compressibility, etc.). In this way, data structure 170 can reduce the workload and complexity for users when requesting provisioned deduplication data storage areas.

[0034] In some implementations, after the deduplication data storage area has been preset and executed for a given period of time, the management system 160 (e.g., Figure 1 (As shown) Historical information regarding the actual execution of the deduplication data storage area can be collected or received. Such historical information may include information about the actual local and remote storage requirements of the deduplication data storage area during operation. Therefore, in some embodiments, data structure 170 can be modified to include an updated storage multiplier reflecting the actual performance of the deduplication data storage area. In such embodiments, management system 160 can use the updated data structure 170 to repeat the process of determining resource allocation and modify or update the resource provisioning for the deduplication data storage area. In this way, resource allocation for the deduplication data storage area can be corrected based on more accurate information, thereby improving the resource utilization efficiency of the deduplication data storage area.

[0035] Please note that, although Figure 3The data structure 170 is illustrated as a lookup table, but other implementations are also possible. For example, it is conceivable that the multiplier data can be hardcoded as program instructions, stored as a delimited text file (e.g., as key-value pairs), stored as an Extensible Markup Language (XML) file, etc.

[0036] Figure 4 -Example input parameters

[0037] Now for reference Figure 4 The diagram illustrates an example input parameter 400 according to some implementations. As shown, input parameter 400 may include daily data volume, maximum local retention period, and maximum remote retention period. Daily data volume may specify the expected amount of data to be protected each day during deduplication data storage operations (e.g., 1TB, 150TB, etc.). Maximum local retention period may specify that any given backup will reside on the local storage device of the deduplication data storage area (e.g., in...). Figure 1 The maximum remote retention period (as shown in the allocation portion of local storage device 130) can specify the maximum time period during which any given backup will be retained in a remote storage device within the deduplication data storage area (e.g., in...). Figure 1 The maximum time period during which the allocation portion of the remote storage device 140 shown is retained.

[0038] In some implementations, input parameter 400 can be associated with a preset new deduplication data storage area 150 (e.g., Figures 1 to 2 The input parameter 400 can be associated with the request (as shown), and the desired characteristics of the requested deduplication data storage area can be specified. The input parameter 400 can be provided by the management system 160 (such as...). Figure 1 (As shown) Input parameter 400 is received from the user, control application, operating system, etc. For example, input parameter 400 can be received via a user interface, application programming interface (API), etc. The management system 160 can use the received input parameter 400 to determine resource allocation 210 (e.g., for deduplication data storage area 150) and resource allocation 210. Figure 2 (as shown in the image).

[0039] Figure 5 -Example constant values

[0040] Now for reference Figure 5 The illustration shows an example constant value 175 according to some implementations. In some implementations, the management system 160 (e.g. Figure 1 (As shown) can include a constant value of 175, and the constant value of 175 can be used to determine the resource allocation of one or more deduplication data storage areas.

[0041] like Figure 5As shown, the constant value 175 can include the following values: basic random access memory (RAM), memory area count, memory area RAM, data session RAM, sparse index RAM, backup window, backup stream, minimum backup stream, copy stream, minimum copy session, session / stream (i.e., session per stream), remote metadata size, virtual central processing unit (CPU), instructions per second (IOPS), and other constants.

[0042] In some implementations, the base RAM constant can specify the amount of RAM memory required to run the operating system (OS), backup applications, etc. (e.g., 10 GB). The storage area count constant can specify the expected number of deduplication processes to be performed. The storage area RAM constant can specify the amount of RAM memory required for housekeeping within the deduplication data storage area (e.g., 1 GB). The data session RAM constant can specify the amount of RAM memory required to process each backup stream (e.g., 250 MB, 500 MB, etc.). The sparse index RAM constant can specify the amount of RAM memory required to represent a sparse index for a given data unit (e.g., 1 TB) within the deduplication data storage area. The sparse index can include a hash of a sample set of data blocks included in the given data unit.

[0043] In some implementations, the backup window constant can specify the expected time period for completing the backup (e.g., 8 hours). The backup stream constant can specify the minimum number of backup streams required to achieve a given amount of backup throughput (e.g., two streams for 1TB / hour backup throughput). The minimum backup stream constant can specify the minimum number of backup streams that the deduplicated data store should support to achieve full parallelism for a single backup (e.g., 16 streams). The replication stream constant can specify the minimum number of replication streams required to achieve a given amount of replication throughput (e.g., two streams for 1TB / hour replication throughput). The minimum replication session constant can specify the minimum number of replication streams that the deduplicated data store should support to achieve full parallelism for a single replication (e.g., 16 streams).

[0044] In some implementations, the session constant for each stream can specify the number of backup sessions that will be opened for each backup stream (e.g., two backup sessions). The remote metadata size constant can specify the average size of metadata stored locally for remote storage data (e.g., 3% of the total remote storage). The virtual CPU constant can specify the number of virtual processors required to achieve a given throughput, assuming a specific average core performance (e.g., two virtual processors are needed to achieve a throughput of 1TB / hour, assuming an average core performance of 1.5GHz). The input / output operations per second (IOPS) value can specify the number of IOPS required by the storage device(s) to achieve a given throughput (e.g., 200 IOPS to achieve a throughput of 1TB / hour, assuming an average IOP size of 100KB).

[0045] Figure 6 - Example procedure for pre-configuring a deduplication data storage area

[0046] Now for reference Figure 6 This illustrates an example process 600 for a preset deduplication data storage area according to some embodiments. Process 600 can be executed by processor(s) 110 (e.g., multiple processors) executing instructions. Figure 1 The process 600 can be executed (as shown). It can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. For ease of illustration, reference can be made below to examples illustrating some implementations. Figures 1 to 5 The details of process 600 will be described below. However, other implementations are also possible.

[0047] Box 610 may include inputs indicating the daily data volume, maximum local retention period, and maximum remote retention period for the deduplication data storage area. For example, refer to... Figure 1 The management system 160 can receive input parameters indicating the desired characteristics of the deduplication data storage area 150 to be preset, including daily data volume, maximum local retention period, and maximum remote retention period.

[0048] Box 620 may include determining the local storage multiplier based on a comparison of the data structures with the maximum local retention and storage. Box 625 may include determining the amount of local storage based on the local storage multiplier and the daily data volume. For example, refer to Figure 1The management system 160 can compare the maximum local retention period with the stored data structure 170, and thereby determine the local storage multiplier. The management system 160 can then calculate the local storage volume as the product of the daily data volume and the local storage multiplier. This calculation can be expressed by the following formula:

[0049] Local storage = Daily protected data * Local storage multiplier

[0050] Box 630 may include determining the remote storage multiplier based on a comparison of the maximum remote retention and storage data structures. Box 635 may include determining the remote storage amount based on the remote storage multiplier and the daily data volume. For example, refer to Figure 1 The management system 160 can compare the maximum remote retention period with the stored data structure 170, and thereby determine the remote storage multiplier. The management system 160 can then calculate the remote storage volume as the product of the daily data volume and the remote storage multiplier. This calculation can be expressed by the following formula:

[0051] Remote storage = Daily protected data * Remote storage multiplier

[0052] Box 640 may include determining the amount of remote metadata based on remote storage and remote metadata size. For example, refer to Figure 1 The management system 160 can calculate the amount of remote metadata as the amount of remote storage multiplied by the remote metadata size constant (e.g., Figure 5 The product of (as shown). This calculation can be expressed by the following formula:

[0053] Remote metadata = Remote storage * Remote metadata size

[0054] Box 645 may include determining the total amount of local storage provisioned for the deduplication data storage area. For example, refer to Figure 1 The management system 160 can calculate the total local storage as the sum of local storage and remote metadata. This calculation can be expressed by the following formula:

[0055] Total storage = Local storage + Remote metadata

[0056] Box 650 may include determining the number of data sessions in the deduplication data storage area. For example, refer to Figure 1 The management system 160 can use the following four formulas to calculate the number of data sessions:

[0057] Throughput = Daily protected data / Backup window

[0058] The maximum value in the equation: Backup stream = ((throughput * backup stream), (minimum backup stream))

[0059] The maximum value among ((throughput * replica flow)) and (minimum replica flow)

[0060] Number of sessions = (Backup streams + Replica streams) * (Sessions / Streams)

[0061] Box 655 may include determining the total amount of memory pre-allocated for the data storage area to be used for deduplicating deduplicated data. For example, refer to Figure 1 The management system 160 can use the following four formulas to calculate the total memory to be preset:

[0062] RAM = Memory Area Count * Memory Area RAM

[0063] Session RAM = Number of sessions * Data Session RAM

[0064] Storage RAM = (Local Storage + Remote Storage) * Sparse Index RAM

[0065] Total memory = Basic RAM + Storage RAM + Session RAM + Storage RAM

[0066] Box 660 may include determining the number of processors pre-configured for the data storage area to be used for deduplication. For example, refer to Figure 1 The management system 160 can calculate the number of processors as throughput multiplied by the virtual CPU constant (e.g., ...). Figure 5 The product of (as shown). This calculation can be expressed by the following formula:

[0067] Number of processors = Throughput * Virtual CPUs

[0068] Box 670 may include determining the number of input / output operations per second (IOPS) preset for the data storage area to be used for deduplication. For example, refer to Figure 1 The management system 160 can calculate the number of IOPS as throughput multiplied by an IOPS constant (e.g., ...). Figure 5 The product of (as shown). This calculation can be expressed by the following formula:

[0069] IOPS = Throughput * IOPS

[0070] Box 680 may include pre-provisioning a deduplication data storage area based on a determined amount of resources. For example, refer to Figure 1 The virtualization manager 155 can receive the determined resource quantities from the management system 160 and automatically pre-configure the deduplication data storage area 150 based on these resource quantities. Following box 680, process 600 can be completed.

[0071] In some implementations, after process 600 has been executed to pre-configure the deduplication data storage area, the management system 160 (e.g., ...) Figure 1(As shown) The system can collect or receive historical information about the actual execution of the deduplication data storage area. Further, the management system 160 can update the data structure 170 to include a new storage multiplier reflecting the actual performance of the deduplication data storage area. In such an implementation, the management system 160 can repeat process 600 using the updated data structure 170, and can thereby modify or reconfigure the resources allocated to the deduplication data storage area. In this way, process 600 can be repeated to improve the resource utilization efficiency of the deduplication data storage area.

[0072] Figure 7 -Example computing device

[0073] Figure 7 A schematic diagram of an example computing device 700 is shown. In some examples, computing device 700 may generally correspond to management system 160 (e.g., Figure 1 Some or all of the components shown are included. As shown, computing device 700 may include hardware processor 702, memory 704, and machine-readable storage device 705 including instructions 710-740. Machine-readable storage device 705 may be a non-transitory medium. Instructions 710 to 740 may be executed by hardware processor 702 or by a processing engine included in hardware processor 702.

[0074] Instruction 710 can be executed to receive input indicating the expected amount of data in the deduplication data storage area and at least one retention period. For example, refer to... Figure 1 The management system 160 can receive input parameters indicating the desired characteristics of the deduplication data storage area 150 to be preset, including daily data volume, maximum local retention period, and maximum remote retention period.

[0075] Instruction 720 can be executed to perform a comparison of at least one retention period with the stored data structure. Instruction 730 can be executed to determine at least one storage multiplier based on the comparison of at least one retention period with the stored data structure. For example, refer to... Figure 1 The management system 160 can compare the maximum local retention period with the stored data structure 170, and thereby determine the local storage multiplier. In another example, the management system 160 can compare the maximum remote retention period with the stored data structure 170, and thereby determine the remote storage multiplier.

[0076] Instruction 740 can be executed to determine the amount of storage to be pre-allocated for the deduplication data storage area based on the expected data volume and at least one storage multiplier. For example, refer to Figure 1 The management system 160 can calculate the local storage volume as the product of the daily data volume and the local storage multiplier.

[0077] Figure 8 -Example Process

[0078] Now for reference Figure 8 An example process 800 according to some implementations is shown. In some examples, a storage controller 110 ( Figure 1 Process 800 is executed as shown. Process 800 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. For ease of illustration, reference can be made below to examples illustrating some implementations. Figures 1 to 3 The details of process 800 will be described below. However, other implementations are also possible.

[0079] Box 810 may include receiving input from the storage management system indicating the expected amount of data in the deduplication data storage area and at least one retention period. Box 820 may include performing a comparison of at least one retention period with the stored data structure via the storage management system. Box 830 may include determining at least one storage multiplier based on the comparison of at least one retention period with the stored data structure via the storage management system. Box 840 may include determining the amount of storage to be provisioned for the deduplication data storage area via the storage management system based on the expected data amount and at least one storage multiplier. Box 850 may include provisioning the deduplication data storage area based at least on the determined storage amount. After box 850, process 800 may be completed.

[0080] Figure 9 -Example machine-readable media

[0081] Figure 9 A machine-readable medium 900 is shown, containing storage instructions 910-940 according to some embodiments. Instructions 910-940 can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. The machine-readable medium 900 can be a non-transitory storage medium, such as an optical storage medium, a semiconductor storage medium, or a magnetic storage medium.

[0082] Instruction 910 can be executed to receive input indicating the expected amount of data in the deduplication data storage area and at least one retention period. For example, refer to... Figure 1 The management system 160 can receive input parameters indicating the desired characteristics of the deduplication data storage area 150 to be preset, including daily data volume, maximum local retention period, and maximum remote retention period.

[0083] Instruction 920 can be executed to perform a comparison of at least one retention period with the stored data structure. Instruction 930 can be executed to determine at least one storage multiplier based on the comparison of at least one retention period with the stored data structure. For example, refer to... Figure 1 The management system 160 can compare the maximum local retention period with the stored data structure 170, and thereby determine the local storage multiplier. In another example, the management system 160 can compare the maximum remote retention period with the stored data structure 170, and thereby determine the remote storage multiplier.

[0084] Instruction 940 can be executed to determine the amount of storage to be pre-allocated for the deduplication data storage area based on the expected data volume and at least one storage multiplier. For example, refer to Figure 1 The management system 160 can calculate the local storage volume as the product of the daily data volume and the local storage multiplier.

[0085] According to the embodiments described herein, a storage management system can determine the amount of computing resources to be provisioned to a specific deduplication data storage area. In some embodiments, the storage management system can receive inputs including daily data volume, maximum local retention period, and maximum remote retention period. The storage management system can determine a local storage multiplier based on a comparison of the maximum local retention period with the stored data structure, and then determine the local storage volume based on the local storage multiplier and the daily data volume. The storage management system can also determine a remote storage multiplier based on a comparison of the maximum remote retention period with the stored data structure, and then determine the remote storage volume based on the remote storage multiplier and the daily data volume. The storage management system can use the remote storage volume to determine the amount of remote metadata to be stored in the local storage device. Further, the storage management system can determine other resource quantities to be provisioned for the deduplication data storage area, including memory, processor, and storage I / O performance. In some embodiments, the deduplication data storage area can be automatically provisioned based on the determined resource quantities. Therefore, the disclosed technology can provide a deduplication data storage area with sufficient computing resources to meet the desired performance level without incurring additional costs due to excessive computing resources. In this way, the disclosed technology can improve the performance and efficiency of a deduplication storage system.

[0086] Note that, although Figures 1 to 9 Various examples are shown, but the implementation methods are not limited in this respect. For example, see references... Figure 1 It is conceivable that storage system 100 may include additional devices and / or components, fewer components, different components, different arrangements, etc. In another example, it is conceivable that the functionality of the aforementioned management system 160 may be included in any other engine or software of storage system 100. Other combinations and / or variations are also possible.

[0087] Data and instructions are stored in a corresponding storage device implemented as one or more computer-readable or machine-readable storage media. Storage media include various forms of non-transitory memory, including: semiconductor memory devices such as dynamic random access memory or static random access memory (DRAM or SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory; magnetic disks such as fixed floppy disks and removable disks; other magnetic media, including magnetic tape; optical media such as optical discs (CDs) or digital video discs (DVDs); or other types of storage devices.

[0088] Note that the instructions discussed above may be provided on a single computer-readable or machine-readable storage medium, or alternatively, on multiple computer-readable or machine-readable storage media distributed across a large system with potentially multiple nodes. Such one or more computer-readable or machine-readable storage media are considered part of an article (or article of manufacture). An article or article of manufacture may refer to any single or multiple manufactured components. The one or more storage media may be located in a machine that executes the machine-readable instructions, or at a remote site from which the machine-readable instructions can be downloaded for execution via a network.

[0089] In the foregoing description, numerous details have been set forth to facilitate understanding of the subject matter disclosed herein. However, embodiments may be practiced without some of these details. Other embodiments may include modifications and variations of the details discussed above. The appended claims are intended to cover such modifications and variations.

Claims

1. A system comprising: processor; Memory; as well as A machine-readable storage device storing instructions that can be executed by the processor to perform the following operations: Receive input indicating the expected amount of data in the deduplication data storage area and at least one retention period; Perform a comparison between the at least one retention period and the stored data structure, wherein the stored data structure includes different combinations of corresponding storage multipliers corresponding to the corresponding maximum retention period; Based on a comparison between the at least one retention period and the stored data structure, at least one storage multiplier among the corresponding storage multipliers is determined; Based on the expected data volume and the determined at least one storage multiplier, determine the amount of storage to be pre-set in the data storage area for deduplication of the duplicate data; The deduplication data storage area is pre-configured based at least on the determined storage capacity; After the deduplication data storage area has been preset and executed for a given period of time, collect or receive historical information about the actual local and remote storage requirements of the deduplication data storage area during the operation; The data structure of the storage is updated based on historical information regarding the actual local and remote storage requirements of the deduplication data storage area during operation; and The updated storage amount, pre-set for the deduplication data storage area, is determined based on the updated data structure.

2. The system as claimed in claim 1, wherein, The stored data structure is populated with the storage multiplier based on expert knowledge about the first retention scheme most commonly used for a particular maximum retention period.

3. The system as described in claim 2, wherein, The stored data structure is further populated with the storage multiplier based on empirical information about storage requirements associated with different parts of the first retention scheme.

4. The system of claim 3, wherein the empirical information includes typical variations between two consecutive backups of a specific portion of the first retention scheme.

5. The system of claim 1, further comprising instructions executable by the processor to perform the following operations: The local storage multiplier is determined based on a comparison between the maximum local retention period and the stored data structure; and The amount of local storage is determined based on the local storage multiplier and the expected amount of data.

6. The system of claim 5, further comprising instructions executable by the processor to perform the following operations: The remote storage multiplier is determined by comparing the maximum remote retention period with the data structure of the storage. The remote storage quantity is determined based on the remote storage multiplier and the expected data volume; The amount of remote metadata is determined based on the remote storage size and the remote metadata size; and The total amount of local storage to be preset is determined based on the amount of local storage and the amount of remote metadata.

7. The system of claim 1, further comprising instructions executable by the processor to perform the following operations: Determine the number of data sessions in the deduplication data storage area; Determine the total amount of memory that should be reserved in the data storage area for deleting the duplicate data; Determine the number of processors pre-configured in the data storage area for deleting the duplicate data; as well as Determine the preset number of input / output operations per second (IOPS) for the data storage area to be used for deleting the duplicate data.

8. The system of claim 1, wherein, The preset storage amount is generated based on the product of one of the determined storage multipliers and the expected data amount, wherein the expected data amount includes the expected amount of data to be protected on a periodic basis.

9. A method, the method comprising: The system receives input from the storage management system, which indicates the expected amount of data in the deduplication data storage area and at least one retention period. The storage management system performs a comparison between the at least one retention period and the stored data structure, wherein the stored data structure includes different combinations of corresponding storage multipliers corresponding to the corresponding maximum retention period; The storage management system determines at least one storage multiplier among the corresponding storage multipliers by comparing the at least one retention period with the stored data structure. The storage management system determines the amount of storage space to be pre-allocated for the deduplication data storage area based on the expected data volume and at least one determined storage multiplier. The deduplication data storage area is pre-configured based at least on the determined storage capacity; After the deduplication data storage area has been pre-configured and executed for a given period of time, the storage management system collects or receives historical information about the actual local and remote storage requirements of the deduplication data storage area during the operation. The storage management system updates the storage data structure based on historical information regarding the actual local and remote storage requirements of the deduplication data storage area during operation; and The storage management system determines the updated storage amount pre-allocated for the deduplication data storage area based on the updated data structure.

10. The method of claim 9, further comprising: The storage multiplier is populated into the storage data structure based on the following: Expert knowledge regarding the first retention scheme most commonly used for a specific maximum retention period, and Empirical information regarding storage requirements associated with different parts of the first retention scheme.

11. The method of claim 10, wherein the empirical information includes typical variations between two consecutive backups of a specific portion of the first retention scheme.

12. The method of claim 9, further comprising: The local storage multiplier is determined by comparing the maximum local retention period with the data structure of the storage. as well as The amount of local storage is determined based on the local storage multiplier and the expected amount of data.

13. The method of claim 12, further comprising: The remote storage multiplier is determined by comparing the maximum remote retention period with the data structure of the storage. The remote storage quantity is determined based on the remote storage multiplier and the expected data volume; The amount of remote metadata is determined based on the remote storage size and the remote metadata size; and The total amount of local storage to be preset is determined based on the amount of local storage and the amount of remote metadata.

14. The method of claim 9, further comprising: Determine the number of data sessions in the deduplication data storage area; Determine the total amount of memory that should be reserved in the data storage area for deleting the duplicate data; Determine the number of processors pre-configured in the data storage area for deleting the duplicate data; as well as Determine the preset number of input / output operations per second (IOPS) for the data storage area to be used for deleting the duplicate data.

15. The method of claim 9, wherein, The preset storage amount is generated based on the product of one of the determined storage multipliers and the expected data amount, wherein the expected data amount includes the expected amount of data to be protected on a periodic basis.

16. A non-transitory machine-readable medium storing instructions that, when executed, cause a processor to perform the following operations: Receive input indicating the expected amount of data in the deduplication data storage area and at least one retention period; Perform the comparison between the at least one retention period and the stored data structure, wherein, The stored data structure includes different combinations of corresponding storage multipliers corresponding to the corresponding maximum retention period; Based on a comparison between the at least one retention period and the stored data structure, at least one storage multiplier among the corresponding storage multipliers is determined; Based on the expected data volume and the determined at least one storage multiplier, determine the amount of storage to be pre-set in the data storage area for deduplication of the duplicate data; The deduplication data storage area is pre-configured based at least on the determined storage capacity; After the deduplication data storage area has been preset and executed for a given period of time, collect or receive historical information about the actual local and remote storage requirements of the deduplication data storage area during the operation; The data structure of the storage is updated based on historical information regarding the actual local and remote storage requirements of the deduplication data storage area during operation; as well as The updated storage amount, pre-set for the deduplication data storage area, is determined based on the updated data structure.

17. The non-transitory machine-readable medium as described in claim 16, in, The stored data structure is filled with the storage multiplier based on the following: Expert knowledge regarding the first retention scheme most commonly used for a specific maximum retention period; Empirical information regarding storage requirements associated with different parts of the first retention scheme.

18. The non-transitory machine-readable medium of claim 17, wherein the empirical information includes typical variations between two consecutive backups of a specific portion of the first retention scheme.

19. The non-transitory machine-readable medium of claim 16, comprising instructions that, upon execution, cause the processor to perform the following operations: The local storage multiplier is determined based on a comparison between the maximum local retention period and the stored data structure; and The amount of local storage is determined based on the local storage multiplier and the expected amount of data.

20. The non-transitory machine-readable medium of claim 19, comprising instructions that, upon execution, cause the processor to perform the following operations: The remote storage multiplier is determined by comparing the maximum remote retention period with the data structure of the storage. The remote storage quantity is determined based on the remote storage multiplier and the expected data volume; The amount of remote metadata is determined based on the amount of remote storage and the size of remote metadata. as well as The total amount of local storage to be preset is determined based on the amount of local storage and the amount of remote metadata.

Images