Provision of a deduplication data store

Abstract

A system that includes the following: a processor; a storage facility; and a machine-readable memory that stores instructions, whereby the instructions can be executed by the processor to: to receive inputs (400) specifying an expected amount of data and a minimum retention period for a deduplication data store (150); to perform a comparison of the at least one retention period with a stored data structure (170), wherein the stored data structure (170) comprises different combinations of respective storage multipliers corresponding to the respective maximum retention periods; to determine at least one storage multiplier of the respective storage multipliers based on the comparison of at least one retention period with the stored data structure (170); to determine a storage quantity to be provided for the deduplication data store (150) based on the expected amount of data and the determined minimum storage multiplier; and to provide the deduplication data store (150) based at least on the specified amount of storage.

Description

background

[0001] Data reduction techniques can be used to decrease the amount of data stored in a storage system. One example of a data reduction technique is data deduplication. Data deduplication identifies duplicate data units and attempts to reduce or eliminate the number of duplicate data units stored in the storage system.

[0002] US 11 308 035 B2 describes data storage operations, including content indexing, containerized deduplication, and policy-driven storage, performed in a cloud environment. The systems support a variety of clients and cloud storage locations that can connect to the system within this cloud environment. Data transmission occurs over wide area networks such as the internet, which can lead to significant latency and / or packet loss. Various network protocols, such as HTTP and FTP, are used. US 11 308 035 B2 describes methods for content indexing data stored in a cloud environment to facilitate subsequent searches, including collaborative searches. Furthermore, US 11 308 035 B2 describes methods for performing containerized deduplication to reduce the load on a system namespace, achieve cost savings, and so on.US 11 308 035 B2 describes methods for identifying suitable storage locations, including suitable cloud storage locations, for data files subject to a storage policy. Furthermore, US 11 308 035 B2 describes systems and methods for deploying a cloud gateway and scalable data object storage within a cloud environment, as well as other functionalities.

[0003] US 2012 / 0215743A1 describes a system and method for monitoring the retention of backups of various systems, servers, files, etc. The system and method may include receiving retention data from a retention monitoring system. The retention data may contain the actual retention period for one or more systems, servers, files, etc., and may also include determining whether the actual retention period is consistent with a predefined retention period. If so, the data may be formatted, compiled, and displayed. If the actual retention period is inconsistent with the predefined retention period, the retention data may be marked as non-standard and then formatted, compiled, and / or displayed.Other aspects of the system and methodology of US 2012 / 0215743A1 may include ensuring the actual amount of storage capacity used by backups of systems, servers, files, etc. If the actual storage capacity does not match the allocated capacity, the data may be classified as non-standard.

[0004] US 2021 / 0216407A1 describes a backup management system that can include a data management server, a warm-tier data store, and a cold-tier data store. Snapshots can be captured from various client devices. A single data block stored in the warm-tier data store can be referenced by multiple backup snapshots and / or one or more users. When the total reference count of a data block equals the cold-tier reference count, or is equal to or less than a threshold for the total reference count, the data management server can determine that the data block is ready for migration to the cold-tier data store. The data management server can then place the data block in a candidate queue. Within the queue, data blocks with similar retention periods or similar expected recovery times can be grouped together.The unit can be transferred to the cold-tier data storage in a single write request. Brief description

[0005] A system according to claims 1 to 7, a method according to claims 8 to 13 and a non-transitory machine-readable medium according to claims 14 to 17 is disclosed. Brief description of the drawings

[0006] Some embodiments are described with reference to the following illustrations. Fig. This is a schematic diagram of an example system in accordance with some implementations. Fig. shows an example of resource allocation in accordance with some implementations. Fig. This is a representation of an exemplary data structure in accordance with some implementations. Fig. This is an illustration of example input parameters in accordance with some implementations. Fig. This is an illustration of examples of constant values ​​in accordance with some implementations. Fig. This is an illustration of a sample process in accordance with some implementations. Fig. is a schematic diagram of an example computer device according to some implementations. Fig. This is an illustration of a sample process in accordance with some implementations. Fig. is a diagram of a machine-readable medium that stores instructions in accordance with some implementations.

[0007] In the drawings, identical reference numbers denote similar, but not necessarily identical, elements. The illustrations are not necessarily to scale, and the size of some parts may be exaggerated to make the example shown clearer. Furthermore, the drawings contain examples and / or embodiments that correspond to the description; however, the description is not limited to the examples and / or embodiments shown in the drawings. Detailed description

[0008] In this disclosure, the use of the terms "a," "an," or "the" includes the plural forms unless the context clearly indicates otherwise. Similarly, the terms "include," "including," "comprises," "comprehensive," "contribute," or "indicating," when used in this disclosure, specify the presence of the indicated elements but do not exclude the presence or addition of other elements.

[0009] In some examples, a storage system can deduplicate data to reduce the storage space required. The storage system can perform a deduplication process, dividing a data stream into discrete data units or "chunks." Furthermore, the storage system can identify identifiers or "fingerprints" of incoming data units and determine which incoming data units are duplicates of previously stored data units. In the case of duplicate data units, the storage system can store references to the previous data units instead of storing the duplicate incoming data units themselves.

[0010] As used here, the term "fingerprint" refers to a value derived by applying a function to the contents of the data unit (where "content" can be all or a subset of the data unit's contents). An example of a function that can be applied is a hash function, which produces a hash value based on the incoming data unit. Examples of hash functions include cryptographic hash functions such as those of Secure Hash Algorithm 2 (SHA-2), e.g., SHA-224, SHA-256, SHA-384, etc. Other examples may use other types of hash functions or other types of fingerprint functions.

[0011] A "storage system" can comprise a single storage device or a multitude of storage devices. A storage system can also include one or more storage controllers that manage access to the storage device(s), such as a storage array. A "data unit" can refer to any portion of data that can be separately identified within the storage system. In some cases, a data unit can refer to a chunk, a collection of chunks, or some other part of data. In some examples, a storage system can store data units in persistent storage. The persistent storage can be one or more persistent (e.g., non-volatile) storage devices, such as disk-based storage devices (e.g., hard disk drives (HDDs)), solid-state devices (SSDs), such as flash memory devices, or a combination thereof.

[0012] A "control unit" can refer to a hardware processing circuit that may include any or a combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or another hardware processing circuit. Alternatively, a "control unit" can refer to a combination of a hardware processing circuit and machine-readable instructions (software and / or firmware) that can be executed on the hardware processing circuit.

[0013] In some examples, data deduplication tasks can be performed by separate storage systems, referred to here as "deduplication data stores." Each deduplication data store can be implemented as a virtualized system or device that represents an abstraction of a physical storage system and contains virtualized components that represent the hardware components of the physical storage system. In some examples, a single physical storage system can host multiple virtual deduplication data stores that operate independently. For instance, a first virtual deduplication data store can protect (i.e., process and store) backup data for a commercial website, and a second virtual deduplication data store can protect backup data for a company's accounting system.

[0014] In some examples, the host system's computing resources can be allocated to one or more deduplication data stores. For instance, each deduplication data store can be allocated a specific share of the host processors, host memory, local host memory, and so on. However, if a deduplication data store is not allocated enough computing resources, it may not be able to process backup data efficiently. In such cases, a backup process may be relatively slow and may not complete before the next backup process is scheduled. If too many computing resources are allocated to the deduplication data store, these resources may be wasted. Accordingly, allocating excessive resources can lead to relatively higher costs without providing any additional benefit.

[0015] In accordance with some implementations of this disclosure, a storage management system can determine the amounts of computational resources that must be allocated to a given deduplication data store. In some implementations, the storage management system can receive inputs that include a daily data volume (i.e., the expected amount of data to be protected each day), a maximum local retention period (i.e., the longest period to store data in local storage after protection of that data has been initiated, e.g., from the creation of a backup of that data), and a maximum remote retention period (i.e., the longest period to store data in remote storage after protection of that data has been initiated, e.g., from the creation of a backup of that data).The storage management system can determine a local storage multiplier based on a comparison of the maximum local retention period with a stored data structure, and then determine the local storage amount based on the local storage multiplier and the daily data volume. The storage management system can also determine a remote storage multiplier based on a comparison of the maximum remote retention period with the stored data structure, and determine the remote storage amount based on the remote storage multiplier and the daily data volume. The storage management system can then use the remote storage amount to determine a remote metadata amount to store in local storage.As described below, the memory management system can also determine other resource amounts to allocate to the deduplication data store, including memory, processor, and memory I / O performance (e.g., I / O operations per second (IOPS)). Furthermore, in some implementations, the deduplication data store can be automatically provisioned according to the determined resource amounts. Accordingly, the disclosed technique can provision a deduplication data store with sufficient computational resources to achieve a desired performance level without incurring additional costs due to excessive computational resources. In this way, the described technique can provide an efficient method for determining appropriate resource amounts for different workloads, which can lead to improved performance and efficiency of the deduplication storage system. Fig. 1 - Example storage system

[0016] Fig. Figure 100 shows an example of a storage system 100, which includes a management system 160, a host system 105, and remote storage 140. The host system 105 can include various computing resources such as processor(s) 110, memory 120, and local storage 130 (e.g., hard disks, flash memory, optical disks, etc.). The processor(s) 110 can be a microprocessor, a microcontroller, a processor module or subsystem, a programmable integrated circuit, a programmable gate array, multiple processors, a microprocessor with multiple processor cores, or another control or computing device. The memory 120 can be dynamic random-access memory (DRAM), static random-access memory (SRAM), etc. The local storage 130 and the remote storage 140 can include hard disks, flash memory, optical disks, etc. The remote storage 140 can also be referred to here as "cloud-based storage".

[0017] In some implementations, the host system 105 can contain a virtualization manager 155 and any number of deduplication datastores 150A-150N (also referred to here as "deduplication datastore 150"). The virtualization manager 155 can be a software application that can be run by the processor(s) 110 to initialize and manage the deduplication datastores 150. For example, the deduplication datastores 150 can be implemented as virtual machines to perform data deduplication, and the virtualization manager 155 can be a hypervisor that manages the deduplication datastores 150. In some implementations, the virtualization manager 155 can manage the allocation of host resources 110, 120, and 130 to the deduplication datastores 150. An example implementation of such resource allocations is described below with reference to Fig. described.

[0018] In some implementations, the management system 160 may include functions for determining or recommending resource allocations for the deduplication data store 150. The management system 160 can receive inputs (e.g., via a web page or other user interface) specifying the desired properties of a deduplication data store 150 to be deployed (e.g., on the host system 105 or another system). The received inputs may include, for example, a daily data volume, a maximum local retention period, and a maximum remote retention period. The management system 160 can compare the local and remote retention periods with a stored data structure 170 that contains storage multipliers (e.g., a lookup table) and thereby determine local and remote storage multipliers. The management system 160 can determine a local storage volume (i.e.,The management system 160 can determine a portion of local storage (i.e., a portion of remote storage 130) based on the daily data volume and the local storage multiplier. It can also determine a remote storage amount (i.e., a portion of remote storage 140) based on the daily data volume and the remote storage multiplier, and then use this remote storage amount to determine a remote metadata amount to store in local storage 130. Furthermore, in some implementations, the management system 160 can use constants 175 to determine other resource amounts to be allocated to the deduplication data store, including the resources of main memory, processor, and memory input / output (I / O).In some implementations, the virtualization manager 155 can obtain this resource information from the management system 160 and automatically provision the deduplication data store 150 according to the received resource information. This functionality of the management system 160 is described below with reference to the... Fig. described.

[0019] In some implementations, the management system 160 can be a computer device comprising one or more processors, memory, and one or more storage devices (in Fig. (not shown). In such implementations, the memory multiplier data structure 170 and the constants 175 can be stored in one or more storage devices of the management system 160. In other implementations, the management system 160 can be implemented as a network service or function, as a web page, as an application running on the host system 105, etc. Fig. 2 - Example of resource allocation

[0020] Fig. Figure 200 shows an example of resource allocations 200A-200N (here also referred to as "resource allocations 200"). As shown, the deduplication data store 150A can be run using the corresponding resource allocation 200A, and the deduplication data store 150N can be run using the corresponding resource allocation 200N. Each resource allocation 200 can specify certain amounts or proportions of resources such as the processors 110, the memory 120, and the local storage 130.

[0021] In some implementations, the virtualization manager can be set to 155 (in Fig. (as shown) receive or otherwise access the resource allocations 200A-200N and automatically provision the deduplication data stores 150A-150N according to the resource allocations 200A-200N. In other implementations, the resource allocations 200A-200N can be provided to a user (e.g., a human system administrator) as recommendations for selecting the system components to provision (e.g., by purchasing computing and storage equipment, by renting cloud services, etc.). Fig. 3 - Example of a data structure

[0022] In Fig. Figure 170 is an example of a storage multiplier data structure according to some implementations. As shown, the data structure can contain multiple rows or records, with each row containing a different combination of a maximum retention period (e.g., from 1 to 7 days, from 8 to 90 days, etc.) and a storage multiplier.

[0023] In some implementations, data structure 170 can be a table used to look up storage multipliers corresponding to specific maximum retention periods. For example, management system 160 might receive input specifying a maximum local retention period of five days, compare this input to data structure 170, and thus determine a local storage multiplier of 0.9 (i.e., as specified in the first row of data structure 170). In another example, management system 160 might receive further input specifying a maximum remote retention period of 120 days, compare this input to data structure 170, and thereby determine a remote storage multiplier of 1.5 (i.e., as specified in the third row of data structure 170).

[0024] In some implementations, the data structure 170 can be padded with memory multipliers to specify the average memory requirements associated with different maximum retention periods. In some examples, the average memory requirements associated with different maximum retention periods might not change in a simple or linear way as a function of the maximum retention periods (i.e., there might be a non-linear relationship between the memory multipliers and the maximum retention periods). Instead, the memory requirements might change depending on different retention schemes, which are typically used for different retention periods and require amounts of memory that are not functions of the retention periods.For example, let's assume that with a maximum retention period of four weeks, the most commonly used retention scheme involves retaining data once a day in weeks 1 and 2, and once a week in weeks 3 and 4. In another example, with a maximum retention period of six months, the most commonly used retention scheme might involve retaining data once a day in month 1, once a week in months 2 and 3, and twice a month in months 4 and 6.

[0025] Accordingly, in some implementations, data structure 170 can be populated with the memory multipliers based on expert information and / or empirical information about the memory requirements associated with different maximum retention periods. The expert information might, for example, include knowledge of the most commonly used retention schemes for different maximum retention periods. Furthermore, the empirical information (e.g., system test results, performance measurements during actual system use, historical user observations, etc.) can specify the memory requirements associated with different parts of a particular retention scheme.For example, empirical information might indicate that two consecutive daily backups typically differ by X percent, that two consecutive weekly backups typically differ by Y percent, that two consecutive monthly backups typically differ by Z percent, and so on. Furthermore, the degree of data compression (e.g., through deduplication) might vary inversely to the typical changes between two consecutive backups. Accordingly, expert information (e.g., the most commonly used retention scheme for a given maximum retention period) and empirical information (e.g.,The storage requirements associated with the various parts of the most commonly used retention scheme are used to determine the relative storage requirements associated with the corresponding maximum retention periods and can thus be used to populate the storage multipliers in Data Structure 170. Furthermore, Data Structure 170 can provide an estimate of storage requirements based on user input of the desired maximum retention period, without requiring additional user input to specify detailed characteristics of the deduplication data store (e.g., retention scheme, data compressibility, etc.). In this way, Data Structure 170 can reduce the effort and complexity for a user when requesting the deployment of a deduplication data store.

[0026] Once a deduplication data store has been deployed and run for a specific period, the management system can 160 (in Fig. (as shown) in some implementations, historical information about the actual performance of the deduplication data store may be collected or received. Such historical information may include information about the actual local and remote storage requirements of the deduplication data store during operation. Accordingly, in some implementations, the data structure 170 may be modified to include updated storage multipliers that reflect the actual performance of the deduplication data store. In such implementations, the management system 160 may use the updated data structure 170 to repeat the process of determining resource allocations and to modify or update the provision of resources to the deduplication data store.In this way, the resource allocation for the deduplication data store can be revised based on more accurate information, thereby improving the efficiency of resource utilization for the deduplication data store.

[0027] Note that the data structure 170 in Fig. Although it is represented as a lookup table, other implementations are also possible. For example, the memory multiplier data can be hard-coded as program instructions, stored as delimited text files (e.g., as key-value pairs), as XML files (Extensible Markup Language), etc. Fig. 4 - Example input parameters

[0028] Fig. This shows an example of the 400 input parameters in accordance with some implementations. As shown, the 400 input parameters can include a daily data volume, a maximum local retention period, and a maximum remote retention period. The daily data volume can specify the expected amount of data to be protected each day during the operation of the deduplication data store (e.g., 1 TB, 150 TB, etc.). The maximum local retention period can specify the maximum length of time a particular backup will be retained in the local storage of the deduplication data store (e.g., in an allocated portion of local storage 130, as shown in [reference]). Fig. (as shown). The maximum remote retention period can specify the maximum length of time a particular backup is kept in the remote storage of the deduplication data store (e.g., in an allocated portion of remote storage 140, as shown). Fig. (shown).

[0029] In some implementations, the input parameter 400 can be combined with a request to provision a new deduplication data store 150 (shown in Fig. ) be connected and specify the desired properties of the requested deduplication data store. The input parameters 400 can be used by the management system 160 (in Fig. (represented) are received from a user, a control application, an operating system, etc. For example, input parameters 400 can be received via a user interface, an application programming interface (API), etc. The management system 160 can use the received input parameters 400 to allocate resources 210 (in Fig. (shown) to determine the deduplication data storage 150. Fig. 5 - Example of constant values

[0030] Fig. This shows an example of the constant value 175 in accordance with some implementations. In some implementations, this (in Fig. The management system 160 (as shown) contains the constant values ​​175 and uses the constant values ​​175 to determine the resource allocations for one or more deduplication data stores.

[0031] As in Fig. As shown, the constant values ​​can include 175 values ​​for Base Random Access Memory (RAM), Memory Count, Storage RAM, Data Session RAM, Sparse Index RAM, Backup Window, Backup Streams, Minimum Backup Streams, Copy Streams, Minimum Copy Sessions, Sessions / Stream (i.e., sessions per stream), Remote Metadata Size, Virtual Central Processing Unit (CPU), Input / Output Instructions per Second (IOPS), and other constants.

[0032] In some implementations, the constant "Base RAM" can specify the amount of RAM required to run the operating system (OS), backup application, etc. (e.g., 10 GB). The constant "Memory Count" can specify the expected number of deduplication processes to be executed. The constant "Storage RAM" can specify the amount of RAM required to manage the deduplication data store (e.g., 1 GB). The constant "Data Session RAM" can specify the amount of RAM required to process each backup stream (e.g., 250 MB, 500 MB, etc.). The constant "Sparse Index RAM" can specify the amount of RAM required for a sparse index representing a given unit of data (e.g., 1 TB) in the deduplication data store. The sparse index can contain hashes for a subset of data blocks contained within the given data unit.

[0033] In some implementations, the constant "Backup Window" can specify the time period in which backups are expected to complete (e.g., 8 hours). The constant "Backup Streams" can specify the minimum number of backup streams required to achieve a given backup throughput (e.g., two streams to achieve a backup throughput of 1 TB / hour). The constant "Minimum Backup Streams" can specify the minimum number of backup streams the deduplication data store should support to achieve full parallelism for a single backup (e.g., 16 streams). The constant "Copy Streams" can specify the minimum number of copy streams required to achieve a given copy throughput (e.g., two streams to achieve a copy throughput of 1 TB / hour).The constant “minimum copy sessions” can specify the minimum number of copy streams that the deduplication data store should support to achieve full parallelism for a single copy (e.g., 16 streams).

[0034] In some implementations, the constant "sessions per stream" can specify the number of backup sessions each backup stream opens (e.g., two backup sessions). The constant "remote metadata size" can specify the average size of locally held metadata for the remote storage data (e.g., 3% of the total remote storage). The constant "virtual CPU" can specify the number of virtual processors required to achieve a specific throughput at a given average core power (e.g., two virtual processors to achieve a throughput of 1 TB / hour at an average core power of 1.5 GHz). The constant "input / output operations per second" (IOPS) can specify the number of IOPS the storage device(s) must provide to achieve a specific throughput (e.g.,IOPS of 200 to achieve a throughput of 1 TB / hour, assuming an average IOP size of 100 KB). Fig. 6 - Example process for deploying a deduplication data store

[0035] In Fig. This is an example process 600 for providing a deduplication data store according to some implementations. Process 600 can be run by processor(s) 110 (shown in Fig. The 600 process can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions that can be executed by one or more processors). The machine-readable instructions can be stored on a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing machine, multiple processing machines, etc. For illustration, details of the 600 process are given below with reference to Fig. The examples described show some implementations. However, other implementations are also possible.

[0036] Block 610 can include the receipt of inputs specifying a daily data volume, a maximum local retention period, and a maximum remote retention period for a deduplication data store. For example, the management system 160 (see Fig. ) Receive input parameters specifying the desired properties of a deduplication data store to be provided, including a daily data volume, a maximum local retention period, and a maximum remote retention period.

[0037] Block 620 may include determining a local storage multiplier based on a comparison of the maximum local retention with a stored data structure. Block 625 may include determining a local storage amount based on the local storage multiplier and the daily data volume. For example, the management system 160, as in Fig. The maximum local retention period is compared with the stored data structure 170 to determine a local storage multiplier. The management system 160 can then calculate the local storage amount as the product of the daily data volume and the local storage multiplier. This calculation can be represented by the following formula: Local storage = Daily protected data * local storage multiplier

[0038] Block 630 may include determining a remote storage multiplier based on a comparison of the maximum remote retention with the stored data structure. Block 635 may include determining a remote storage amount based on the remote storage multiplier and the daily data volume. For example, the management system 160 (see Fig. The maximum remote retention period is compared with the stored data structure 170 to determine a remote storage multiplier. The management system 160 can then calculate the amount of remote storage required as the product of the daily data volume and the remote storage multiplier. This calculation can be represented by the following formula: Remote storage = Daily protected data * Remote storage multiplier

[0039] Block 640 can include the determination of a remote metadata set based on the remote storage volume and the remote metadata size. For example, management system 160 can determine a remote metadata set as the product of the remote storage volume and the constant for the remote metadata size (as in Fig. (shown) calculate (see Fig. This calculation can be represented by the following formula: Remote metadate=remote storage*remote metadata size

[0040] Block 645 may include the determination of a total amount of local storage to be allocated for the deduplication data store. For example, the management system 160, as in Fig. The total amount of local storage can be calculated as the sum of the local storage amount and the remote metadata amount. This calculation can be represented by the following formula: Total storage = Local storage + Remote metadata

[0041] Block 650 can include determining the number of data sessions for the deduplication data store. For example, the management system 160 can calculate the number of data sessions using the following four formulas (see Fig. ): Throughput = Daily protected data / backup window Fuse currents = Max of ((Throughput * Fuse currents), (Min. Fuse currents)) Copy currents = Max of ((Throughput * Copy currents), (Min. Copy currents)) Number of sessions = (Fuse currents + Copy currents) * (Sessions / Current)

[0042] Block 655 may include the determination of the total amount of storage to be provided for the deduplication data store. For example, the management system 160 may calculate the total storage to be provided using the following four formulas (see below). Fig. ): RAM storage = Number of memory locations * RAM storage Sessions−RAM=Number of sessions*Data session−RAM Memory−RAM=(Local Memory+Remote−Memory)*Sparse−Index−RAM Total memory = Base RAM + Stored RAM + Session RAM + Stored RAM

[0043] Block 660 may include the determination of a processor quantity to be allocated to the deduplication data store. For example, management system 160 may determine the processor quantity as the product of the throughput and the virtual CPU constant (as in Fig. (shown) calculate (see Fig. This calculation can be represented by the following formula: Amount of processors = Throughput * Virtual CPU

[0044] Block 670 may include the determination of a number of input / output operations per second (IOPS) to be provided for the deduplication data store. For example, management system 160 may determine the IOPS quantity as the product of the throughput and the IOPS constant (in Fig. (shown) calculate (see Fig. This calculation can be represented by the following formula: IOPS - Quantity = Throughput * IOPS

[0045] Block 680 can include the provisioning of the deduplication data store based on the determined resource quantities. For example, the virtualization manager 155 can receive the determined resource quantities from the management system 160 and automatically provision the deduplication data store 150 according to these resource quantities (see...). Fig. After block 680, process 600 can be completed.

[0046] In some implementations, the management system can be 160 (shown in Fig. After process 600 has been executed to provision the deduplication data store, management system 160 can collect or receive historical information about the actual execution of the deduplication data store. Additionally, management system 160 can update data structure 170 to include new memory multipliers that reflect the actual performance of the deduplication data store. In such implementations, management system 160 can rerun process 600 using the updated data structure 170, thereby modifying or reallocating the resources allocated to the deduplication data store. This allows process 600 to be rerun to improve the efficiency of resource utilization for the deduplication data store. Fig. 7 - Example computer device

[0047] Fig. shows a schematic diagram of an example computer device 700. In some examples, the computer device 700 may generally represent part or all of the management system 160 (shown in Fig. ). As shown, the computer device 700 can comprise a hardware processor 702, a memory 704, and a machine-readable memory 705 containing instructions 710-740. The machine-readable memory 705 can be a non-transferable medium. The instructions 710-740 can be executed by the hardware processor 702 or by a processing unit contained within the hardware processor 702.

[0048] Instruction 710 can be executed to receive inputs specifying an expected data volume and a minimum retention period for a deduplication data store. For example, the management system 160 (see Fig. ) Receive input parameters specifying the desired properties of a deduplication data store to be provided, including a daily data volume, a maximum local retention period, and a maximum remote retention period.

[0049] Instruction 720 can be executed to compare the minimum retention period with a stored data structure. Instruction 730 can be executed to determine at least one storage multiplier based on the comparison of the minimum retention period with the stored data structure. For example, the management system 160, as in Fig. In one example, the management system 160 compares the maximum local retention period with the stored data structure 170 and thereby determines a local storage multiplier. In another example, the management system 160 can compare the maximum remote retention period with the stored data structure 170 and thereby determine a remote storage multiplier.

[0050] Instruction 740 can be executed to determine the amount of memory to allocate to the deduplication data store based on the expected amount of data and a minimum memory multiplier. For example, the management system 160 (see Fig. ) calculate a local storage amount as the product of the daily data volume and the local storage multiplier. Fig. 8 - Example process

[0051] In Fig. This is an example process 800, shown in accordance with some implementations. In some examples, process 800 can be run using memory controller 110 (shown in Fig. Process 800 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions that can be executed by one or more processors). The machine-readable instructions can be stored on a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing machine, multiple processing machines, etc. For illustration, details of Process 800 are given below with reference to the Fig. The examples described show some implementations. However, other implementations are also possible.

[0052] Block 810 may involve the receipt of input by a storage management system specifying an expected data volume and a minimum retention period for a deduplication data store. Block 820 may involve the storage management system comparing the minimum retention period with a stored data structure. Block 830 may involve the storage management system determining a minimum storage multiplier based on the comparison of the minimum retention period with the stored data structure. Block 840 may involve the storage management system determining a storage volume to be allocated to the deduplication data store based on the expected data volume and the minimum storage multiplier. Block 850 may involve allocating the deduplication data store based on the determined storage volume.After block 850, process 800 can be completed. Fig. 9 - Example of a machine-readable medium

[0053] Fig. Figure 900 shows a machine-readable medium on which instructions 910-940 are stored in accordance with some implementations. Instructions 910-940 can be executed by a single processor, multiple processors, a single processing machine, multiple processing machines, etc. The machine-readable medium 900 can be a non-transient storage medium, such as an optical, semiconductor, or magnetic storage medium.

[0054] Instruction 910 can be executed to receive inputs specifying an expected data volume and a minimum retention period for a deduplication data store. For example, the management system 160, as in Fig. shown, input parameters received that specify the desired properties of a deduplication data store to be provided, including a daily data volume, a maximum local retention period, and a maximum remote retention period.

[0055] Instruction 920 can be executed to compare the minimum retention period with a stored data structure. Instruction 930 can be executed to determine at least one storage multiplier based on the comparison of the minimum retention period with the stored data structure. For example, the management system 160, as in Fig. In one example, the management system 160 compares the maximum local retention period with the stored data structure 170 and thereby determines a local storage multiplier. In another example, the management system 160 can compare the maximum remote retention period with the stored data structure 170 and thereby determine a remote storage multiplier.

[0056] Instruction 940 can be executed to determine the amount of memory to allocate to the deduplication data store based on the expected amount of data and a minimum memory multiplier. For example, the management system 160 (see Fig. ) calculate a local storage amount as the product of the daily data volume and the local storage multiplier.

[0057] In accordance with the implementations described here, a memory management system can determine the amount of computational resources that must be allocated to a given deduplication data store. In some implementations, the memory management system can receive inputs including a daily data volume, a maximum local retention period, and a maximum remote retention period. The memory management system can determine a local memory multiplier based on a comparison of the maximum local retention period with a stored data structure and then determine the amount of local memory based on the local memory multiplier and the daily data volume.The memory management system can also determine a remote memory multiplier based on a comparison of the maximum remote retention time with the stored data structure and determine the amount of remote memory based on the remote memory multiplier and the daily data volume. The memory management system can use the remote memory amount to determine a remote metadata amount to store in local memory. Furthermore, the memory management system can determine other resource amounts to allocate for the deduplication data store, including memory, processor, and memory I / O performance. In some implementations, the deduplication data store can be automatically provisioned according to the determined resource amounts.Accordingly, the disclosed technique can provide a deduplication data storage system with sufficient computing resources to achieve a desired performance level without incurring additional costs due to excessive computing resources. In this way, the described technique can offer improved performance and efficiency of the deduplication storage system.

[0058] Note that the Fig. Various examples show that implementations are not limited in this respect. For example, the storage system can have 100, as in Fig.The configuration shown may include additional devices and / or components, fewer components, different components, different arrangements, etc. Another example is that the functionality of the management system 160 described above may be contained in a different machine or software of the storage system 100. Other combinations and / or variations are also possible.

[0059] Data and instructions are stored in appropriate storage devices, which are implemented as one or more computer-readable or machine-readable storage media. Storage media include various forms of non-transient memory, including semiconductor memory such as dynamic or static random-access memory (DRAM or SRAM), erasable and programmable read-only memory (EPROM), electrically erasable and programmable read-only memory (EEPROM), and flash memory; magnetic disks such as hard disks, floppy disks, and removable disks; other magnetic media including tapes; optical media such as compact discs (CDs) or digital video discs (DVDs); or other types of storage devices.

[0060] It should be noted that the instructions discussed above can be provided on a single computer-readable or machine-readable storage medium, or alternatively, on multiple computer-readable or machine-readable storage media distributed throughout a large system, possibly with multiple nodes. Such computer-readable or machine-readable storage medium or media are considered part of an article (or article of manufacture). An article or article of manufacture can refer to any single component manufactured or to multiple components. The storage medium or media can be located either in the machine on which the machine-readable instructions are executed or at a remote location from which machine-readable instructions can be downloaded for execution over a network.

[0061] The foregoing description includes numerous details to provide an understanding of the subject matter disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and deviations from the details described above. It is intended that the accompanying claims cover such modifications and variations.

Claims

[1] A system that includes the following: a processor; a storage facility; and a machine-readable memory that stores instructions, whereby the instructions can be executed by the processor to: to receive inputs (400) specifying an expected amount of data and a minimum retention period for a deduplication data store (150); to perform a comparison of the at least one retention period with a stored data structure (170), wherein the stored data structure (170) comprises different combinations of respective storage multipliers corresponding to the respective maximum retention periods; to determine at least one storage multiplier of the respective storage multipliers based on the comparison of at least one retention period with the stored data structure (170); to determine a storage quantity to be provided for the deduplication data store (150) based on the expected amount of data and the determined minimum storage multiplier; and to provide the deduplication data store (150) based at least on the specified amount of storage. [2] The system according to claim 1, wherein the stored data structure (170) is filled with the storage multipliers based on expert knowledge about a first retention scheme that is most frequently used for a given maximum retention period. [3] The system according to claim 2, wherein the stored data structure (170) is further populated with the storage multipliers based on empirical information about storage requirements that are assigned to different parts of the first storage scheme. [4] The system according to claim 1, comprising instructions that can be executed by the processor to: to determine a local storage multiplier based on a comparison of a maximum local retention time with the stored data structure (170); and to determine a local storage amount based on the local storage multiplier and the expected amount of data. [5] The system according to claim 4, comprising instructions that can be executed by the processor to: to determine a remote storage multiplier based on a comparison of a maximum remote retention time with the stored data structure (170); to determine a remote storage quantity based on the remote storage multiplier and the expected amount of data; to determine a remote metadata set based on the remote storage volume and a remote metadata size; and to determine a total amount of local storage (130) to be provided, based on the amount of local storage and the amount of remote metadata. [6] The system according to claim 1, comprising instructions that can be executed by the processor to: to determine a number of data sessions for the deduplication data store (150); to determine a total storage amount to be provided for the deduplication data store (150); to determine a number of processors to be allocated to the deduplication data store (150); and to determine the number of input / output operations per second (IOPS) to be provided for the deduplication data store (150). [7] The system according to claim 1, wherein the amount of storage to be provided is determined based on a product of the specified at least one storage multiplier and the expected amount of data, wherein the expected amount of data comprises an expected amount of data that is to be protected regularly. [8] A method (800) comprising the following: Received (810), by a storage management system, of inputs (400) specifying an expected amount of data and a minimum retention period for a deduplication data store (150); Performing (820) a comparison of the minimum retention period with a stored data structure (170), wherein the stored data structure (170) comprises different combinations of respective storage multipliers corresponding to the respective maximum retention periods; Determine (830), by the memory management system, at least one of the respective memory multipliers based on comparing the at least one retention period with the stored data structure (170); Determine (840), by the memory management system, a storage quantity to be allocated to the deduplication data store (150), based on the expected amount of data and the determined at least one storage multiplier; and Provision (850) of the deduplication data store (150) based at least on the specified amount of storage. [9] The method (800) according to claim 8, further comprising: Filling the memory multipliers in the stored data structure (170) based on: Expert knowledge about an initial storage scheme that is most commonly used for a specific maximum storage period, and empirical information about storage requirements that are assigned to different parts of the first storage scheme. [10] The method (800) according to claim 8, further comprising: Determining a local storage multiplier based on a comparison of a maximum local retention time with the stored data structure (170); and Determining a local storage amount based on the local storage multiplier and the expected amount of data. [11] The method (800) according to claim 10, further comprising: Determining a remote storage multiplier based on a comparison of a maximum remote retention time with the stored data structure (170); Determining a remote storage quantity based on the remote storage multiplier and the expected amount of data; Determining a remote metadata set based on the remote storage volume and remote metadata size; and Determine a total amount of local storage (130) to provide, based on the amount of local storage and the amount of remote metadata. [12] The method (800) according to claim 8, further comprising: Determine the number of data sessions for the deduplication data store (150); Determine the total amount of storage to be allocated to the deduplication data store (150); Determine the number of processors to be allocated to the deduplication data store (150); and Determine the number of input / output operations per second (IOPS) to be provided for the deduplication data store (150). [13] The method (800) according to claim 8, wherein the storage quantity to be provided is determined based on a product of the specified at least one storage multiplier and the expected amount of data, wherein the expected amount of data comprises an expected amount of data that is to be protected regularly. [14] A non-transitory machine-readable medium (705; 900) that stores instructions which, when executed, cause a processor (702) to: To receive inputs (400) (710; 910) specifying an expected amount of data and a minimum retention period for a deduplication data store (150); to perform a comparison of the minimum retention period with a stored data structure (170) (720; 920), wherein the stored data structure (170) comprises different combinations of respective storage multipliers corresponding to the respective maximum retention periods; to determine at least one storage multiplier of the respective storage multipliers based on the comparison of at least one retention period with the stored data structure (170) (730; 930); to determine a storage quantity (740; 940) to provide for the deduplication data store (150) based on the expected data quantity and the determined minimum storage multiplier; and to provide the deduplication data store (150) based at least on the specified amount of storage. [15] The non-transitory machine-readable medium (705; 900) according to claim 14, where the stored data structure (170) is filled with the storage multipliers based on: Expert knowledge about an initial storage scheme that is most commonly used for a specific maximum storage period; empirical information about storage requirements that are assigned to different parts of the first storage scheme. [16] The non-transitory machine-readable medium (705; 900) according to claim 14, comprising instructions which, when executed, cause the processor (702) to: to determine a local storage multiplier based on a comparison of a maximum local retention time with the stored data structure (170); and to determine a local storage amount based on the local storage multiplier and the expected amount of data. [17] The non-transitory machine-readable medium (705; 900) according to claim 16, comprising instructions which, when executed, cause the processor (702) to: to determine a remote storage multiplier based on a comparison of a maximum remote retention time with the stored data structure (170); to determine a remote storage quantity based on the remote storage multiplier and the expected amount of data; to determine a remote metadata set based on the remote storage volume and a remote metadata size; and to determine a total amount of local storage (130) to be provided, based on the amount of local storage and the amount of remote metadata.

Images