Migration of Log Groups in a Storage System

Abstract

The present disclosure relates to migrating log groups in a storage system. Example embodiments relate to metadata operations in the storage system. Examples include: receiving, by a storage controller of a deduplication storage system, a plurality of data streams to be stored in a persistent storage device of the deduplication storage system; identifying, by the storage controller, a group of logs in a first log group that were modified during a first backup process; determining, by the storage controller, a count of the group of logs that were modified during the first backup process; comparing, by the storage controller, the determined count with a migration threshold; and migrating, by the storage controller, at least one log in the group of logs to a second log group based at least on the comparison of the determined count with the migration threshold.

Description

Background Art

[0001] Data reduction techniques can be applied to reduce the amount of data stored in a storage system. Example data reduction techniques include deduplication. Deduplication identifies duplicate data units and attempts to reduce or eliminate the number of instances of the duplicate data units stored in the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Some embodiments are described with respect to the following figures.

[0003] Figure 1A to Figure 1B is a schematic diagram of an example storage system according to some embodiments.

[0004] Figure 2 is an illustration of an example data structure according to some implementations.

[0005] Figure 3 is an illustration of an example data structure according to some implementations.

[0006] Figures 4A to 4D is an illustration of an example log migration according to some implementations.

[0007] Figure 5 is an illustration of an example process according to some implementations.

[0008] Figures 6A to 6C is an illustration of an example data structure according to some implementations.

[0009] Figure 7 is an illustration of an example process according to some implementations.

[0010] Figure 8 is an illustration of an example process according to some implementations.

[0011] Figure 9 is a diagram of an example machine-readable medium having instructions stored thereon, according to some implementations.

[0012] Figure 10 is a schematic diagram of an example computing device in accordance with some embodiments.

[0013] Throughout the drawings, like reference numerals refer to similar, but not necessarily identical, elements. The drawings are not necessarily drawn to scale, and the dimensions of some parts may be exaggerated to more clearly illustrate the examples shown. Furthermore, the drawings provide examples and / or implementations consistent with the description; however, the description is not limited to the examples and / or implementations provided in the drawings. DETAILED DESCRIPTION

[0014] In the present disclosure, the use of the terms "a," "an," or "the" is intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, when used in the present disclosure, the terms "includes / including / comprises / comprising" or "have / having" specify the presence of stated elements, but do not preclude the presence or addition of other elements.

[0015] In some examples, a storage system may back up a collection of data (referred to herein as a "stream" or "data stream") of data in a deduplicated form, thereby reducing the amount of storage space required to store the data stream. The storage system may create a "backup item" to represent the data stream in a deduplicated form. The data stream (and the backup item representing it) may correspond to (multiple) user objects (e.g., (multiple) files, file systems, (multiple) volumes, or any other suitable data collection). For example, the storage system may perform a deduplication process that includes breaking the data stream into discrete data units (or "chunks") and determining "fingerprints" of these incoming data units (as described below). Further, the storage system may compare the fingerprints of the incoming data units with the fingerprints of the stored data units, and may thereby determine which incoming data units are duplicates of previously stored data units (e.g., when the comparison indicates matching fingerprints). In the event that a data unit is a duplicate, the storage system may store a reference to the previously stored data unit instead of storing the duplicate incoming data unit. In this way, the deduplication process may reduce the amount of space required to store the received data stream.

[0016] As used herein, "fingerprint" refers to a value obtained by applying a function to the contents of a data unit (where "content" may include all or a subset of the contents of a data unit). Examples of functions that may be applied include hash functions that generate a hash value based on the contents of an incoming data unit. Examples of hash functions include cryptographic hash functions, such as the Secure Hash Algorithm 2 (SHA-2) hash function (e.g., SHA-224, SHA-256, SHA-384, etc.). In other examples, other types of hash functions or other types of fingerprint functions may be employed.

[0017] A "storage system" may include a storage device or an array of storage devices. A storage system may also include (multiple) storage controllers that manage access to (multiple) storage devices. A "data unit" may refer to any portion of data that can be individually identified in a storage system. In some cases, a data unit may refer to a chunk, a collection of chunks, or any other portion of data. In some examples, a storage system may store data units in a persistent storage device. The persistent storage device may be implemented using one or more (multiple) persistent (e.g., non-volatile) storage devices, such as (multiple) disk-based storage devices (e.g., (multiple) hard disk drives (HDDs)), (multiple) solid-state devices (SSDs) (e.g., (multiple) flash memory devices), etc., or a combination thereof.

[0018] A "controller" may refer to a hardware processing circuit, which may include any one or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or other hardware processing circuits. Alternatively, a "controller" may refer to a combination of a hardware processing circuit and machine-readable instructions (software and / or firmware) executable on the hardware processing circuit.

[0019] In some examples, a storage system may use stored metadata to process an original data stream and reconstruct the original data stream from stored data units. Such stored metadata may include a data recipe (also referred to herein as a "manifest") that specifies the order in which particular data units are received (e.g., in a data stream). In order to retrieve stored data (e.g., in response to a read request), the storage system may use the manifest to determine the order in which the data units are received, thereby recreating the original data stream. The manifest may include a series of records, each record representing a specific set (or multiple) of data units. The records in the manifest may include one or more fields (also referred to herein as "pointer information") that identify container indexes that include storage information for the data units. For example, the storage information may include one or more index fields that specify location information (e.g., container, offset, etc.) of the stored data units, compression and / or encryption characteristics of the stored data units, etc.

[0020] In some examples, the storage system can use a log associated with a container index. When an event occurs that will result in changes to the metadata stored in the container index, these changes can be recorded in a log associated with the container index. Subsequently, when the container index needs to be written to a persistent storage device, the log can be written to the persistent storage device instead of the container index. Further, because the log only records changes to the container index, the processing time and bandwidth consumed by writing the log to the persistent storage device are relatively small compared to what would be consumed when writing the container index to the persistent storage device. In some examples, a log group can be formed from multiple logs. The log group can be written to the persistent storage device as a whole to reduce the total number of write operations (i.e., compared to performing a separate write operation for each log).

[0021] In some examples, a storage system may receive data streams at different points in time and may generate backup items to represent the received data streams in deduplicated form. The process of generating backup items may be referred to herein as a "backup process." In some examples, after completing multiple backup processes, logs to be accessed during subsequent backup processes may be scattered across multiple log groups. Thus, executing subsequent backup processes may involve a relatively large number of input / output (I / O) operations to load multiple log groups from a persistent storage device to a memory and to write multiple log groups from the memory to the persistent storage device. Therefore, the relatively large amount of dispersion of logs across log groups may degrade the performance of the storage system.

[0022] According to some embodiments of the present disclosure, a storage system may migrate logs between log groups to reduce or control the dispersion of logs. In some embodiments, a storage controller may identify a set of logs in a first log group that are modified during a first backup process. Further, the storage controller may determine a count of logs in the identified set of logs, and may compare the determined count with a migration threshold. In some embodiments, the migration threshold may specify a minimum number of modified logs (i.e., logs modified during the backup process) that should be included in a given log group (e.g., to avoid migration). The storage controller may migrate the logs to a second log group (referred to herein as performing "log migration") based at least on a comparison of the count of modified logs with the migration threshold. In this way, logs that may be accessed during subsequent backup processes may be merged into fewer log groups. Therefore, some embodiments may reduce the number of I / O operations for the logs, thereby improving the performance of the deduplication storage system. Referring to Figures 1 to 3 below, Figure 9 Describes various details of log migration.

[0023] Figure 1A to Figure 1B - Sample Storage System

[0024] Figure 1A 1 shows an example of a storage system 100 including a storage controller 110, a memory 115, and a persistent storage device 140 according to some embodiments. The persistent storage device 140 may include one or more non-transitory storage media, such as a hard disk drive (HDD), a solid-state drive (SSD), an optical disk, etc., or a combination thereof. The memory 115 may be implemented using a semiconductor memory such as a random access memory (RAM). In some examples, the storage controller 110 may be implemented via hardware (e.g., electronic circuitry) or a combination of hardware and programming (e.g., including at least one processor and instructions executable by the at least one processor and stored on at least one machine-readable storage medium).

[0025] In some embodiments, storage system 100 may receive a data stream and may execute a backup process to create a backup project to represent the received data stream. For example, storage controller 110 may divide the data stream into data units and may store at least one copy of each data unit in data container 170 (e.g., by appending the data unit to the end of data container 170). In some examples, each data container 170 may be divided into multiple parts (also referred to herein as "entities").

[0026] In one or more embodiments, the storage controller 110 may generate a fingerprint for each data unit. For example, the fingerprint may include a complete or partial hash value based on the data unit. To determine whether an incoming data unit is a duplicate of a stored data unit, the storage controller 110 may compare the fingerprint generated for the incoming data unit with the fingerprint of the stored data unit. If the comparison results in a match, the storage controller 110 may determine that the storage system 100 already stores a duplicate of the incoming data unit.

[0027] like Figure 1A As shown, the persistent storage device 140 can store a manifest 150, a container index 160, a data container 170, and a log group 120. In some embodiments, the storage controller 110 can generate the manifest 150 to record the order in which the data units are received. Further, the manifest 150 can include a pointer or other information indicating the container index 160 associated with each data unit. In some embodiments, the associated container index 160 can indicate the storage location of the data unit. For example, the associated container index 160 can include information that specifies that the data unit is stored in the entity at a specific offset and the entity is stored in the data container 170 at a specific offset. Further, the container index 160 can include a reference count indicating the number of manifests 150 that reference each data unit.

[0028] In some embodiments, storage controller 110 may receive a read request to access stored data and, in response, may access manifest 150 to determine a sequence of data units that constitute at least a portion of an original data stream. Storage controller 110 may then use pointer data included in manifest 150 to identify a container index 160 associated with the data unit. Further, storage controller 110 may use information included in the identified container index 160 (as well as information included in manifest 150) to determine a storage location of the data unit (e.g., data container 170, entity, offset, etc.) and may then read the data unit from the determined location.

[0029] Figure 1B An example implementation of a log grouping 120 is shown. As shown, a log grouping 120 can be a data structure for grouping multiple logs 130. Each log 130 can be associated with a different container index 160. Further, each log 130 can include information indicating changes to data stored in the container index 160 associated with the log 130. For example, when the copy of the container index 160 present in the memory 115 is modified to reflect a change to metadata, the change can also be recorded as an entry in the associated log 130. Each log grouping 120 can be transferred as a whole between the memory 115 and the persistent storage device 140. In some embodiments, each container index 160 can include information identifying the log 130 and the log grouping 120 associated with the container index 160.

[0030] like Figure 1B As shown, each log grouping 120 may include a candidate list 180, a transfer list 182, and a backup identifier 186. The backup identifier 186 may be set or updated to identify the backup process of the log grouping 120 that is currently being used (e.g., being modified). The candidate list 180 and the transfer list 182 may be data structures for identifying the logs 130 in the log grouping 120 that were modified during the backup process. For example, each of the candidate list 180 and the transfer list 182 may be an ordered array of "flags" to indicate, for each log 130 in the log grouping 120, whether the log 130 was modified during the backup process. Setting such a flag to indicate modification of the corresponding log may be referred to herein as "flagging" a particular log 130 in the candidate list 180 or the transfer list 182. In the examples described herein, flags may be implemented using Boolean values, binary values (0 or 1), or any other suitable type of data.

[0031] In some embodiments, the storage controller 110 may include functionality for reducing or limiting the dispersion of logs 130 across various log groups 120. The storage controller 110 may populate a candidate list 180 to mark logs 130 in the first log group 120 that were modified during the first backup process. In some embodiments, upon detecting the initiation of the second backup process, the storage controller 110 may determine a count of the logs 130 marked in the candidate list 180 (i.e., indicating the total number of logs 130 in the first log group 120 that were modified during the first backup process). The storage controller 110 may detect the initiation of the second backup process by detecting a new or changed backup identifier 186.

[0032] In some embodiments, if the determined count of logs 130 marked in the candidate list 180 is below the migration threshold, the storage controller 110 may copy the markers of the candidate list 180 to the transfer list 182 (i.e., identifying a set of logs that were modified during the first backup process) and may clear the candidate list 180 (e.g., reset all markers in the candidate list 180). The storage controller 110 may determine whether any logs 130 in the first log grouping 120 meet the migration condition. As used herein, a "migration condition" refers to at least one condition that must be met before a log 130 is migrated to another log grouping 120. For example, in some embodiments, the migration condition may be met if the logs 130 that were modified during the second backup process are also marked in the transfer list 182. However, in other examples, any log 130 marked in the transfer list 182 may meet the migration condition.

[0033] In some embodiments, if any logs 130 in the first log grouping 120 meet the migration conditions, the storage controller 110 may migrate these logs 130 to the second log grouping 120 (e.g., a new log grouping 120). In this way, the logs 130 that are modified during a given backup process can be merged into fewer log groups 120. Therefore, the number of I / O operations for the log groups 120 can be reduced, thereby improving the performance of the storage system 100. The disclosed migration technology will be described below with reference to Figures 4A to 10 Further discussion.

[0034] In some examples, when a log 130 is migrated to a different log grouping 120, the corresponding container index 160 may be modified to update information identifying its associated log 130 and log grouping 120. Thus, the migration of a log 130 may involve a write I / O to the container index 160 (i.e., to store the modified version of the container index 160 in the persistent storage 140). Furthermore, if the storage controller 110 performs more than a given number of log migrations, the performance cost of writing to the associated container index 160 may offset or exceed the performance benefit of reducing the number of read I / O operations required to transfer the log grouping 120 to the storage 115. Therefore, even if a log 130 meets the migration criteria, it may be desirable to prevent the migration of that log 130 to limit the total number of log migrations.

[0035] In some embodiments, the storage controller 110 can use the migration budget 184 to control the number of log migrations performed during the backup process, and thereby prevent the performance cost of writing to the associated container index 160 from offsetting or exceeding the performance advantage of log migration. The migration budget 184 can be implemented as an unsigned integer counter and can be set to a predefined value when the migration budget 184 is initialized. For example, the predefined value of the migration budget 184 can be specified by a configuration setting of the storage system 100, by user input, etc. The migration budget 184 can be initialized in response to a system event (e.g., restart, reset, etc.), a user command, etc.

[0036] In some embodiments, for each log migration performed in the storage system 100 (e.g., across multiple data streams, across defined data stream groups, for a single data stream, etc.), the storage controller 110 can reduce the migration budget 184 by a first amount (e.g., one). When the migration budget 184 decreases below a minimum threshold (e.g., to a value of zero in the case where the minimum threshold is one), the storage controller 110 can pause the execution of any log migrations. Further, if the migration budget 184 increases above the minimum threshold, the storage controller 110 can resume the execution of log migrations. In this manner, the migration budget 184 can be used to control (e.g., limit) the number of log migrations performed during a backup process.

[0037] In some embodiments, the storage controller 110 may increase the migration budget 184 by a second amount (e.g., 0.5) for each write I / O to the container index 160 that is not caused by log migration (e.g., a write I / O performed in response to a save command for the container index 160). In some embodiments, the migration budget 184 may be limited to below a maximum level. For example, the maximum level of the migration budget 184 may be a function of the number of data streams being processed by the storage system 100 (e.g., equal to the number of streams multiplied by a constant value). Figure 7 The disclosed techniques using migration budget 184 are further described.

[0038] Figure 2 - Example data structure

[0039] Now refer to Figure 2 , shows a diagram of an example data structure 200 for use in data deduplication according to some embodiments. As shown, data structure 200 may include backup item metadata 202, manifest record 210, container index 220, and container 250. In some embodiments, backup item metadata 202 may include multiple manifest records 210. In some examples, manifest record 210, container index 220, and container 250 may generally correspond to example embodiments of manifest 150, index 160, and data container 170 (shown in FIG. 1 ), respectively. In some examples, data structure 200 may be generated and / or managed by storage controller 110 (shown in FIG. 1 ).

[0040] like Figure 2 As shown, in some examples, manifest record 210 may include various fields, such as an offset, length, container index, and unit address or fingerprint. In some embodiments, each container index 220 may include any number of data unit records 230 and entity records 240. Each data unit record 230 may include various fields, such as a fingerprint (e.g., a hash of the data unit), a unit address, an entity identifier, a unit offset (i.e., the offset of the data unit within the entity), a reference count value, a unit length, the like, or a combination thereof. In some examples, the reference count value may indicate the number of manifest records 210 that reference the data unit record 230. Furthermore, each entity record 240 may include various fields, such as an entity identifier, an entity offset (i.e., the offset of the entity within the container), a stored length (i.e., the length of the data unit within the entity), a decompressed length, a checksum value, and compression / encryption information (e.g., compression type, encryption type, etc.). In some embodiments, each container 250 may include any number of entities 260, and each entity 260 may include any number of stored data units.

[0041] In one or more embodiments, the data structure 200 can be used to retrieve stored deduplicated data. For example, a read request can specify the offset and length of the data in a given file. These request parameters can be matched with the offset and length fields of a particular manifest record 210. The container index and unit address of the particular manifest record 210 can then be matched with a particular data unit record 230 included in the container index 220. Further, the entity identifier of the particular data unit record 230 can be matched with the entity identifier of the particular entity record 240. In addition, one or more other fields of the particular entity record 240 (e.g., entity offset, storage length, checksum, etc.) can be used to identify the container 250 and entity 260, and the data unit can then be read from the identified container 250 and entity 260.

[0042] Although for simplification Figure 2 In the illustration of FIG, only one of each data structure is shown, but data structure 200 may include multiple instances of backup item metadata 202, each instance including or pointing to one or more manifest records 210. In this example, data structure 200 may include multiple manifest records 210. Manifest records 210 may reference multiple container indexes 220, each corresponding to one of multiple containers 250. Each container index 220 may include one or more data unit records 230 and one or more entity records 240.

[0043] Figure 3 - Example data structure

[0044] Figure 3 A diagram of memory 115 is shown including a log grouping 310 and a plurality of container indexes 330. As shown, log grouping 310 includes a plurality of logs 320. In some examples, log grouping 310, logs 320, and container index 330 may generally correspond to example implementations of log grouping 120, logs 130, and container index 160 (shown in FIG. 1 ), respectively.

[0045] In some embodiments, each log 320 may be associated with a corresponding container index 330 and may record changes to metadata stored in the corresponding container index 330. Furthermore, each log 320 may include or be associated with a version number 325. Each container index 330 may include or be associated with a version number 335. In some embodiments, version number 325 may be compared with version number 335 to determine whether the log 320 or the associated container index 330 reflects the latest version of the metadata. For example, if version number 325 is greater than version number 335, it may be determined that the change data included in the log 320 reflects a more recent metadata state than the metadata stored in the container index 330. If so, the container index 330 may be updated to include the changes recorded in the log 320. However, if version number 325 is less than version number 335, it may be determined that the change data included in the log 320 reflects an older metadata state than the metadata stored in the container index 330. In this case, the log 320 may be cleared without updating the container index 330. In some implementations, the comparison of version number 325 to version number 335 may be performed in response to loading log 320 or associated container index 330 from persistent storage into memory (eg, from persistent storage 140 into memory 115, as shown in FIG. 1).

[0046] In one or more embodiments, the number of logs 320 included in the log group 310 can be specified in a stored parameter (e.g., a user setting, a configuration variable, etc.). In some examples, the parameter can be adjusted or regulated to modify the performance characteristics of input / output (I / O) operations in the storage system. For example, the parameter can be increased to attempt to obtain a relatively lower frequency of relatively larger write I / O operations. In another example, the parameter can be decreased to attempt to obtain a relatively higher frequency of relatively smaller write I / O operations.

[0047] Figures 4A to 4D -Example log migration

[0048] Figures 4A to 4D FIGURE 1 illustrates an example log migration according to some embodiments. Specifically, Figures 4A to 4D A plurality of log groups 410A-410E (also generally referred to as "log groups 410") are shown at different points in time during log migration. Figures 4A to 4D In the example of , each log group 410 includes four logs 420, and each log group 410 corresponds to a different backup process. Further, in this example, if the number of modified logs in the log group 410 (i.e., during the backup process) is equal to or less than a threshold of two, the log group 410 may undergo log migration. In some embodiments, Figures 4A to 4DThe illustrated functions and / or actions may be performed using storage system 100 (shown in FIG. 1 ) (eg, via storage controller 110 executing instructions stored on a machine-readable storage medium).

[0049] exist Figures 4A to 4D In FIG, log 420 with slash shading indicates that the log was modified during a given backup process, while log 420 without shading indicates that the log was not modified during a given backup process. Figure 4A , first log grouping 410A includes one log 420A that was modified during the backup process (shown with diagonal shading). Further, first log grouping 410A also includes three logs that were not modified during the backup process (shown without shading). Second log grouping 410B includes one log 420B that was modified during the backup process (shown with shading) and three unmodified logs, and third log grouping 410C includes one log 420C that was modified during the backup process (shown with shading) and three unmodified logs. Finally, fourth log grouping 410D includes four logs 420D that were modified during the backup process (shown with shading).

[0050] Now refer to Figure 4B , the first log grouping 410A includes only one log 420A that was modified during the backup process, and thus the number of modified logs (i.e., one) is less than the threshold of two. Therefore, a new log grouping 410E is initialized or otherwise generated, and the log 420A is migrated 430 from the first log grouping 410A (e.g., by the storage system 110) to the new log grouping 410E.

[0051] Now refer to Figure 4C , the second log grouping 410B includes only one log 420B that was modified during the backup process. Therefore, the log 420B is migrated 440 from the second log grouping 410B to the new log grouping 410E.

[0052] Now refer to Figure 4D , the third log grouping 410C includes only one log 420C that was modified during the backup process. Therefore, the log 420C is migrated 450 from the third log grouping 410C to the new log grouping 410E. After the migration 450, the example log migration can be completed.

[0053] like Figures 4A to 4D As shown, the example log migration divides the logs accessed during a given backup process into four log groups 410 (e.g., Figure 4A As shown) into two log groups (as Figure 4D1 ), so that a later backup process similar to the given backup process may be able to be executed with access to a smaller number of log packets. Furthermore, after the log migration, the migrated logs 420 are now stored in the order in which they were used during the given backup process. Thus, the example log migration can reduce the number of I / O operations required during the later backup process, thereby improving the performance of the deduplication storage system (e.g., storage system 100 shown in FIG. 1 ).

[0054] Figure 5 and Figures 6A to 6C -Example log migration process

[0055] Now refer to Figure 5 , shows an example log migration process 500 according to some embodiments. The process 500 can be performed by the storage controller 110 executing instructions. The process 500 can be implemented in hardware or a combination of hardware and programming (for example, machine-readable instructions that can be executed by (multiple) processors). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. For ease of explanation, reference is made below to an example log migration process 500 according to some embodiments. Figures 1A to 6C 5. Details of process 500 are described below. However, other implementations are possible.

[0056] Block 510 may include: detecting the initiation of a first backup process in a deduplication storage system. Block 520 may include: updating a candidate list to mark a first set of logs in a log group that are modified during the first backup process. For example, referring to Figure 1A to Figure 1B , the storage controller 110 detects the initiation of a first backup process in the storage system 100 (e.g., by detecting a new backup identifier 186 in the log packet 120). The first backup process may include storing a copy of the data stream in the persistent storage device 140. Further, as Figure 6A As shown, storage controller 110 populates candidate list 180 to mark the logs 130 in first log group 120 that were modified during the first backup process. In some examples, candidate list 180 may be an ordered array of flags (e.g., bit values), each flag corresponding to a different log 130 in log group 120. In such an example, setting a flag (e.g., setting the flag to a bit value of "1") indicates that the corresponding log 130 was modified during the backup process.

[0057] Block 530 may include detecting the initiation of a second backup process in the deduplication storage system. Block 540 may include determining a count of the first set of logs in the candidate list in response to detecting the initiation of the second backup process. For example, referring to Figure 1A to Figure 1B, the storage controller 110 detects the initiation of the second backup process in the storage system 100 (e.g., based on the new backup identifier 186). In response to the detection, the storage controller 110 selects the candidate list 180 that is marked (in Figure 6A The count indicates how many logs 130 in the first log group 120 are modified from the start of the first backup process to the end of the first backup process.

[0058] Block 550 may include copying the contents of the candidate list to the transfer list if the count (determined at block 540) is below the migration threshold. Block 560 may include clearing the candidate list. For example, referring to Figures 1A to 6B , the storage controller 110 determines that the count of the logs 130 marked in the candidate list 180 is below the migration threshold. In response to this determination, the storage controller 110 performs a copy 610 of the marked array in the candidate list 180 to the transfer list 182. The storage controller 110 clears the contents of the candidate list 180 (e.g., resets all flags to a value of "0" indicating that the logs have not been modified). Alternatively, if it is determined that the count of the logs 130 marked in the candidate list 180 is not below the migration threshold, the storage controller 110 does not perform the copy 610 and instead clears the existing contents of the transfer list 182 (e.g., any flags copied to the transfer list 182 before initiating the second backup process).

[0059] Block 570 may include updating the candidate list to mark each log in the log group that is not marked in the transfer list and that was modified during the second backup process. Figures 1A to 6C , the storage controller 110 determines that the first log 130 was modified during the second backup process, and also determines that the flag 640 for the first log 130 is not set in the transfer list 182 (i.e., this indicates that the first log 130 was not modified during the first backup process). Therefore, in response to these determinations, the storage controller 110 sets the flag 630 in the candidate list 180 (in Figure 6C 180 ) to indicate that the first log 130 was modified during the second backup process. After the second backup process is completed (e.g., if the count of the marked logs 130 in the candidate list 180 is again below the migration threshold), the set flag 630 can be transferred to the transfer list 182.

[0060] Block 580 may include, for each log in the log group that is marked in the transfer list and that was modified during the second backup process, migrating the log to the second log group. Figures 1A to 6C, storage controller 110 determines that second log 130 was modified during the second backup process, and also determines that a flag 620 for second log 130 is set in transfer list 182 (i.e., this indicates that second log 130 was previously modified during the first backup process). Therefore, storage controller 110 determines that second log 130 has met the migration condition (i.e., it was modified in both the first and second backup processes in a row), and in response, migrates second log 130 to a second log group 120 (e.g., a new log group 120). This migration process may be repeated for other logs 130 that were modified during the second backup process and are also marked in transfer list 182. In some embodiments, each log 130 is migrated only once during a given backup process. After block 580, process 500 may be completed.

[0061] exist Figure 5 In the example embodiment shown, each log 130 can be migrated individually when accessed during the second backup process. However, other embodiments are possible. For example, if it is determined that the count of logs 130 marked in the candidate list 180 is below the migration threshold (i.e., after completing the first backup process), the storage controller 110 can migrate all logs 130 marked in the candidate list 180 as a group. In such an example, log migration can be performed as a batch process and may not include populating the transfer list 182.

[0062] Figure 7 -Example log migration control process

[0063] Now refer to Figure 7 , shows an example log migration control process 700 according to some embodiments. The process 700 can be performed by the storage controller 110 executing instructions. The process 700 can be implemented in hardware or a combination of hardware and programming (for example, machine-readable instructions that can be executed by (multiple) processors). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. For illustration purposes, reference may be made below to Figures 1 to 2, which show examples according to some embodiments. Figure 6C Details are described for process 700. However, other implementations are possible.

[0064] Block 710 may include initializing a migration budget in the storage system. For example, referring to FIG. 1 , the storage controller 110 may initialize the migration budget 184 in response to a system event, a user command, or the like. Upon initialization, the migration budget 184 may be set to a specified value to prevent the performance cost of writing to the container index 160 during log migration from reaching an unacceptable level (e.g., outweighing the performance benefit of log migration). In some examples, the migration budget 184 may be implemented as an unsigned integer counter.

[0065] Block 720 may include executing a backup process in the storage system. For example, referring to FIG. 1 , storage system 100 receives multiple data streams to be stored in persistent storage 140. In some examples, each data stream (or group of data streams) may be processed as a separate backup process. In some examples, storage controller 110 may perform data deduplication on each backup process.

[0066] Block 730 may include: if the migration conditions are met and the migration budget is not exceeded, performing log migration. Figure 6C , the storage controller 110 populates the candidate list 180 to mark or otherwise identify the logs 130 in the first log group 120 that were modified during the first backup process. Upon detecting the initiation of the second backup process, the storage controller 110 determines the count of the marked logs 130 in the candidate list 180 and whether the count is below the migration threshold. If so, the storage controller 110 copies the contents of the candidate list 180 to the transfer list 182 and clears the candidate list 180. Subsequently, the storage controller 110 determines whether the migration condition for the second log 130 has been met (e.g., the second log 130 was modified during the second backup process and the flag 620 for the second log 130 is set in the transfer list 182). If the migration condition has been met, the storage controller 110 determines whether the migration budget 184 is currently exceeded. If the migration budget 184 is not currently exceeded, the storage controller 110 migrates the second log 130 to the second log group 120. Otherwise, if the migration budget 184 is currently exceeded, the storage controller 110 does not migrate the second log 130.

[0067] Block 740 may include reducing the migration budget by a first amount for each log migration performed. Block 750 may include increasing the migration budget by a second amount for each container index write performed that is not caused by log migration. For example, referring to FIG. 1 , the storage controller 110 reduces the migration budget 184 by a first amount (e.g., 1) for each log migration performed in the storage system 100. Further, the storage controller 110 may increase the migration budget 184 by a second amount (e.g., 0.5) for each write I / O to the container index 160 that is not caused by log migration (e.g., a write I / O performed in response to a save command for the container index 160). After block 750, the process 700 may return to block 720 (i.e., continue executing the backup process in the storage system).

[0068] Figure 8 -Example log migration process

[0069] Now refer to Figure 8 , shows an example log migration process 800 according to some embodiments. The process 800 can be performed by the storage controller 110 executing instructions. The process 800 can be implemented in hardware or a combination of hardware and programming (for example, machine-readable instructions that can be executed by (multiple) processors). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. For ease of explanation, reference is made below to Figures 1 to 3, which show examples according to some embodiments. Figure 6C 8. Details of process 800 are described below. However, other implementations are possible.

[0070] Block 810 may include receiving, by a storage controller of a deduplication storage system, a plurality of data streams to be stored in a persistent storage device of the deduplication storage system. Block 820 may include identifying, by the storage controller, a set of logs in a first log group that were modified during a first backup process. For example, referring to Figures 1A to 6A , the storage controller 110 detects the initiation of the first backup process in the storage system 100. The storage controller 110 identifies which logs 130 in the first log group 120 are modified during the first backup process, and marks the identified logs 130 in the candidate list 180 (in Figure 6A (Illustrated with an "X" in the figure).

[0071] Block 830 may include determining, by the storage controller, a count of logs modified during the first backup process. Figures 1A to 6A, the storage controller 110 detects the initiation of the second backup process in the storage system 100 (eg, by detecting the changed backup identifier 186 ). In response to the detection, the storage controller 110 counts the marked logs 130 in the candidate list 180 .

[0072] Block 840 may include comparing, by the storage controller, the determined count to a migration threshold. Block 850 may include migrating, by the storage controller, at least one log from the identified set of logs to a second log group based at least on the comparison of the determined count to the migration threshold. For example, referring to Figures 1A to 6B , the storage controller 110 determines that the count of the marked log 130 in the candidate list 180 is below the migration threshold, and in response, performs a copy 610 of the mark array of the candidate list 180 to the transfer list 182. Subsequently, the storage controller 110 determines that the second log 130 was modified during the second backup process, and further determines that the mark 620 of the second log 130 is set in the transfer list 182. In response to these determinations, the storage controller 110 migrates the second log 130 to the second log group 120. After block 850, the process 800 may be completed.

[0073] Figure 9 - Example machine-readable medium

[0074] Figure 9 A machine-readable medium 900 storing instructions 910-940 according to some embodiments is shown. The instructions 910-940 may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. The machine-readable medium 900 may be a non-transitory storage medium, such as an optical storage medium, a semiconductor storage medium, or a magnetic storage medium.

[0075] Instructions 910 may be executed to receive a plurality of data streams to be stored in a persistent storage device of a deduplication storage system. Instructions 920 may be executed to identify a set of logs in a first log grouping that were modified during a first backup process. Instructions 930 may be executed to determine a count of logs modified during the first backup process. Instructions 940 may be executed to migrate at least one log in the identified set of logs to a second log grouping based at least on a comparison of the determined count with a migration threshold.

[0076] Figure 10 - Sample computing device

[0077] Figure 10A schematic diagram of an example computing device 1000 is shown. In some examples, computing device 1000 may generally correspond to some or all of storage system 100 (shown in FIG. 1 ). As shown, computing device 1000 may include a hardware processor 1002, memory 1004, and machine-readable storage 1005 including instructions 1010-1040. Machine-readable storage 1005 may be a non-transitory medium. Instructions 1010-1040 may be executed by hardware processor 1002 or by a processing engine included in hardware processor 1002.

[0078] Instructions 1010 may be executed to receive a plurality of data streams to be stored in a persistent storage device of a deduplication storage system. Instructions 1020 may be executed to identify a set of logs in a first log grouping that were modified during a first backup process. Instructions 1030 may be executed to determine a count of logs modified during the first backup process. Instructions 1040 may be executed to migrate at least one log in the identified set of logs to a second log grouping based at least on a comparison of the determined count with a migration threshold.

[0079] According to some embodiments of the present disclosure, a storage system can migrate logs between log groups to reduce or control the dispersion of logs. In some embodiments, a controller can identify a group of logs in a first log group that are modified during a first backup process and determine a count of the identified group of logs. The storage controller can migrate the logs to a second log group based at least on a comparison of the count of modified logs with a migration threshold. In this way, logs to be accessed during subsequent backup processes can be merged into fewer log groups. Therefore, some embodiments can reduce the number of I / O operations for reading logs, thereby improving the performance of the deduplication storage system.

[0080] Note that although Figures 1A to 10 Various examples are shown, but the embodiments are not limited in this respect. For example, referring to Figure 1A It is contemplated that the storage system 100 may include additional devices and / or components, fewer components, different components, a different arrangement, etc. In another example, it is contemplated that the functionality of the storage controller 110 described above may be included in any other engine or software of the storage system 100. Other combinations and / or variations are also possible.

[0081] Data and instructions are stored in corresponding storage devices implemented as one or more computer-readable or machine-readable storage media. Storage media include various forms of non-transitory memory, including: semiconductor memory devices such as dynamic random access memory or static random access memory (DRAM or SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory; magnetic disks such as fixed floppy disks and removable disks; other magnetic media, including magnetic tape; optical media such as compact disks (CDs) or digital video disks (DVDs); or other types of storage devices.

[0082] Note that the instructions discussed above may be provided on one computer-readable or machine-readable storage medium, or alternatively, may be provided on multiple computer-readable or machine-readable storage media distributed in a large system having potentially multiple nodes. Such one or more computer-readable or machine-readable storage media are considered part of an article (or product). An article or product may refer to any manufactured single component or multiple components. The one or more storage media may be located in the machine that runs the machine-readable instructions, or at a remote site from which the machine-readable instructions may be downloaded over a network for execution.

[0083] In the foregoing description, numerous details have been set forth to facilitate an understanding of the subject matter disclosed herein. However, embodiments may be practiced without some of these details. Other embodiments may include modifications and variations of the details discussed above. The appended claims are intended to cover such modifications and variations.

Claims

1. A data deduplication storage system, comprising: processor; Memory; as well as A machine-readable storage medium storing instructions, the instructions being executable by the processor to: receiving a plurality of data streams to be stored in a persistent storage device of the deduplication storage system to perform a first backup process of the plurality of data streams, wherein the first backup process of the plurality of data streams causes a change in data in a container index; identifying a set of logs in a first log grouping that were modified during the first backup process, wherein the first log grouping includes a candidate list and a transfer list, the candidate list and the transfer list including an ordered array of tags, each tag corresponding to a different log in the first log grouping, each log including information indicating changes to data stored in a container index associated with the log, and each container index including information identifying the log and log grouping associated with the container index; determining a count of logs modified during the first backup process; and At least one log in the set of logs marked in the transfer list is migrated to a second log grouping based at least on a comparison of the determined count and a migration threshold.

2. The storage system of claim 1 , comprising instructions executable by the processor to: The candidate list is updated to mark logs that were modified during the first backup process.

3. The storage system of claim 2 , comprising instructions executable by the processor to: detecting initiation of a second backup process of the plurality of data streams; In response to detecting the initiation of the second backup process: determining a count of the number of marked logs in the candidate list; comparing the count of the marked logs to the migration threshold; as well as In response to determining that the count of the marked logs is below the migration threshold, the contents of the candidate list are copied to the transfer list.

4. The storage system of claim 3 , comprising instructions executable by the processor to: In response to determining that the at least one log is modified during the second backup process and marked in the transfer list, the at least one log is migrated to the second log grouping.

5. The storage system of claim 3 , comprising instructions executable by the processor to: In response to determining that the at least one log is modified during the second backup process and marked in the transfer list: Determine if your migration budget is currently exceeded; In response to determining that the migration budget is not currently exceeded, migrating the at least one log; as well as In response to determining that the migration budget is currently exceeded, migration of the at least one log is prevented.

6. The storage system of claim 5 , comprising instructions executable by the processor to: For each log migration performed, reducing the migration budget by a first amount; and For each container index write performed that is not caused by log migration, the migration budget is increased by a second amount.

7. The storage system according to claim 6, wherein: The first amount is greater than the second amount.

8. The storage system of claim 3, comprising instructions executable by the processor to: detecting said initiation of said second backup process based on a change of a backup identifier, in, The backup identifier, the candidate list, and the transfer list are all stored in the first log packet.

9. The storage system of claim 3, comprising instructions executable by the processor to: In response to detecting the initiation of the second backup process, clearing the candidate list; and After the candidate list is cleared, in response to determining that a specific log is modified during the second backup process and is not marked in the transfer list, the specific log is marked in the candidate list.

10. A method comprising: receiving, by a storage controller of a deduplication storage system, a plurality of data streams to be stored in a persistent storage device of the deduplication storage system to perform a first backup process of the plurality of data streams, wherein the first backup process of the plurality of data streams causes a change to data in a container index; identifying, by the storage controller, a set of logs in a first log grouping that were modified during the first backup process, wherein the first log grouping includes a candidate list and a transfer list, the candidate list and the transfer list including an ordered array of tags, each tag corresponding to a different log in the first log grouping, each log including information indicating changes to data stored in a container index associated with the log, and each container index including information identifying the log and log grouping associated with the container index; determining, by the storage controller, a count of logs modified during the first backup process; comparing, by the storage controller, the determined count to a migration threshold; and At least one log in the set of logs marked in the transfer list is migrated, by the storage controller, to a second group of logs based at least on a comparison of the determined count and the migration threshold.

11. The method of claim 10, further comprising: updating the candidate list to mark the set of logs modified during the first backup process; detecting initiation of a second backup process of the plurality of data streams; In response to detecting the initiation of the second backup process: determining a count of the number of marked logs in the candidate list; comparing the count of the marked logs to the migration threshold; as well as In response to determining that the count of the marked logs is below the migration threshold, copying the contents of the candidate list to the transfer list, The candidate list and the transmission list are both stored in the first log group.

12. The method of claim 11, further comprising: In response to determining that the at least one log is modified during the second backup process and marked in the transfer list, the at least one log is migrated to the second log grouping.

13. The method of claim 11, further comprising: In response to determining that the at least one log is modified during the second backup process and marked in the transfer list: Determine if your migration budget is currently exceeded; as well as In response to determining that the migration budget is not currently exceeded, the at least one log is migrated.

14. The method of claim 13, further comprising: For each log migration performed in the deduplication storage system, reducing the migration budget by a first amount; as well as For each container index write performed that is not caused by log migration, increase the migration budget by a second amount, Wherein, the first amount is greater than the second amount.

15. A non-transitory machine-readable medium having stored thereon instructions that, when executed, cause a processor to: receiving a plurality of data streams to be stored in a persistent storage device of a deduplication storage system to perform a first backup process of the plurality of data streams, wherein: The first backup process of the plurality of data streams causes a change in data in a container index; identifying a set of logs in a first log grouping that were modified during the first backup process, wherein the first log grouping includes a candidate list and a transfer list, the candidate list and the transfer list including an ordered array of tags, each tag corresponding to a different log in the first log grouping, each log including information indicating changes to data stored in a container index associated with the log, and each container index including information identifying the log and log grouping associated with the container index; determining a count of logs modified during the first backup process; and At least one log in the set of logs marked in the transfer list is migrated to a second log grouping based at least on a comparison of the determined count and a migration threshold.

16. The non-transitory machine-readable medium of claim 15, comprising instructions that, when executed, cause the processor to: updating the candidate list to mark the set of logs modified during the first backup process; detecting initiation of a second backup process of the plurality of data streams; In response to detecting the initiation of the second backup process: determining a count of the number of marked logs in the candidate list; comparing the count of the marked logs to the migration threshold; as well as In response to determining that the count of the marked logs is below the migration threshold, copying the contents of the candidate list to the transfer list, The candidate list and the transmission list are both stored in the first log group.

17. The non-transitory machine-readable medium of claim 16, comprising instructions that, when executed, cause the processor to: In response to determining that the at least one log is modified during the second backup process and marked in the transfer list, the at least one log is migrated to the second log grouping.

18. The non-transitory machine-readable medium of claim 16, comprising instructions that, when executed, cause the processor to: In response to determining that the at least one log is modified during the second backup process and marked in the transfer list: Determine if the migration budget is currently exceeded; and In response to determining that the migration budget is not currently exceeded, the at least one log is migrated.

19. The non-transitory machine-readable medium of claim 18, comprising instructions that, when executed, cause the processor to: For each log migration performed, reducing the migration budget by a first amount; and For each container index write performed that is not caused by log migration, the migration budget is increased by a second amount.

Images