TREE STRUCTURE NODE DENSITY PRIORITIZATION

Abstract

A non-transitory, machine-readable storage medium (500) comprising instructions that, when executed, cause a system to: arranging (502) data in a plurality of nodes in a tree structure, the plurality of nodes comprising higher-level nodes and lower-level nodes, a given node of the plurality of nodes being associated with a probabilistic data structure that provides a probabilistic indication of whether data is present in the given node; for reads of the given node, determine (504) a false positive measure of false positives indicated by the probabilistic data structure, wherein a false positive occurs when the probabilistic data structure indicates that data is present in the given node when the data is not present in the given node; and prioritize (506) compaction of the given node based on the determined false positive measure, wherein compaction of the given node comprises transferring data in the given node to another node in the tree structure.

Description

background

[0001] Data reduction techniques can be applied to reduce the amount of data stored in a storage system. One example of a data reduction technique is data deduplication. Data deduplication identifies duplicate data units and attempts to reduce or eliminate the number of duplicate data units stored in the storage system.

[0002] US 2020 / 0 057 782 A1 refers to key-value stores with logarithmically structured merge trees (“LSM trees”).

[0003] US 2020 / 0 250 148 A1 refers to key-value stores with log-structured file systems.

[0004] US 2020 / 0 175 070 A1 describes the use of flash memory (or other secondary storage media), RAM-based data structures and mechanisms for accessing key-value pairs stored in flash memory using only a small amount of RAM space.

[0005] US 2015 / 0 310 053 A1 describes a method and apparatus for generating and storing a secondary index. Short description

[0006] A non-transitory, machine-readable storage medium according to claims 1 to 14, a computer according to claims 15 to 18 and a method according to claims 19 and 20 are disclosed. Brief description of the drawings

[0007] Some embodiments of the present disclosure are described with reference to the following figures. Fig. is a block diagram of an arrangement including a memory system according to some examples. Fig. is a block diagram of a key-value index according to some examples. Fig. illustrate an example of compacting an indirect node in a key-value index according to some examples. Fig. is a flowchart of a compaction process according to some examples. Fig. is a block diagram of a storage medium that stores machine-readable instructions according to some examples. Fig. is a block diagram of a system according to some examples. Fig. is a flowchart of a method according to further examples.

[0008] In the drawings, identical reference numbers indicate similar, but not necessarily identical, elements. The illustrations are not necessarily to scale, and the size of some parts may be exaggerated to clarify the example shown. Furthermore, the drawings contain examples and / or embodiments consistent with the description; however, the description is not limited to the examples and / or embodiments shown in the drawings. Detailed description

[0009] In this disclosure, the use of the term "a," "a" or "which" includes the plural forms unless the context clearly indicates otherwise. Likewise, the term "including," "contains," "comprises," or "has," when used in this disclosure, specifies the presence of the specified elements but does not preclude the presence or addition of other elements.

[0010] In some examples, storage systems use indexes to indicate relationships or mappings between keys and values ​​(also referred to herein as "key-value pairs"). A key-value index containing key-value pairs can be used to perform data deduplication in a storage system based on "fingerprints" of incoming data units, where each fingerprint is associated with a specific data unit. Such a key-value index is also referred to as a "fingerprint index." More generally, a "fingerprint index" can refer to an index that maps the fingerprints of data units to the respective location information of the data units.

[0011] A "storage system" may include a collection of storage devices (a single storage device or multiple storage devices). A storage system may also include one or more storage controllers that manage access to the storage device(s). A "unit of data" may refer to any piece of data that can be separately identified in the storage system. In some cases, a unit of data may refer to a block, a collection of blocks, or another piece of data. In some examples, a storage system may store units of data in persistent storage. Persistent storage may be achieved using one or more persistent (e.g., non-volatile) storage devices, such as disk-based storage devices (e.g., hard disk drives (HDDs)), solid-state devices (SSDs), such as flash memory devices, or the like, or a combination thereof.

[0012] A "controller" may refer to a hardware processing circuit, which may include any one or a combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or other hardware processing circuit. Alternatively, a "controller" may refer to a combination of hardware processing circuitry and machine-readable instructions (software and / or firmware) executable on the hardware processing circuitry.

[0013] A "fingerprint" refers to a value derived by applying a function to the contents of a piece of data (where the "contents" may include all or a subset of the contents of the piece of data). An example of a function that can be applied is a hash function, which generates a hash value based on the incoming piece of data. Examples of hash functions include cryptographic hash functions such as Secure Hash Algorithm 2 (SHA-2), e.g., SHA-224, SHA-256, SHA-384, etc. Other types of hash functions or other types of fingerprint functions may also be used in other examples.

[0014] In this discussion, a "fingerprint" can refer to a full fingerprint or a partial fingerprint. A full fingerprint includes the entire value calculated by the fingerprint function based on the contents of a data unit. A partial fingerprint includes a portion (less than all bits) of the full fingerprint.

[0015] A fingerprint of an incoming data unit is compared to a fingerprint index, which can be a key-value index where fingerprints are the keys and the corresponding data locations are the values. A "location indicator" can refer to any piece of information that provides a clue to the location of a data unit in persistent storage. A "location indicator" can also be referred to as "location information" of a data unit.

[0016] In some examples, a memory location indicator may be in the form of a sequential block number (SBN). An SBN can be used to identify where a unit of data is physically stored in persistent storage. However, in some examples, the SBN does not identify the physical memory location; instead, the SBN can be used to derive a physical address or other value that identifies a physical memory location. In other examples, other types of memory location indicators may also be used.

[0017] During a data deduplication operation performed on an incoming data unit at a storage system, a match between a fingerprint generated for the incoming data unit and a fingerprint stored in the fingerprint index indicates that the incoming data unit may be a duplicate of a data unit already stored in the storage system. If the incoming data unit is a duplicate of a previously stored data unit, instead of storing the duplicate incoming data unit, a reference counter stored in the storage system may be incremented to indicate the number of received instances of the data unit, and a reference to the data unit is stored in the storage system in place of the incoming data unit.

[0018] A key-value index can be stored as a tree structure of nodes, starting with a root node that has subnodes. The subnodes of a root node can include internal nodes (referred to here as "indirect" nodes) or leaf nodes. Each indirect node can also have subnodes, which can include child indirect nodes or leaf nodes.

[0019] Each leaf node of a key-value index stores key-value data (e.g., key-value pairs). In some examples, a particular leaf node stores key-value pairs for keys within a specific range (e.g., fingerprint values). For example, a first leaf node stores key-value pairs with keys in a first range, a second leaf node stores key-value pairs with keys in a different second range, and so on.

[0020] Each root node and each indirect node of the key-value index may contain a node buffer, which is a storage structure (e.g., in memory). A "memory" may be implemented with non-volatile random-access memory (NVRAM) and / or volatile memory. NVRAM may be implemented with electrically erasable programmable read-only memory (EEPROM). In other examples, NVRAM may be implemented with battery-backed dynamic random-access memory (DRAM) or battery-backed static random-access memory (SRAM). Volatile memory may be implemented with DRAM devices, SRAM devices, or any other type of memory where the data stored in the memory is lost when the memory's power supply is interrupted.

[0021] The node buffer of the root node or the indirect node of a key-value index can store key-value pairs. The node buffer of a root node can store key-value pairs for the entire possible range of key values ​​(e.g., all possible values ​​of fingerprints). The node buffer of a specific indirect node stores key-value pairs belonging to a specific range of keys. The node buffers of different indirect nodes store key-value pairs for different key ranges. In other words, each of the indirect nodes can be connected to a different part of the key range in the node buffer of the root node.

[0022] The node buffer of a root node or an indirect node stores the respective key-value pairs until a node buffer fullness threshold is reached, which may result in the stored key-value pairs in the node buffer being propagated to subnodes (propagating key-value pairs in the key-value index tree structure involves a gradual update of the key-value pairs in the tree structure). The "fullness threshold" may refer to a threshold of the used node buffer space consumed by storing data (e.g., key-value pairs) (referred to as "node buffer utilization"). The used space threshold may be defined as a percentage threshold of the total node buffer space or as a threshold of the size of the node buffer consumed for storing data.

[0023] The transfer of key-value pairs in the tree structure occurs as part of a compression of the nodes of the key-value index. When the node buffer of a node of the key-value index is compressed, the key-value pairs in the node buffer are distributed among the subnodes according to the key range of each subnode.

[0024] In some examples, each root node or indirect node of the key-value index may contain a probabilistic data structure and fence pointers (fence pointers are discussed further below). In some examples, the probabilistic data structure includes a Bloom filter. A Bloom filter is an example of a probabilistic data structure used to check whether an element is a member of a set, which in some examples of the present disclosure means checking whether a fingerprint computed for a piece of data is contained in a node buffer of a node of the key-value index. A Bloom filter may return a false positive match, incorrectly indicating that an element is a member of the set (i.e., the Bloom filter indicates that the element is a member of the set, when in fact, the element is not a member of the set).In the context of a fingerprint index, a Bloom filter may falsely indicate that a fingerprint is present in the node buffer of a particular node of the fingerprint index, even though the fingerprint is not actually present in the node buffer of that particular node. More generally, the Bloom filter may falsely indicate that data is present at a particular node of a tree structure, even though the data is not present at that given node.

[0025] However, a Bloom filter does not produce false negatives. In other words, the Bloom filter can either produce the result that the data might be located in a node of the tree structure, or the data definitely isn't located in that node of the tree structure.

[0026] When the nodes (including indirect nodes) of a key-value index fill with key-value pairs and exceed a node fill level threshold, compaction processes can be triggered to compact the nodes that have filled beyond the fill level threshold. Note that the fill level threshold can be the same or different for each node buffer in the key-value index.

[0027] A problem related to reading the key-value index is read amplification. If a Bloom filter of a given node's key-value index returns a positive result in response to a search of the given node for a fingerprint computed based on an incoming data unit, a memory controller can read the contents of the given node's node buffer into memory. Note that in some cases, the contents of the node buffer loaded into memory include a portion of the given node's node buffer. The portion of the node buffer read into memory may be a portion smaller than the entire contents of the node buffer, or the portion may be the entire contents of the node buffer. Reading a portion of the node buffer smaller than the entire contents of the node buffer relies on the use of fence pointers, which are explained below.The memory controller then searches the portion of the node buffer loaded into memory for a key (fingerprint) that matches the key (fingerprint) of the incoming data unit. If a matching key (fingerprint) is indeed present in the key-value index, this means that the incoming data unit is a duplicate of another data unit previously stored in the memory system, so the incoming data unit does not need to be stored again.

[0028] If the positive result returned by the Bloom filter for a given node turns out to be a false positive, then the portion of the given node's node buffer read into memory would not contain the key (fingerprint) that matches the key (fingerprint) of the incoming data unit. The search continues down to the leaf nodes. If the memory controller confirms that the key (fingerprint) is not contained in the key-value index, the memory controller inserts the fingerprint of the incoming data unit into the key-value index and stores the incoming data unit in permanent memory. This false positive would result in unnecessary reading of data from the node's node buffer into memory, consuming memory resources and consuming the storage system's bandwidth.A high number of false positives when performing key-value index lookups can lead to read amplification, where many reads of data from key-value index nodes into memory do not yield productive data useful for data deduplication. Therefore, it is desirable to reduce read amplification associated with key-value index lookups.

[0029] The above statements refer to examples where querying the key-value index is associated with writing incoming data units. In other examples, queries against the key-value index can be performed as part of read operations to search for the corresponding fingerprints. False positives can also be associated with such searches.

[0030] To reduce false positives, compaction of the indirect nodes of the key-value index can be performed to reduce the amount of data stored in the respective node buffers. The more data a given node buffer contains (e.g., a Bloom filter for the given node buffer represents a larger set of fingerprints in the given node buffer), the more likely it is that the Bloom filter will return a false positive in response to a search for a fingerprint. Reducing the amount of data stored in the given node buffer reduces the probability that a Bloom filter of the respective node will return a false positive in response to a search for the respective node for an incoming data unit. As mentioned earlier, compaction of a node refers to the distribution of the node's key-value pairs among the node's subnodes.

[0031] However, compacting node buffers of key-value index nodes consumes processing and storage resources because data is moved from a higher-level node to a lower-level node. With a large number of compaction operations to be applied to multiple key-value index nodes, competition for processing and storage resources may cause some compaction operations to be delayed while others are performed.

[0032] In accordance with some implementations of the present disclosure, prioritizing the compaction of nodes of a key-value index may be based on observed false positive measures associated with reading the respective nodes of the key-value index. Prioritizing the compaction of nodes based on observed false positive measures may include one or both of the following steps: 1) prioritizing the selection of a node of the key-value index for compaction based on observed false positive measures (among multiple factors in some examples) of the nodes, or 2) once compaction jobs have been selected for corresponding nodes, prioritizing the compaction jobs based on observed false positive measures (among multiple factors in some examples) of the corresponding nodes.For example, for nodes of the key-value index, one node (out of the multiple nodes) selected for compaction may be the one with a higher priority, e.g., based on a higher observed false positive measure and possibly one or more other factors. In some examples, the nodes considered for compaction are those with a node buffer utilization that exceeds the threshold, and prioritizing the selection of nodes for compaction based on the observed false positive measures may be for those nodes with a node buffer utilization that exceeds the threshold. In other examples, all nodes of the key-value index are considered for compaction, and the selection of all such nodes for compaction may be based on the nodes' observed false positive measures.

[0033] As another example, once nodes have been identified and scheduled for compaction because their node buffer utilization has exceeded the threshold, the priorities of the compaction jobs for the nodes can be assigned based on the observed false positive measures (and possibly other factors). For example, a first compaction job to compact a first node of the key-value index will be assigned a higher priority than a second compaction job to compact a second node of the key-value index if a false positive on lookups of the first node is greater than a false positive on lookups of the second node.

[0034] Fig. shows an example of a memory system 102 that includes a memory 104 (a volatile memory and / or NVRAM) and a persistent memory 107. Although in Fig. While a particular arrangement of components is illustrated, it should be understood that the storage system 102 may include a different arrangement of components in other examples.

[0035] The storage system 102 also includes a storage controller 106, which includes various modules, including a deduplication engine 108, an update engine 110, a merge engine 112, and a compaction engine 150. Although in the example of Fig. While specific engines are depicted, in other examples, storage controller 103 may include fewer or more engines. Note that storage controller 106 may be part of storage system 102 or may be separate from storage system 102 (e.g., connected to storage system 102 via a network).

[0036] Each engine may refer to a portion of hardware processing circuitry of the storage controller 106, or alternatively, to machine-readable instructions (software and / or firmware stored on at least one machine-readable storage medium) that may be executed by the hardware processing circuitry of the storage controller 106. In other examples, some of the engines may also be separate from the storage controller 106.

[0037] When incoming data units 114 (from write requests) are received by the storage system 102, fingerprint index updates 109 may be created for the incoming data units 114. The write requests may be received from a requester device (or multiple requesters) connected to the storage system 102 via a network, such as a local area network (LAN), a wide area network (WAN), a storage area network (SAN), etc. A requester device may refer to a server computer, a desktop computer, a notebook computer, a tablet computer, a smartphone, or another type of electronic device.

[0038] After data deduplication performed by the deduplication engine 108, the data units 117 of the write requests may be written to a data store 116 in the persistent storage 107. Due to the application of data deduplication, the data units 117 stored in the data store 116 are unique data units (i.e., the data units 117 do not contain duplicate data units). In further examples, the data deduplication applied by the deduplication engine 108 reduces (but does not eliminate) the occurrence of duplicate data units stored in the data store 116.

[0039] A “fingerprint index update” generated by the deduplication engine 108 in response to an incoming data unit 114 may contain update information for a fingerprint index (in the form of a Fig. illustrated key-value index 120). The key-value index 120 comprises a hierarchical arrangement of nodes in a tree structure and is stored in the persistent store 107. In one example, a fingerprint index update 109 for an incoming data unit 114 may include a fingerprint (e.g., a hash value) calculated by applying a fingerprint function (e.g., a hash function) to the incoming data unit 114 and a location indicator (e.g., an SBN) for the incoming data unit 114.

[0040] The update engine 110 receives fingerprint index updates 109 from the deduplication engine 108. The update engine 110 temporarily stores each fingerprint index update 109 in an update buffer 118 in the memory 104. Although in Fig. While a single update buffer 118 is shown, multiple buffers may be used to store fingerprint index updates 109. For example, update buffer 118 may include an active update buffer and a synchronization buffer, where the active update buffer is used to receive fingerprint index updates 109, while the fingerprint index updates 109 in the synchronization buffer are merged (flushed) into key-value index 120 by merge engine 112. Merging a fingerprint index update 109 into key-value index 120 may refer to moving a key-value pair of fingerprint index update 109 from update buffer 118 into a corresponding node of key-value index 120.

[0041] The fingerprint index updates 109 are accumulated in the update buffer 118 before being merged (inserted) into nodes of the key-value index 120. The fingerprint index updates 109 are inserted by the merge engine 112 into the node buffers of the nodes closest to a root node of the key-value index 120.

[0042] In some examples, the compaction of nodes of the key-value index 120 is performed independently (asynchronously with respect to) each merge of fingerprint index updates 109 from the update buffer 118 to the key-value index 120.

[0043] Asynchronous compaction of nodes of key-value index 120 contrasts with synchronous compaction of nodes, which occurs when fingerprint index updates are merged into the nodes of key-value index 120. With synchronous compaction, when fingerprint index updates are merged into the nodes of key-value index 120, a compaction of nodes can be triggered to distribute the fingerprint index updates to lower-level nodes of key-value index 120 until the key-value pairs reach leaf nodes of key-value index 120.

[0044] During synchronous compaction, the compaction of nodes in the key-value index 120 may trigger further compaction of lower-level nodes as the lower-level nodes fill up. Such compaction is a recursive process in which the compaction of an initial collection of indirect nodes (including a single indirect node or multiple indirect nodes) of the key-value index 120 may trigger compaction in a lower-level collection of indirect nodes until the updates reach node buffers that can accommodate the key-value pairs of the fingerprint index updates without further compaction, or until the key-value pairs reach the leaf nodes of the key-value index 120. The synchronous compaction process, which is synchronous with (i.e.inline with) a merge operation may be inefficient in terms of using processing and storage resources of the storage system 102.

[0045] According to some implementations of the present disclosure, the merge processes and the compaction processes are decoupled, such that the compaction processes are deferred until resources are available in the storage system 102 to perform compaction. The compaction processes are performed independently of the merge processes.

[0046] In some examples, compaction of each indirect node can be performed in isolation to avoid recursive compaction; in other words, compaction of a higher-level indirect node does not trigger compaction of a lower-level indirect node because the key-value pairs of the higher-level indirect node are moved to the lower-level indirect node.

[0047] In some examples, a portion of the key-value index 120 may be retrieved into the memory 104 of the memory controller 106 to improve performance when accessing portions of the key-value index 120. In the present discussion, the term "key-value index 120" may refer to the key-value index 120 in the persistent store 107 and / or any portion of the key-value index 120 that has been copied into the memory 104 of the memory controller 106.

[0048] When a compaction criterion is met for a particular node of the key-value index 120 (e.g., when the node buffer utilization of the particular node exceeds the threshold), a compaction job may be inserted into a compaction work stack 122, which in some examples may be stored in memory 104. A "compaction work stack" may refer to a temporary storage structure (a "compaction job scheduling structure") used to store information about compaction jobs to be executed ("CJ" in Fig. ) is used. Inserting a compaction job into the compaction work batch 122 refers to inserting information about the compaction job into the compaction work batch 122. The compaction job information may identify a corresponding node of the key-value index 120 for which compaction is to be performed, as well as other information associated with the compaction job and / or the corresponding node.

[0049] The compaction work stack 122 contains a collection of compaction jobs ("CJs") to be scheduled for operation. A compaction job may be executed by the compaction engine 150, which compacts a corresponding node of the key-value index 120 by moving key-value pairs in the node buffer of the corresponding node to subnodes of the corresponding node.

[0050] The compaction engine 150 includes false positive measure determination logic 152 and node compaction prioritization logic 154. A "logic" of the compaction engine 150 may refer to a portion of the hardware processing of the compaction engine 150 or, alternatively, to machine-readable instructions (software and / or firmware stored on at least one machine-readable storage medium) that may be executed by the hardware processing circuitry of the compaction engine 150.

[0051] In some examples, false positive measure determination logic 152 determines a false positive measure for each node in key-value index 120. False positive measure determination logic 152 may count the number of false positives returned by a Bloom filter for a respective node during a particular time interval (e.g., a sampling time interval). Dividing this number of false positives by the time interval yields a false positive rate. In some examples, the false positive measure for the respective node determined by false positive measure determination logic 152 is the false positive rate. In other examples, the false positive measure may include the number of false positives. In other examples, other false positive measures may be used by false positive measure determination logic 152.

[0052] In some examples, node compaction prioritization logic 154 may assign priorities to the respective nodes associated with compaction jobs. Assigning a priority to a node associated with a compaction job may refer to assigning the priority to the compaction job. Fig. the priorities (“P”) assigned to the respective compaction jobs can be added to the compaction work stack 122 in association with the respective compaction jobs.

[0053] In some examples, node compaction prioritization logic 154 calculates a priority for a compaction job based on a corresponding false positive measure of false positives returned by a Bloom filter of a node associated with the compaction job. For example, the priority may be a value within a range that correlates proportionally to the false positive measure (e.g., a higher priority is assigned for a higher false positive measure). In other examples, node compaction prioritization logic 154 may calculate a priority for a compaction job based on the corresponding false positive measure as well as one or more other factors, such as the fullness of the corresponding node buffer, and possibly other factors.

[0054] In the latter examples, the priority assigned to a node associated with a compaction job may be a weighted sum of a false positive measure computed for the node and a node buffer fullness level (e.g., a percentage of the node buffer consumed, an amount of data stored in the node buffer, etc.), and possibly other factors.

[0055] Based on the priorities ("P") assigned to the respective compaction jobs ("CJ") in the compaction work batch 122, the compaction engine 150 may select a compaction job (from the compaction jobs in the compaction work batch 122) for execution. For example, the compaction engine 150 may compare the priorities of the compaction jobs in the compaction work batch 122 and select the compaction job with the highest priority for execution to compact a corresponding node of the key-value index 120.

[0056] The foregoing refers to examples in which the compaction jobs in the compaction work stack 122 are assigned priorities based on the observed false positive measures of the respective nodes. Alternatively or additionally, the node compaction prioritization logic 154 may prioritize a selection of a node of the key-value index 120 to be added to the compaction work stack 122 based on the observed false positive measures of the nodes in the key-value index 120. For example, for each node of the key-value index 120 (e.g.,a node with node buffer usage that exceeds the fullness threshold, or any node in the key-value index 120), the node may be assigned a priority based on the node's observed false positive measure (and possibly other factors such as the node's node buffer fullness) for use in selecting among multiple nodes to add to the compaction work stack 122. Selecting a node for addition to the compaction work stack 122 refers to adding a compaction job for the node to the compaction work stack 122.

[0057] An example of the key value index 120 is in Fig. In the example of Fig. the key-value index 120 contains a root node 202 which has various indirect child nodes 204-1, 204-2, 204-3, etc.

[0058] Each indirect node can in turn have further subnodes. In the example of Fig. the indirect node 204-1 has child indirect nodes 204-11, 204-12, 204-13, etc.

[0059] Each of the indirect nodes 204-11, 204-12, 204-13, etc. can in turn have further subnodes, e.g., leaf nodes.

[0060] In the example of Fig. The key-value index 120 has four hierarchical levels, with the highest level being the root node of 202 and the lowest level being the leaf nodes. The indirect nodes 204-11, 204-12, and 204-13 are at a lower level in the hierarchy than the indirect node 204-1.

[0061] A level of the key-value index 120 that is closer to the root node 202 is at a higher level than another level that is farther from the root node 202 in the hierarchy of the key-value index 120.

[0062] Each node of the key-value index, except leaf nodes, can contain the following elements: child pointers, fence pointers, a Bloom filter, and buffer blocks that together form a node buffer.

[0063] For example, the root node 202 contains subordinate pointers 202-CP, fence pointers 202-FP, a Bloom filter 202-BF, and buffer blocks 202-BC. For each of the Fig. For the indirect nodes 204-1, 204-2, 204-3, 204-11, 204-12 and 204-13 shown, the child pointers are identified by the suffix "-CP" after the reference number of the indirect node (e.g., 204-1-CP), the fence pointers by the suffix "-FP" after the reference number of the indirect node (e.g., 204-1-FP), the Bloom filter by the suffix "-BF" after the reference number of the indirect node (e.g., 204-1-BF) and the buffer blocks by the suffix "-BC" after the reference number of the indirect node (e.g., 204-1-BC).

[0064] Child pointers are values ​​that point to or otherwise reference immediate child nodes of the respective non-leaf node (a root node or an indirect node) in the key-value index 120. For example, the child pointers 202-CP of the root node 202 point to the indirect nodes 204-1, 204-2, 204-3, etc., which are immediate child nodes of the root node 202. Similarly, the child pointers 204-1-CP of the indirect node 204-1 point to the indirect nodes 204-11, 204-12, 204-13, etc., which are immediate child nodes of the indirect node 204-1.

[0065] In some examples, the child pointers of an indirect node may be created when the indirect node is first compacted (e.g., when an amount of data in the node buffer of the indirect node exceeds a fullness threshold) and the compaction job for the indirect node has been selected for execution by the compaction engine 150.

[0066] In some examples, the child pointers and / or the fence pointers and / or the Bloom filters of the root node 202 and the indirect nodes of the key-value index 120 may be copied to the memory 104 of the memory controller 106 to enable faster access.

[0067] The Bloom filter of a non-leaf node is used to determine whether a key (a fingerprint of an incoming data unit 114) is contained in a corresponding node buffer of the non-leaf node (with the possibility of false positives). The Bloom filter indicates which keys are not contained in the respective node buffer and indicates keys that might be contained in the node buffer, with the possibility that such an indication is a false positive.

[0068] If the Bloom filter indicates that a particular key generated for an incoming data unit 114 is contained in a node buffer of a particular indirect node, a portion of the node buffer is retrieved into memory 104 ( Fig. ), and the deduplication engine 108 may search the portion of the node buffer retrieved into the node buffer of the memory 104 to find the particular key. In some examples, the fence pointers of the particular indirect node may be used to identify a particular buffer block likely to store key-value pairs containing the given key. In some examples, a fence pointer may identify the lowest key and / or the highest key of each buffer block. For example, each fence pointer may identify a lower bound (and / or an upper bound) of keys contained in the corresponding buffer block (in other words, each fence pointer indicates a key range for a buffer block, such that key-value pairs with keys in the key range are stored in the buffer block).The fence pointers can be used to determine which buffer block contains a key range into which the respective key falls. Accordingly, instead of loading the entire contents of the node buffer for the particular indirect node into memory 104, only the identified buffer block is loaded into memory 104, reducing the amount of data read into memory 104 and thus reducing the memory resource consumption and the consumption of persistent storage associated with key-index node lookups.

[0069] Fig. show an exemplary compaction process 300 performed with respect to the indirect node 204-1. Fig. shows part of the key value index 120. Fig. shows a state of the key-value index 120 before the compaction of the indirect node 204-1, and Fig. shows a state of the key-value index (represented as "120") after the compaction process 300, in which the indirect node 204-1 was compacted.

[0070] It is assumed that the node buffer of the indirect node 204-1 (consisting of four buffer blocks 204-1-BC) is filled to the extent that the threshold is exceeded. It is also assumed that the compaction job for the indirect node 204-1 was selected by the compaction engine 150 based on priorities assigned to the corresponding nodes associated with compaction jobs by the node compaction prioritization logic 154 ( Fig. ).

[0071] The compaction process 300 includes the transfer of key-value pairs from a parent node to one or more child nodes, in this case from the parent indirect node 204-1 to the child indirect nodes 204-11, 204-12, and 204-13. The arrows 302, 304, and 306 in Fig. show the transfer of key-value pairs from the node buffer of the parent indirect node 204-1 to the child indirect nodes 204-11, 204-12 and 204-13. In Fig. It is assumed that the subordinate indirect node 204-11 has two buffer blocks 204-11-BC before the compaction process 300, the subordinate node 204-12 has one buffer block 204-12-BC, and the subordinate indirect node 204-13 has two buffer blocks 204-13-BC.

[0072] After the compaction process 300, as in Fig. As shown, the child indirect node 204-11 has three buffer blocks 204-11-BC', the child indirect node 204-12 has three buffer blocks 204-12-BC', and the child indirect node 204-13 has three buffer blocks 204-13-BC'.

[0073] After the compaction process 300, no buffer blocks are allocated to the parent node 204-1 (i.e., the node buffer of the parent indirect node 204-1 is empty). The node buffer of the indirect node 204-1 can then be refilled when the storage system 102 receives additional fingerprint index updates.

[0074] Fig. is a flowchart of a process 400 performed by the compaction engine 150 according to some examples. The false positive measure determination logic 152 collects (at 402) over time the number of false positives returned by Bloom filters of the respective indirect nodes of the key-value index 120. For example, the false positive counts may be tracked by the false positive measure determination logic 152 during a sampling interval. After the sampling interval, the false positive measure determination logic 152 may calculate the false positive measure for each indirect node of the key-value index 120 (at 404) by dividing the count by the time duration of the sampling interval.

[0075] Note that the observed false positive measures may be running measures that are iteratively updated at successive sampling intervals.

[0076] Once the false positive measures have been calculated by the false positive measure determination logic 152, the node compaction prioritization logic 154 determines (at 406) priorities for compacting nodes of the key-value index 120 based on factors, including the observed false positive measure. Determining priorities for compacting nodes based on the observed false positive measures may include one or both of the following steps: 1) determining priorities for selecting nodes of the key-value index 120 for compaction based on observed false positive measures (among multiple factors in some examples) of the nodes, or 2) once compaction jobs are identified for corresponding nodes, prioritizing the compaction jobs based on observed false positive measures (among multiple factors in some examples) of the corresponding nodes.

[0077] Based on the assigned priorities, the compaction engine 150 selects (at 408) a node with the highest priority for compaction. In some examples, the selection (at 408) includes selecting a node to add to the compaction work stack 122 based on a comparison of the priorities assigned to the nodes in the key-value index 120. In other examples, the selection (at 408) includes selecting a compaction job from the compaction work stack 122 to execute based on the priorities assigned to the compaction jobs in the compaction work stack 122.

[0078] After moving a compaction job for a node into the compaction work stack 122 or after completing a compaction job selected from the compaction work stack 122, the compaction engine 150 determines (at 410) whether there are more nodes of the key-value index 120 eligible for addition to the compaction work stack 122 or whether there are more compaction jobs to be selected (from the compaction work stack 122). In response to determining that there are more nodes or compaction jobs to be selected, the compaction engine 150 repeats tasks 406, 408, and 410. The process continues for the remaining nodes or compaction jobs. If additional compaction jobs are triggered, the compaction jobs and their respective priorities may be added to the compaction work stack 122.Nodes may also be added that are eligible for selection for inclusion in the compaction work stack 122, for example, when a new node is added to the key-value index 120 or a node has a node buffer usage that exceeds the fullness threshold.

[0079] Fig. is a block diagram of a non-transitory machine-readable or computer-readable storage medium 500 that stores machine-readable instructions that, when executed, cause a system to perform various tasks. The system may include one or more computers, such as the memory controller 106 of Fig. .

[0080] The machine-readable instructions include tree-arranging instructions (502) for arranging data in a plurality of nodes in a tree structure (e.g., the key-value index 120 tree structure of Fig. ). The plurality of nodes includes higher-level nodes and lower-level nodes, wherein a given node of the plurality of nodes is associated with a probabilistic data structure (e.g., a Bloom filter) that provides a probabilistic indication of whether data is present at the given node.

[0081] The machine-readable instructions include instructions 504 for determining the false positive measure (e.g., instructions of the false positive measure determination logic of Fig. ) to determine a false positive measure of false positives indicated by the probabilistic data structure for reads of the given node, where a false positive occurs when the probabilistic data structure indicates that data is present in the given node when the data is not present in the given node.

[0082] The machine-readable instructions include compaction prioritization instructions 506 (e.g., instructions of the node compaction prioritization logic 154 of Fig. ) to prioritize compaction of the given node based on the determined false positive measure, wherein compaction of the given node comprises transferring data in the given node to another node in the tree structure.

[0083] In some examples, prioritizing compaction of the given node includes setting a higher priority for compaction of the given node in response to a higher false positive measure and setting a lower priority for compaction of the given node in response to a lower false positive measure. In this way, a node with a higher false positive measure may be selected with higher priority for compaction, after which the false positive values ​​for the node should be reduced.

[0084] In some examples, the data in the plurality of nodes in the tree structure includes key-value pairs, where each of the key-value pairs includes a fingerprint of a data unit and location information (e.g., the location indicator described above) of the data unit. Compacting the key-value pairs of the given node moves the key-value pairs to the child node(s) of the given node, thereby reducing the amount of data in a node buffer of the given node and thus reducing false positives generated by the given node's probabilistic data structure.

[0085] In some examples, prioritizing compaction of the given node based on the determined false positive measure comprises setting a first priority for compaction of the given node, and the machine-readable instructions are executable to compare a plurality of priorities (including the first priority) set for respective ones of the plurality of nodes and select a node of the plurality of nodes based on the comparison of the plurality of priorities.

[0086] In some examples, the compaction job for a first node is assigned a higher priority than a priority assigned to a compaction job for a second node of the plurality of nodes in response to the false positive measure determined for the first node being higher than a false positive measure determined for the second node, even though the node buffer of the second node contains more data than the node buffer of the first node. For example, although the second node may have more key-value pairs, such key-value pairs in the second node may be accessed less frequently than key-value pairs in the first node, and thus the second node would be associated with fewer false positives.

[0087] In further examples, prioritizing the compaction of the given node includes selecting the given node from the plurality of nodes for which a compaction job is to be added to a compaction job scheduling structure.

[0088] In some examples, each respective node of the plurality of nodes includes a respective node buffer for storing the respective node's respective data. The machine-readable instructions are executable to create a compaction job for the respective node responsive to an amount of data at the respective node exceeding a threshold (e.g., the above-mentioned fullness threshold). Prioritizing compaction of the given node includes prioritizing a compaction job created for the given node in response to an amount of data at the given node exceeding the threshold.

[0089] In some examples, machine-readable instructions are executable to retrieve information about compaction jobs in a compaction job scheduling structure (e.g., the compaction work stack 122 of Fig. ) and to schedule the compaction jobs based on the priorities assigned to the compaction jobs. By using the compaction job scheduling structure, the compaction jobs in the compaction job scheduling structure can be executed asynchronously with respect to each merge operation that requires fingerprint index updates in the update buffer 118 ( Fig. ) into nodes of the tree structure.

[0090] Fig. is a block diagram of a computer 600 (e.g., the memory controller 106 of Fig. ) that includes a hardware processor 602 (or multiple hardware processors). A hardware processor may include a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or other hardware processing circuitry.

[0091] Computer 600 includes a non-transitory storage medium 604 storing machine-readable instructions executable on hardware processor 602 to perform various tasks. Machine-readable instructions executable on a hardware processor may refer to instructions executable on a single hardware processor or on multiple hardware processors.

[0092] The machine-readable instructions in the storage medium 604 include index information creation instructions 606 to create index information for incoming data units to be stored in a storage system (e.g., the fingerprint index updates 109 of Fig. ) to insert them into an index tree structure (e.g. the key-value index 120 of Fig. ). The index tree structure comprises a plurality of nodes, including higher-level nodes and lower-level nodes, wherein a particular node of the plurality of nodes is associated with a probabilistic data structure that provides a probabilistic indication of whether data is present at the particular node.

[0093] The machine-readable instructions in storage medium 604 include compaction performance instructions 608 to perform compactions of selected nodes of the plurality of nodes asynchronously with the merging of the index information into the index tree structure.

[0094] The machine-readable instructions in storage medium 604 include false positive measure determination instructions 610 for determining a false positive measure of false positives indicated by the probabilistic data structure for reads of the given node, wherein a false positive occurs when the probabilistic data structure indicates that data is present in the given node when the data is not present in the given node.

[0095] The machine-readable instructions in the storage medium 604 include compaction prioritization instructions 612 to prioritize compaction of the given node based on the determined false positive measure, wherein compaction of the given node comprises transferring data in the given node to another node in the index tree structure.

[0096] The performance of the compactions depends on the prioritization of the compactions of the selected nodes.

[0097] Fig. is a flowchart of a process 700 according to some examples. The process 700 includes, for incoming data units to be stored in the storage system, generating (at 702) index information for merging into an index tree structure comprising a plurality of nodes with higher-level nodes and lower-level nodes, wherein each respective node of the plurality of nodes is associated with a respective probabilistic data structure providing a probabilistic indication of whether data is present at the respective node, and the index information includes fingerprints of the incoming data units and associated location information of the incoming data units.

[0098] The process 700 includes, for each respective node of the plurality of nodes, determining (at 704) a respective false positive measure of false positives indicated by the respective probabilistic data structure of the respective node, wherein a false positive occurs when the respective probabilistic data structure indicates that data is present in the respective node when the data is not present in the respective node.

[0099] Process 700 includes assigning (at 706) priorities to the respective nodes of the index tree structure based on the determined false positive measures. Assigning priorities to the respective nodes may refer to 1) assigning priorities for selecting nodes of the key-value index 120 for compaction based on the false positive measures (among several factors in some examples) of the nodes, or 2) assigning priorities to the compaction jobs in the compaction work stack 122 based on the observed false positive measures (among several factors in some examples) of the respective nodes.

[0100] The process 700 includes selecting (at 710) nodes of the index tree structure for compaction according to the assigned priorities.

[0101] A storage medium (e.g., 500 in Fig. in Fig.) may include any one or a combination of the following elements: a semiconductor storage device such as dynamic or static random access memory (DRAM or SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory; a magnetic disk such as a hard disk, floppy disk, and removable disk; other magnetic medium including tape; an optical medium such as a compact disk (CD) or digital video disk (DVD); or another type of storage device. It should be noted that the instructions described above may be provided on a single computer- or machine-readable storage medium, or alternatively, on multiple computer- or machine-readable storage media distributed throughout a large system, possibly with multiple nodes.Such computer-readable or machine-readable storage medium or media are considered part of an article (or article of manufacture). An article or article of manufacture may refer to each manufactured component or multiple components. The storage medium or media may be located either in the machine on which the machine-readable instructions are executed or at a remote location from which machine-readable instructions can be downloaded over a network for execution.

[0102] In the foregoing description, numerous details are set forth to provide an understanding of the subject matter disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and variations from the details described above. The appended claims are intended to cover such modifications and variations.

Claims

[1] A non-transitory, machine-readable storage medium (500) comprising instructions that, when executed, cause a system to: arranging (502) data in a plurality of nodes in a tree structure, the plurality of nodes comprising higher-level nodes and lower-level nodes, a given node of the plurality of nodes being associated with a probabilistic data structure that provides a probabilistic indication of whether data is present in the given node; for reads of the given node, determine (504) a false positive measure of false positives indicated by the probabilistic data structure, wherein a false positive occurs when the probabilistic data structure indicates that data is present in the given node when the data is not present in the given node; and prioritize (506) compaction of the given node based on the determined false positive measure, wherein compaction of the given node comprises transferring data in the given node to another node in the tree structure. [2] The non-transitory, machine-readable storage medium (500) of claim 1, wherein the probabilistic data structure comprises a Bloom filter (202-BF). [3] The non-transitory machine-readable storage medium (500) of claim 1, wherein prioritizing compaction of the given node comprises: Setting a higher priority for compacting the given node in response to a higher false positive measure, and Setting a lower priority for compaction of the given node in response to a lower false positive measure. [4] The non-transitory machine-readable storage medium (500) of claim 1, wherein compacting the given node comprises transferring the data in the given node to at least one subnode of the given node. [5] The non-transitory machine-readable storage medium (500) of claim 1, wherein the data in the plurality of nodes in the tree structure comprises key-value pairs, each key-value pair of the key-value pairs comprising a fingerprint of a data unit and location information of the data unit. [6] The non-transitory, machine-readable storage medium (500) of claim 1, wherein prioritizing compaction of the given node based on the determined false positive measure comprises setting a first priority for compaction of the given node, and wherein the instructions, when executed, cause the system to: compare a plurality of priorities set for the respective nodes of the plurality of nodes, the plurality of priorities including the first priority; and select a node of the plurality of nodes based on comparing the plurality of priorities. [7] The non-transitory machine-readable storage medium (500) of claim 1, wherein each respective node of the plurality of nodes comprises a respective node buffer for storing respective data of the respective node, and wherein the instructions, when executed, cause the system to: to generate a compaction job for the respective node in response to a data volume in the respective node exceeding a threshold, wherein prioritizing compaction of the given node comprises prioritizing a compaction job generated for the given node, the compaction job generated for the given node being generated in response to an amount of data in the given node exceeding the threshold. [8] The non-transitory machine-readable storage medium (500) of claim 7, wherein the instructions, when executed, cause the system to: to store information about consolidation orders in a consolidation order planning structure; and to plan the compaction jobs based on priorities assigned to the compaction jobs. [9] The non-transitory, machine-readable storage medium (500) of claim 7, wherein the instructions, when executed, cause the system to: assigning the compaction job for the given node a higher priority than a priority assigned to a compaction job for another node of the plurality of nodes in response to the false positive measure determined for the given node being higher than a false positive measure determined for the another node. [10] The non-transitory machine-readable storage medium (500) of claim 7, wherein the instructions, when executed, cause the system to: assigning the compaction job for the given node a higher priority than a priority assigned to a compaction job for another node of the plurality of nodes in response to the false positive measure determined for the given node being higher than a false positive measure determined for the another node, even if the node buffer of the another node has more data than the node buffer of the given node. [11] The non-transitory machine-readable storage medium (500) of claim 7, wherein the instructions, when executed, cause the system to: in response to a read request from the given node: to determine whether the probabilistic data structure returns a positive indication that probabilistically indicates that data for the read request is present in the node buffer of the given node, and in response to the positive indication, to read a portion of the data in the node buffer of the given node into a memory. [12] The non-transitory machine-readable storage medium (500) of claim 11, wherein if the positive indication is a false positive, the portion of the data read into the memory does not include data for the read request. [13] The non-transitory machine-readable storage medium (500) of claim 1, wherein prioritizing compaction of the given node comprises selecting the given node from the plurality of nodes for which a compaction job is to be added to a compaction job scheduling structure. [14] The non-transitory machine-readable storage medium (500) of claim 1, wherein the data in the plurality of nodes in the tree structure comprises fingerprints of data units and associated location information of the data units, and wherein the instructions, when executed, cause the system to: Merge updates for incoming data units (114) into the tree structure, wherein the compactions of nodes of the plurality of nodes are performed independently of any merging of updates for incoming data units (114) into the tree structure. [15] Computer (600) comprising: a processor (602); and a non-transitory storage medium (604) comprising instructions executable on the processor (602) to: generating index information for incoming data units (114) to be stored in a storage system (102) to aggregate them into an index tree structure comprising a plurality of nodes including higher-level nodes and lower-level nodes, wherein a given node of the plurality of nodes is associated with a probabilistic data structure providing a probabilistic indication of whether data is present in the given node; Perform compactions of selected nodes of the plurality of nodes asynchronously with the merging of the index information into the index tree structure; for reads of the given node, determine a false positive measure of false positives specified by the probabilistic data structure, where a false positive occurs when the probabilistic data structure specifies that data is present in the given node when the data is not present in the given node; and prioritize a compaction of the given node based on the determined false positive measure, wherein the compaction of the given node comprises transferring data in the given node to another node in the index tree structure, whereby the compactions are performed according to the prioritization of the compactions of the selected nodes. [16] The computer (600) of claim 15, wherein the instructions are executable on the processor (602) to: calculate the false positive measure based on the number of false positives in a time interval associated with the reads of the given node. [17] The computer (600) of claim 15, wherein the index information comprises fingerprints of the incoming data units (114) and associated location information of the incoming data units (114), and wherein the compaction of the given node transmits fingerprints and associated location information in a node buffer of the given node to subnodes of the given node. [18] The computer (600) of claim 15, wherein the instructions are executable on the processor (602) to: assigning the given node a higher priority than a priority assigned to another node of the plurality of nodes in response to the false positive measure determined for the given node being higher than a false positive measure determined for the another node, even if a node buffer of the another node has more data than a node buffer of the given node. [19] A method (700) performed by a memory controller (106) of a memory system (102) comprising: for incoming data units (114) to be stored in the storage system (102), generating (702) index information for merging into an index tree structure comprising a plurality of nodes including higher-level nodes and lower-level nodes, each respective node of the plurality of nodes being associated with a respective probabilistic data structure providing a probabilistic indication of whether data is present in the respective node; for each corresponding node of the plurality of nodes, determining (704), for reads of the corresponding node, a corresponding false positive measure of false positives indicated by the respective probabilistic data structure of the corresponding node, wherein a false positive occurs when the respective probabilistic data structure indicates that data is present in the corresponding node when the data is not present in the corresponding node; Assigning (706) priorities to respective nodes of the index tree structure based on the determined false positive measure; and Selecting (708) nodes of the index tree structure for compaction according to the assigned priorities, wherein compaction of a node comprises transferring data in the node to a lower node in the index tree structure. [20] The method (700) of claim 19, comprising: Generating a compaction job for a given node of the selected nodes in response to a node buffer of the given node being filled with index information exceeding a threshold.

Images