Three-dimensional probabilistic data structures

Abstract

Technology related to probabilistic data structures is disclosed. A database node can maintain a probabilistic data structure capable of encoding a database key. The probabilistic data structure can include multiple levels, each level capable of storing indicators of transitions between consecutive characters in the database key. The database node can insert a specific database key into the probabilistic data structure, and the specific database key can include a series of characters. The insertion operation can include, for each transition between consecutive characters in the series of characters, setting an indicator indicating the transition in a corresponding level in the multiple levels. The database node can also maintain pedigree information specifying one or more pedigrees corresponding to the transition.

Description

Background Art Technical Field

[0001] The present invention relates generally to database systems and, more particularly, to techniques for implementing probabilistic data structures.

[0002] Description of Related Technology

[0003] Modern database systems generally implement management systems that enable users to store a set of information in an organized manner and to efficiently access and manipulate the set of information. In some cases, these management systems maintain a log-structured merge tree (LSM tree) having multiple levels, each of which stores information as a "key-value" pair. The LSM tree typically includes two high-level parts: an in-memory cache and a permanent memory. In operation, a database system includes a database node that initially writes database records to its in-memory cache and then dumps and clears the database records to permanent memory. The database node can receive a request for database records associated with a specific database key. In many cases, database records associated with a specific database key are not stored in the database in-memory cache. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 is a block diagram illustrating exemplary elements of a system having a database node including a probabilistic data structure according to some embodiments.

[0005] Figure 2 is a block diagram illustrating example elements of a database key according to some embodiments.

[0006] Figures 3A to 3C is a block diagram illustrating exemplary elements of a probabilistic data structure having multiple levels including a bitmap according to some embodiments.

[0007] Figure 4 is a block diagram illustrating exemplary elements of pedigree information with a pedigree according to some embodiments.

[0008] Figure 5 is a block diagram illustrating exemplary elements of interaction between database nodes having a probabilistic data structure according to some embodiments.

[0009] Figure 6 and Figure 7 is a flow chart illustrating an exemplary method involving maintaining a probabilistic data structure according to some embodiments.

[0010] Figure 8 is a flow chart illustrating an exemplary method involving performing operations to look up a database key in a probabilistic data structure according to some embodiments.

[0011] Fig. 9 is a block diagram illustrating an exemplary computer system according to some embodiments.

[0012] The disclosure includes references to "one embodiment" or "an embodiment." The appearances of the phrase "in one embodiment" or "in an embodiment" do not necessarily refer to the same embodiment. The particular features, structures, or characteristics may be combined in any suitable manner consistent with the disclosure.

[0013] In the disclosure, different entities (which may be referred to by different names, such as "units", "circuits", other components, etc.) may be described or qualified as being "configured to" perform one or more tasks or operations. This fixed expression - [the entity] is configured to [perform one or more tasks] - refers to a structure (i.e., a tangible object, such as an electronic circuit) in this document. More specifically, the fixed expression is used to indicate that the structure is set to perform one or more tasks during operation. Even if the structure is not currently being operated, it can be said that the structure is "configured to" perform certain tasks. "A network interface configured to communicate over a network" is intended to cover, for example, an integrated circuit having circuits that perform this function during operation, even if the integrated circuit is not currently in use (e.g., not connected to a power source). Therefore, an entity described or described as being "configured to" perform a certain task refers to some physical entity, such as a device, a circuit, a memory storing program instructions that can be executed to perform the task, etc. This phrase does not refer to an intangible thing here. Therefore, the fixed expression "configured to" is not used here to refer to software entities such as application programming interfaces (APIs).

[0014] The term “configured to” does not mean “configurable to.” For example, an unprogrammed FPGA would not be considered “configured to” perform some specific function, although it may be “configurable to” perform that function and may be “configured to” perform that function after being programmed.

[0015] As used herein, the terms "first", "second", etc. are used as labels for the nouns that follow them, and do not imply any order of precedence (e.g., spatial, temporal, logical order, etc.) unless otherwise specified. For example, in a processor with eight processing cores, the terms "first" and "second" processing cores may be used to refer to any two of the eight processing cores. In other words, for example, the first processing core and the second processing core are not limited to processing cores 0 and 1.

[0016] As used herein, the term "based on" is used to describe one or more factors that affect the determination result. The term does not exclude the possibility that other factors may affect the determination result. That is, the determination result may be based only on specified factors or on specified factors and other unspecified factors. Consider the phrase "determine A based on B". The phrase stipulates that B is a factor used to determine A or affect the determination result of A. The phrase does not exclude that the determination result of A may also be based on some other factors, such as C. The phrase is also intended to cover embodiments in which A is determined only based on B. Therefore, as used herein, the phrase "based on" is synonymous with the phrase "based at least in part on". DETAILED DESCRIPTION

[0017] In some implementations of a database system, the database system includes a plurality of database nodes that are capable of processing transactions in which a database node writes a database record to a cache in the database node's own local memory and then dumps the data record to a persistent memory shared among the database nodes. In many cases, a database node needs to obtain the latest version of a database record and therefore issues a request to another database node to determine whether the latest version of the record is located in the cache of the other database node. Often, the record does not exist in the cache of the other node, and therefore the requesting node wastes network bandwidth sending requests to the other node.

[0018] In some cases, probabilistic data structures such as Bloom filters can be used to reduce the number of requests between database nodes because such probabilistic data structures can provide information on whether a record for a specific database key exists in the cache of a database node. As used herein, the term "probabilistic data structure" refers to a data structure that stores information indicating that a specific item does not exist or may exist in a specific location within a system. For example, a probabilistic data structure can store information indicating that a record for a specific database key does not exist or may exist in a cache in the memory of a database node. The inventors have recognized that current methods for implementing probabilistic data structures are insufficient for various reasons. First, known probabilistic data structures cannot check database keys in parallel, but must be checked in serial. For example, a probabilistic data structure may store database keys in a tree structure. In order to check the existence of a specific database key, the branches of the tree structure (e.g., a linked list) must be traversed serially (e.g., by traversing the linked list). Second, known probabilistic data structures are designed for accurate key lookups, but are not effective for key lookups in open or closed ranges. Third, known probabilistic data structures are not efficient in memory usage relative to the false positive result rate they provide when performing key lookups.

[0019] Public text describes various techniques for realizing probabilistic data structures, and its mode overcomes some or all shortcomings of existing methods.In various embodiments described below, database node maintains a group of probabilistic data structures, and each probabilistic data structure includes a three-dimensional data structure, and this three-dimensional data structure includes multiple bitmaps, and these bitmaps are stacked to form a hierarchical structure of the level that can encode the database key--in some cases, these bitmaps can not be stacked, to create a three-dimensional data structure.The bitmap in the "level" of the three-dimensional data structure can correspond to the "radix to radix" matrix, and its intersection can be used to store the information of the transition between the continuous characters of the database key.Therefore, in various embodiments, by storing the information of the transition between a series of characters forming the database key, the database key is encoded in the probabilistic data structure.(through the full text of the public text, when describing the concept of storing information about the database key in the probabilistic data structure, the terms "insert" and "encode" are used interchangeably).As an example, the database key "SLOT" is inserted into the probabilistic data structure. When inserting the database key, the database node may use at least three levels of the hierarchy: one level that stores information about the transition S→T; another level that stores information about the transition L→O; and yet another level that stores information about the transition O→T. In some cases, more levels may be used to store information about the transition from the root character to the first character of the database key, and about the transition from the last character of the database key to the terminal character.

[0020] In order to determine whether a database key has been encoded in the probabilistic data structure, the database node may check different levels of the three-dimensional data structure to obtain corresponding information about transitions between a series of characters that make up the database key. Continuing with the previous example, if a database node wants to determine whether the database key "SLOT" has been encoded in the probabilistic data structure that the node has received, the database node may check, for example, the third layer (or another layer, depending on the implementation) for information about the transition O→T. If the third layer does not store information about the transition, the database node can determine that the database key is not encoded. In various embodiments, the check to see whether the database key is encoded can be performed in a parallel manner in which the database node checks each layer of the probabilistic data structure relatively simultaneously.

[0021] In some cases, when multiple database keys are inserted into a probabilistic data structure, certain information may be lost. Consider the example that the keys "ROOM" and "SLOT" have been encoded in the probabilistic data structure. If a database node checks the database key "ROOT", the database node will incorrectly determine that the database key has been encoded. This is because there are three transitions in the key "ROOT": R→O, O→O, and O→T. Individually, each transition will appear in the data structure - transition R→O will form a setting due to "ROOM", transition O→O will form a setting due to "ROOM", and transition O→T will form a setting due to "SLOT" - therefore, a lookup for the database key "ROOT" will result in a false positive result. Therefore, the information that may be lost when the transitions for a particular database key are encoded are those transitions associated with that particular database key.

[0022] To remedy this problem, in various embodiments, pedigree information is maintained for a specified transition that identifies one or more pedigrees, each pedigree corresponding to a set of characters from which the transition "comes from." Consider the example of inserting database keys "SLOT" and "ROOT" into a probabilistic data structure. Pedigree information may be stored for a transition O→T that identifies two pedigrees: pedigree [RO] from "ROOT" and pedigree [SL] from "SLOT." Continuing with the aforementioned example involving a false positive result, the pedigree information may indicate that pedigree [L] exists for the transition O→T—in some cases, the complete pedigree [SLO] may not be stored. Thus, when a database node checks for the existence of transition O→T from "ROOT," the database node will see that transition O→T forms a setup, but its setup corresponds to pedigree [L] from "SLOT" rather than pedigree [O] from "ROOT." Thus, maintaining pedigree information in this manner may reduce the occurrence of false positive results.

[0023] After the database keys of the threshold number are inserted into the probabilistic data structure, the database node can provide the probabilistic data structure to another database node.Before providing the probabilistic data structure to other database nodes, in various embodiments, the original database node performs a compression operation on the probabilistic data structure to reduce the memory usage of the data structure.When creating the probabilistic data structure at the beginning, the database node can allocate a fixed amount of memory for the bitmap of each layer of the probabilistic data structure.In some cases, after inserting the database key, the database node can determine that the memory of certain levels is over-allocated.Therefore, the database node can compress these levels.For example, if all database keys share a common part (for example, an organization ID) and therefore share a group of common transitions, the database node can replace the bitmap with the information of the different formats specifying the group of common transitions.After the other database node receives the probabilistic data structure, the other database node can use the probabilistic data structure to determine whether to request a record from the original database node for a specific database key.

[0024] These techniques can be superior to existing methods because they allow for the implementation of a probabilistic data structure that can check database keys in a parallel manner and can be compressed to reduce the memory footprint of the probabilistic data structure. In addition, as described in more detail below, these techniques enable the implementation of a probabilistic data structure that provides more efficient range lookups than existing methods. As a result of these techniques, database nodes can process database transactions at a faster rate than existing methods. Reference will now be made to Figure 1 We begin by discussing exemplary applications of these techniques.

[0025] Now go to Figure 1 , a block diagram of system 100 is shown. System 100 includes a group of components that can be implemented by hardware or a combination of hardware and software routines. In the illustrated embodiment, system 100 includes database node 110, and database node 110 includes transaction engine 120, in-memory cache 130 and probabilistic data structure 140. As shown, probabilistic data structure 140 includes encoded database key 145. In some embodiments, the implementation of system 100 may be different from the diagram. For example, system 100 may include multiple database nodes 110 in communication, one or more databases (which can implement distributed storage), and / or other components interacting with database node 110, such as application servers.

[0026] In various embodiments, the system 100 implements a platform service that allows users of the service to develop, run, and manage applications. As an example, the system 100 can be a multi-tenant system that provides various functions to multiple users / tenants hosted by the multi-tenant system. Therefore, the system 100 can execute software routines from various different users (e.g., providers and tenants of the system 100) and provide code, web pages, and other data to users, databases, and other entities associated with the system 100. As shown, the system 100 includes a database node 110 that can store and access data of users associated with the system 100.

[0027] In various embodiments, database node 110 is hardware, software or a combination thereof that can provide database services (e.g., data storage, data retrieval, and / or data manipulation). Database services may be provided to other components within system 100 or components outside system 100. As shown, database node 110 receives database transaction request 105--the request may be received from an application server (not shown) attempting to access a set of database records 135. For example, database transaction request 105 may specify an SQL SELECT command that selects one or more rows from one or more database tables. The content of the row may be specified in database record 135, so database node 110 may return one or more database records 135 corresponding to one or more selected table rows to the requester. In some cases, database transaction request 105 may indicate that database node 110 writes one or more database records 135. Therefore, in various embodiments, database node 110 writes database records 135 to in-memory cache 130 at the beginning, and then dumps and clears these database records to the database. In various embodiments, the database is shared with other database nodes 110 that may be accessed by database records 135 dumped by database node 110.

[0028] In various embodiments, the transaction engine 120 is a set of software routines executable to process database transactions, which may involve inserting database records 135 into the in-memory cache 130, and inserting corresponding database keys 145 into the probabilistic data structure 140. When a database record 135 is written to the in-memory cache 130 as part of processing a database transaction, in various embodiments, the transaction engine 120 inserts (or encodes) the corresponding database key 145 into the illustrated probabilistic data structure 140. In some cases, rather than inserting the database key 145 when writing the database record 135 to the in-memory cache 130, the transaction engine 120 may perform a batch insert of the database keys 145 for a database transaction prior to initiating a transaction commit—in some cases, the batch insert may be performed as part of a pre-commit phase in which the database node 110 performs a set of actions prior to ultimately committing the database transaction.

[0029] In various embodiments, the in-memory cache 130 is a buffer that stores data in the memory (e.g., random access memory (RAM)) of the database node 110. TM Memstore can be used as an example of an in-memory cache 130. As described above, a database node 110 can initially write a database record 135 (e.g., in the form of a "key-value" pair) in its in-memory cache 130. In some cases, the latest / latest version of a row in a database table can be found in the database record 135 stored in the in-memory cache 130. However, in some embodiments, the database record 135 written to the in-memory cache 130 of the database node 110 is not visible to other database nodes 110. That is, other database nodes 110 do not know what information is stored in the in-memory cache 130 of the database node 110 without asking. In order to determine whether the in-memory cache 130 is storing a database record 135 associated with a particular database key 145, in various cases, another database node 110 may issue a database record request (not shown) to the database node 110. Such a request may include a particular database key 145, and if there is a database record corresponding to the key in the in-memory cache 130, the database node 110 may return the database record 135.

[0030] In various cases, the database node 110 may return a database record response to the other database nodes 110, where the response does not include the database record 135. This may be because, in various embodiments, the in-memory cache 130 is relatively small in size (e.g., 2GB), and therefore, the probability of the existence of a database record 135 corresponding to a particular key in the in-memory cache 130 may be relatively low. Therefore, in various embodiments, the database node 110 generates a probabilistic data structure 140 and provides it to the other database nodes 110 - this can reduce the amount of database record requests that the database node 110 receives from those other database nodes 110. Figure 5 , discusses exemplary interactions between two database nodes 110 in more detail.

[0031] In various embodiments, the probabilistic data structure 140 is a data structure that stores information indicating the probability that a database record 135 exists in the in-memory cache 130 for a database key 145. The probabilistic data structure 140 may include a three-dimensional data structure that includes a hierarchy of bitmaps, each bitmap capable of storing information about transitions between consecutive characters within the database key 145. Figure 3B , discusses exemplary representations of three-dimensional data structures in more detail. In some embodiments, the bitmap corresponds to a "radix-to-radix" matrix, where a given intersection between two characters can be used to store information corresponding to the transition between the two characters in database key 145. Figure 3A , discusses exemplary layouts of bitmaps in more detail.

[0032] As described above, database node 110 may receive database transaction request 105 specifying a transaction involving writing one or more database records 135 to its in-memory cache 130. In various cases, database record 135 may be associated with a database key 145 comprising a series of characters - Figure 2, discusses examples of database keys 145 in more detail. After writing a database record 135 to the in-memory cache 130, in various embodiments, the database node 110 inserts the corresponding database key 145 into the probabilistic data structure 140. To insert the database key 145 into the probabilistic data structure 140, the database node 110 may store information related to transitions between consecutive characters of the database key 145 in different levels of the three-dimensional data structure. In some embodiments, the database node 110 starts at the highest level and stores an indicator of a transition from a root character to the first character of the database key 145. The database node 110 may proceed downward through the levels, storing an indicator of the next transition between consecutive characters of the database key 145 in each subsequent level. As a result, each transition of the database key 145 may be stored in a separate level of the three-dimensional data structure.

[0033] As about Figure 4 Discussed in more detail, in some embodiments, which database keys 145 correspond to transitions, information about which is maintained, associated with an indicator corresponding to the transition. This information can identify one or more pedigrees associated with a transition between consecutive characters - a pedigree can specify a series of characters for the corresponding database key before the transition. Thus, in various cases, if two database keys 145 share a common transition at the same point in their series of characters, two pedigrees can be stored, each pedigree specifying the characters of that database key before the transition for a corresponding one of the two database keys 145. This pedigree information can be stored in a probabilistic data structure, but separate from the three-dimensional data structure.

[0034] In various embodiments, the database node 110 provides the probabilistic data structure 140 to another database node 110 to enable the other database node 110 to determine whether to request the database record 135 from the in-memory cache 130. The other database node 110 can check the probabilistic data structure 140 to determine whether the corresponding database key 145 of the database record has been encoded in the probabilistic data structure 140. If the database key 145 appears to have been encoded, the other database node 110 can request the database record 135 for the database key. Figure 5 , an example of this process is discussed in more detail.

[0035] Now go to Figure 2, a block diagram of an exemplary database key 145 is shown. In the illustration, database keys 145 "ROCK", "ROOM", and "SLOT" are linked to a root character 202 and a terminal character 204 to form a "prefix tree" of database keys 145. In some embodiments, database keys 145 may not be linked to a root character 202 or a terminal character 204. In some embodiments, the implementation of the prefix tree of database keys 145 may be different from the illustration. For example, the prefix tree of database keys 145 may include different database keys 145, more or fewer database keys 145, database keys 145 of different lengths, etc.

[0036] As shown, each database key 145 includes a corresponding series of characters 200. As a result, there is a character transition 210 between the consecutive characters 200 in the series of characters 200 that form these database keys 145. In some cases, the database keys 145 can share a common character transition 210 that occurs at the same position in their series of characters 200. For example, as shown, the database keys "ROCK" and "ROOM" share a common character transition 210A (R→O). As discussed in more detail below, the same information (e.g., a certain bit set in the bitmap of the probabilistic data structure 140) can be used to represent the transition 210A in the database keys "ROCK" and "ROOM". The same transition 210 can occur in two or more database keys 145, however, the transition can occur in different positions within their series of characters 200. As an example, the transition T→A can occur towards the beginning of the database key "001TA1IAE34" and towards the end of the database key "001LK3IETA3". As discussed in more detail below, information may be stored in one level of the probabilistic data structure 140 representing the transition T→A in database key “001TA1IAE34,” and information may be stored in another level representing the transition T→A that also appears in database key “001LK3IETA3.”

[0037] Now go to Figure 3A , a block diagram of an exemplary bitmap 300 is shown. In the illustrated embodiment, the bitmap 300 corresponds to a "character-to-character" matrix including a plurality of intersections 310, each intersection 310 being defined by the intersection of two corresponding characters 200. As further shown, the characters 200 of one dimension of the matrix include a terminal character 204. In some embodiments, the implementation of the bitmap 300 may be different from that shown. For example, one dimension of the matrix may correspond to, for example, Figure 3C Root character 202 is shown.

[0038] As discussed herein, the database key 145 includes a series of character transitions 210 between consecutive characters 200 in a series of characters 200 that define the database key. In various embodiments, the database key 145 is encoded into the probabilistic data structure 140 by storing information about the character transitions 210 of the database key 145 in the bitmap 300. With the illustrated embodiment in mind, consider an example where the character transition 210Z→A of the database key 145 "PIZZA" is being encoded into the bitmap 300. To encode the character transition 210, the transaction engine 120 may store an indicator 315 of the character transition 210 at the intersection 310A. As shown, the intersection 310A corresponds to the intersection of the characters "A" and "Z" - note that the character 200 on the left side of the matrix may correspond to the starting character of the character transition 210, while the character 200 on the top side of the matrix corresponds to the ending character. In various embodiments, the indicator 315 is a bit value that is set to indicate the occurrence of the character transition 210. As shown, a bit value of "1" is set at intersection 310A to indicate the occurrence of character transition 210Z→A. Figure 4 As discussed in greater detail, indicator 315 may correspond to pedigree information or a pointer to pedigree information.

[0039] In some embodiments, when encoding a database key 145, the transaction engine 120 may store an indicator 315 of the character transition 210 between the last character 200 and the terminal character 204 of the database key in the bitmap 300. This may reduce the false positive result rate of the probabilistic data structure 140 because the probabilistic data structure 140 may be encoded with database keys 145 of different lengths. Keeping in mind the illustrated embodiment, consider an example where the terminal character transition 210 of the database key 145 "JFDLIA" is encoded in the bitmap 300. As shown, the transaction engine 120 may store the transition at intersection 310B. Indicator 315.

[0040] Now go to Figure 3B , a block diagram of an exemplary probabilistic data structure 140 is shown. In the illustrated embodiment, the probabilistic data structure 140 includes a bitmap 300 that defines a series of levels 320 forming a three-dimensional structure. In some embodiments, the implementation of the probabilistic data structure 140 may be different from that shown. As an example, the highest level of the probabilistic data structure 140 may be as follows: Figure 3C The "root character to character" matrix shown.

[0041] As discussed herein, the database key 145 includes a series of character transitions 210. These character transitions 210 may form an ordering in which each consecutive character transition 210 corresponds to a consecutive level 320 included in the probabilistic data structure 140. Thus, when encoding the database key 145, in various cases, multiple levels 320 may be used to store indicators 315 of the character transitions 210 of the database key 145. Consider the example that the database key 145 "ABC" is being encoded. The bitmap 300 of the first highest level 320 may store indicators 315 of the character transition 210 {}→A, the bitmap 300 of the next second level 320 may store indicators 315 of the character transition 210A→B, the bitmap 300 of the third level 320 may store indicators 315 of the character transition 210B→C, and the bitmap 300 of the fourth level 320 may store indicators 315 of the character transition 210A→B. Indicator 315. In some embodiments, the character transition 210 from the root character 202 to the terminal character 204 may not be stored.

[0042] In various embodiments, when encoding a database key 145 that involves storing indicators 315 within a bitmap 300 at multiple different levels 320, the transaction engine 120 may store these indicators 315 in a parallel manner. That is, rather than initially storing an indicator 315 for a first character transition 210 and then proceeding to store an indicator 315 for a second character transition 210 for a particular database key 145, the transaction engine 120 may store these two indicators 315 relatively simultaneously (i.e., in parallel). Continuing with the previous example, the transaction engine 120 may write the indicators 315 for all four character transitions 210 to the bitmap 300 in parallel.

[0043] Now go to Figure 3C , a block diagram of an exemplary top-level bitmap 300 is shown. In the illustrated embodiment, the bitmap 300 corresponds to a "character-to-character" matrix that includes a plurality of intersection points 310 defined by the intersection of characters 200 and root characters 202. In various embodiments, the transaction engine 120 may store an indicator 315 of the character transitions 210 between the root character 202 and the start character 200 for each database key 145 that is encoded. Thus, the bitmap 300 of the top level 320 included in the three-dimensional structure of the probabilistic data structure 140 may correspond to the bitmap 300 shown.

[0044] Now go to Figure 4, a block diagram of exemplary pedigree information and bitmap 300 is shown. In the illustrated embodiment, bitmap 300 includes intersections 310A and 310B storing pointer information 420A and 420B, respectively. As further shown, pedigree information 410A includes pedigrees 415A and 415B, and pedigree information 410B includes pedigrees 415C and 415D. In some embodiments, the implementation of bitmap 300 and pedigree information 410A and 410B may be different from the illustrated embodiment. For example, pedigree 415 may include more or fewer characters 200.

[0045] As described above, when database keys 145 are encoded, some information about which database key 145 caused indicator 315 to be stored may be lost. Consider an example where transaction engine 120 performs a database key lookup on database key 145 "LTTZA", which involves checking Figure 3A The character transition 210Z→A in the bitmap 300 is shown in FIG. Figure 3A As shown, the indicator 315 is set for the character transition 210 Z→A, but confusion may result if the indicator 315 was set for the database key 145 "LTTZA" or another database key 145 such as "PIZZA". Therefore, in various embodiments, the lineage information 410 may be stored in association with the character transition 210 to identify the database key 145 (or a portion thereof) that was encoded into the probabilistic data structure 140 and associated with the character transition.

[0046] In various embodiments, the pedigree information 410 identifies the pedigree 415 of the character transition 210. In various embodiments, the pedigree 415 identifies one or more characters 200 that precede the character transition 210 and are associated with the corresponding database key 145. For example, consider the illustrated embodiment. As shown, the intersection 310A (which corresponds to the character transition 210Z→A) includes pointer information 420A pointing to the pedigree information 410A. The pedigree information 410A includes two pedigrees 415A and 415B for the character transition 210Z arrow A. The pedigree 415A identifies the pedigree of [PIZ], and the pedigree 415B identifies the pedigree of [TTT]. Therefore, the pedigrees 415A and 415B may indicate that the database keys 145 "PIZZA" and "TTTZA" were once encoded in the probabilistic data structure 140-as described below, the pedigree 415 may specify a predefined number of characters 200, which does not include all characters 200 before the character transition 210. As a result, if the transaction engine 120 checks Figure 4 As part of a lookup performed for database key 145 "LTTZA", transaction engine 120 may determine that database key 145 "LTTZA" has not been encoded.

[0047] In various embodiments, the transaction engine 120 may store lineage information 410 for each character transition 210 set in the bitmap 300. As shown, for example, the character transitions 210Z→A and The intersection points 310A and 310B of the character transition 210 include pointer information 420A and 420B pointing to the pedigree information 410A and 410B, respectively. In some embodiments, the pedigree information 410A and 410B are stored in separate locations, while in other embodiments, they are stored in the same location. In some embodiments, instead of storing the pedigree 415 as a series of characters 200, the pedigree 415 is stored as a hash value derived by performing a hash function on the series of characters 200. For example, instead of storing [PIZ] for the pedigree 415A, a hash value derived by hashing [PIZ] can be stored for the pedigree 415A. In some embodiments, if multiple pedigrees 415 associated with a character transition 210 do not meet a defined threshold (e.g., more than three), these pedigrees 415 can be stored in the bitmap 300 instead of being stored separately. Thus, in some cases, hash values ​​for certain lineages 415 may be stored at intersections 310 in place of pointer information 420 for associated character transitions 210 .

[0048] As discussed herein, the probabilistic data structure 140 may include a plurality of bitmaps 300 defining a hierarchical structure of levels 320 forming a three-dimensional structure. Thus, the lineage information 410 may be specific to a character transition 210 in a bitmap 300 of a particular level 320 of the three-dimensional structure. As an example, a character transition 210Z→A may be disposed in two different bitmaps 300 at different levels 320. The lineage information 410 associated with the character transition may be different for each bitmap 300. That is, the pointer information 420 in the bitmap 300 of the first level 320 may point to the lineage information 410 including the lineage 415[OOZ], while the pointer information 420 for the same character transition 210, but in the bitmap 300 of the second level 320, may point to the lineage information 410 including the lineage 415[KLZ].

[0049] The amount of information included in the pedigree 415 can vary depending on the pedigree sampling scheme used. Various pedigree sampling schemes are discussed below, and an exemplary database key 145 "AFDFJKL" is used to help simplify these discussions. In some embodiments, a complete pedigree sampling scheme is used, where the pedigree 415 corresponds to all characters 200 of the database key 145 prior to the character transition 210 associated with the database key. For example, for the character transition 210K→L, the pedigree 415 associated with "AFDFJKL" can specify [AFDFJ]. For the character transition 210D→F, the pedigree 415 can specify [AF]. In some embodiments, a fixed-length pedigree sampling scheme is used, where each pedigree 415 consumes a maximum fixed size of space (e.g., 524KB). Thus, in various cases, each pedigree 415 specifies up to a maximum number of characters 200. As an example, for the character transition 210K→L, the lineage 415 associated with "AFDFJKL" may specify [FJ], and for the character transition 210D→F, the lineage 415 may specify [AF]. In some embodiments, a variable-length lineage sampling scheme is used, in which the lineages 415 of different groups of levels 320 consume different maximum fixed sizes of space. As an example, the lineages 415 corresponding to the character transitions 210A→F, F→D, and D→F may specify a maximum of two characters 200, while the character transitions 210F→J, J→K, and K→L may specify a maximum of four characters 200. That is, the lineages 415 associated with the character transitions 210 in the first half of the levels 320 may specify up to a first quota of characters 200, while the lineages 415 associated with the character transitions 210 in the second half of the levels 320 may specify up to a second quota of characters 200.

[0050] Now go to Figure 5 , a block diagram showing an exemplary interaction between two database nodes 110 is shown. In the illustrated embodiment, database node 110A includes a probabilistic data structure 140, while database node 110B includes a transaction engine 120 having a lineage engine 510. In some embodiments, the implementation of database nodes 110 may differ from that shown. For example, database node 110B may include multiple probabilistic data structures 140 that it may share with database node 110A.

[0051] As previously mentioned, during its entire operation, database node 110A can receive database transaction requests 105 to perform database transactions. Performing database transactions can include writing database records 135 to the in-memory cache 130 of database node 110A. In addition to writing records 135 to the in-memory cache 130, database node 110A can also insert / encode database keys 145 corresponding to those database records into a set of probabilistic data structures 140. Database node 110A can share one or more of these probabilistic data structures 140 with database node 110B. In various embodiments, database node 110A inserts database keys 145 into probabilistic data structures 140 until a specified amount of keys have been inserted. Then, database node 110A can create another probabilistic data structure 140 and continue to insert database keys 145.

[0052] Prior to providing the probabilistic data structure 140, in various embodiments, the database node 110A performs a compression operation on the probabilistic data structure 140 to reduce its memory footprint. In some embodiments, when the database node 110A creates the probabilistic data structure 140, the database node 110A allocates a fixed amount of memory for each level 320 included in the probabilistic data structure 140—e.g., a fixed amount of memory for each bitmap 300. During the compression operation, the database node 110A may identify a manner in which to compress one or more levels 320. In various instances, the database keys 145 may share common character transitions 210—e.g., a group of database keys 145 may share common characters 200 corresponding to the same organization ID. As a result, the amount of information (e.g., the number of bits set) stored in the bitmaps 300 corresponding to the character transitions 210 of those common characters 200 may be limited (e.g., only one bit may be set). Therefore, in various cases, the database node 110A may replace those bitmaps 300 with smaller, differently formatted information (e.g., string objects). This information may specify character transitions 210 without using a matrix structure that may be included in bitmap 300 .

[0053] After compressing the probabilistic data structure 140 into a more space-efficient format, the database node 110A may send the probabilistic data structure 140 to the database node 110B. In some embodiments, the probabilistic data structure 140 is included in a database record response 530 to the database record request 520. That is, the database node 110B may send a database record request 520 to the database node 110A for any database record 135 corresponding to a particular database key 145. If the database node 110A locates a database record 135 corresponding to the particular database key 145 in its in-memory cache 130, the database node 110A may return a database record response 530 including the database record 135. As part of the response, the database node 110A may include one or more "new" probabilistic data structures 140 that have not yet been provided to the database node 110B. In some cases, the database record response 530 may not include the database record 135, but only include the probabilistic data structure 140 if no database record 135 corresponding to the specified database key 145 is found.

[0054] In various cases, the database node 110B may use the probabilistic data structure 140 to determine whether to issue a database record request 520 for a database record 135. To make this determination, in various embodiments, the database node 110B performs a lookup of the database key to determine whether the database key 145 corresponding to the database record 135 has been encoded in the probabilistic data structure 140. In some cases, the database node 110B may perform a single key lookup. To perform a single key lookup, the pedigree engine 510 may be executed to calculate all pedigrees 415 of the relevant database key 145. The database node 110B may then perform a vectorized pedigree check on all bitmaps 300 in the level 320 associated with the database key. The check on the bitmap 300 may be performed in parallel. That is, the database node 110B may check the bitmap 300 in each relevant level 320 in parallel to determine whether the indicator 315 has been set for the expected character transition 210. If the indicator 315 is present and it identifies the pedigree information 410, the database node 110B may check the pedigrees 415 specified in the information to determine whether one of them matches the pedigree 415 calculated by the pedigree engine 510 for the corresponding character transition 210. If the database node 110B determines that the database key 145 is encoded in the probabilistic data structure 140, the database node 110B may send a database record request 520 to the database node 110A to request the database record 135 associated with the database key 145.

[0055] In some cases, the database node 110B may perform a database key range search. The database node 110B may initially perform a single key search using the starting database key 145 of the specified key range. If the starting database key 145 has been encoded, the database node 110B may determine that the range is non-empty and may send one or more database record requests 520 to the database node 110A for the database records 135 associated with the key range. If the starting database key 145 is not encoded, the database node 110B may determine the longest common prefix between the database key 145 encoded in the probabilistic data structure 140 and the starting database key 145. To make this determination, the database node 110B may continue to advance through the character transitions 210 of the starting database key 145 until a character transition 210 of the starting database key 145 is not set in the probabilistic data structure 140. Based on the longest common prefix, the database node 110B may determine the smallest database key 145 that is larger than the starting key 145. To make this determination, if there is a valid extension of the prefix by introducing a new character 200 that is lexicographically greater than the starting key 145, then the database node 110B may continue to extend the prefix (by adding characters 200) until the terminal character 204 is reached to form the key 145. If there is no valid extension, then the database node 110B may truncate the prefix by one least significant character 200 and may then repeat the previous extension process to form the key 145 until the key 145 is formed or the prefix of characters 200 is empty. If the smallest key 145 greater than the starting key 145 is less than or equal to the ending key 145 of the specified key range, then the database node 110B may determine that the range is non-empty and may send one or more database record requests 520 to the database node 110A for the database records 135 associated with the key range.

[0056] Now go to Figure 6 , a flow chart of method 600 is shown. Method 600 is one embodiment of a method performed by a database node (e.g., database node 110) to maintain a probabilistic data structure (e.g., probabilistic data structure 140). In some embodiments, method 600 may include other steps than those shown. As an example, a database node may receive a transaction request (e.g., transaction request 105) that specifies a transaction involving writing a set of database records to an in-memory cache (e.g., in-memory cache 130) of the database node.

[0057] Method 600 begins at step 610, where a database node maintains a probabilistic data structure capable of encoding a database key (e.g., database key 145). In various embodiments, the probabilistic data structure includes a plurality of levels (e.g., level 320), each level capable of storing an indicator (e.g., indicator 315) of a transition (e.g., transition 210) between consecutive characters (e.g., character 200) in a given database key.

[0058] In step 620, the database node inserts the database key into the probabilistic data structure. The database key includes a series of characters. Therefore, when inserting the database key, for each transition between consecutive characters in the series of characters, the database node may store an indicator indicating the transition in a corresponding level of the plurality of levels (e.g., the first transition may be stored in the highest level). In some cases, the database node may store an indicator in a corresponding level of the plurality of levels (e.g., the lowest level) indicating a transition from the last character of the series of characters to a terminal character (e.g., terminal character 204) that exits the series of characters.

[0059] In some cases, a specific indicator may be stored for a transition between specific consecutive characters of a database key, and the specific indicator that specifies a set of database key pedigrees (e.g., pedigree 415) may be associated to maintain pedigree information (e.g., pedigree information 410). A given database key pedigree may identify one or more characters preceding a given transition between consecutive characters of a corresponding database key (e.g., pedigree [SLO] of transition "0"→"T" of "SLOT"). In various embodiments, the pedigree information is stored separately from multiple levels including the specific indicator. Thus, the specific indicator may be a pointer (e.g., pointer information 420) identifying the location where the pedigree information is stored. In some embodiments, each database key pedigree maintained in the probabilistic data structure is associated to identify the same number of characters (i.e., the same character length). In some cases, the specific indicator corresponds to at least two different database keys that have been inserted into the probabilistic data structure. Thus, the set of database key pedigrees may include a corresponding database key pedigree for each of the at least two different database keys.

[0060] The database node may send a probabilistic data structure to a second database node to enable the second database node to determine whether to request a database record from the database node. The database node may also receive a second probabilistic data structure from the second database node, which enables the database node to determine whether to request a database record from the second database node. Therefore, the database node may determine whether to request a database record for a specific database key by determining whether the second probabilistic data structure includes a hierarchy of indicators storing transitions between consecutive characters included in the specific database key. If it is determined that the second probabilistic data structure includes a hierarchy of indicators storing transitions between consecutive characters included in the specific database key, then in response, the database node may send a request for a database record associated with the specific database key to the second database node. Determining whether to request a database record may also include calculating a plurality of database key pedigrees corresponding to transitions between consecutive characters included in the specific database key, and comparing the plurality of database key pedigrees with a database key pedigree associated with the indicator to be maintained.

[0061] Now go to Figure 7 , a flow chart of method 700 is shown. Method 700 is an embodiment of a method performed by a computer system (e.g., database node 110) to maintain a data structure (e.g., probabilistic data structure 140). Method 700 can be performed by executing a set of program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 700 may include other steps in addition to those shown. For example, a database node may receive a request (e.g., transaction request 105) that specifies a transaction involving writing a set of database records to an in-memory cache (e.g., in-memory cache 130) of the database node.

[0062] Method 700 begins at step 710 where a computer system writes a database record (eg, database record 135 ) to a particular location (eg, in-memory cache 130 ).

[0063] In step 720, the computer system inserts a database key (e.g., database key 145) associated with a database record into a data structure including a bitmap hierarchy (e.g., bitmap 300). The database key includes a series of characters. For a given transition (e.g., transition 210) between consecutive characters in the series of characters (e.g., characters 200), inserting the database key may include storing data indicating the given transition (e.g., indicator 325) in a corresponding bitmap of the hierarchy. The data stored in the corresponding bitmap may identify pedigree information (e.g., pedigree information 410) that specifies one or more pedigrees (e.g., pedigree 415). Therefore, inserting the database may include storing in the pedigree information a pedigree indicating one or more characters that appear before a given transition in a series of characters. In some embodiments, the computer system performs a hash operation on the one or more characters that appear before the given transition to derive a hash value, and stores the pedigree in the pedigree information as the hash value.

[0064] In step 730, the data structure is sent to another computer system so that the other computer system can determine whether to request a database record from the specific location, and before inserting the database key into the data structure, the data structure is allocated so that a specific bitmap in the bitmap hierarchy has a specific memory size; before sending the data structure to the other computer system, a compression operation is performed on the data structure, wherein the compression operation includes: determining that the memory size of the data written to the specific bitmap does not consume a threshold amount of the specific memory size; and replacing the specific bitmap with specific data in another format, wherein the specific data represents data written to the specific bitmap in response to inserting a threshold number of database keys into the data structure, creating a second data structure in which subsequent database keys will be inserted.

[0065] Now go to Figure 8 , a flow chart of method 800 is shown. Method 800 is one embodiment of a method performed by a database node (e.g., database node 110) to determine whether to request a database record (e.g., database record 135) from another database node for a database key (e.g., database key 145). In some embodiments, method 800 may include other steps than those shown. For example, a database node may generate a probabilistic data structure (e.g., probabilistic data structure 140) and send it to another database node.

[0066] Method 800 begins at step 810, where a database node receives a probabilistic data structure comprising a set of levels (e.g., level 320), each level capable of storing indicators (e.g., indicator 325) of transitions (e.g., transition 210) between consecutive characters (e.g., character 200) in a database key.

[0067] In step 820, the database node determines whether to request a database record from another database node for a specific database key. The specific database key includes a set of transitions between characters of the specific database key. Determining whether the request can include, for each transition in the set of transitions, determining whether a corresponding level in the set of levels includes an indicator indicating the transition. In some cases, a specific one of the set of transitions may be associated with a specific pedigree (e.g., pedigree 415) that specifies all characters of the specific database key before the specific transition. Therefore, the database node can determine whether a specific pedigree is included in a set of pedigrees identified by a probabilistic data structure for a specific transition. In some embodiments, for a corresponding database key, each pedigree in the set of pedigrees identifies all characters of the corresponding database key before the specific transition in the corresponding database key.

[0068] In response to determining that a particular level in the set of levels does not include an indicator indicating a corresponding transition in the set of transitions, the database node may access a database record for the particular database key from a distributed memory separate from another database node.

[0069] In some cases, the database node may perform a database key range check to determine whether at least one database key within the database key range has been encoded into the probabilistic data structure. Thus, for each transition in a set of transitions associated with the starting key of the database key range, the database node may determine whether a corresponding level in the set of levels includes an indicator indicating the transition. If it is determined that the starting key is not encoded in the probabilistic data structure, in response, the database node may determine a longest prefix of the starting key that has been encoded in the probabilistic data structure. The database node may then add one or more characters to the longest prefix so that the longest prefix is ​​lexicographically greater than the starting key. The database may determine whether the longest prefix with the one or more characters added has been encoded in the probabilistic data structure.

[0070] Exemplary Computer System

[0071] Now go to Fig. 9, a block diagram of an exemplary computer system 900 in which database node 110 may be implemented is shown. Computer system 900 includes a processor subsystem 980 and an I / O interface 940 coupled to system memory 920 via an interconnect 960 (e.g., a system bus). I / O interface 940 is coupled to one or more I / O devices 950. Computer system 900 may be any of a variety of types of devices, including, but not limited to, a server system, a personal computer system, a desktop computer, a laptop or notebook computer, a mainframe computer system, a tablet computer, a handheld computer, a workstation, a network computer, and a consumer device such as a mobile phone, a music player, or a personal digital assistant (PDA). Although for convenience in Fig. 9 A single computer system 900 is shown in FIG. 9 , but system 900 may also be implemented as two or more computer systems operating together.

[0072] Processor subsystem 980 may include one or more processors or processing units. In various embodiments of computer system 900, multiple instances of processor subsystem 980 may be coupled to interconnect 960. In various embodiments, processor subsystem 980 (or each processor unit within 980) may include a cache or other form of onboard memory.

[0073] The system memory 920 may be used to store program instructions that may be executed by the processor subsystem 980 so that the system 900 performs various operations described herein. The system memory 920 may be implemented using different physical memory media, such as hard disk memory, floppy disk memory, removable disk memory, flash memory, random access memory (RAM-SRAM, EDORAM, SDRAM, DDRSDRAM, RAMBUSRAM, etc.), read-only memory (PROM, EEPROM, etc.), etc. The memory in the computer system 900 is not limited to a main memory such as the memory 920. Instead, the computer system 900 may also include other forms of storage devices, such as a cache in the processor subsystem 980 and an auxiliary memory (e.g., a hard disk drive, a storage array, etc.) on the I / O device 950. In some embodiments, these other forms of memory may also store program instructions that may be executed by the processor subsystem 980. In some embodiments, program instructions that implement the transaction engine 120 and the probabilistic data structure 140 when executed may be included / stored in the system memory 920.

[0074] According to various embodiments, the EO interface 940 can be any of various types of interfaces configured to be connected to other devices and communicate with other devices. In one embodiment, the I / O interface 940 is a bridge chip (e.g., a south bridge) from the front side to one or more backside buses. The EO interface 940 can be connected to one or more EO devices 950 via one or more corresponding buses or other interfaces. Examples of I / O devices 950 include storage devices (hard drives, optical drives, removable flash drives, storage arrays, SANs, or controllers associated with the foregoing), network interface devices (e.g., connected to a local area network or a wide area network), or other devices (e.g., graphics, user interface devices, etc.). In one embodiment, the computer system 900 is connected to a network via a network interface device 950 (e.g., configured to communicate via WiFi, Bluetooth, Ethernet, etc.).

[0075] Implementations of the subject matter of the present application include, but are not limited to, the following Examples 1 to 20.

[0076] 1. A method comprising:

[0077] maintaining, by the database node, a probabilistic data structure capable of encoding a database key, wherein the probabilistic data structure includes a plurality of levels, each level capable of storing an indicator of a transition between consecutive characters in a given database key; and,

[0078] The database node inserts a database key into the probabilistic data structure, wherein the database key comprises a series of characters, and wherein the inserting comprises: for each transition between consecutive characters in the series of characters, storing an indicator indicating the transition in a corresponding level in the plurality of levels.

[0079] 2. The method of embodiment 1, wherein a specific indicator is stored for a transition between a plurality of specific consecutive characters of the database key, wherein the method further comprises:

[0080] Pedigree information specifying a set of database key pedigrees is maintained by the database node in association with the particular indicator, wherein a given database key pedigree identifies one or more characters preceding a given transition between consecutive characters of the corresponding database key.

[0081] 3. The method of embodiment 2, wherein the pedigree information is stored separately from the multiple levels including the specific indicator, and wherein the specific indicator is a pointer that identifies the storage location of the pedigree information.

[0082] 4. The method of embodiment 2, wherein each database key spectrum maintained in association with the probabilistic data structure identifies a same number of characters.

[0083] 5. The method of embodiment 2, wherein, for the transition between the particular consecutive characters, the particular indicator corresponds to at least two different database keys that have been inserted into the probabilistic data structure.

[0084] 6. The method of embodiment 5, wherein the set of database key genealogies includes a corresponding database key genealogy for each of the at least two different database keys.

[0085] 7. The method according to embodiment 1, further comprising:

[0086] The database node sends the probabilistic data structure to a second database node to enable the second database node to determine whether to request a database record from the database node; and

[0087] The database node receives a second probabilistic data structure from the second database node, the second probabilistic data structure enabling the database node to determine whether to request a database record from the second database node.

[0088] 8. The method according to embodiment 7, further comprising:

[0089] The database node determines whether to request a database record for a particular database key, wherein the determination comprises:

[0090] determining whether the second probabilistic data structure includes a level storing indicators indicating transitions between consecutive characters included in the particular database key; and

[0091] If it is determined that the second probabilistic data structure includes a level storing indicators indicating transitions between consecutive characters included in the particular database key, in response, the database node sends a request to the second database node for a database record associated with the particular database key.

[0092] 9. The method of embodiment 8, wherein determining whether to request a database record further comprises:

[0093] computing a plurality of database key pedigrees corresponding to transitions between consecutive characters included in a particular database key; and

[0094] The plurality of database key histories are compared to a database key histories maintained in association with the indicator.

[0095] 10. The method of embodiment 1, wherein the inserting comprises:

[0096] In a corresponding level of the plurality of levels, an indicator is stored indicating a transition from a last character in the series of characters to a terminal character exiting the series of characters.

[0097] 11. A non-transitory computer readable medium having program instructions stored thereon, the program instructions enabling a computer system to:

[0098] Write database records to a specific location;

[0099] Inserting a database key associated with the database record into a data structure comprising a bitmap hierarchy, wherein the database key comprises a series of characters, and wherein the inserting comprises:

[0100] For a given transition between consecutive characters in the series of characters, storing data indicating the given transition in a corresponding bitmap in the hierarchical structure; and sending the data structure to another computer system to enable the other computer system to determine whether to request a database record from the particular location.

[0101] 12. The non-transitory computer readable medium of embodiment 11, wherein the data stored in the corresponding bitmap identifies pedigree information specifying one or more pedigrees, and wherein the inserting further comprises:

[0102] In the pedigree information, a pedigree indicating one or more characters appearing before a given transition in the series of characters is stored.

[0103] 13. The non-transitory computer readable medium of embodiment 12, wherein storing the pedigree comprises:

[0104] performing a hash operation on the one or more characters that occur before the given transition to derive a hash value; and

[0105] The pedigree is stored in the pedigree information as the hash value.

[0106] 14. The non-transitory computer readable medium of embodiment 11, wherein the operations further comprise:

[0107] Prior to inserting the database key into the data structure, allocating the data structure so that a particular bitmap in the bitmap hierarchy has a particular memory size;

[0108] Before sending the data structure to other computer systems, a compression operation is performed on the data structure, wherein the compression operation includes:

[0109] determining that a memory size of data written to the particular bitmap does not consume a threshold amount of the particular memory size; and

[0110] The specific bitmap is replaced with specific data in another format, wherein the specific data represents data written in the specific bitmap.

[0111] 15. The non-transitory computer readable medium of embodiment 11, wherein the operations further comprise:

[0112] In response to a threshold number of database keys being inserted into the data structure, a second data structure is created into which subsequent database keys are to be inserted.

[0113] 16. A method comprising:

[0114] A database node receives a probabilistic data structure comprising a set of levels, each of the levels capable of storing an indicator of a transition between consecutive characters in a database key; and

[0115] The database node determines whether to request a database record for a particular database key from another database node, wherein the particular database key includes a set of transitions between characters of the particular database key, and wherein the determination includes:

[0116] For each transition in the set of transitions, it is determined whether a corresponding level in the set of levels includes an indicator indicating the transition.

[0117] 17. The method of embodiment 16, further comprising:

[0118] If it is determined that a particular level in the set of levels does not include an indicator indicating a corresponding transition in the set of transitions, in response, the database node accesses a database record for the particular database key from a distributed memory separate from another database node.

[0119] 18. The method of embodiment 16, wherein a particular transition in the set of transitions is associated with a particular spectrum of all characters preceding the particular transition that specify the particular database key, wherein the determining comprises:

[0120] The database node determines whether the particular pedigree is included in a set of pedigrees identified by the probabilistic data structure for the particular transition, wherein each pedigree in the set of pedigrees identifies, for a corresponding database key, all characters of the corresponding database key prior to the particular transition in the corresponding database key.

[0121] 19. The method of embodiment 16, further comprising:

[0122] The database node performs a database key range check to determine whether at least one database key within the database key range has been encoded into the probabilistic data structure, wherein the performing comprises:

[0123] For each transition in a set of transitions associated with the starting key of the database key range, it is determined whether a corresponding level in the set of levels includes an indicator indicating the transition.

[0124] 20. The method of embodiment 19, further comprising:

[0125] If it is determined that the starting key is not encoded in the probabilistic data structure, in response, the database node:

[0126] determining a longest prefix of the start key that has been encoded in the probabilistic data structure;

[0127] adding one or more characters to the longest prefix so that the longest prefix is ​​lexicographically larger than the starting key; and

[0128] A determination is made as to whether the longest prefix appended with the one or more characters is encoded in the probabilistic data structure.

[0129] Although specific embodiments have been described above, these embodiments are not intended to limit the scope of the disclosure, even if only a single embodiment is described for a particular feature. The examples of features provided in the disclosure are intended to be illustrative rather than limiting, unless otherwise stated. The above description is intended to encompass alternatives, modifications, and equivalents that are obvious to those skilled in the art having benefit of the disclosure.

[0130] The scope of the disclosure includes any feature or combination of features disclosed herein (whether explicitly or implicitly disclosed), or any generalization thereof made, whether or not it mitigates or solves any or all problems. Accordingly, new claims may be made during the prosecution of the present application (or an application claiming priority thereto) for any such combination of features. In particular, with reference to the appended claims, features of dependent claims may be combined with features of independent claims, and features of individual independent claims may be combined in any appropriate manner, not limited to the specific combinations listed in the appended claims.

Claims

1. A method for implementing a probabilistic data structure, comprising: maintaining, by a database node, a probabilistic data structure capable of encoding a database key, the database key having a sequence of characters, the sequence of characters having a plurality of transitions between two consecutive characters, wherein the probabilistic data structure comprises a three-dimensional data structure, the three-dimensional data structure comprising a plurality of bitmaps, the plurality of bitmaps stacked to form a hierarchical structure of levels capable of encoding the database key, wherein each bitmap corresponds to a character-to-character matrix, and wherein each consecutive transition corresponds to a consecutive level from the hierarchy of levels; and The database key is inserted into the probabilistic data structure by the database node, wherein the inserting includes: for a given transition among the multiple transitions, storing an indicator of the given transition at an intersection of two characters of a corresponding character-to-character matrix of a level corresponding to the given transition between two consecutive characters of the database key.

2. The method according to claim 1, further comprising: Pedigree information specifying a set of database key pedigrees is maintained by the database node in association with the indicator, wherein a particular database key pedigree identifies one or more characters of an inserted database key preceding the two consecutive characters involved in the given transition.

3. The method of claim 2, wherein the pedigree information is stored separately from a hierarchical structure of the levels including the indicator, and wherein the indicator is a pointer identifying a location where the pedigree information is stored.

4. A method according to claim 2 or 3, wherein each database key pedigree maintained in association with the probabilistic data structure identifies the same number of characters.

5. The method of claim 2, wherein for the two consecutive characters involved in the given transition, the indicator corresponds to at least two different database keys that have been inserted into the probabilistic data structure. 6 . The method of claim 5 , wherein the set of database key pedigrees includes a respective database key pedigree for each of the at least two different database keys.

7. The method according to claim 1, further comprising: the database node sending the probabilistic data structure to a second database node to enable the second database node to determine whether to request a database record from the database node; and The database node receives a second probabilistic data structure from the second database node, the second probabilistic data structure enabling the database node to determine whether to request a database record from the second database node.

8. The method according to claim 7, further comprising: The database node determines whether to request a database record for a particular database key, wherein the determination comprises: determining whether the second probabilistic data structure includes a hierarchy storing indicators indicating transitions between consecutive characters included in the particular database key; and If it is determined that the second probabilistic data structure includes a level storing indicators indicating the transitions between consecutive characters included in the particular database key, in response, the database node sends a request to the second database node for database records associated with the particular database key.

9. The method of claim 8, wherein the determining whether to request a database record further comprises: calculating a plurality of database key pedigrees corresponding to the transitions between consecutive characters included in the particular database key; and The plurality of database key pedigrees are compared to a database key pedigree maintained in association with the indicator.

10. The method of claim 1, wherein the inserting comprises: In a corresponding level of the hierarchical structure of levels, an indicator is stored indicating a transition from a last character in the series of characters to a terminal character exiting the series of characters.

11. A computer system comprising: at least one processor; and A memory having program instructions stored therein, wherein the program instructions can be executed by the at least one processor to perform the method according to any one of claims 1 to 10.

12. A computer-readable medium having program instructions stored thereon, wherein the program instructions can be executed by a computer system to perform the method according to any one of claims 1 to 10.

Images