Token matching in large document corpora

Abstract

A method includes receiving a dictionary comprising a plurality of entities, wherein each entity has a length between 1 token and n tokens; constructing a probabilistic data representation model comprising n pairs of Bloom filters (BFs), the pairs of BFs being indexed from 1 to n; automatically populating the probabilistic data representation model with data representations of the entities, wherein, with respect to each pair of BFs indexed as i: (i) a first BF is populated with the first i tokens of all the entities having at least i+1 tokens, and (ii) a second BF is populated with all the entities having exactly i tokens; receiving a corpus of text, wherein the corpus of text is segmented into tokens; and automatically matching each token in the corpus of text with the populated probabilistic data representation model, wherein the matching comprises: sequentially querying each of the pairs of BFs, in the order of the index, to determine a match.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 271,839, filed February 10, 2019, which is a continuation of U.S. Patent Application No. 16 / 108,497, filed August 22, 2018, which are incorporated by reference herein in their entirety as if fully set forth below and for all applicable purposes. BACKGROUND

[0003] The present invention relates to the field of machine learning.

[0004] Text analytics or extracting structured information from large amounts of unstructured data is becoming an important aspect of today’s enterprises. A common task in text analytics is “Dictionary Matching” (DM), which is the detection of specific sets and patterns of words in unstructured text. As the number of unstructured text data, such as emails, web entities, and machine data logs, continues to grow, it is becoming increasingly important to perform DM tasks in a computationally efficient manner. However, it is often the case that DM operators spend a significant amount of time performing text analytics because these operators scan through a corpus of text, with subsequent steps only acting on the results and partial documents. Thus, it would be valuable to reduce the computational and memory requirements in DM tasks.

[0005] The foregoing examples of the related art and their associated limitations, along with other drawbacks, will become evident to one of ordinary skill in the art upon reading and understanding the present specification. SUMMARY

[0006] The following embodiments and various aspects thereof are described and illustrated by way of non-limiting examples in connection with the systems, tools, and methods described and illustrated, which are intended to be exemplary and illustrative, and not limiting.

[0007] In some embodiments, a method is provided that includes: in a population phase, receiving a dictionary comprising a plurality of entities, wherein each entity has a length between 1 token and n tokens; automatically constructing a probabilistic data representation model comprising n pairs of Bloom filters (BFs), wherein each pair of BFs is indexed from 1 to n; automatically populating the probabilistic data representation model with data representations of the entities, wherein, with respect to each pair of BFs indexed as i: (i) a first BF in the pair of BFs is populated with the first i tokens of all the entities having at least i+1 tokens, and (ii) a second BF in the pair of BFs is populated with all the entities having exactly i tokens; in a matching phase, receiving a corpus of text, wherein the corpus of text is segmented into tokens; and automatically matching each token in the corpus of text with the populated probabilistic data representation model, wherein the matching comprises: sequentially querying each of the pairs of BFs in the order of the indices to determine a match.

[0008] In some embodiments, a system is also provided that includes: at least one hardware processor; and a non-transitory computer-readable storage medium having stored thereon program instructions executable by the at least one hardware processor to: in a population phase, receive a dictionary comprising a plurality of entities, wherein each entity has a length between 1 token and n tokens; automatically construct a probabilistic data representation model comprising n pairs of Bloom filters (BFs), wherein each pair of BFs is indexed from 1 to n; automatically populate the probabilistic data representation model with data representations of the entities, wherein, with respect to each pair of BFs indexed as i: (i) a first BF in the pair of BFs is populated with the first i tokens of all the entities having at least i+1 tokens, and (ii) a second BF in the pair of BFs is populated with all the entities having exactly i tokens; in a matching phase, receive a corpus of text, wherein the corpus of text is segmented into tokens; and automatically match each token in the corpus of text with the populated probabilistic data representation model, wherein the matching comprises: sequentially querying each of the pairs of BFs in the order of the indices to determine a match.

[0009] In some embodiments, there is also provided a computer program product comprising: a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by at least one hardware processor to: in a population phase, receive a dictionary comprising a plurality of entities, wherein each entity has a length between 1 token and n tokens; automatically construct a probabilistic data representation model comprising n pairs of Bloom Filters (BFs), wherein each pair of BFs is indexed from 1 to n; automatically populate the probabilistic data representation model with data representations of the entities, wherein, with respect to each pair of BFs indexed as i: (i) populate a first BF in the pair of BFs with the first i tokens of all the entities having at least i+1 tokens, and (ii) populate a second BF in the pair of BFs with all the entities having exactly i tokens; in a matching phase, receive a corpus of text, wherein the corpus of text is segmented into tokens; and automatically match each token in the corpus of text with the populated probabilistic data representation model, wherein the matching comprises: sequentially querying each pair of BFs in the order of the indices to determine a match.

[0010] In some embodiments, each of the pairs of BFs has a specified number of hash functions, and wherein the specified number is equal to an optimal number calculated to minimize a probability of error.

[0011] In some embodiments, the specified number is less than the optimal number. In some embodiments, the specified number also decreases as the index of the pair of BFs increases.

[0012] In addition to the exemplary aspects and embodiments described above, further aspects and embodiments will become apparent to those skilled in the art upon study of the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0013] Exemplary embodiments are illustrated in the accompanying drawings. The size of the components and features shown in the drawings is generally chosen for convenience and clarity of presentation and is not necessarily to scale. The following listing outlines the drawings.

[0014] Figure 1A A document tokenization process is illustrated;

[0015] Figure 1B A multivariate token matching process is illustrated;

[0016] Figure 2 Operation of a Bloom filter is schematically illustrated;

[0017] Figure 3 A sequential probabilistic data representation comprising pairs of Bloom filters is illustrated in accordance with an embodiment; and

[0018] Figure 4 is a flowchart of the functional steps in a method for automatically matching a dictionary of tokens to a large text corpus using sequential probabilistic data representations according to embodiments. DETAILED DESCRIPTION

[0019] Disclosed herein are methods, systems, and computer program products for automatically, quickly, and efficiently matching a dictionary of entities to a large text corpus using sequential probabilistic data representations of the entities. The disclosed methods are very accurate and enable large-scale matching operations while reducing time, memory, and computational resource requirements.

[0020] As used herein, "Dictionary Matching" (DM), also known as "Token Matching", is the task of finding all occurrences of a given token in a text. The most basic instance of this problem is the exact token matching problem, i.e., the problem of finding all exact occurrences of a pattern in a text. Approximate token matching (also known as fuzzy token matching) is a technique that finds approximate matches of a pattern. Typically, a preprocessing phase of "tokenization" is performed prior to performing DM on a text corpus, which splits the text into, e.g., words and sentences. A token can be defined as an instance of a sequence of characters in a document that are grouped together as a useful semantic unit for processing. The tokenization process can be performed by, e.g., splitting the text at white spaces (as shown in Figure 1A ), while removing certain punctuation marks. In other cases, tokenization can include additional strategies for handling special characters or more complex patterns (e.g., apostrophes, hyphens, placeholders, abbreviations, and contractions). The output of the token generator can be a set of tuples, each consisting of a start offset and an end offset of a token within the text data.

[0021] A pattern consisting of a sequence of tokens is referred to as a multi-token pattern. For example, as illustrated in Figure 1B , in a dictionary comprising a full name of a person, two separate dictionary entities can be "Samuel Adams" and "Samuel Adams Johns". When a text input comprises the string "Samuel Adams Johns", the DM task is expected to return only the match of the full name, while rejecting "Samuel Adams" as a false positive possible match.

[0022] As mentioned above, DM tasks typically consume a large amount of memory, since the operation must keep and scan a large text corpus in memory for all possible matches. As illustrated in Figure 1B , the memory requirements of the DM task become more complex when multi-token pattern matching is required, and when the string comprises substrings that can match separate dictionary entities.

[0023] Therefore, in some embodiments, the present invention uses a combination of spatially efficient probabilistic data representations to maintain low memory requirements and low error rates as the size of the dictionary and / or text corpus increases proportionally. In some embodiments, a specific sequential structure of multiple Bloom filters and / or similar filters can be used to store a probabilistic data representation for each token in the dictionary, wherein the filter pairs are correlated with each other in such a way that they are configured to reduce computation and the number of false positives during queries to the data structure.

[0024] The following discussion will focus on using multiple pairs of Bloom filters (BFs) that are sequentially related to each other to store probabilistic representations of a large number of entities. However, in various embodiments, Bloom filters with different structures, different types of Bloom filters, and / or other probabilistic data representation filters and / or models may be used.

[0025] Bifurcation Fission (BF) is a method that uses hash transformation to determine set membership and is particularly useful in situations requiring fast set membership testing on large datasets (such as text analysis and DM tasks). Because BF is a probabilistic method, it has an error rate that only occurs in cases involving (i.e., false positives) – it will never report true members as not belonging to the set, but may report some non-members as members.

[0026] BF uses hash transformations to compute vectors (filters) representing the dataset. Membership in a set can be tested by comparing the result of hashing a potential member with the vector. For example, given a set S of n elements...

[0027] S = {x1, x2, ..., x} n}

[0028] It can be represented by an m-bit B array

[0029] B=[b1, b2, b3, b4,..., b m-1 b m ].

[0030] BF uses k independent hash functions h1...h k These independent hash functions h1...h k It is assumed to be uniformly distributed within the range {1, ..., m}. (Refer to...) Figure 2 To populate set S, the following condition is required to include all elements in S. x ∈S added to BF B:

[0031] For each 1≤i≤k, B[h] i [x] = 1.

[0032] During a query, to check whether an element y is in the BFB:

[0033] For each 1≤i≤k, return true if all B[h i (y)]==1, else return false.

[0034] BF provides an inherent improvement in time efficiency as a function of the false positive rate allowed by the task. More specifically, the expected complexity of a simple BF is measured for insertion and query, and depends on the number of hash functions k assigned by the filter for "storage". A simple BF relying on k hash functions will require a fixed constant (O(k)) amount of time to populate or query an item in the set, where the time constant is independent of the number of items in the set.

[0035] Given a single BF using k hash functions, with m bits storing n elements, then the expected false positive probability is:

[0036]

[0037] Thus, given a particular value of acceptable false positive rate, the optimal number of hash functions k and the length of the bit vector m can be derived as follows:

[0038]

[0039] To illustrate this result, a BF representing a set of one million items with a false positive probability of 0.01 would require only 9,585,059 bits (1.14 MB) and 7 hash functions.

[0040] However, for the DM task, using a single BF and / or similar probabilistic data representations to maintain all entities simultaneously incurs several limitations:

[0041] • significantly higher query time for matching all possible tokens found in the text against all possible tokens of different lengths in the BF; and

[0042] • a large amount of redundant computation if not accounting for the existence of tokens that include sub-tokens (as illustrated above). Figure 1B

[0043] Thus, in some embodiments, the present application provides a structure that employs sequential BFs, where sub-tokenization is considered when populating the BFs. In comparison to the single BF storage model, the multi-BF, i.e., the sequential query model of the present application, provides faster population and query times and fewer false positive errors in matching.

[0044] Reference will be made to Figure 3 ​To describe in detail the working principle of the present invention. In the exemplary case where the dictionary has entities of length between, for example, 1 token and 4 tokens, the present invention is configured for building a BF structure comprising 4 pairs of BFs. For a dictionary comprising entities with different lengths, the number of pairs of BFs will match the longest token length expected. In some embodiments, the first BF in each pair can be designated as BF i Cont and the second BF in each pair can be designated as BF i Stop where i is an index number equal to 1 to n. As illustrated in Figure 3 , during the filling of the BFs:

[0045] • The first BFs in all pairs with index i between 1 and (n-1) will be filled with the first i tokens of all entities with at least n+1 tokens. Thus, for example, in the case of the entity "Bank of England" (n=3), BF1 Cont will store "Bank", and BF2 Cont will store "Bank of"; and

[0046] • The second BFs in the pair indexed as i=n will store all entities comprising exactly n tokens. Thus, for example, BF3 Stop will store "Bank of England" (n=3).

[0047] In some embodiments, once the BFs have been filled with all entities as described above, for a given token matching task, the BFs will be sequentially queried to match all possible entities starting with the current token. For example, as illustrated in Figure 3 , matching the 3-gram "Bank of England" will involve sequentially querying the following BFs:

[0048] (i) "Bank" in BF1 Cont ;

[0049] (ii) "Bank of" in BF2 Cont ; and

[0050] (iii) "Bank of England" in BF3 Stop .

[0051] Similarly, matching the 4-gram "United Nations Secretary General" will involve sequentially querying the following BFs:

[0052] (i) "United" in BF1 Cont"United" in "United States";

[0053] (ii) BF2 Cont "United Nations" in "United Nations";

[0054] (iii) BF3 Cont "United Nations Secretary" in "United Nations Secretary General"; and

[0055] (iv) BF4 Stop "United Nations Secretary General" in "United Nations Secretary General".

[0056] As can be seen, due to the relative sparsity of the multi-gram tokens, "higher order" BFs (i.e., those representing long entities, such as BF4 Cont and BF4 Stop ) can be queried less frequently.

[0057] In some embodiments, the average token length q * of a given set of tokens can be related, at least in part, to both the BF population time and the BF query time, as follows:

[0058] • Populate : For an exact match on a single token of length q, the matching process will query the BF Cont filters indexed 0 to q-2 for the expected q-1 BF insertions, and the all-BF Stop filter for 1 insertion, for a total of q insertions to all BFs in the structure.

[0059] • Query : For an exact match on a token of length q, the matching process will query all BF Cont and all BF Stop from 0 to q-1, for a total of 2q queries. For a partial match, the matching process will query all BF Cont and all BF Stop until the n-gram is no longer similar. In any event, the total number of queries will be no more than 2q-1.

[0060] Thus, the length q, derived directly as the average of all tokens q, bounds the complexity of populating and querying the tokens. Furthermore, q also depends on the distribution of tokens in the dictionary; multi-length tokens can have different rates of distribution in various dictionaries, some lengths can occur more frequently than others, etc. Similarly, to some extent, the query time can vary by the distribution of tokens found in the text corpus during tokenization. Ultimately, the expected complexity for both population and query is O(k·q *). Since q * The expected complexity can be presented as O(k) since the number of fixed, typically expected to be low, for example between 2 and 3, and in most use cases, no higher than 5.

[0061] In some embodiments, the sequentially paired BF structures can also be configured to allow a significant reduction in the number of hash functions k required to represent all entities. Since the primary purpose of multiple hash functions in a BF is to reduce the false positive rate, it can be expected that a query process including a sequence of related queries, as disclosed by the present invention, can result in fewer false positives.

[0062] For example, if the false positive rate of two individual BFs is 1% each, the probability of a false positive return in a query including both structures would be 0.1% X 0.1% or ~0.01%. For example, as mentioned above, with reference to Figure 3 , matching the 3-gram "Bank of England" would involve sequentially matching the term through three separate BFs: BF1 Cont , BF2 Cont , BF3 Cont In such a case, for a false positive return, all three BFs must return a false positive match. Since each of the BFs has a certain probability of returning a false positive, the likelihood of all three BFs returning a false positive simultaneously is significantly reduced.

[0063] To take advantage of this overall reduced false positive probability, in some embodiments, a small number of hash functions can be used to construct the BFs Cont As mentioned above, this can result in low fill time and query time.

[0064] Furthermore, as the order of the BF Cont filter increases, a higher acceptable false positive rate can be assigned to it. For example, when matching a 6-gram token, a less stringent false positive rate can be used for BF4 Cont and BF5 Cont Since the likelihood of a false match would have already been reduced by the chain of previous sequential queries, and will be further reduced by the false positive rate of BF6 Cont Therefore, the overall reduced number of hash functions, as well as the reduced number of tokens to be queried by each of the structures, compensates for the increased number of structures to be queried during token matching.

[0065] Figure 4 is a flowchart of a method for automatically matching a dictionary of tokens to a large text corpus using sequential probabilistic data representations. At 400, a dictionary including tokens having n-gram lengths 1 to n is received. At 402, sequential probabilistic data representing the structures is constructed, for example using n pairs of BFs.

[0066] At 404, the first i tokens of all entities having at least i+1 tokens are stored in a first BF in the BF pair indexed as i. At 406, all entities including exactly i tokens are stored in a second BF in the BF pair indexed as i.

[0067] In some embodiments, exemplary populating computer program code of the present application can include the following computer instructions:

[0068]

[0069] At 408, tokens from a tokenized corpus of text are received. At 410, each text corpus token is sequentially matched against the BF structure to identify a match. In some embodiments, exemplary querying computer program code of the present application can include the following computer instructions:

[0070]

[0071] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0072] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire cable, because the computer readable storage medium is a non-transitory computer readable medium that can be accessed by a computer system.

[0073] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0074] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or source code and object code written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0075] Various aspects of the present application are described in terms of flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0076] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including

[0077] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0078] The flow and block diagrams in the drawings show architectural, functional, and operational aspects of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions (s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession can in fact be executed substantially concurrently or can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or combinations of special purpose hardware and computer instructions.

[0079] The descriptions of various embodiments of the present application have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method comprising: receiving, by a server, a dictionary comprising a plurality of entities, wherein each entity has a length between 1 token and n 1 token. automatically constructing by the server a probabilistic data representation model comprising n a Bloom filter BF pair, where each BF pair is indexed from 1 to n n. The server automatically populates the probabilistic data representation model using the data representations of the multiple entities, wherein the indexed entity is... i Each BF pair, utilizing from having at least i +1 tokens for the preceding entities of the plurality of entities i Tokens are used to populate the first BF in the BF pair, and with exactly Tokens i All of the multiple entities of the token are used to populate the second BF in the BF pair; receiving, by the server, a corpus of text segmented into tokens; and automatically matching, by the server, each token in the corpus of text to a populated probabilistic data representation model.

2. The method of claim 1, wherein the matching further comprises: in order from 1 to n each BF pair sequentially to determine a match.

3. The method of claim 1, wherein each BF pair has a specified number of hash functions, each specified number equal to an optimal number computed to minimize a probability of error.

4. The method of claim 1, wherein each BF pair has a specified number of hash functions, each specified number less than an optimal number computed to minimize a probability of error.

5. The method of claim 1, wherein each BF pair has a specified number of hash functions, each specified number less than an optimal number computed to minimize a probability of error, the specified number further decreasing as the index to the BF pair increases.

6. A computing device comprising: a memory containing a machine-readable medium, the machine-readable medium including machine executable code, the machine-readable medium having instructions stored thereon for performing a method of token matching; a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to: receiving a dictionary comprising a plurality of entities, wherein each entity has a length between 1 token and n 1 token. automatically constructing includes n probabilistic data representation model comprising n indexed; populating the probabilistic data representation model automatically with data representations of the plurality of entities, wherein for each BF pair indexed as i the first BF in the BF pair is populated with the first tokens from the plurality of entities having at least i+1 i the second BF in the BF pair is populated with the plurality of entities having exactly i tokens;​ receive a corpus of text segmented into tokens; and automatically match each token in the corpus of text to a populated probabilistic data representation model.

7. The computing device of claim 6, the machine executable code further causing the processor to: in order from 1 to n each BF pair sequentially to determine a match.

8. The computing device of claim 6, wherein each BF pair has a specified number of hash functions computed to minimize a probability of error.

9. The computing device of claim 6, wherein a BF pair has a specified number of hash functions, each specified number less than an optimal number computed to minimize a probability of error.

10. The computing device of claim 6, wherein each BF pair has a specified number of hash functions, each specified number less than an optimal number computed to minimize a probability of error, the specified number further decreasing as the index to the BF pair increases.

11. A non-transitory machine-readable medium having stored thereon instructions for performing a method, the non-transitory machine-readable medium including machine executable code that, when executed by at least one machine, causes the machine to: receiving a dictionary comprising a plurality of entities, wherein each entity has a length between 1 token and n 1 token. automatically constructing includes n probabilistic data representation model comprising n indexed; populating the probabilistic data representation model automatically with data representations of the plurality of entities, wherein for each BF pair indexed as i the first BF in the BF pair is populated with the first tokens from the plurality of entities having at least i+1 i the second BF in the BF pair is populated with the plurality of entities having exactly i tokens;​ receive a corpus of text segmented into tokens; and automatically match each token in the corpus of text to a populated probabilistic data representation model.

12. The non-transitory machine-readable medium of claim 11, further including machine executable code that causes the machine to: in order from 1 to n each BF pair is sequentially queried in order to determine a match.

13. The non-transitory machine-readable medium of claim 11, wherein each BF pair has a specified number of hash functions, each specified number equal to an optimal number computed to minimize a probability of error.

14. The non-transitory machine-readable medium of claim 11, wherein each BF pair has a specified number of hash functions, each less than an optimal number computed to minimize a probability of error.

15. The non-transitory machine-readable medium of claim 11, wherein each BF pair has a specified number of hash functions, each specified number less than an optimal number computed to minimize a probability of error, the specified number further decreasing as the index of the BF pair increases.

Images