System and method for generating filters for k mismatch searches

Abstract

Systems and methods of generating filters for k mismatch searches are disclosed. A computer system for performing k mismatch searches includes at least one processor configured to: receive a size of a search term; receiving a similarity threshold; and generating a set of filters by forming the set of filters according to a similarity threshold, a set of combinations of matching locations and vacant locations based on a search term size, where each of the combinations contains at least one filter in the set of filters.

Description

[0001] This application is a divisional application of the patent application with application number 201880032748.6 and invention name “System and method for generating filters for k unmatched searches”, which has an international application date of March 19, 2018 and international application number PCT / IL2018 / 050311, and entered the national stage on November 18, 2019. The entire contents of the patent application are incorporated herein by reference. Technical Field

[0002] The present invention generally relates to approximate string matching (k mismatches), and to generating filter sets to optimize available computer resources. Background Art

[0003] The entirety of provisional application 62473400, filed on March 19, 2017, and entitled "System and Method for Generating a Filter Set for K Unmatched Searches," is incorporated herein by reference.

[0004] Searching for strings on large databases is a time-consuming task. A "naive algorithm" that searches directly can take days, weeks, or even longer. Biological databases, such as the Genome Database or the Protein Database, collect large amounts of genome sequences or proteins, annotated, and analyzed. Other large databases store pictures, audio recordings, videos, 3D structures, and more.

[0005] Detecting similarities in nucleotide and amino acid sequences helps annotate the sequence of interest. For example, if the sequence is a gene, similar sequences can reveal similarities in function, regulation, interactions with other molecules, three-dimensional structure of the protein product, or a common evolutionary origin.

[0006] The technique of finding occurrences of a pattern string that approximately matches a given text is called the "k-mismatch problem". This technique is also known as "approximate pattern matching", "approximate string matching" or "fuzzy string search". Several algorithms are known in the art to solve the k-mismatch problem. These algorithms perform a two-stage process in which the search is performed after preprocessing one or more patterns.

[0007] Landau and Vishkin disclosed a method for solving the K mismatch problem, also known as the kangaroo method, whose time complexity of the preprocessing stage is O(n+m), while the time complexity of searching for patterns in the text is O(nk), G.M. Landau, U. Vishkin, "Efficient string matching in the presence of errors", Proc. 26th IEEE FOCS, pp. 126-136, 1985.

[0008] Fisher and Patterson disclosed an algorithm for solving the k-mismatch problem in "String matching and other products", Proceedings of the 7th SIAM-AMS Complexity of Computation, R. Karp, editor, pp. 113-125, 1974, which has a time complexity of O(min|Σ|, m)nlogm).

[0009] Abrahamson and Kosaraju in "Generalized string matching", SIAM journal on Computing, 16 (6): 1039-1051, 1987, disclosed a method using O Running time for a method to solve the K mismatch problem.

[0010] Amir, Lewenstein, and Porat published another algorithm with running time O( The method of Landau and Vishkin and the method of Amir, Levenstein, and Porat are linear for small k.

[0011] Filtering methods can speed up the running time of the approximate string matching process. Filtering type algorithms discard large parts of the text, aiming to keep the important parts that contain potential matches. A common filtering method is based on q-grams. These are substrings of length q. A q-gram of a sequence is all overlapping substrings of length q. Most research in this field focuses on non-upgrading (i.e., usual) q-gram filters.

[0012] Applying filtering methods can be best performed by considering necessary but not sufficient conditions for word similarity at a chosen threshold. For example, being a mammal is a necessary condition for being recognized as a human, but it is not a sufficient condition.

[0013] Applying filtering methods provides improvements to the naive algorithm. The following two examples are improvements to the naive algorithm for searching for all words in a given text that are similar to a given word W of 20 letters that contains only one mismatch (ie k=1).

[0014] In the first example, a necessary (but not sufficient) similarity condition is imposed as follows: When word W is split into two parts: W1 (positions 1-10) and W2 (positions 11-20), there may be a mismatch in W1 (W2 matches) or in W2 (W1 matches). In this case, the necessary condition for similarity is an exact match with either W1 or W2. For a search with a mismatch threshold, a word is found in the text that either matches W1 or matches W2, and then for each exact match with W1, the last ten characters of the exact match are compared with W2. Similarly, for an exact match with W2, the first ten characters are compared with W1 (allowing for one mismatch). By preprocessing the text, an exact search for W1 and W2 can be done in O(1) time complexity. This will map all ten-character words and require memory of size O(n), where n is the size of the text. The probability of W1 and W2 appearing in the text is (1 / d) 10 , where d is the size of the alphabet. The search time is (n / d) 10 × p, where p is the expected number of random comparisons with two mismatches.

[0015] The second example has a lower running time, but requires more memory (i.e., random access memory ("RAM")) during the preprocessing phase. After mapping all 19-letter words with a "wildcard" at all possible locations. The size of this map is about 19×n, and the speed of finding a solution by extracting from all 19-letter words W using a wildcard is O(1). The disadvantage of this approach is that it does not work for lower similarity thresholds. For example, with a threshold of 8 mismatches and 20-letter words, the number of possible words that contain this many wildcards is very high (20 8 This requires a lot of memory resources to store the preprocessed text (more than 10 5 × n). For larger values ​​of n, this is not feasible.

[0016] US Patent 7,756,847 discloses an iterative method for searching a plurality of output strings, each of which has a partial or complete match with a query string in a database consisting of a plurality of long strings or a long string.

[0017] U.S. Patent 7,444,326 discloses approximate substring indexing by decomposing each string in a database into overlapping "positional q-grams", sequences of predetermined length q, and containing information about the "position" of each q-gram in the string (i.e., the first q-gram, the fourth q-gram, etc.).

[0018] U.S. Patent 7,584,173 discloses a process for determining which, if any, string in a text list is used for a search string that has an edit distance from the search string that is less than a threshold. The process uses dynamic programming on a grid where the search string characters correspond to rows and the text characters correspond to columns.

[0019] A generalized approach to q-gram filters is to use gapped q-grams (also called "gap q-grams"), fixed subsets of non-contiguous shapes. The following references describe how to apply this approach.

[0020] Randomly selected shapes are disclosed in "Califano A., Rigoutsos 1., FLASH: "A fast look-up algorithm for string homology", in: Proc. 1st International Conference on Intelligent Systems for Molecular Biology (ISMB), pp. 56-64 (1993)". The use of regular shapes is disclosed in "Pevzner, PA, Waterman, MS," Multiple Filtration and Approximate Pattern Matching", Algorithmica, 13 (1 / 2), 135-154 (1995)" and "Lehtinen, O., Sutinen, E., Tarhio, J.," Experiments on Block Indexing", Proceedings of the 3rd South American Workshop on String Processing (WSP'96), Carleton University Press (1996)".

[0021] The advantages of gapped q-grams over non-gapped ones were considered and their application in lossy filtering was disclosed in: "Ma, B., Tromp, J., Li, M.: Pattern Hunter: "Faster and More Sensitive Homology Search", Bioinformatics, 18, 440-445 (2002)" and "Buhler, J., Keich, U., Sun, Y.: "Designing Seeds for Similarity Search in Genomic DNA", Proceedings of the 7th Annual International Conference on Computational Molecular Biology (RECOMB-03) (April 2003)".

[0022] For the case of a small number of errors, some methods for grammar development have been proposed: "Farach-Colton M., Landau G., Sahinalp S., Tsur D.,"Optimal spaced seeds for faster approximate string matching", J. Comput. System Sci. 73(7)1035-1044(2007); Francois Nicolas, Eric Rivals,"Hardness of optimal spaced seed design", J. Comput. System Sci. 74(5)831-849(2008); Egidi L., Manzini G.:"Better spaced seeds using quadratic residues", Journal of Computer and System Sciences 79(7), pp.1144-1155(2013)". Summary of the invention

[0023] The disclosed technology aims to provide a novel method that generates an optimal set of filters for a given search task based on the combined properties of shapes, taking into account the available amount of computer resources.

[0024] Therefore, according to the disclosed technology, a computer system for performing k-unmatched searches is provided, which includes at least one processor configured to: receive the size of a search term; receive a similarity threshold; and generate a filter set by the following steps: according to the similarity threshold, forming a filter set based on a set of combinations of matches for the size of the search term and the position of the vacancy, wherein each of the combinations contains at least one filter from the filter set.

[0025] In some embodiments, the processor is further configured to create a minimum filter set by, for each filter in the filter set, determining that the filter is a retainable filter when there is at least one combination in a set that contains only combinations of filters in the filter set, and deleting the filter from the filter set when the filter is not a retainable filter.

[0026] In some implementations, each combination in the set of combinations includes a match at a first position of the corresponding combination.

[0027] In some implementations, each combination in the set of combinations includes a match at a last position of the corresponding combination.

[0028] In some embodiments, the processor is configured to form a filter set by: initializing the filter set; and for each selected combination of the set of combinations: searching the selected combination for an occurrence of a filter in the filter set; when an occurrence of a filter in the filter set is not found in the selected combination, deriving a new filter from the selected combination; and adding the new filter to the filter set.

[0029] In some implementations, the new filter is part of a selected combination.

[0030] In some implementations, a randomization process is utilized to determine any new filter and multiple matches for the new filter.

[0031] In some implementations, the number of matches is a predetermined number.

[0032] In some embodiments, the processor is configured to derive a plurality of new filters from the selected combination, and assign a score to each of the plurality of new filters and add the new filter to the filter set when the score of the new filter is higher than the score of any other new filter of the plurality of new filters.

[0033] In some embodiments, the processor is further configured to assign a score to each filter in the set of filters based on a parameter selected from the group consisting of: the expected number of comparisons between the search terms and the text when the filters are applied separately; the frequency of occurrence of the filter in the combined set; the number of matches in the filter; and the scope of the filter.

[0034] In some embodiments, the processor is further configured to: create a second filter set by: selecting filters from the filter set; replacing the selected filters with at least one different filter; selecting and replacing in an order sorted from the highest score of the selected filters to the lowest score of the selected filters, wherein each combination contains at least one filter from the second filter set; and selecting the second filter set from the filter set based on the reasonable scores.

[0035] In some implementations, the reasonableness score is calculated using a parameter selected from the group consisting of: an expected search computation speed determined based on the sum of expected comparison times for all filters in the filter set; an amount of memory required to perform a search of the filter set.

[0036] In some embodiments, the processor is further configured to generate a plurality of filters in a filter set, wherein: each position in the search term corresponds to at least one of the plurality of filters in the filter set, wherein each of the combinations contains at least one filter of the plurality of filters in the filter set at a position in a corresponding one of the combinations, and the position of the search term corresponds to the position in the corresponding one of the combinations, i.e., the processor creates a plurality of filter sets. Each position in the search term corresponds to at least one filter set. Each combination contains at least one filter from one of the plurality of filter sets. The position in the combination where the filter is found corresponds to the position in the search term, which corresponds to the filter set containing the filter (i.e., the filter in the combination).

[0037] In some embodiments, the processor is configured to form each of the multiple filters of the filter set by: designating the combined set as the current set of the combination; and designating the filter set as the current filter set based on the current set of the combination; for each position, obtaining a sequence of positions in the search terms; generating a filter set associated with the position based on the sequence; when the combination contains a filter in the generated filter set associated with the position and the filter exists in the combination at the position, updating the combination from the current set of the combination; when the filter is not retainable relative to the current set of the combination, deleting the filter from the current filter set, wherein the filter is retainable relative to the current set of the combination when there is at least one combination in the current set of the combination that only contains the filter.

[0038] In some embodiments, the processor is configured to form a filter set associated with a location by: obtaining a set of combinations, a current filter set, and a location; for each filter in the current filter set: when the filter is found in at least one combination in the set of combinations of the location, adding the filter to the filter set associated with the location, and deleting the combination from the set of combinations.

[0039] In some implementations, the processor is further configured to: receive a lossy threshold, create a combined subset based on the lossy threshold and the similarity threshold with respect to the search term, wherein generating the set of filters includes generating filters with respect to the combined subset.

[0040] In some implementations, the size of the search term corresponds to a plurality of possible sizes of the search term, and wherein the similarity threshold corresponds to a plurality of similarity thresholds, each similarity threshold corresponding to one of the plurality of possible sizes of the search term.

[0041] In some implementations, the at least one hardware processor is further configured to form a plurality of sets of filters, wherein each of the plurality of sets of filters corresponds to a different sequence of the plurality of position sequences in the search terms.

[0042] In some embodiments, at least one hardware processor is further configured to select a set of filter sets from the plurality of sets of filter sets based on an expected number of comparisons to be performed when the selected set of filter sets is applied to perform a k-unmatched search, wherein the expected number of comparisons is a function of a given text and a given term to be searched.

[0043] In some implementations, the at least one hardware processor is further configured to apply the execution of the k unmatched searches to cluster a plurality of words of the text.

[0044] In some implementations, the at least one hardware processor is further configured to apply clustering to speed up the text indexing step of the k-no-match search. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The disclosed technology will be more fully understood and appreciated through the following detailed description in conjunction with the accompanying drawings, in which:

[0046] Figure 1 is a flow chart illustrating the steps of a method of generating a filter set;

[0047] Figure 2 is a flow chart illustrating the steps of another method for populating a filter set;

[0048] Figure 3 It is a schematic diagram of the use of this filter set;

[0049] Figure 4 is an illustration of examples of minimal filter sets obtained by embodiments of the present invention and their advantages over naive algorithms;

[0050] Figure 5A is a flow chart illustrating the steps of a method for a k-unmatched search system to create and select a filter set from a plurality of generated filter sets;

[0051] Figure 5B is a flow chart illustrating the steps of a method for a k-unmatched search system to create and select a filter set from a plurality of generated filter sets;

[0052] Figure 6 is a flow chart illustrating the steps of a k-unmatched search system to generate a set of filters associated with a location;

[0053] 7A to 7C illustrating generation of a set of filters associated with a location; and

[0054] Fig. 8A and Figure 8B Each shows a computer system for performing any of the steps disclosed herein, constructed and operated in accordance with an embodiment of the disclosed technology. DETAILED DESCRIPTION

[0055] The present invention provides a method for generating an optimal set of filters for a given search task taking into account the available computer memory (ie, random access memory ("RAM")) available for the task.

[0056] The present invention can be applied to the computation of all or part of a protein connectivity network ("PCN"). PCN is a graph that can be used to solve different problems in computational biology, primarily to aid in the prediction of protein structure and function. PCN consists of nodes, which are small fragments of protein sequences, and the edges between nodes reflect the high similarity between the fragments.

[0057] For sufficiently large protein databases, computing all or part of the PCN is a critical process that requires significant computational resources. Therefore, it is important to improve the process through efficient algorithms. Finding all edges of such a graph is an example of the k-mismatch search problem.

[0058] The task of the "k-mismatch problem" is of great significance to many systems that support word searches on large databases. In addition to bioinformatics (i.e., searches of DNA and protein subsequences), the invention can be applied to document or Internet searches by representing information such as pictures, sounds, videos, and 3D structures as linear text built from corresponding letters.

[0059] Embodiments of the present invention can be applied to systems that include different types of large-scale databases. For example, the database can be distributed across multiple servers, or the database can be partitioned on a large scale. As the amount of stored information grows exponentially, it becomes increasingly important to develop an efficient algorithm for fast searching on large databases.

[0060] The term "text" refers hereinafter to digital text, which is an electronic version of a written text. The text may be provided in bulk via electronic media or accessed remotely via online streaming over the web.

[0061] The term "similarity" hereinafter refers to the matching of a search term in the searched text or data stream to a specified mismatch threshold.

[0062] The term "hammer distance" refers hereinafter to the distance between a string or a substring and a pattern.

[0063] Approximate string matching is a technique for finding strings that approximately match a pattern by searching for all locations in the text where the distance between the substring at that location and the pattern is less than or equal to a boundary k.

[0064] The term "string" in this document refers to any sequence of letters. A bit sequence may be part of another bit sequence, for example, the term string also includes any substring of a string.

[0065] The term "shape" refers to the form of the filter below. The shape includes matching positions and has a size and a span. The size is the amount or number of matches, while the span is the length of the shape, including the matching positions and the gaps between them (if any). For example, the shape is {0,1,3,6}, ##-#--#. Its size is 4 and its shape is 7. Gaps indicate positions where matches may not occur. The order of matches in a shape without gaps is also called a seed, and gaps are also called intervals.

[0066] The term "gap q-gram filter" refers hereinafter to a filter of defined shape, whose size is q, i.e. the amount or number of matches is q. A gap filter is also called a gap seed.

[0067] The term "filter" may refer to a leave-gap q-gram filter hereinafter.

[0068] The term "indexing" refers hereinafter to searching for all similar terms for each term in a given text.

[0069] The term "minimum filter set" is hereinafter referred to as a set of filters that are necessary for a given similarity threshold and can be used by a search term. In other words, no filter can be removed from the set.

[0070] The term "term similarity graph" refers hereinafter to a graph in which nodes are repetitions (overlaps) of terms in a text of a chosen size and edges reflect similarities between connected terms greater than or equal to a defined similarity threshold.

[0071] The term "combination" refers herein to the possible combinations of matching and gap positions that satisfy the conditions of the similarity threshold for a search term of a given size, and is represented as a sequence of "0" and "1", where for a position in a given sequence, "0" indicates no match and "1" indicates a match at the corresponding position.

[0072] In general, the term "exhausted" means that the iterative process has run out of input to continue, whether that input is a set of filters or a set of combinations. For example, the term "the set of all combinations has been exhausted" hereafter refers to the case where the set of all combinations is empty or completely marked as examined.

[0073] The term "fragment of a combination" refers hereinafter to a series of matches and mismatches starting at a certain position of the combination. A filter considered as a fragment of the combination will start from the position of the match and continue until a predetermined number of matches is found (i.e., there should be a match at the end of the fragment). For example, for the combination 10011010100111, the filters with 4 matches are: 1001101, 110101, 10101001, 1010011, and 100111. Optionally, the number of matches may not be defined, or may be received as user input. In this case, the number of matches may be randomly selected from a predetermined interval.

[0074] Larger texts (e.g., greater than 1G) may have some limitations on RAM. Embodiments of the present invention may be implemented in different computing environments, such as distributed on one or more computers, collectively having considerable memory resources (e.g., 10-100 times larger than the text to be searched), or distributed on a network of computers (nodes), with each node having very limited memory resources (e.g., less than or equivalent to the size of the text to be searched).

[0075] The disclosed techniques provide improvements in memory usage and search time over prior art techniques under a range of conditions: such as searching text content, alphabetic letters, number of mismatches allowed, and word size, to name a few.

[0076] An exemplary embodiment of the present invention provides a computer system for generating a set of filters for performing one or more k-unmatched searches on text. The computer system includes at least one hardware processor. The computer system can be any of the following: a distributed network of multiple hardware processors; a local network of multiple hardware processors; or a stand-alone computer. The computer system receives the following parameters: the size of the search term, a similarity threshold, and the number of matches in the filter. The number of matches in the filter can be automatically defined, for example based on the size of the text to be searched and the amount of available RAM.

[0077] The computer system generates a set of filters based on the complete set of possible combinations of matches and positions of gaps for the size of the search terms and according to a similarity threshold. For example, for a search of length 5, in addition to a perfect match (e.g., zero mismatches), there is a similarity threshold of 4 matches (e.g., at least four of the five positions must match, so at most 1 position may not match), and thus there are 5 possible combinations, since mismatches may occur anywhere from positions 1 to 5 of the search terms. The set of filters thus generated can then be reduced to improve memory usage and computational efficiency.

[0078] More formally, the computer system may repeat the following steps to reduce the size of the filter set to achieve a minimum set, i.e., produce a minimum filter set: a) selecting a filter from the filter set, and b) determining the filter as a retainable filter if the filter is the only filter in the filter set that is included in at least one combination in the set of combinations. Otherwise, when the filter is not determined to be a retainable filter, the computer system removes the filter from the filter set. Although the filter set has been reduced to a minimum set, it can still be used for lossless search, i.e., finding each term in the text.

[0079] The memory requirements for implementing the techniques disclosed herein are determined by the number of filters and other parameters. The following description relates to techniques for determining a minimum set of filters, i.e., a number of filters sufficient to cover all possible combinations but no more. The effectiveness of determining a minimum set of filters is determined by the following parameters:

[0080] 1) The memory required to perform the mapping step. This is evaluated in terms of the number of filters in the minimum filter set (MCS). For a search text size of n and the number of filters in the minimum set given by |MCS|, the complexity of this parameter can be estimated as O(n*|MCS|).

[0081] 2) The search speed can be determined based on at least two factors: the number of comparisons between the search query and the search text and the number of "idle" filter comparisons. The latter relates to the situation where no subsequence of the filter is found in the search text. For larger search texts and correspondingly a large number of filter occurrences, the main factor determining the search speed is the number of comparisons. In contrast, when the size of the search text is relatively small and the frequency of occurrence of the filters is correspondingly low, then the limiting factor determining the search speed is mainly the number of "idle" checks.

[0082] The number of comparisons of the search query to the search text can be calculated as the sum of the comparisons, estimated for each filter in the MCS as follows:

[0083] Number of comparisons = P*L*n

[0084] where P is the probability of the filter appearing, L is the number of positions in the search query where the filter exists, and n is the size of the search text.

[0085] The probability P of a filter appearing can be estimated as:

[0086] P = 1 / Σ letter m in the alphabet

[0087] Where m is the number of matches in the filter.

[0088] The number of "idle" checks can be estimated as:

[0089] Idle check times = (1-P) n *L*MCS|

[0090] Where L = (query size) - (filter span) + 1.

[0091] In general, the higher the required number of matches (threshold), the larger the corresponding minimum filter set (MCS). Therefore, although choosing filters with very low probability of occurrence (e.g., with a relatively high required number of matches) reduces the number of comparisons, this requires greater memory resources to resolve larger MCSs and increases the number of "idle" checks. The selection of the most appropriate MCS for a given problem needs to be optimized given the availability of memory. The optimal number of matches (m) in the filter can be determined experimentally for each application, which determines the probability of occurrence P and |MCS|. To handle low memory situations, the mapping of the full size of the search text can be divided into multiple parts, and the search can be implemented in parts sequentially or in parallel. The size n of each part can be chosen based on the RAM required to perform the MCS mapping for the corresponding part. s .

[0092] For relatively small search texts, or situations where you want to break the search text into smaller parts and search each separately, you can use the following techniques:

[0093] 1) Create multiple MCSs, each of which meets the given search requirements, such as the size of the search query and the similarity threshold (maximum number of mismatches). Arrange the set based on the number of matches in the filter. The general rule is: the larger the number of matches, the fewer comparisons between the search query and the search text, requiring more memory (RAM) to perform the mapping and resulting in more "idle" checks.

[0094] 2) Measure the search speed of each MCS.

[0095] 3) Select the MCS that provides search results in the shortest time (ie, maximum speed).

[0096] The following table illustrates the above technique for calculating the minimum filter set (MCS) for a query term of size 10 characters. The table illustrates the above technique for finding all similar words in a text that contain no more than four mismatches.

[0097] First, consider all combinations of matches and non-matches and create a minimum filter set (MCS) so that each combination contains at least one filter from the filter set. Mark matches as "1" and non-matches (or don't care) as "0", and:

[0098] ABCDEFGHIJ-Query

[0099] 1111110000 - matching unmatched combination

[0100] ABCDEFQQQQ-text

[0101] Table A below illustrates the creation of an MCS corresponding to the above criteria, where the filters are shown in bold. When selecting a filter from the combination, the first filter that contains four matches is taken:

[0102] Table A

[0103]

[0104]

[0105] Starting with the empty set {}, the first filter is derived from the combination of the first occurrence of the four matches to the search term, in this case {"1111"}, and added to the MCS set. This filter appears in the next two combinations and is therefore not added again. The fourth combination does not contain any filters in the current filter set (e.g., the current filter set only contains the filter "1111", which is not found in the fourth combination "1110111000"). Therefore, new filters are derived to ensure that each combination includes at least one filter (e.g., "covered"). For example, a new filter can be obtained from the non-covered combinations by taking the first fragment of this combination with a number of matches corresponding to a threshold, defining the number of matches in this filter. In the example above, this fragment corresponds to the sequence "11101", with four 1s, corresponding to the threshold of four matches, and located in positions 1 to 5 of the fourth combination (with 1 non-match at position 4). This fragment is added to the filter set as the second filter. The next four combinations include filters that are already included in the MCS and are therefore "covered". The second to last combination provides a third filter: "101101", which is not yet included in the MCS. This filter is added as the third filter. The final combination includes a filter that was already included in the MCS, namely "1111", and is ignored. The result is an MCS with three filters that cover all the combinations considered in the table.

[0106] According to another embodiment of the invention, the complete set of combinations consists of combinations having a match in the first position or alternatively in the first and last positions.Such sets are advantageous because they can significantly reduce the overall set without losing any relevant matches.

[0107] Exemplary embodiments of the present invention will be described below with reference to the accompanying drawings.

[0108] Figure 1 1 is a flow chart of a method for generating a minimum filter set 100. The generation of the minimum filter set is a pre-processing stage, which precedes the search for the search terms in a given text. For word similarities with only a predetermined threshold, the filter set can be reduced by considering only necessary but not sufficient conditions.

[0109] The computer system is used to perform the following steps:

[0110] At 102, an empty filter set is defined. At 104, a set of all combinations of matches and empty positions for each search term in at least one search term is generated. Due to memory size limitations, combinations may be dynamically generated and then deleted from memory.

[0111] After the set of all combinations is generated, at 106, each combination in the set is checked to see if it contains a filter in the current filter set.

[0112] If no filter in the updated filter set is found in the combination, a filter is derived from the combination and the newly derived filter is added to the filter set (108). The newly derived filter may be a fragment of the combination, or may be derived randomly. If the checked combination contains at least one filter in the filter set, the next combination is checked. The size of the filter may be based on the number of matches, which may be received as input. Alternatively, the number of matches may be randomly selected. A fragment of a combination may also be referred to as a shape.

[0113] The amount of memory required to preprocess the text is proportional to the size of the text to be searched, multiplied by the size of the filter set. If the size of the preprocessed text is too large, you can consider a smaller shape size, i.e. a smaller number of matches. The greater the number of matches, the larger the filter set, which will increase the amount of memory required for the preprocessing text stage.

[0114] Alternatively, the text may be divided into smaller portions, and each portion may be processed separately in a parallel computing environment (eg, two or more computer processors may be used simultaneously).

[0115] After all combinations in the set of all combinations have been checked, the filter set is reduced according to the following steps: Each filter in the filter set is checked (112) by checking each combination in the set of all combinations (110) to confirm whether the filter is a single occurrence in the combination. A single occurrence means that although a combination may contain multiple occurrences of the same filter, if different filters in the filter set appear in all combinations, the checked filter is not a single occurrence. If the checked filter is not a single occurrence, it is deleted from the filter set (114). Then, the next filter in the updated filter set is checked until all current filters in the filter set have been checked. The resulting filter set only includes filters that are a single occurrence in at least one combination. This is a minimum filter set (116).

[0116] The embodiment of the present invention provides detailed steps for creating a filter set, as shown above. Figure 1 Alternative methods for these steps are described in Figure 2, and are described below. All filters that appear in each combination are identified (202). If the number of all possible filters is too large, other conditions may be imposed, such as a fixed upper limit on matching, a maximum span, and the selection of filters is limited to a fragment of the combination, to name a few possible limitations. A weight or score is assigned to each filter. For example, at 204, a score is determined based on the sum of the combinations in which the filter appears. The filter with the highest score is added to the filter set (206). The combination containing this filter can be marked as checked (208) or removed from the set of combinations being checked. Process 200, i.e., steps 202-208, can be repeated for any unmarked combinations that have not been removed until a good filter set is found. A "good" filter set is a filter set that can be used to conduct faster searches or produce higher quality results compared to prior art methods. For certain search conditions, executing process 200 provides an effective filter set in a shorter time compared to prior art methods. To produce a minimal filter set, steps 202-206 may be repeated for any unmarked (or non-removed) combinations until a satisfactory filter set is established, or the set of combinations is empty or completely marked as checked.

[0117] refer to Figure 3 , shows an exemplary diagram of using a filter set. For the search term, W = "AARRCCDDEE", and MCS, for S = {XXX, XX-X}, corresponds to two filters. The first filter corresponds to three consecutive matches (XXX), and the second filter corresponds to two consecutive matches, followed by no match and the third match (XX-X). The extracted set S W The 15 terms listed in Table 300 are included, of which the 12th term is RC-D. Given a portion of text: "GGGGGGGRCGDGGGGGGGGGG", this portion contains only one instance of the term RC-D, corresponding to configuration XX-X, and no other terms. Therefore, the comparison with the term RC-D occurs only in one position, where the character "G" in the text is aligned with the character "C" in the search term, resulting in a mismatch:

[0118]

[0119] According to another embodiment of the present invention, instead of generating or calculating a filter set for each user, a filter set may be calculated for a wide range of parameters and stored in an accessible public database.

[0120] Reference now Figure 4 , Figure 4is a diagram of several exemplary minimum filter sets obtained by embodiments of the present invention. The advantages of these minimum sets over naive algorithms are described as follows.

[0121] Word size |W|=20, similarity threshold T=60% (i.e., up to 8 mismatches are allowed). |S|=size of the minimum filter set S, n C = the number of terms generated by searching the term in all filters S, M = the number of matches in the filter. The advantage (A) compared to the naive algorithm is calculated as d M / n C , where d = the size of the alphabet, d = 20.

[0122] According to another embodiment of the disclosed technology, a heuristic method is used to find a good filter set. The heuristic method is applied by forming multiple filter sets according to a predetermined order and assigning a score to each filter set. The score can be assigned to each filter set based on (i) its size, i.e., the number of filters in the set; and / or (ii) effectiveness, i.e., the number of terms generated from the filter set. In general, the length of the filter affects the number of terms generated from the filter. That is, as the filter length increases, the number of terms decreases, thereby increasing the search speed.

[0123] Reference Figure 5A , a good set of filters can be obtained by comparing the scores assigned to each of the plurality of filter sets. Each filter set can be formed after first performing the following two actions: i) generating a complete set of filters and assigning a weight or score to each filter (502), and ii) sorting the filters according to the assigned scores (504). The sorting can be performed in descending order.

[0124] Each filter set (506) may be formed by the following steps: generating and initializing the filter set. If it is not the first filter set, one or more filters in the complete set of filters are ignored (508). The filter with the highest score is added to the generated filter set. Any combination containing the filter with the highest score is marked as checked and can be removed from the set of combinations to be checked. The above operation is repeated for the unmarked combinations that are not removed until a satisfactory filter set is established, or until the set of combinations is empty or the set of combinations is exhausted (all marked as checked) (510). All non-retainable filters are removed from the filter set, and reasonable scores are assigned to the filter set according to one or more predetermined criteria (512). The filter set with the best reasonable score is retained (514). The best reasonable score can be the highest score or the lowest score, depending on the optimization criteria. The above steps for forming the filter set can be repeated until the step of ignoring one or more filters 508, which includes removing all filters except the filter with the highest score. In some cases, the loop may not terminate due to the large number of filter sets that may be ignored. In this case, these steps can be repeated according to the available computer power and time to obtain a satisfactory solution.

[0125] The predetermined criteria for scoring each filter set may be the number of filters in the filter set and the number of terms generated from the filter set for a given search term.

[0126] The criteria for including / excluding any of the one or more filters in 508 may be performed based on elements in a predetermined series of arithmetic sequences. The predetermined series of arithmetic sequences may be such that each element in the series is an arithmetic sequence having a common difference d. In a non-limiting example, in step 508, the formation of the second filter set may exclude the second highest filter, in step 508, the formation of the third filter set may exclude the second and third highest filters, in step 508, the formation of the fourth filter set may exclude the second, third, and fourth highest filters, and so on. Thus, according to this example, d=1 and the elements in the series of arithmetic sequences are ({2}, {2,3}, {2,3,4}, etc.).

[0127] Reference now Figure 5B , which illustrates a heuristic method for finding a satisfactory set of filters constructed and operated according to another embodiment of the disclosed technology. The computer system performs the following steps:

[0128] Generate the complete set C of filters.

[0129] i. Assign a weight (or score) to each filter; and

[0130] ii. Sort the filters in descending order according to their respective weights (502b). For each filter i in C (starting from the most frequent, i.e., highest weight) (504b)

[0131] The filters in aC (whose frequency is lower than that of filter i) are marked (506b) according to the order of selection (described in detail below).

[0132] b.According to Figure 2 The method shown in creates a minimal set of filters, starting from i, and repeating steps 206-208 for unmarked filters (508b).

[0133] c. Calculate the effectiveness of the minimum filter set.

[0134] d. If the effectiveness is better than the effectiveness of the best minimum filter set, the minimum filter set is retained as the best minimum filter set (510b).

[0135] The filter in eC is unmarked and the next marked filter is checked (512b).

[0136] f. Repeat the above steps (514b) for the next filter i. The output of these steps is the optimal minimum filter set (516b).

[0137] Reference now Figure 6 , Figure 6 A heuristic method is shown for reducing the number of terms generated from a filter set constructed and operated according to another embodiment of the disclosed technology. Reducing the number of terms generated from the filter set can increase the search speed. The method takes into account the fact that filter relevance exists or does not exist at different positions in the term, and constructs one or more filters for each position of the search term. Thus, position-dependent filter sets are formed, and each filter set is associated with a specific position in a given search term. The filters can have different sizes. One optimization disclosed is to include only filters that are actually found at specific positions in the search term.

[0138] The computer system performs the following steps: Form a filter set (MCS) (602) (for example, by implementing one of the above algorithms to create a filter set that covers all possible combinations). Set the current position to the first position (604). Associate the formed filter set with the current position (606). The first position can be the leftmost position for a given search term, or a "low occurrence" position, as described below. Alternatively, select some sequence of positions S in the search terms (the simplest sequence can be S = {1, 2, ..., n-k+1} (k is the span of the minimum filter). According to the present invention, any other order can be used, and the first position in S is set to the current position (604).

[0139] All combinations in the set of all combinations of filters at the current position are removed, or marked as checked (608). The next position in S is set as the current position (610). The current position can be moved to the right by 1. After step 610 as described above, the size of the filter set is reduced to a minimum, that is, a minimum filter set is generated. Any filter that is not determined to be a retainable filter in the set of unremoved or unmarked combinations will be removed, or marked as checked, and the updated filter set will be set as the current filter set (611). Steps 606 to 611 can be repeated until all combinations are used up. The obtained filter set is the largest for the first position of a word, decreasing from the left position to the right position in the sequence S.

[0140] Figure 6 The output of the method described in is a set of sets. Each set in the set is associated with a certain position in the search term. One set (relative to the starting position) is maximal (and the form of this set should be used for the mapping).

[0141] The effectiveness of a location-dependent filter set can be calculated as the sum of the comparisons of the filters associated with each location (n c ). The application of the filter set associated with the position can be performed by the following steps: mapping all the words in all the filters in the text; extracting all the words for each position of the term according to the counterparts associated with the filter set for that position; comparing the term with the corresponding position in the text according to the mapping of the extracted words.

[0142] Table B below illustrates the results of scoring the filter sets. Scores are assigned based on the number of terms generated from the filter set for a given search term. Consider Figure 4The first set in: S = {XXX, XX-X}. This set is found for a search term size of |W| = 20 and a similarity threshold T = 60% (i.e., a maximum of 8 mismatches are allowed). The number of terms generated from the filter set for a given search term is 35. Consider a search term: "abcdefghijklmnopqrst". The following table shows that for the filter XXX, the terms will be:

[0143] Table B:

[0144]

[0145]

[0146] The number of words per filter n C( i ) It can be calculated as:

[0147] n C(i) =|W|-|C(i)|+1

[0148] Where: |W| = size of search term W, |C(i)| = span of filter i, including spaces.

[0149] Reference now Figure 7A-7C , which shows several position-dependent filter sets constructed and operated according to another embodiment of the disclosed technology. By using position-dependent filter sets suitable for a given search term, the effectiveness of the algorithm can be significantly improved. This can take into account the occurrence of each extracted term in the text in the formed position-dependent filter set. Thus, "low occurrence" positions of a given search term can be associated with the largest set, while "high occurrence" positions can be associated with the smallest set. Figure 4 The filter set is calculated using the search conditions described in (search term size |W| = 20, and similarity threshold T = 60% (i.e., up to 8 mismatches are allowed)). The position order is S = {1, 2, 3, ...}. The starting set is Figure 4 The set given in .

[0150] See also Fig. 7A , which shows an example of a location-dependent filter set for the number of matches in the filter M=3, and Figure 4 This means that from the beginning of the search term until the 13th position, it should be compared with the "XXX" form. Figure 7B , which shows an example of a location-dependent filter set for the number of matches in the filter M=4, and Figure 4 The second row of Table 400 corresponds to Figure 7C Table C in the figure compares the scores of position-dependent sets with Figure 4 , where the score is determined based on the number of terms generated from the filter set for a given search term.

[0151] According to another exemplary embodiment of the present invention, an array of sets may be prepared as follows: for a given search term, starting from different positions, the lowest occurrence position is selected. A set of sets starting from this lowest occurrence position is used. Similarly, the array of sets may contain a set of sets with different assignments for the next position in S. In the latter case, the first and second lowest occurrence positions may be selected, and a set of sets with the corresponding first and second lowest occurrence positions may be used. The process continues for a larger number of positions.

[0152] The "low occurrence" position is calculated as the sum of all word occurrences starting from that position for all filters in the filter set associated with that position. All words of all filters in the filter set can be mapped in the text to provide word occurrences (text pre-processing).

[0153] Embodiments of the present invention can be used in several implementations: (i) searching for a word or a group of words among all similar words in a given text; (ii) searching for a word or a group of words among all similar words in an online stream; and (iii) searching for each word in the first text's similar words in a second text; (iv) indexing, i.e. searching for each word of text-similar words in the same text (which can be viewed as the construction of a word similarity graph for a given text).

[0154] The use of the complete set of all filters for searching is described in “Frenkel, ZMand Trifonov, EN “Evolutionary networks in the formatted protein sequence space”. Journal of Computational Biology, 14(8): 1044-57, Oct. 2007”. The complete set of all filters is generated by the exhaustive combination operator.

[0155] Exemplary embodiments of the present invention provide a method that uses the above-mentioned minimal set of filters to search for words similar to a word or a group of words in a given text, thereby providing an improvement over the exhaustive method. Text indexing is similar to constructing a similarity graph for text. The similarity graph is characterized by multiple nodes connected by multiple edges. The nodes represent multiple repeated character sequences, corresponding to the size of the search terms; each edge connecting a pair of nodes is based on a respective similarity threshold between the nodes. The similarity graph can be compressed according to conventional methods such as the method described in [PCT / IL2016 / 051220]. Clustering of words can significantly reduce the time required to complete a search.

[0156] This can be explained as follows. Consider the following lemma:

[0157] If the following conditions are met:

[0158] 1. The size of words A, B, C, and D are all m.

[0159] 2. Word A is similar to word B and has the maximum number of mismatches x, that is, the distance d between A and B is ((d(A,B)≤x))

[0160] 3. Word C is similar to word D and has the maximum number of mismatches y, that is, the distance d between C and D is (d(C,D)≤y)

[0161] 4. Word A is different from word C and has the maximum number of matches z, that is, at least mz mismatches. Therefore, the distance d between A and C is (d(A,C)≥mz)

[0162] Then, within the maximum number of mismatches given by max{0;m-(x+y+z)}, word B is similar to D and the following relationship holds:

[0163] d(B,D)≤max{0;m-(x+y+z)}

[0164] The proof of this result is as follows. Consider two circles, the first with radius x, center A, and B on the side. The second circle with radius y, center C, and D on the side. The circles can overlap for the case of minimum distance between A and C, or can be spaced apart for the case of maximum distance between A and C. Two possible cases are: 1) the minimum distance between B and D is given by: m-(x+y+z), and 2) the minimum distance between B and D is 0 (i.e., the circles overlap).

[0165] As described below, the above lemma can be applied to perform a search. To check whether two clusters match within 60% similarity (i.e., a maximum of eight mismatches or at least 12 matches for a sequence of 20 characters), clusters are constructed to meet a similarity criterion. For example, one such criterion may be that all sequences in each cluster are similar to the corresponding cluster seed sequence within at least 90% of the similarity measure (i.e., a maximum of two mismatches).

[0166] If the maximum similarity of each seed sequence is 35% (i.e. a maximum of 7 matches), the maximum similarity between any two sequences of the two clusters cannot exceed 11 matches (i.e. 55%). In other words, the minimum distance between the clusters is at least 9 mismatches (20-(7+2+2)=9). In this case, it is impossible to meet the 60% similarity requirement between the two clusters. Therefore, there is no need to check such cluster pairs, thereby optimizing memory, time and computing resources. If the first cluster has m sequences and the second cluster has n sequences, the economy is m*n-1 comparisons. It can be noted that the probability of finding a sequence fragment with a similarity of 35% in the database is relatively low (for an alphabet of 20 letters, this ratio is about 10 -5 ).

[0167] The following indexing algorithm can be applied, where th1 represents the threshold for the minimum number of matches required for indexing and th2 represents the maximum number of mismatches required for clustering:

[0168] 1) Divide the text into a set of clusters of words. The clusters are formed as follows:

[0169] a) For each word not in a cluster:

[0170] b) finding all similar words at the chosen threshold th2, i.e., by using the k mismatch search described above;

[0171] c) Sort the words in the cluster: words that are more similar to the seed word are assigned smaller indices;

[0172] 2) Check whether the connection between clusters meets the following conditions: if at least one word in the first cluster is similar to at least one word in the second cluster with similarity threshold th1, then connect the clusters,

[0173] a) For the seed words (determined according to the k mismatch search technique above), find all seed words that are similar within at least (K1=th1-2*th2) matches. That is, similar words can only appear in this type of cluster;

[0174] b) Compare only the words of cluster index i and j, where K1+K i +K j≥thl (i.e. K1 is the number of matches between seed words, K i and K j are the number of mismatches between words i and j and the corresponding seed words, respectively).

[0175] c) If a connection is detected, the process terminates.

[0176] 3) Complete the indexing. This step is optional, as for many applications, identifying connections between clusters is sufficient.

[0177] a) Compare all nodes in each cluster;

[0178] b) Compare the words of the connected clusters (similarly, only compare the words of cluster index i and j, where K1+K i +K j ≥thl, where K1, K i and K j Same as above.

[0179] The text indexing algorithm is now given as follows: i) All the words present in all filters are mapped to the text. ii) For each search term, all the words in all filters are extracted and the term is compared with the search text based on the mapping of the extracted words. For fast access, the address of the extracted word can be used.

[0180] According to another exemplary embodiment of the present invention, under certain conditions, the words in all filters in the search terms are mapped and stored in a suffix tree data structure for a set of search terms. The comparison with the entire set of filters is performed in linear time. These conditions may include situations where preprocessing of the text cannot be performed. For example: when the text is not available, or the text stream is online or the amount of available memory is insufficient.

[0181] In one exemplary embodiment, "low occurrence" positions of a given search term may be associated with the largest set, while "high occurrence" positions may be associated with the smallest set. Thus, two position-dependent filter sets are calculated for two sequences of positions S1 and S2.

[0182] Referring to Table C below, two position-dependent filter sets are shown for two possible sequences of positions S1, S2 in the search term, where XXX represents three consecutive matches, and XX-X represents two consecutive matches, followed by a wildcard and a match:

[0183] Table C

[0184]

[0185] Consider a search term W = 'ABCDEFGHIJKLMNOPQRST'. Assume that the occurrences of all three-letter words in the text are the same - equal to zero, but do not include "BCD" (i.e. XXX) and "BC-E" (i.e. XX-X), which are equal to 1000 occurrences each. In order to find all words in the text that are similar to the search term W (similarity threshold is 60%), according to the set of sets on the left (calculated as S1), using the sequences "BCD" and "BC-E", this requires 2000 comparisons, while using the set of sets on the right (calculated as S2) (where "BC-E" is not used), finding all similar words with 60% similarity to W will require 1000 comparisons. Therefore, for such a search term and such a text, the set on the right is more effective.

[0186] When preparing a position-dependent set of filter sets as described herein, the resulting position-dependent set depends on the order or "sequence" of the positions in the search terms, such as shown in Table C above.

[0187] Although the expected number of comparisons obtained using different groups of filter sets can be the same, the number of filters corresponding to specific positions may be substantially different, affecting performance. After mapping filters to specific text for searching, the occurrences of the search terms in all filter forms are known and can be used to further optimize the search. For example, the occurrences of "BCD", "BC-E", and any other terms in the search terms in the above examples are known in the search text. Therefore, for a given search term, a set of corresponding sets that provide the minimum number of comparisons with the text can be selected to increase the search speed. The position sequence that provides the minimum set of comparisons with the search terms is selected. However, the initial filter set for the "universal" filter set or all filter sets is the same. This allows the same mapping to be used for all possible position sets of the filter set.

[0188] According to yet another exemplary embodiment of the present invention, a search for a word or more words may be performed by initially constructing a similarity graph for a given text. The similarity graph may be compressed according to a higher similarity threshold, where connections between "supernodes" indicate that there is at least one connection between corresponding subgraphs in the main graph. A minimum set of filters may be applied in the process of discovering connections between supernodes.

[0189] Here is an example of indexing using a compressed similarity graph:

[0190] Task: Build a similarity graph for the text T, that is, find all similar words with at least a threshold thr1 for each word.

[0191] i. Create clusters (of a chosen maximum size) as subgraphs where the join condition is a threshold thr2 that is higher than thrl. For example, thr1 is 60% match and thr2 can be 80-90% match. It can be shown that this requires fewer filter sets and also less time and RAM for computation.

[0192] ii. Compute connections between clusters (this is much faster than full network computation. This means that a cluster is called “connected” if there is at least one pair of nodes (connected by thrl) (the first one is from one cluster and the second one is from another cluster).

[0193] iii. Compute the similarity graph based on the computed connected clusters. This task is also much faster than direct computation (since only comparisons should be made within clusters and between connected clusters).

[0194] According to yet another exemplary embodiment of the present invention, the k-unmatched search may be lossy, in the sense that matching words will be found in the text within a probability threshold. The algorithm described above is "lossless", i.e., if the set of combinations considered contains all possible combinations of matching and spacing positions for a given threshold and search term size, any words similar to the search terms within the selected threshold will not be lost. However, the disclosed technology can also be used as a "lossy" algorithm, i.e., similar words are found with at least some predetermined probability. One of the advantages of this approach is that the search speed is improved and the RAM required for preprocessing is reduced.

[0195] In order to adapt this technique to a "lossy" algorithm, a set of filters is constructed that covers only a portion, i.e., a subset, of the complete set of combinations of positions for matches and gaps. The ratio of the size of the subset to the complete set should be greater than or equal to the predetermined probability of finding the search term. In this case, the algorithm is repeated until all subsets of the complete set of combinations of positions for matches and gaps are marked or removed, as opposed to performing this operation until the entire set of combinations is exhausted.

[0196] In some applications, the search terms may not be restricted to a selected size (with a corresponding number of allowed mismatches). Instead, the search terms may have an allowed size (S) that may, for example, be within a predetermined lower bound (S). L ) to a predetermined upper boundary (S U ). The number of mismatches allowed can also vary between a high and low mismatch threshold. For example, the number of mismatches allowed might be proportional to the size of the search terms, the context in which they are applied, the characteristics of the text, to name a few. In such cases, it might be useful to create "generic" sets of filters that can be applied to a variety of different contexts across a range of search term sizes and / or mismatch numbers.

[0197] Although for the allowed search term size S, a corresponding set of filters is needed to provide optimal search speed, it is not necessary to repeat the mapping (which requires RAM resources) for each different size in the allowed range of S. An implementation of an algorithm that can calculate a "generic" set of filters is given below:

[0198] I is S L Construct a filter set (via one of the above algorithms);

[0199] ii. Add this collection to the set of filters (SFM) used for mapping;

[0200] iii. For S L+1 To S u Each S in i , for example, using SFM as an initial set (ie, replacing the initial empty set used in the algorithm), a filter set is constructed by the algorithm described in paragraph 1 of the claims.

[0201] iv. If a new filter is added to the collection, the filter retention check should start from the new filter.

[0202] v. Add new (retainable) filters to SFM.

[0203] As a result, filter sets are constructed that contain sets for each possible size of search terms (with corresponding similarity conditions). However, only filters from SFM are present in these sets. Therefore, only these filters should be mapped in the pre-process.

[0204] For search terms |W| that are larger than S U In the case of left and W right , for each word, all similar words (with the same similarity threshold (%)) should be detected in the text. right The previous flow text of similar words should be consistent with W left For comparison, and with W left The subsequent flow text of similar words should be the same as W right Make a comparison.

[0205] The techniques described herein can be applied to any number of contexts. For example, when searching for words in a dictionary, the search terms typically must go through a "normalization" phase to account for differences in spelling, tense, singular / plural, etc. Thus, the techniques disclosed herein can be used in place of such a normalization phase, or in conjunction with normalization to provide better results.

[0206] The set of filters obtained as described above can be suitable for searching in a list of words (dictionary). In this case, the position of each word in each filter should be taken into account in the mapping. Only those words in the dictionary that contain the corresponding filter in the same position as the search term should be selected for comparison with the search term.

[0207] Other techniques known in the art can be used in conjunction with the systems and methods disclosed herein to enhance performance. In some cases, the size of the alphabet can affect the performance or effectiveness of the algorithm. A smaller alphabet can reduce the combinatorial complexity of the search, so in some cases, two or more letters can be clustered into one letter to reduce the size of the alphabet, thereby improving the effectiveness of the algorithm. A possible alternative is to cluster two or more letters from the initial alphabet to one letter of the new alphabet. Alternatively, in some cases, increasing the size of the alphabet may reduce the number of occurrences per filter. For example, DNA text has a 4-letter alphabet {a, t, c, g}, corresponding to the four nucleotides present in DNA. Therefore, the probability of occurrence of any three-letter word is 1 / 4*1 / 4*1 / 4=1 / 64. A size of 16 (=4) can be constructed from all two-letter combinations. 2 ) is as follows: {aa, at, ac, ag, ta, tt, tc, tg, ...}. Now, the probability of any three-letter word appearing will be 1 / 16*1 / 16*1 / 16=1 / 4096, which is much smaller. This can greatly improve the search efficiency. The alphabet size can be increased or decreased accordingly.

[0208] Other implementations: A database of filters can be created for different applications. Genetic algorithms can be used to improve the quality (effectiveness) of the filter set.

[0209] Reference now Fig. 8A, which shows a computer system 800 constructed and operated according to an embodiment of the disclosed technology for performing any of the steps disclosed herein. The computer system 800 includes at least one processor 802 (e.g., CPU), a random access memory (RAM) unit 804, a disk storage unit 806, a transceiver 808, and a power supply 810. The at least one processor 802, the RAM unit 804, the disk storage unit 806, and the transceiver 808 are electrically coupled to the power supply 810. The RAM unit 804, the disk storage unit 806, and the transceiver 808 are communicatively coupled to the at least one processor 802. The transceiver 808 is used to send and receive data, including at least one or more program code instructions, and data related to the k-unmatched search technology disclosed herein. The transceiver 808 can communicate with another computer system (not shown) using any suitable wired or wireless protocol, for example, through electrical or optical cables, short-range and medium-range communication methods (e.g., Bluetooth, Wifi, IR), and longer-range communication methods (e.g., RF and cellular networks), etc.

[0210] The RAM unit 804 and the disk storage unit 806 are each operable to store one or more program code instructions and data related to the k-unmatched search techniques disclosed herein, such as any of which may be received via the transceiver 808. The at least one processor 802 is operable to read the one or more program code instructions and data related to the k-unmatched search techniques and execute the program code instructions on the data to perform any of the steps described herein. The at least one processor 802 is operable to write the results of the execution to the RAM unit 804 and the disk storage unit 806. The at least one processor 802 is operable to request data from and provide data to another computer system (not shown) via the transceiver 808. The at least one processor 802 is operable to provide the results of the execution to a user, such as via a user interface (not shown).

[0211] Reference now Figure 8B , while still referring to Fig. 8A , which shows a networked computer system 820 constructed and operated according to another embodiment of the disclosed technology for performing any of the steps disclosed herein above. The computer system 820 includes a plurality of nodes 800A, 800B, ..., 800N and a communication network 822. Each of the nodes 800A, 800B, ..., 800N corresponds to Fig. 8A Therefore, each of the nodes 800A, 800B, ..., 800N includes at least the above Fig. 8AThe components mentioned, such as at least one processor 802 (e.g., CPU), a random access memory (RAM) unit 804, a disk storage unit 806, a transceiver 808, and a power supply 810. The nodes 800A, 800B, ..., 800N are operable to communicate with each other via a communication network 822 and the respective transceivers 808 of each of the nodes 800A, 800B, ..., 800N. The computer system 820 is operable to perform any steps described herein in a distributed manner on the nodes 800A, 800B, ..., 800N using conventional techniques known in the art of distributed computing. The steps may be performed in any combination of parallel and serial computing.

[0212] Computer system 800 and computer system 820 are each configured to perform at least the following steps:

[0213] Receive a size of a search term and a similarity threshold. Generate a filter set. Form the filter set based on a set of combinations of matches of the size of the search term and the position of the slot according to the similarity threshold. Each combination includes at least one filter from the filter set. In some embodiments, each combination in the set of combinations includes a match at a first position of the respective combination. In some embodiments, each set of combinations also includes a match at a last position of the respective combination. Reference Fig. 8A , the processor 802 receives the search terms and the similarity threshold, for example, via the transceiver 808. The processor 802 forms a filter set based on a set of combinations of the size of the search terms and the position of the slots according to the similarity threshold. Each of the combinations contains at least one filter in the filter set.

[0214] For each filter in the filter set, when there is at least one combination in the set of combinations that contains only the filter, the corresponding filter is determined to be a retainable filter. When the filter is not a retainable filter, the filter is removed from the filter set. Fig. 8A When at least one combination in the set of combinations contains only the filter, the processor 802 determines that the filter is a retainable filter. If the processor 802 determines that the filter is not a retainable filter, the processor 802 removes the filter from the filter set.

[0215] The filter is formed according to the following steps: Initialize the filter set. For each selected combination of the combined set: search the selected combination for the occurrence of the filter in the filter set; when the occurrence of the filter is not found in the selected combination, a new filter is derived from the selected combination. The new filter is added to the filter set. In some embodiments, the new filter is a fragment of the selected combination. In some embodiments, a randomization process is used to determine any one of the new filter and the number of matches of the new filter. In some embodiments, the number of matches is a predetermined number. In some embodiments, multiple new filters are derived from the selected combination, and scores are assigned to each of the multiple new filters. When the score of the new filter is higher than the score of any other new filter of the multiple new filters, the new filter is added to the filter set. In some embodiments, scores can be assigned to each filter based on any combination of the following parameters: the expected number of comparisons between the search terms and the text corresponding to the application filter; the frequency of occurrence of the filter in the combined set; the number of matches in the filter; and the span of the filter. Reference Fig. 8A , the processor 802 forms a filter set as follows: the processor 802 initializes the filter set. For each selected combination in the set of combinations: the processor 802 searches the selected combination for the filter in the filter set. If the processor 802 does not find the filter in the selected combination, the processor 802 derives a new filter from the selected combination and adds the new filter to the filter set.

[0216] The second filter set may be created according to the following steps: one or more filters may be selected from the filter set. Each selected filter may be replaced by at least one different filter, wherein the selection and replacement are performed in an order sorted from the highest score of the selected filters to the lowest score of the selected filters, wherein each combination may contain at least one filter from the second filter set. The second filter set is selected from the filter set according to a reasonable score. The reasonable score may be calculated using one or more of the following parameters: an expected search calculation speed determined based on a sum of an expected number of comparisons of all filters in the filter set; and an amount of storage required to search each of the corresponding filter sets. Reference Fig. 8A , the processor 802 creates a second filter set according to the above steps. The processor 802 selects the second filter set from the original filter set according to the reasonable score.

[0217] Multiple filter sets may be generated as follows. Each position in the search term corresponds to at least one of the multiple filter sets. Each of the combinations contains at least one filter from at least one of the multiple filter sets at a position in the corresponding combination. The position of the search term corresponding to at least one of the multiple filter sets corresponds to the position in each combination. Fig. 8A , the processor 802 generates multiple filter sets according to the above steps.

[0218] Each of the plurality of filter sets is formed as follows: a combined set is designated as the current set of the combination, and a filter set based on the current set of the combination is set as the current filter set. A sequence of positions in the search term is obtained. Thereafter, for each position, a filter set associated with the position is generated according to the sequence. If the combination contains a filter in the generated filter set associated with the position, the combination in the current set of the combination is updated. Reference Fig. 8A , processor 802 creates multiple filter sets. Each position in the search term corresponds to at least one of the filter sets. Each combination contains at least one filter from one of the multiple filter sets. The position in the combination where the filter is found corresponds to the position in the search term that corresponds to the filter set containing the filter (i.e., the filter in the combination).

[0219] When a filter is not retainable with respect to the current set of combinations, the filter is removed from the current set of filters, wherein the filter is retainable with respect to the current set of combinations when there is at least one combination in the current set of combinations that contains only the filter. Fig. 8A When the filter cannot be retained, the processor 802 removes the filter from the current filter set.

[0220] The set of filters associated with a position is formed as follows: Get the set of combinations, and the current set of filters and the position. For each filter of the current set of filters: When the filter is found in at least one combination in the set of combinations for the position, add the filter to the set of filters associated with the position, and delete, remove, or otherwise mark the at least one combination from the set of combinations accordingly. Fig. 8A , the processor 802 associates the filter set with the location according to the above steps.

[0221] In some embodiments, a plurality of sets of filters are formed, wherein each of the plurality of sets of filters corresponds to a different sequence of the plurality of sequences of positions in the search terms. A set of filters can be selected from the plurality of sets of filters based on an expected number of comparisons to be performed when the selected set of filters is applied to perform a k-unmatched search. The expected number of comparisons is a function of a given text and a given term being searched.

[0222] In some implementations, a lossy threshold is received. A combined subset is created based on the lossy threshold and the similarity threshold for the search terms, wherein generating the set of filters includes generating filters for the combined subset. Fig. 8A , processor 802 receives the lossy threshold, such as via transceiver 808 .

[0223] In some implementations, the size of the search term corresponds to a plurality of possible sizes of the search term, and the similarity threshold corresponds to a plurality of similarity thresholds, each similarity threshold corresponding to one of the plurality of possible sizes of the search term.

[0224] In some implementations, the execution of the k-unmatched search is applied to cluster multiple words of the text. Clustering can be applied to speed up the text indexing step of the k-unmatched search.

[0225] The present invention is not limited to the above-described embodiments, but includes any and all embodiments within the scope of the appended claims. It will be understood by those skilled in the art that the disclosed technology is not limited to the technology specifically shown and described above. Instead, the scope of the disclosed technology is limited only by the claims, and its possible representative approximation is as follows.

Claims

1. A computer system for performing a k-unmatched search, comprising at least one processor, wherein the at least one processor is configured to: receiving the size of the search term; receiving a similarity threshold; and The filter set is generated by forming the filter set based on a set of combinations of positions of matches and positions of gaps of the size of the search terms according to the similarity threshold, wherein: Each of the combinations contains at least one filter from the set of filters.

2. The computer system according to claim 1, wherein: The at least one hardware processor is further configured to create a minimum set of filters by: For each filter in the filter set, when there is at least one combination in the set of combinations that only includes the filter in the filter set, determining that the filter is a retainable filter, and When the filter is not a retainable filter, the filter is deleted from the filter set.

3. The computer system according to claim 1, wherein: Each combination in the set of combinations includes a match at a first position of the respective combination.

4. The computer system according to claim 3, wherein: Each combination in the set of combinations includes a match at a last position of the respective combination.

5. The computer system according to claim 1, wherein: The at least one hardware processor is configured to form the filter set by: Initializing the filter set; For each selected combination of the set of combinations: searching the selected combination for occurrences of filters in the set of filters, When the occurrence of a filter in the filter set is not found in the selected combination, a new filter is derived from the selected combination, and the new filter is added to the filter set.

6. The computer system according to claim 5, wherein: The new filter is a combined fragment of the selections.

7. The computer system according to claim 5, wherein: Any one of the new filter and the matching number of the new filter is determined through a randomization process.

8. The computer system according to claim 7, wherein: The number of matches is a predetermined number.

9. The computer system according to claim 5, wherein: The at least one hardware processor is configured to derive a plurality of new filters from the selected combination, assign a score to each of the plurality of new filters, and add the new filter to the filter set when the score of the new filter is higher than the score of any other new filter in the plurality of new filters.

10. The computer system according to claim 9, wherein: The at least one hardware processor is further configured to assign the score to each filter in the set of filters based on a parameter selected from the group consisting of: an expected number of comparisons between the search term and text, respectively, when the filter is applied; The frequency of occurrence of the filter in the combined set; the number of matches in the filter; and the span of the filter.

11. The computer system according to claim 10, wherein: The at least one hardware processor is further configured to: Create a second filter set by: Select a filter from the filter set, replacing each of said selected filters with at least one different filter, said selecting and said replacing being performed in an order sorted from a highest score of said scores of said selected filters to a lowest score of said scores of said selected filters, wherein each of said combinations contains at least one filter from said second set of filters, and The second filter set is selected from the filter sets according to the rationality score.

12. The computer system according to claim 11, wherein: The reasonableness score is calculated using a parameter selected from the group consisting of: an expected computational speed of the search determined based on a sum of expected comparison times of all of the filters in the filter set; an amount of memory required to perform the search of the filter set.

13. The computer system according to claim 1, wherein: The at least one hardware processor is further configured to generate a plurality of filters in the filter set, wherein: Each position in the search term corresponds to at least one of the plurality of filters in the filter set, Each of the combinations includes at least one of the multiple filters in the filter set at a position in each of the combinations, the position of the search term corresponding to the at least one of the multiple filters in the filter set, and the filter set corresponds to the position in each of the combinations.

14. The computer system of claim 13, wherein: The at least one hardware processor is configured to form each of the plurality of filters in the filter set by: designating the combined set as a current set of combinations, and designating the filter set based on the current set of combinations as a current filter set; obtains a sequence of positions within the search term, According to the sequence, for each of the positions, generating a set of filters associated with the position, When the combination contains at the position a filter in the generated set of filters associated with the position, updating a combination in the current set of combinations, When the filter is not retainable with respect to the current set of combinations, removing the filter from the current set of filters, When there is at least one combination in the current set of combinations that only includes the filter, the filter can be retained relative to the current set of combinations.

15. The computer system of claim 14, wherein: The at least one hardware processor is configured to form the set of filters associated with a location by: obtaining the combined set, the current filter set and the position; For each filter in the current filter set: When a filter is found at the position in at least one combination in the set of combinations, adding the filter to the set of filters associated with the position, and The at least one of the combinations is deleted from the set of combinations.

16. The computer system of claim 1, wherein: The at least one hardware processor is further configured to: receive a lossy threshold; create a subset of the combinations based on the lossy threshold and the similarity threshold of the search terms, wherein generating the set of filters includes generating the filters with respect to the subset of the combinations.

17. The computer system of claim 1, wherein: The size of the search term corresponds to a plurality of possible sizes of the search term, and wherein the similarity threshold corresponds to a plurality of similarity thresholds, each similarity threshold corresponding to one of the plurality of possible sizes of the search term.

18. The computer system according to claim 15, wherein: The at least one hardware processor is further configured to form a plurality of sets of filters, wherein each of the plurality of sets of filters corresponds to a different sequence of the plurality of position sequences in the search terms.

19. The computer system of claim 18, wherein: The at least one hardware processor is also configured to select a set of filters from the multiple sets of filters, the selection being performed based on an expected number of comparisons to be performed when the k unmatched searches are performed applying the selected set of filters, wherein the expected number of comparisons is a function of a given text and a given term to be searched.

20. The computer system of claim 1, wherein: The at least one hardware processor is further configured to apply the performing of the k unmatched searches to cluster a plurality of terms of a text.

21. The computer system of claim 20, wherein: The at least one hardware processor is further configured to apply the clustering to accelerate a text indexing step of the k no-match search.

Images