Stable set-similarity joining method

Abstract

A method for set similarity binding is provided, where each set represents a process and each token represents a process step. The process includes a sequence of process steps executed in at least one source computer system. Thus, similar sets represent similar processes in a collection of processes. The method is based on a two-level signature scheme. After indexing the sets into an inverted list using a first signature, selected lists, especially long lists, are re-indexed using a second signature. As a result, the number of candidates and therefore the number of distance calculations required can be effectively reduced. Its experimental evaluation shows that the method consistently outperforms state-of-the-art algorithms for datasets with diverse characteristics, suggesting a stable solution for a wide range of applications.

Description

Technical Field

[0001] The present invention relates to a method for determining sets that are similar to a selected query set within a set of sets, and more generally, to the problem of set similarity join.

Background Art

[0002] In search of more information, an increasingly large amount of data is currently being recorded. In order to access relevant information within a dataset, especially a large dataset, a search operation is essential. However, finding exact matches within a dataset is not always sufficient, for example, when considering spelling or typing errors that frequently occur in text data. Therefore, a rather practical feature for matching records within a dataset is a similarity operator. A similarity operator finds similar records in the same or two different datasets based on a function that determines the distance between two records.

[0003] Set similarity join determines all pairs of similar sets from within the same set of sets or from two sets of sets. Set elements are usually called tokens, and tokens within each set are unique. Two sets are similar if the token overlap, i.e., the number of tokens that match in both sets, exceeds a predefined threshold.

[0004] The performance of state-of-the-art methods for determining set similarity join substantially depends on the dataset to which the method is applied. Since the characteristics of datasets vary widely, a broad landscape of methods for the set similarity join problem has emerged.

[0005] Using the Hamming distance selected for distance measurement, the landscape of known methods for set similarity join problems can be subdivided into two main groups. The first group includes methods that work well in set similarity join (SSJ) scenarios, and the second group includes methods that work well in Hamming space join (HSJ) scenarios.

[0006] Datasets classified as SSJ scenarios are characterized by a large token universe, i.e., a large total number of unique tokens in the dataset, a widely distributed set within the vector space, and a highly skewed distribution of tokens with common and non-common tokens. The most suitable method for the set similarity joining problem in SSJ scenarios is typically pruning the sets without token overlap between the prefix and the prefix of the search / query set, based on a variation of prefix filtering.

[0007] In contrast, datasets classified as HSJ scenarios are characterized by a small token universe and a substantially uniform token distribution. A second group of methods for the set similarity joining problem typically derives from a variation of the partition enumeration framework by partitioning each set and searching each partition individually using a threshold distance smaller than a predefined threshold distance.

[0008] Applying methods adapted for SSJ scenarios to datasets originating from HSJ scenarios results in significant drawbacks. Prefix filtering is based on the assumption that the tokens forming the prefixes of the sets are not common and result in many different tokens. However, in the case of a typical dataset from an HSJ scenario, the number of non-common tokens is limited due to both the small universe size and the more uniform distribution of tokens. Therefore, the first group of methods generates long inverse lists for datasets from HSJ scenarios, requiring numerous candidate validations. Note that each candidate validation involves calculating the distance between the candidate set and the search / query set.

[0009] Similarly, applying methods tailored to HSJ scenarios to datasets resulting from SSJ scenarios is a second-best approach. Due to the large universe size of the datasets, query-side enumeration becomes computationally expensive.

[0010] In reality, large datasets typically do not fully belong to either one or the other scenario. Therefore, any known method will at least partially operate outside the scenario in which they are best suited.

[0011] Therefore, the object of the present invention is to provide a method for solving the set similarity joining problem in a more stable and efficient manner for large real-world datasets, thereby overcoming the aforementioned problem at least partially.

[0012] According to the present invention, this objective is solved by the method described in the independent claim. Preferred embodiments and further developments of the present invention are defined in the dependent claims.

[0013] Therefore, a computer implementation method is provided for determining at least one similar set for a selected queryset within a set of sets. It should be noted, of course, that this method can also report an empty set if no similar sets are found for the selected queryset within the set of sets.

[0014] Each set represents a process, each token represents a process step, and a process comprises a set of process steps executed in at least one source computer system.

[0015] This method, a) A step of determining the first signature for each set, b) A step of indexing each set based on a determined first signature, wherein each indexed set is stored in at least one first record, and each of the at least one first record is recorded in a first data structure representing the first index, corresponding to the determined first signature, c) The step of selecting a subset of the first data structure based on a predefined cost function, d) For each first record in a selected subset of the first data structure, the step of determining a second signature for each indexed set, e) A step of re-indexing an indexed set based on a determined second signature, wherein each re-indexed indexed set is stored in at least one second record, and each of the at least one second record is recorded in a second data structure representing at least one second index, corresponding to the determined second signature. f) A step of generating a hybrid index from the complement of a selected subset of the first index and at least one second index, g) A step of exploring at least a subset of a hybrid index by calculating distance values ​​between each set of each record in the subset and the selected queryset, wherein the explored set is identified as one of at least one similar set if the distance values ​​are less than or equal to a predefined threshold.

[0016] At least one similar set represents at least one similar process to a selected process within the set of processes. In other words, a similar set represents similar processes within the set of processes, specifically process instances (executions of processes).

[0017] The method according to the present invention has the advantage of effectively reducing the number of distance calculations required in step g). It should be noted that a large number of required distance calculations makes set similarity joins computationally expensive. Therefore, despite the overhead of re-indexing a subset of the first index and generating a hybrid index, the runtime and possibly even memory usage for set similarity joins on large datasets can be reduced compared to the methods known from the first and / or second groups outlined above.

[0018] The reduction in the number of required distance calculations is achieved by re-indexing a subset of the first records in the first data structure into multiple second records in the second data structure. The purpose of re-indexing the first records in the first data structure is to distribute the indexed set of the first records into two or more second records in the corresponding second data structure so that the number of re-indexed sets in each of the two or more second records is less than the number of sets in the original first record.

[0019] A predefined cost function determines which first records of the first data structure should be re-indexed. Based on the characteristics of the second signature, only a subset of at least one second record of the second data structure remains as the (pre)candidate set, which must be validated according to step g). In other words, in step g), several second records can be selected based on the second signature, thereby reducing the number of sets for which the distance to the selected queryset is calculated, compared to the number of sets stored in the selected first records of the first data structure. Thus, the number of memory accesses required is reduced.

[0020] The inventors have found that using this two-level signature scheme for the set similarity coupling problem, by adjusting a predefined cost function and the selection of the first and second signatures, performs better than its peers across a wide range of datasets from SSJ scenarios, HSJ scenarios, and any mixture thereof. Therefore, the two-level signature scheme stabilizes the performance of the set similarity coupling problem on real-world datasets.

[0021] A queryset can be selected from the same set of records or from different sets of records in which similar records are searched. The former, a set similarity join, is also called a self-join.

[0022] Preferably, the first record of the first data structure is the first linked list, and / or the second record of the second data structure is the second linked list. The first linked list and / or the second linked list can be implemented by a reverse list.

[0023] The cost function and the second signature may be adjusted in step e) such that the number of sets stored in the selected first linked list is greater than each of the number of sets stored in the second linked list.

[0024] Preferably, the first signature includes a first prefix, the first prefix includes at least one token of the corresponding set up to a predefined number of tokens, and the predefined number of tokens in the first prefix is calculated based on a predefined distance function and a threshold value.

[0025] Preferably, the method is repeated, and in each repetition, different sets of the set of sets are selected as the query set until all sets of the set of sets are processed.

[0026] By repeating the method for each set of the set of sets in a self-join setting, it is guaranteed to find all pairs of similar sets within the set of sets.

[0027] In one embodiment, the first data structure and the second data structure are stored in the volatile memory of the storage device, particularly the main memory.

[0028] The selected subset of the first data structure is preferably stored in an allocation data structure representing an allocation vector. As a result, the selected first record can be processed, particularly merged, before being re-indexed independently of the complement of the selected subset of the first data structure.

[0029] In one embodiment, in step c), the selected subset of at least one first record is removed from the first index, and preferably, the corresponding memory space is deallocated.

[0030] Clearing the first record to be re-indexed later using the second signature has the advantage of reducing memory usage. Technically, the first record of the first data structure stored in the allocation data structure for re-indexing can be deleted from the first data structure.

[0031] In one embodiment, in step c), a predefined cost function is calculated independently for each of at least one first record.

[0032] Preferably, a predefined cost function compares a first cost of searching for one indexed set of at least one first record using a predefined distance function with a second cost of re-indexing the indexed set to generate a second signature for the indexed set and distribute the number of re-indexed indexed sets to the corresponding at least one second record, and searching for the re-indexed indexed set of at least one second record, and if the first cost exceeds the second cost, the first record is selected for the subset.

[0033] Preferably, the indexed set of at least one first record is sorted in ascending order of the difference between the first cost and the second cost.

[0034] Preferably, in step d), a selected subset of the first data structure is merged before determining a second signature (E2) for each indexed set (I) using one of a group of heuristic methods, -Merge each first record into a common record, - Processing each first record separately, - Merging each first record into a common record based on predefined criteria, where first records that do not meet the predefined criteria are skipped. - Merging each first record into at least one merged record based on predefined criteria, wherein each skipped first record is stored and re-evaluated in subsequent processing. -These consist of combinations of them.

[0035] The determination of which of the first records in the first data structure should be selected for re-indexing is NP-hard. Therefore, the optimal solution of the cost function for determining which of the first records to select is too difficult to solve in a reasonable amount of time. As a result, the inventors have established a group of heuristic methods for establishing an approximate solution to this NP-hard problem. Each of these heuristic methods is based on a simple decomposition of the first index, where the first records either remain in the first data structure or are re-indexed into their corresponding second data structure. However, a simple decomposition can result in a large number of second data structures and high associated re-indexing and signature generation costs. Therefore, by merging multiple first records before re-indexing, these costs can be reduced at the slight expense of higher candidate verification costs that may result from signature collisions between the second records forming the second index.

[0036] In one embodiment, in step g), each set of records in the subset of the hybrid index is processed as the number of tokens increases.

[0037] This processing order allows for the application of further filters to reduce the number of distance calculations required in step g).

[0038] Preferably, in step g), a predefined length filter is applied so that each set having tokens of the first signature that match the tokens of the first signature determined for the selected queryset is skipped if the matching tokens are located in a position in the set where fewer tokens remain than a predefined lower bound, the predefined lower bound being determined from a threshold, the size of the selected queryset, and the position of the matching tokens in the set.

[0039] This extended length filtering has the advantage that a selected set of subsets of a hybrid index can be skipped in a computationally less expensive way than by calculating their distances to the selected queryset using a predefined distance function.

[0040] Preferably, the predefined cost function is based on a filter of a predefined length, and the first signature further includes the position of the matching token in the set and the number of tokens in the set.

[0041] In one embodiment, the first and second indices are determined using a complementary method, preferably AllPairs and FastCoveringLSH.

[0042] The inventors have found that it is advantageous when the methods for determining the first and second indices complement each other in terms of performance. In other words, if the first index performs poorly for a dataset belonging to an HSJ / SSJ scenario, the second index is constructed using a method that performs better for that dataset. Therefore, no dataset is treated significantly worse than the best of the first and second indices. AllPairs uses a relatively small prefix index but works well for many real-world datasets. However, some datasets with small token universes generate a large number of candidates that must be validated later. In these cases, it is advantageous to use algorithms designed for HSJ scenarios, such as FastCoveringLSH.

[0043] In one embodiment, a predefined distance function determines the Hamming distance. The Hamming distance between two sets is the number of tokens that appear only in one set. The two sets are similar if their Hamming distance is less than or equal to a given distance threshold. [Brief explanation of the drawing]

[0044] Details and features of the present invention, as well as specific embodiments of the present invention, can be derived from the following description relating to the drawings. [Figure 1] Figure 1 shows an overview of a two-level signature scheme according to one embodiment of the present invention. [Figure 2] Figure 2 shows a flowchart of one embodiment of the method according to the present invention. [Figure 3] Figure 3 shows a comparison of runtimes for different threshold distances between one embodiment of the method according to the present invention and known prior art methods for datasets of SSJ and HSJ scenarios. [Figure 4] Figure 4 shows a comparison of memory consumption for different threshold distances between one embodiment of the method according to the present invention and a known prior art method for datasets of SSJ and HSJ scenarios. [Modes for carrying out the invention]

[0045] A set similarity join finds all pairs P of similar sets between two sets of sets, based on a distance function D and a threshold T. If the two sets of sets are identical, the set similarity join is called a self-join. In particular, two sets R of sets are considered similar if the distance value calculated by the distance function D is less than or equal to a predefined threshold T, and are then labeled as threshold distance ∈.

[0046] To determine the similarity between two sets R in a set, there must exist a clear measure between all sets R within the set. This measure is given by a predefined distance function D. The distance function D used in the embodiments of the present invention described below is the Hamming distance function.

[0047] In practice, set similarity joins often serve as a preparatory step for clustering datasets across different domains, for example, using DBSCAN. In particular, set similarity joins can be used to determine similar sets in a process mining domain dataset. These datasets contain information about processes executed within or using external computer systems.

[0048] The data describing these processes is preferably stored in a process protocol that can be modeled as a set. This process protocol includes at least three attributes. The first attribute stores cases, i.e., process instances. The second attribute stores events, i.e., process steps that form part of a corresponding case. The third attribute stores traces, which are sequences of events. For each trace, the set is formed from transitions between events or activities belonging to the case. For example, the trace "Start," "Invoice Approval," "Quote," "Order," "Receipt," "Invoice Approval," and "Payment" can be modeled using the set {(S,R),(R,Q),(Q,O),(O,C),(C,I),(I,P),(P,E)}. In this way, the similarity between two cases, i.e., two traces, can be measured using the Hamming distance of the set representation.

[0049] The performance of known methods for the set similarity joining problem typically differs primarily in the way they generate signatures for each set. A signature is a type of hash value for a set. Below, the signature of set R will also be denoted as Sign(R).

[0050] This signature scheme must generally be constructed to ensure that any two similar sets share at least one signature. The signature scheme may be symmetric, i.e., the signature of the indexed set is computed in the same way as the signature of the selected queryset Q. The signature scheme may also be asymmetric, i.e., a different signature is computed for the selected queryset Q than for the indexed set. However, in an asymmetric signature scheme, the signatures for the indexed set and the selected queryset Q are related and cannot be constructed independently. To avoid computing the union of the signatures of the indexed set and the signatures of the selected queryset for every pair, the index is constructed using the signature of set R.

[0051] According to the present invention, a first index is generated based on a first signature E1. The first index is also hereafter referred to as the primary index. Depending on the first signature scheme and dataset, some indexing (and query) signatures may appear more frequently than others. However, common signatures are problematic because the number of candidates that must be validated is proportional to the Cartesian product of a first inverse list L1 containing the indexed set I and a selected queryset Q that shares the corresponding first signature E1 of the first inverse list L1.

[0052] The inventors have found that the number of candidates can be reduced at the expense of the cost model, more complex indexing, and additional overhead resulting from the search, by removing the first record L1 of the first data structure which contains a large (reverse) list of indexed set I having a second signature E2 generated by a different, more selective second signature scheme.

[0053] Figure 1 shows a general overview of a two-level index structure according to one embodiment of the present invention.

[0054] Sign p The primary index using has several long reverse lists L1 containing several indexed sets I. These lists are then given a second signature scheme Sign s The system is re-indexed using multiple secondary indexes. Afterward, the list L1 of the re-indexed primary index can be removed.

[0055] The classic join algorithm for filter validation follows these three steps: (1) return a set of pre-selected candidates from an index lookup, (2) deduplication and filtering of the pre-selected candidates, and (3) validate the resulting candidate pairs to produce the final result.

[0056] According to the present invention, instead of filtering out pre-candidates for a long first inverse list L1, the two-level signature method re-indexes the selected first inverse list L1 into its own second index. Despite the overhead cost of re-indexing the selected first inverse list L1, the inventors have shown that the total number of candidates to be validated can be reduced to a degree that significantly improves the overall runtime and sometimes even the memory usage of set similarity joins across a wide range of datasets from various domains.

[0057] To achieve this significant performance optimization, two questions need to be answered after establishing the first index: (1) Which first inverse list L1 will be re-indexed? (2) Which second signature scheme will be used?

[0058] Figure 2 shows a flowchart of one embodiment of the method according to the present invention.

[0059] If queryset Q can be selected from the same set of set R or any second set of set R, the set of set R is first indexed in order to determine a similar set as the selected queryset Q in the set of set R. The first index on the set of set R is generated based on the first signature E1.

[0060] The first signature E1 may be, for example, a prefix. This prefix is ​​defined as a first predefined number of tokens in an ordered set of sets R, and this predefined number typically depends on the set size and the similarity function D or dissimilarity distance used.

[0061] The first index contains several first records L1, which are typically represented by a reverse list L1, i.e., a list L1 linked to the unique tokens in the token universe to which the first signature E1 (prefix) extends.

[0062] In practice, some lists L1 can become very long, especially for datasets containing small and / or skewed token universes. Therefore, long inverse lists L1 contain many sets that should be considered pre-candidates and validated using a predefined distance function D.

[0063] Advanced methods for the set similarity join problem typically aim to reduce the number of sets included in the reverse list by using sophisticated filtering techniques that are often adapted to specific characteristics of the dataset. However, we have found that the total number of pre-candidates can also be reduced by decomposing a long reverse list of a first index into multiple new lists of the second index using a second, preferably opposite, signature scheme. Since we can select which long reverse lists to re-index, and despite each re-indexing sacrificing additional performance and memory, the reduction in the total number of pre-candidates has proven to be highly effective, resulting in overall performance and sometimes even reduced memory costs compared to known (advanced) methods for the set similarity join problem.

[0064] According to one embodiment of the present invention, the list L1 of the first index is divided into two classes: a selected list of the first class that is considered for re-indexing, and a complementary list of the second class that remains unchanged within the first data structure. The decision of which list L1 to re-index is based on the result of a predefined cost function.

[0065] This predefined cost function evaluates a first cost, a second cost, and the difference between the first and second costs. The first cost represents the independent cost of determining similar sets within a selected long reverse list L1 for a selected queryset Q. The second cost estimates the independent cost of determining similar sets if the selected long reverse list L1 were to be re-indexed. Thus, the reverse list L1 is considered re-indexable only if the first cost exceeds the second cost. Since the pre-candidates for the second index form a subset of the primary index with respect to independent costs, this condition typically arises only if the cost of generating and re-indexing a second signature E2 outweighs the lesser validation benefits. The result of this step is a simple decomposition, in which the reverse list L1 either remains in the primary / first index or is re-indexed into its own secondary / second index. In other words, for each list L1 of the first data structure that is considered re-indexable, a second data structure is generated in which two or more second records L2 exist, each containing its own reverse list L2 that is shorter than the original reverse list L1 of the first index.

[0066] Next, a hybrid index H is generated from multiple secondary indexes and the remaining primary record L1 of the primary index.

[0067] The inventors propose a method for merging the first record before re-indexing. Depending on the resulting structure of the second index, the merge can reduce the overall verification cost. All four methods begin with a simple decomposition as described above. This simple decomposition of a selected long reverse list L1 of the first index can be extended in two dimensions. (1) This is a so-called Δ check, which accesses the predicted cost for each merge of the first record L1, ensuring that the cost does not increase when merging two first records. On the other hand, calculating the predicted cost increases the overhead of the cost model, and as a result, omitting the Δ check can improve the efficiency of the method if almost all merges of selected first records L1 are beneficial. (2) Using multiple second indexes may increase the memory usage of the method because some indexed sets I may have to be duplicated in multiple second indexes. However, limiting the number of indexes may result in a large second index containing a common second signature E2, which, due to a large predictive cost (i.e., low benefit of merging), locks other selected first records L1 from being merged and therefore causes the Δ check to fail.

[0068] The four methods for merging the first record L1 of a selected subset of the first data structure arise from all four possible combinations.

[0069] In any case, the inverse list L1 of the first index is first sorted in descending order of the difference between the first cost and the second cost. During the re-indexing process, the first records L1 of a selected subset of the first data structure are merged using one of a group of heuristic methods.

[0070] One of these methods is the SingleSimple method, which merges all selected lists L1 of the first index into a common record and re-indexes it into the same second index. This method works well under the assumption that the sum of the benefits of the re-indexed lists L2 is greater than the number of collisions between lists L2 in the second index. This condition can occur, for example, when a pure secondary index also performs better than a primary index.

[0071] Another method in the group of heuristic methods, MultiSimple, uses the first simple decomposition as its second index without accessing any predicted costs.

[0072] Therefore, these two simple methods work without re-evaluating any of the first records L1 as processing according to the first data structure. The resulting difference in computational cost is modeled using a basic example.

[0073] In this example, the set R={s1,...,s18} is considered to contain four lists at its first index. These four lists are summarized in Table 1. The quadratic signature scheme is given by its pre-image, which is summarized in Table 2 below.

[0074] For simplicity, we assume a symmetric signature scheme where the validation cost of the first index is equal to the validation cost of the second index, and the index creation cost for the second index is equal to 1. All other costs are set to 0. In this example, all inverse lists IL1-IL4 are considered re-indexable in the second index. The first costs of the inverse lists of the first index are also summarized in Table 1. For the first list IL1, the first cost is calculated as 45, and the corresponding indexing validation costs for the second index are calculated as 20 and 10, respectively, i.e., the second cost is equal to 30. All other lists have a first cost of 6 and a second cost of 5, due to an indexing cost of 4 and a validation cost of 1.

[0075] TIFF0007842897000001.tif32170

[0076] TIFF0007842897000002.tif44170

[0077] In the SingleSimple method, lists IL1 to IL4 are merged into a common record before the common record is indexed into the same second index. This method incurs a total cost of 50 (32 + 18).

[0078] In the MultiSimple method, all reverse lists are indexed within their own second index. Therefore, the total cost is summarized as 45 (30 + 5 + 5 + 5).

[0079] Two further methods within the group of heuristic methods follow a reassessment strategy. In short, the reassessment-based methods follow the same pattern as their simpler counterparts, but perform a delta check before merging the two first records. In other words, merging the two first records is performed based on predefined criteria.

[0080] The SingleReassessment method processes the lists in descending order of the difference between the first cost and the second cost, through the inverse list L1 of the first index. In other words, the list L1 with the largest difference between the first and second costs is selected first, and for each subsequent list, if it passes a delta check, the list is merged; otherwise, the list is skipped.

[0081] Applying this to the example above, SingleReassessment starts with IL1. The cost of generating the second index of IL1 is 30, since it only contains IL1. A Δ check is performed before adding the second list IL2. Since Δ is equal to 5, the list is skipped. The third list IL3 is skipped for the same reason. For IL4, Δ is equal to -2, so that the second index is constructed using the union of the first list IL1 and the fourth list IL4. Therefore, the total cost is 45(33+6+6).

[0082] In the case of MultiReassessment, the process proceeds similarly to SingleReassessment, but the skipped list is stored. After the first iteration through the re-indexable list L1 of the first index, a second iteration is performed through the skipped list. That is, the first skipped list is processed and merged with subsequent skipped lists using the MultiReassessment strategy. This procedure is repeated until all re-indexable lists L1 are indexed in the second index. It is possible to construct two or more second indexes using MultiReassessment. The resulting second index can be viewed as a left deep tree where all Δ checks have succeeded. Since only one second index is constructed at a time by constructing a left deep tree, the second index can be constructed using only linearly added memory.

[0083] In the example above, the MultiReassessment method starts with the first list IL1. As with SingleReassessment, lists IL2 and IL3 are skipped, and IL4 is merged with IL1 to form a merged list. Next, a Δ check is performed on IL2 and IL3, which were skipped in the first iteration. Since the second signatures of the sets in IL2 and IL3 are separated, Δ is equal to 0, and the indices can be merged. Thus, the total cost is 43(33+10). Therefore, the MultiReassessment strategy requires the lowest cost in this basic example.

[0084] Generating a hybrid index H allows for further filtering of the pre-selection set before validating candidates against the selected queryset Q.

[0085] In addition to prefix and length filters, a bilateral variation of a position-enhanced length filter is preferably applied. The position filter enhances the prefix filter by utilizing the positional information of tokens to further reduce the number of pre-candidates. The key idea is to find duplicate tokens in prefix E1 of the search set Q and the hybrid index set R, and to verify whether the remaining number of tokens can still reach the minimum amount of duplication required to consider the two sets Q and R similar. If the remaining number of tokens cannot reach the minimum required amount of duplication, the corresponding set R of the hybrid index can be safely pruned.

[0086] In other words, the position filter uses the position of the first matching token in the search set R to find an upper bound on the length of the selected queryset Q. If the position of the corresponding token is also known for the selected queryset Q, a tighter lower bound can be derived and predefined, just like the upper bound. For example, the lower bound l min This can be determined using the following formula. l min =|R|+2p s -∈ In the formula, |R| is the length of set R, p s is the location of the first matching token in the selected queryset Q, and ∈ represents a predefined threshold distance.

[0087] Introducing such a lower bound to a positional filter offers certain advantages, particularly when set similarity is tested for a set of querysets Q. That is, since the dataset is typically processed while increasing the size of the set R, the lower bound increases monotonically. Therefore, if an index entry is skipped once due to the lower bound, that index entry can be updated during the search using a monotonically increasing offset for the first non-failed set of all groups, skipped by all further applications of those positional filters. After the initial failure to meet the upper bound, the remainder of groups with the same position can be skipped.

[0088] Furthermore, one embodiment of the present invention, here referred to as TwoL, is compared to known methods for set similarity coupling problems that use Hamming distance as a measure of similarity.

[0089] The known methods used for this comparison are as follows: GroupJoin is a prefix filter-based set similarity joining algorithm that extends PPJoin. Sets with the same prefix are grouped together to allow for faster filtering, and the groups are expanded during validation. SizeAware is a set similarity joining algorithm that focuses on overlap as a measure of similarity. The dataset is split into short sets and long sets. The long sets are processed using an arbitrary algorithm, such as ScanCount. For overlaps of a predefined size, the small sets are processed by enumerating all subsets of a predefined size and storing the sets in a reverse list indexed by these subsets. AllPairs is the first main-memory algorithm to use prefix filters. Furthermore, filter enhancement is achieved by using length filters. PartAlloc is based on a partition enumeration framework. Each set is divided into a predefined number of parts, and all dimensional deletion distributions are enumerated for each part. An inverse list index is constructed for partitions and deletions. During the search, a cost model is used for each partition to determine whether to search for a partition or a deletion. fcLSH is a fast-covering LSH method based on a set of bitmasks and their corresponding hash functions. SkipJoin is a prefix filter-based set similarity joining algorithm that employs two skipping techniques. (1) Index level skipping groups index entries into skippable blocks based on their location. (2) Answer level skipping utilizes the similarity of the explored sets to the indexed set in order to find other sets that are similar to the indexed set. The choice between calculating candidate indexed sets from scratch or utilizing the similarity of explored sets to the indexed sets is based on the cost model.

[0090] Figure 3 shows a comparison of runtimes for different threshold distances between two embodiments of the present invention and the known prior art methods described above.

[0091] The runtime efficiency shown in Figure 3 is measured as the time to enumerate all pairs of similar sets, excluding the overall runtime, i.e., the time to load the dataset or to perform typical preprocessing steps such as tokenization, sorting, and deduplication. The algorithm was run with the threshold distance ∈ varied in the range of 2 to 5. Embodiments of the present invention, compared to known prior art methods, are based on the SingleSimple method (represented as SS) and the MultiReassessment method (represented as MR).

[0092] The datasets used to compare these algorithms are labeled CELONIS2 and LIVEJ.

[0093] CELONIS2 is a dataset from the domains of process mining and model processes. Sets within CELONIS2 consist of trace activity transitions, i.e., sequences of process steps. Compared to typical set-similarity-joined datasets, CELONIS2 has a smaller token universe.

[0094] LIVEJ is a typical set-similarity join dataset containing a large token universe that is even more heavily skewed. Therefore, compared to LIVEJ, the CELONIS2 dataset is an interesting edge case, and the immediate classification to SSJ vs. HSJ does not seem clear.

[0095] For both datasets, the two embodiments of the present invention yield the best runtime efficiency across the entire range of distance thresholds tested. Note the logarithmic scale of the y-axis. For the CELONIS2 dataset, only the fcLSH dataset yields runtime efficiency comparable to the two embodiments of the present invention. In particular, the SingleSimple method yields the best runtime experience for the CELONIS2 dataset. Notably, the embodiments of the present invention perform significantly better than modern techniques such as AllPairs. This observation also applies to the LIVEJ dataset.

[0096] In the case of the LIVEJ dataset, the two methods according to the present invention clearly outperform their equivalents, although in this case there is no significant difference between the SingleSimple method and the MultiReassessment method. Therefore, the two-level signature method according to the present invention not only improves runtime efficiency for datasets that are stereotypical for similarity joining, but also improves runtime efficiency for real-world datasets that pose a strong challenge to previously known methods for the set similarity joining problem.

[0097] Figure 4 extends the comparison of the method in Figure 3 to memory usage for the datasets CELONIS2 and LIVEJ.

[0098] For the CELONIS2 dataset, the memory usage of the two embodiments of the present invention increases with increasing threshold distance ∈. The methods according to the present invention are found not to be the most memory-efficient, particularly due to the additional storage required for the second data structure. However, the methods according to the present invention outperform certain other methods, especially the fcLSH and SizeAware methods, which require more than twice the memory compared to the SingleSimple and MultiReassessment methods. In particular, for small threshold distances ∈, the memory usage of the methods according to the present invention is comparable to that of methods from the prior art for the set similarity coupling problem.

[0099] For the LIVEJ dataset, the memory usage of the two embodiments of the present invention remains relatively constant even as the threshold distance ∈ increases. Compared to their peers, the memory usage of the two embodiments of the present invention lies at the lower end of the spectrum and is clearly superior in performance to the fcLSH, PartAlloc, and SizeAware methods.

[0100] In practice, improvements in runtime efficiency are particularly beneficial, while slight increases in memory usage can often be safely ignored. However, it is worth noting that embodiments of the present invention perform better with respect to memory usage than methods tuned to HSJ settings.

[0101] In summary, the inventors have demonstrated that re-indexing a selected inverse list of a first index into at least one second index results in significantly improved runtime efficiency across datasets exhibiting different characteristics compared to known methods for set similarity joining problems. In some cases, even memory usage is improved compared to known methods for set similarity joining problems. Details on how subsets of the first index are further processed, particularly merged, before re-indexing appear to have only a minor impact on the observed runtime efficiency and / or memory usage.

Claims

1. A computer implementation for determining at least one similar set for a selected query set (Q) within a set of sets, wherein each set (R) comprises several tokens, Each set (R) represents a process, each token represents a process step, and the process includes a series of process steps executed in at least one source computer system. The aforementioned method, a) A step of determining a first signature (E1) for each set (R), b) A step of indexing each set (R) based on the determined first signature (E1), wherein each indexed set (I) is stored in at least one first record (L1), each of the at least one first record (L1) is recorded in a first data structure representing a first index, and the indexing step corresponds to the determined first signature (E1), c) The step of selecting a subset of the first data structure based on a predefined cost function, d) For each first record (L1) of the selected subset of the first data structure, a second signature (E2) is determined for each indexed set (I), e) A step of re-indexing the indexed sets (I) based on the determined second signature (E2), wherein each re-indexed indexed set (I2) is stored in at least one second record (L2), and each of the at least one second record (L2) is recorded in a second data structure representing at least one second index, and the re-indexing step corresponds to the determined second signature (E2), f) A step of generating a hybrid index (H) from the complement of a selected subset of the first index and the at least one second index, g) A step of exploring at least a subset of the hybrid index by calculating distance values ​​between each set (I, I2) of each record in the subset and the selected query set (Q), wherein the explored set (C) is identified as one of the at least one similar set if the distance value is less than or equal to a predefined threshold (T), The aforementioned set of at least one analogous process represents at least one analogous process to a selected process within the set of processes.

2. The method according to claim 1, wherein the first record (L1) of the first data structure is a first linked list (L1), and / or the second record (L2) of the second data structure is a second linked list (L2).

3. The method according to claim 1 or 2, wherein the first signature includes a first prefix, the first prefix includes at least one token of the corresponding set up to a predefined number of tokens, and the predefined number of tokens in the first prefix is ​​calculated based on a predefined distance function (D) and a threshold (T).

4. The method according to claim 1, wherein the method is repeated, and in each iteration, a different set (R) of the set of sets is the selected query set (Q) until all sets (R) of the set of sets have been processed.

5. The method according to claim 1, wherein in step c), a selected subset of the at least one first record (L1) is removed from the first index and the corresponding memory space is deallocated.

6. The method according to claim 1, wherein in step c), the predefined cost function is calculated independently for each of the at least one first record (L1).

7. The method according to claim 6, wherein the predefined cost function compares a first cost of searching for one of the at least one first record (L1) in the indexed set (I) using the predefined distance function with a second cost of generating a second signature for the indexed set (I), re-indexing the indexed set (I) to the corresponding at least one second record (L2), and searching for the re-indexed indexed set (I2) of the at least one second record (L2), and if the first cost exceeds the second cost, the first record (L1) is selected for the subset.

8. The method according to claim 7, wherein the indexed set (I) of at least one first record is sorted in descending order of the difference between the first cost and the second cost.

9. In step d), the selected subset of the first data structure is merged before determining the second signature (E2) for each indexed set (I) using one of the group of heuristic methods, - Merge each first record into a common record. - Processing each first record separately, - Merging each first record into the common record based on predefined criteria, wherein first records that do not meet the predefined criteria are skipped during the merging process. - Merging each first record into at least one merged record based on the predefined criteria, wherein each skipped first record is stored and re-evaluated in subsequent processing, and - The method according to claim 8, comprising a combination thereof.

10. The method according to claim 1, wherein in step g), each set of records (I, I2) of the subset of the hybrid index is processed as the number of tokens increases.

11. The method according to claim 10, wherein in step g), a predefined length filter is applied so that each set (R) whose first signature token matches the first signature token determined for the selected query set (Q) is skipped if the matching token is located in a position in the set (R) where fewer tokens remain than a predefined lower limit, the predefined lower limit being determined from the predefined threshold (T), the size of the selected query set (Q), and the position of the matching token in the set (R).

12. The method according to claim 11, wherein the predefined cost function, based on a filter of the predefined length, further comprises the position of the matching token in the set (R) and the number of tokens in the set (R).

13. The method according to claim 1, wherein the first index and the second index are determined using a complementary method including AllPairs and FastCoveringLSH.

14. The method according to claim 1, wherein the predefined distance function determines the Hamming distance.

Images