Methods for large set similarity joining

Abstract

A computer-implemented method is provided for finding sets similar to a selected query set in a collection of sets, where each set represents a process. Each set is converted to a representation in a vector space. Furthermore, each set includes a prefix. The method includes creating a data structure representing an inverse metric index and storing it in a storage device. Similar sets to the selected query set are identified by a search step followed by a candidate validation step. In the search step, the inverse metric index is filtered by a predefined subspace of the vector space. In the candidate validation step, each set of filtered inverse metric indexes is identified as a similar set if its distance value is less than or equal to a predefined distance threshold.

Description

[Technical field]

[0001] The present invention relates to a method for finding sets that are similar to a selected query set in a collection of sets, and more generally to the problem of set similarity joining. [Background technology]

[0002] In search of more information, ever-increasing amounts of data are currently being recorded. Search operations are essential to access relevant information in datasets, especially in large datasets. However, finding exact matches in a dataset is not always sufficient, for example when considering spelling or typing errors that frequently occur in text data. Therefore, a fairly practical feature for matching records in a dataset is the similarity operator. Similarity operators find similar records based on a function that determines the distance between two records.

[0003] One approach to find similar records in a dataset is given by set similarity join. Set similarity join reports all pairs of similar records within a collection of sets and / or between multiple sets of sets. A collection of sets contains sets, and each set in a collection of sets contains some tokens. Sets can represent user behavior in social media networks, different types of documents (e.g. tweets or abstracts), photo metadata, or process activities related to process mining domains. Illustrative applications of finding similar pairs in a collection of sets are e.g. plagiarism detection, data cleaning, document clustering, process mining, and data mining in social networks.

[0004] Finding similar set pairs in a collection of sets is a difficult task even for moderately large data sets. Using brute force, the number of distance calculations required is quadratic in the collection size, i.e., the number of sets in the collection. Thus, set similarity joining quickly becomes computationally infeasible.

[0005] The performance of state-of-the-art solutions for set similarity joins depends heavily on the characteristics of the input data. To date, efficient set similarity join algorithms are based on a filter validation framework, in which candidates are retrieved from an inverted list index over the signatures. Candidates must be validated to prune non-qualifying result pairs. Unfortunately, the inverted list of candidates in the index is often large for large data sets, e.g., when the number of tokens is small compared to the size of the set and / or when the distribution of tokens is close to a uniform distribution. Traversing a long inverted list is computationally expensive, strongly limiting the performance of existing solutions. Summary of the Invention

[0006] It is therefore an object of the present invention to provide a method for computing set similarity joins that at least partially overcomes the aforementioned problems. [Problem to be solved by the invention]

[0007] According to the invention, this object is solved by the methods defined in the independent claims. Preferred embodiments and further developments of the invention are defined in the dependent claims.

[0008] Thus, a computer-implemented method is provided for finding (i.e., determining) at least one similar set to a selected query set in a collection of sets. Of course, it should be noted that the method can also report an empty set if no similar set to the selected query set is found in the collection of sets.

[0009] Each set includes a number of tokens. Each set represents a process, and each token represents a process step. The process includes a sequence of process steps that are executed on at least one source computer system.

[0010] Each set is converted to a representation in the vector space and the representation is stored in a storage device. Each set includes a prefix, where the prefix includes at least one token of the corresponding set, up to a predefined number of tokens.

[0011] A data structure is created and stored in a storage device. The data structure includes a plurality of unique elements, each unique element representing a token in a collection of sets. Each set of at least a subset of the collection of sets is linked to at least one unique element of the data structure representing at least one token of its prefix. The linked sets are grouped by number of tokens into at least one group for each unique element.

[0012] Similar sets for a selected query set are identified according to the following steps. In a search step, for each group in the data structure, a spatial index is constructed from the linked sets. Each linked set is inserted into the set of potentially similar sets exclusively if its representation in the vector space lies within a predefined subspace of the vector space. The set of potentially similar sets for each group is stored in a storage device such that the set of potentially similar sets includes a subset of the linked sets of each group.

[0013] In many cases, the subset of linked sets of each group is smaller than the set of linked sets of each group. Thus, the total number of potentially similar sets can be reduced. However, depending on the set of sets being analyzed, the subset of linked sets can also have the size of the set of linked sets, i.e., if all linked sets are similar to the selected query set. In a candidate validation step, for each potentially similar set of spatial indexes, a distance value to the selected query set is calculated using a predefined distance function, and if the distance value is less than or equal to a predefined threshold, the potentially similar set is identified as one of the at least one similar set.

[0014] The at least one similar set represents at least one similar process to the selected process in the collection of processes, in other words, the similar set represents similar processes, particularly process instances, in the collection of processes.

[0015] The method according to the invention has the advantage that the inverted list for each unique token can already be effectively reduced in advance, i.e. before evaluating the predefined distance function for each member of the inverted list. It should be noted that distance calculation is a computationally expensive operation. Therefore, in the candidate validation step, a much reduced number of potentially similar sets must be tested by calculating actual distance values ​​for the selected query set using the predefined distance function.

[0016] The reduction in the number of potentially similar sets for each unique element of the data structure is achieved by exploiting the metric properties of linked sets. In general, converting a set of sets of sets into a representation in a vector space and a range query therein is less computationally expensive than calculating distance values ​​between each set of the set of sets and a selected query set using a predefined distance function. In combination with an appropriately predefined subspace of the vector space for filtering the representations of potentially similar sets, the number of distance calculations required in the candidate validation step can be reduced, and therefore the overall execution time of the set similarity join can be reduced.

[0017] The inventors have found that the reduction of potentially similar sets using a metric range query filter works particularly well for a collection of sets that represent a collection of processes. In practice, a collection of processes is usually a very large data set, and the token universe, i.e., the total number of unique tokens in the collection of sets, is small compared to the number of sets in the collection of sets. In the field of process mining, tokens represent process steps and sets represent process instances, which are executions of processes on or with a source computer system.

[0018] Preferably, the predefined subspace is a query rectangle, the size of which is related to a predefined threshold.

[0019] Preferably, the predefined number of tokens in the prefix is ​​calculated based on a predefined distance function and a predefined threshold.

[0020] Depending on the choice of the predefined distance function, further parameters may be included in the calculation of the number of tokens in the prefix, for example, if the distance function is not the Hamming distance function, the number of tokens in the prefix further depends on the number of tokens per set, i.e., the prefix length may vary for different set sizes.

[0021] Preferably, the method according to the invention is repeated and in each iteration a different set of the collection of sets is the selected query set until all sets of the collection of sets have been processed.

[0022] By repeating the method for each set in the set of sets, we are guaranteed to find all pairs of similar sets in the set of sets.

[0023] In one embodiment, the data structure and collection of potentially similar sets are stored in volatile memory, particularly main memory, of the storage device.

[0024] Preferably, a predefined number of pivot elements are selected from the collection of sets, the predefined number of pivot elements spanning the vector space.

[0025] In this embodiment, the vector space is the same for each group of data structures, ie, the vector space is global.

[0026] In a preferred embodiment, for each group in the data structure, a predefined number of pivot elements are selected from the linked set of the corresponding group.

[0027] In a preferred embodiment, a vector space is generated separately for each group of the data structure based on the linked sets of the corresponding group. It has been found that local vector spaces can allow for a sparser distribution of the linked set representations within these local vector spaces, such that metric query range filters that use the (same) predefined subspaces of the local vector spaces work even more effectively.

[0028] Preferably, the spatial index is constructed only for groups that contain at least a predefined number of linked sets. For each group that contains less than the predefined number of linked sets, the linked sets are sorted according to a random dimension of their representation.

[0029] Creating and populating a spatial index can be inefficient for filtering only small linked sets, but sorting the linked sets according to a random dimension allows filtering of small linked sets in an efficient manner, for example by using one-dimensional range queries.

[0030] In one embodiment, upper and lower bounds are calculated for the selected query set, and each set that contains more tokens than the upper bound of the selected query set or fewer tokens than the lower bound of the selected query set is skipped.

[0031] Preferably, in the searching step, the groups are processed according to the increasing number of tokens in each linked set: each set containing more tokens than the number of tokens in the selected query set is skipped.

[0032] The advantage of processing sets in ascending order is that once a set passes the lower bound criterion it can be permanently removed from the data structure, thus avoiding the examination of ineligible sets and further freeing up memory.

[0033] Preferably, in the searching step, each set having a token in its prefix that matches a token in the prefix of the selected query set is skipped if the matching token is located in a position in the set where there are fewer tokens remaining than are required for similarity between the set and the selected query set according to a predefined threshold.

[0034] In one embodiment, each group of the data structure includes at least one linked set.

[0035] In other words, a group is added to the data structure only if at least one linked set is found for that group. Technically, a group can be represented by a bucket linked to each unique element of the data structure. Thus, avoiding the creation of empty buckets in the data structure has the advantage of occupying less memory.

[0036] Preferably, the predefined distance function is one of the group comprising: - a bounded distance function, in particular a Jaccard distance function, where the distance value is scaled to an unbounded distance value; -Unrestricted distance functions, especially the Hamming distance function a non-metric distance function, in particular a Dice distance function or a cosine distance function, in which the predefined thresholds are scaled by a parameter, the parameter being calculated based on the number of tokens in each set;

[0037] In one embodiment, the collection of sets includes a majority of sets that contain a similar number of tokens.

[0038] In other words, our method works well when many of the sets in the collection of sets contain similar sizes, which results in fewer groups in the data structure and therefore fewer range queries on the spatial index.

[0039] In one embodiment, the number of unique elements in the data structure is small compared to the cardinality of the collection of sets.

[0040] In other words, the token universe is small compared to the number of sets in the collection of sets, which is usually the case for process mining datasets. The method according to the invention has been found to work particularly well for datasets with this property, since these datasets are expected to generate long inverted lists in state-of-the-art methods for set similarity joining. Thus, while other algorithms spend their time scanning those long inverted lists, the solution according to one aspect of the invention considers only a subset of each inverted list, which is often small relative to the original inverted list (it can even be an empty set in the best case). [Brief description of the drawings]

[0041] Further details and features of the invention as well as specific embodiments of the invention can be derived from the following description in conjunction with the drawings. [Figure 1] FIG. 1 is a diagram illustrating an inverse metric index according to one embodiment of the present invention based on an example from Table 2. [Diagram 2] FIG. 2 illustrates a range query according to one embodiment of the present invention during the searching step. [Diagram 3] FIG. 3 is a diagram illustrating an exemplary regular bucket representing a length of three for a given token as part of an inverse metric index according to one embodiment of the present invention. [Figure 4] FIG. 4 is a representation of (a) globally mapped buckets and (b) locally mapped buckets according to one embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram illustrating an exemplary underload bucket representing a length of three for a given token as part of an inverse metric index in accordance with one embodiment of the present invention. [Figure 6]FIG. 6 shows the performance of methods according to two embodiments of the present invention compared to two prior art methods, each evaluated on two different datasets. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] Set similarity joining finds all pairs of similar sets P in a collection of sets based on a distance function D and a threshold T. In particular, two sets R in a collection of sets are considered to be similar if their distance value calculated by using the distance function D is less than or equal to a predefined threshold T.

[0043] To determine the similarity between two sets R of collections, there must be a well-defined measure between all sets R in the collection. The measure is given by a predefined distance function D. Distance functions D commonly used in the field of process mining are, among others, the Jaccard, Cosine, Dice and Hamming distance functions. Corresponding similarity functions can be used as well.

[0044] A simple nested loop join to find all pairs of similar sets P in a collection of sets scales quadratically with the number of distance calculations. Clearly, the quadratic run-time complexity becomes infeasible as soon as the number of sets becomes large enough. Therefore, a filtering framework that reduces the complexity is very important. Two filtering techniques for set similarity joins, in particular prefix filtering and length filtering, have been known for some time.

[0045] A prefix filter is a technique that prunes non-qualified sets a priori, i.e., before computing their distance values ​​to a selected query set Q. A prefix filter is based on the observation that a subset of a set and a selected query set Q must share at least one token to qualify as a candidate or potentially similar set C. The predefined subset is called a prefix and is formed by the first few tokens of a set R. Q is computed under the assumption that all sets R are totally ordered. Thus, a prefix filter allows finding potentially similar sets P by investigating only a small number of tokens instead of the entire set.

[0046] The prefix filter can be combined with a length filter: for every distance function D, a lower bound of binding partners R for a selected query set Q and an upper bound of binding partners R for a selected query set Q are defined. The length filter then defines the criterion for which potentially similar sets C are pruned, i.e., when the number of elements is not between the lower and upper bounds of the selected query set Q. Thus, the length filter further reduces the number of potentially similar sets C in advance.

[0047] A state-of-the-art algorithm for exact similarity joining based on prefix filtering is the AllPairs algorithm combined with length filtering.

[0048] Essentially, the AllPairs algorithm is based on the generation of a prefix index. The prefix index is an array that contains, for each token in the universe of sets, a pointer to an inverted list of sets that share the corresponding token in the prefix. The sets in the inverted list are sorted by increasing set size. Thus, for large data sets with small token universes, the inverted list can be very long. Once the index is built, the selected query set can be searched against the index. In doing so, for each token of the prefix of the selected query set, the set of inverted lists pointed to by the corresponding token in the prefix index is selected as pre-candidates. Furthermore, a length filter may be applied and duplicated sets may be removed from the pre-candidate set. For the resulting candidates, similar sets to the selected query set can be identified by evaluating the distance value between each candidate set and the selected query set. However, for large data sets, the linked list becomes large and the effective number of candidates remains large, leading to validation of many, often unnecessary, candidates.

[0049] We first find that the limitations of previous filtering techniques in the case of large datasets can be overcome by extending the list of filters with a filtering method that exploits the metric properties of a set R in combination with a predefined distance function D. While the prefix and length filters are custom-tuned to the set, the metric-based filter exploits the fact that the set R in combination with the distance function D forms a metric space.

[0050] Therefore, the set R can be transformed into a vector space V and bounded by the maximum distance using a set of pivot elements O selected from the collection of sets. The vector space V can be n-dimensional, where n is the number of selected pivot elements O. Once the set R of collections of sets is mapped into the n-dimensional vector space V, a range query can be used to find potentially similar sets C. More specifically, for a selected query set Q, the metric range query aims to find a candidate set C located within a predefined subspace A of the n-dimensional vector space V, where the subspace O is also denoted as the ε-range of the representation Φ of the selected query set Q in the vector space.

[0051] Metric range queries are based on triangle inequalities. Therefore, the predefined distance function D needs to be metric. However, our method also works for non-metric distance functions such as Dice and Cosine distances by projecting Dice or Cosine candidate generations to a metric distance, e.g., Jaccard distance.

[0052] In its simplest form, the projection may be as follows: The combination is calculated using the Jaccard distance with an adjusted threshold T such that a superset C of candidates is received. This superset of candidates C includes all similar sets P received by calculating the combination using the Dice distance or the cosine distance, respectively. Finally, in a verification step, unnecessary candidates C are removed.

[0053] In the following, an embodiment of the method according to the invention is presented by means of a concrete example drawn from the area of ​​process mining.

[0054] The dataset used in this example assumes process steps such as "order", "pay", "delivery", and "return" in the execution of a particular customer's process. A process instance, which is an execution of a customer's process, can be represented by forming a 2-gram, e.g., {("order", "pay"), ("pay", "delivery"), ("delivery", "return")}. The 2-gram can be mapped to a set containing some tokens, and duplicate tokens are moved so that a process instance can be represented by an integer-based set, e.g., {1, 2, 3}.

[0055] For this example, a more simple data model is assumed: each process step is assigned to a unique token. Table 1 summarizes the assignment of process steps to tokens. For example, the process step "order" is assigned to token "1", the process step "withdraw" is assigned to token "2", etc. Thus, the exemplary process {"order", "pay", "delivery", "return"} can be represented by the set {1, 7, 9, 8}.

[0056] TIFF2025506308000002.tif38142

[0057] Table 2 shows a collection of records or sets that form a subset of the process mining dataset and that are used to demonstrate one embodiment of the method according to the invention. In particular, it is shown below how a similar set P for a selected query record Q is found from the collection of sets in Table 2.

[0058] TIFF2025506308000003.tif68170

[0059] A predefined distance function D determines the similarity between two records R, i.e., two sets R of a collection of sets. Intuitively, the larger the distance value calculated by the distance function D, the less similar the two records R are to each other.

[0060] The method according to the invention generally requires that the distance function D is metric, i.e. additional properties such as a triangle inequality must be satisfied. However, this requirement can be overcome by using suitable approximations so that non-metric distance functions D can also be used.

[0061] In this example, the Hamming distance is used. For example, to calculate the Hamming distance between records R1 and R4 from Table 2, we obtain a distance value of 2.

[0062] In this example, for simplicity, we assume that all sets R are of equal size, i.e., size 6, and the selected query record Q is of size 7. From the collection of sets, a set of pivot elements O is selected as {R1, R8}. The exact prefix size, i.e., the number of elements in the prefix, can be derived from the distance function D in combination with a predefined threshold T. In this example, the prefix size is 2.

[0063] FIG. 1 illustrates an inverse metric index according to one embodiment of the present invention based on an example from Table 2.

[0064] The inverse metric index M shown in FIG. 1 is constructed using one embodiment of a method according to the present invention.

[0065] Each set R of the collection of sets is transformed into a representation Φ in the vector space V. The transformation can be performed beforehand in a preprocessing step, or on the fly when processing through the collection of sets. The representation Φ of the set R, Q, is stored in a storage device, in particular a main memory.

[0066] The vector space V is spanned by a predefined number of pivot elements O, which are selected from a collection of sets. Preferably, the pivot elements O are selected very fast and randomly. However, any existing method can be used to find pivot elements O that are optimized in terms of pruning ability. In this example, the global pivot elements O are selected as R1 and R8.

[0067] The inverse metric index M abstracts prefix, length and metric range filtering into a single data structure. The index structure M can be represented as a fixed size array. The data structure M contains multiple unique elements E, where each unique element E represents a token in a collection of sets.

[0068] During processing, each set R, L is linked to at least one unique element E of the data structure M representing at least one token of its prefix. The linked sets L are grouped into at least one group B for each unique element by their number of tokens, i.e. their set length. In one embodiment, the groups may be stored in an inverted list of buckets linked to the corresponding unique element E. A bucket B stores all sets L of a certain length that share a corresponding token in their prefix. The set length or set size is given by the number of tokens in the set R. Each bucket B can therefore be identified by a combination of token and set length. In the inverted list linked to the unique element E, the buckets B are preferably sorted in ascending order by set length.

[0069] In this example, for each record R, the first two tokens form a prefix. For every token in the prefix, the corresponding bucket list in the inverse metric index M is accessed and the record R is inserted into the bucket B representing the set length 6. Since all records R1 to R8 in Table 2 are of equal size, every token in the inverse metric index M shown in Figure 1 points to a single bucket B representing the length 6. The relationship between the unique element E representing the token and the bucket B is represented by the arrow in Figure 1. With the records R in Table 2 being distributed among the buckets B in the inverse metric index M, every record L in the bucket B, i.e., every linked set L of the group, is bulk loaded into its corresponding spatial index.

[0070] Often, a bucket B, which represents a particular combination of tokens and set lengths, will only contain a very small number of sets R. Since spatial indexes can carry additional computational overhead, such as in the case of tree structures, it can be beneficial to avoid computing such complex structures only for a very small number of sets. For example, in the case of an R-tree, the number of records in bucket B is small, but a minimum number of nodes must be pre-allocated. Buckets B that are assigned a small number of linked sets L are then also defined as low-load buckets.

[0071] If many buckets B in the inverse metric index M are found to be low-load, the cost of building the metric index M may rise and eventually exceed its benefits. Therefore, preferably, the spatial index for bucket B is created and populated only if there are at least a predefined number of records to insert, the predefined number of records to insert being a hyperparameter further referred to as the cutoff parameter. Otherwise, a low-load bucket B is created.

[0072] However, in this example, the cutoff parameter is chosen to convey a value of 0, i.e., for every bucket B, a spatial index is created and populated.

[0073] A similar set P for a selected query set Q is identified according to the following steps: a search step S1 and a candidate validation step S2.

[0074] In a search step S1, the selected query record Q is searched against the reverse metric index M. For every token of the prefix of the selected query record Q, in this case tokens "3" and "4", the reverse metric index M is accessed at the position of the corresponding unique element E and a linked list of buckets is retrieved. In this example, the lists for both tokens "3" and "4" only contain a single entry, namely bucket B of length 6. For different set lengths, a length filter can be applied. Using the length filter, it is only necessary to consider buckets of the linked list representing set lengths that are within the lower and upper size bounds of the selected query record Q, respectively.

[0075] The linked set L of the retrieved bucket B is filtered by range querying the corresponding spatial index. In this example, the vector space V is two-dimensional and the range query is performed by placing a query rectangle A around a representation Φ of the selected query record Q. The size of the query rectangle A is related to a predefined threshold T, which in this case is equal to 2. Thus, the query rectangle A is given by the representation 〈[1,5], [1,5]〉 on a two-dimensional grid. All representations Φ of the linked set L of the corresponding bucket B that are not included in this query rectangle A can be pruned.

[0076] FIG. 2 visualizes a range query according to an embodiment of the present invention, in search step S1.

[0077] In Figure 2, records R, Q are labeled in lower case for readability.

[0078] As shown in panel (a) of Figure 2, in this example, only record R4 is returned as a candidate for the bucket linked to the unique token "3". Instead, records R1, R7, and R8 can be pruned and do not need to be evaluated using a predefined distance function D.

[0079] For the bucket list at position "4" in the inverse metric index M, which corresponds to the second token in the prefix of the selected query record Q, the same query rectangle A is used to place it on the grid of the spatial index of bucket B retrieved from index M. As shown in panel (b) of Figure 2, only a single record is located within rectangle A, and record R5 is added to the set of potentially similar sets C. Records R2, R3, and R6 in bucket B linked to the unique token "4" in the inverse metric index M are pruned instead.

[0080] Since the prefix size is 2 in this example, the search step ends and finally the generated candidates are verified, for example by calculating the actual distance value to the selected query record Q.

[0081] In the candidate validation step S2, for each potentially similar set C of the spatial index, a distance value to the selected query set Q is calculated using a predefined distance function D. If the distance value is less than or equal to a predefined threshold T, the potentially similar set C is identified as a similar set P. In this example, the Hamming distance between the selected query record Q and record R4 is equal to 5, thus exceeding the predefined threshold of 2. The Hamming distance between the selected query record Q and record R5, a second candidate set, is equal to 1, which satisfies the criteria for R5 to be considered as a set P similar to the selected query set Q. Thus, the pair (Q, R5) is reported as a result pair.

[0082] This simple example already demonstrates the technical advantage of the method according to the invention over the AllPairs algorithm. In contrast to AllPairs, which reports eight candidates for this particular example, i.e. all linked sets in the list for tokens "3" and "4" in the corresponding prefix index, the method according to the invention reports only two candidates R4 and R5. Instead of calculating eight distance values ​​for the eight candidates found with the AllPairs algorithm, the method according to the invention calculates only two distance values ​​using a predefined distance function D. This reduction of the candidate set is achieved by transforming the generic metric space of the set R into a two-dimensional vector space V in combination with the predefined distance function D by using two pivot elements O from the collection of sets.

[0083] Naturally, the pivot-based approach generates a certain computational overhead since the intermediate distances based on the pivot element O must be precomputed. Once the intermediate distances are precomputed, they can be utilized to determine whether two sets R, Q are similar before computing the expensive distance value between them. By doing so, the number of candidate sets C can be effectively reduced, which gains an overall advantage over the AllPairs algorithm, especially when the token universe is small compared to the number of sets in the collection of sets.

[0084] To a certain extent, the pivot-based approach continues to improve as more pivot elements O are used, but more pivot elements O also increase the amount of memory space required. It has been found by the inventors that to efficiently obtain a reduction in the set of candidates C by leveraging pre-computed distances using the pivot elements O, a suitable index structure, such as an inverse metric index M, is essential.

[0085] In this example, only a single selected query record Q was searched against the inverse metric index M. To obtain the set similarity join result, all sets R, Q need to be searched against index M, and each of the resulting candidates C needs to be validated.

[0086] FIG. 3 sketches an exemplary regular bucket representing length 3 for a given token as part of the inverse metric index according to one embodiment of the present invention.

[0087] A regular bucket as shown in FIG. 3 includes a B “spatial_index” attribute, and the linked set L assigned to bucket B is stored in such a way that n-dimensional range queries can be performed. In this particular example of FIG. 3, the linked set L is stored in a two-dimensional R-tree. In one embodiment, when the spatial index in bucket B representing a particular combination of token and set length is queried, only potentially similar sets C that (1) share a corresponding token with the selected query record Q in the prefix, (2) pass the length filter, and (3) lie within the query rectangle of the vector space are returned. Ideally, a bucket B representing a particular combination of token and set length is only created in the inverse metric index M if at least one set R exists for a particular token of a particular size.

[0088] In one preferred embodiment, the selected query record Q is searched against the inverse metric index M in ascending order of set size, such that bucket B can be removed from the inverse metric index M if the length it represents is no longer needed.

[0089] In the example shown above, the global pivot element O is selected once at the beginning from the collection of sets. Preferably, the pivot element O is not selected once at the beginning globally but is used to transform the entire data set based on the assumption of all records R. The collection of sets Q resides in a single vector space V. Instead, the inverse metric index M reveals that the linked sets L grouped in individual buckets B form their own local vector space V. Thus, the pivot element O is preferably selected locally, i.e., selected for each bucket B separately from the linked records L assigned to the bucket B. As shown in the example of FIG. 3, the local pivot element O(R1, R4) may be stored in the "local_pivot" attribute of the bucket B. In doing so, all sets L in each bucket are mapped to their vector space V representation using the corresponding local pivot element O.

[0090] Nevertheless, this procedure requires that bucket B contains at least n records, where n represents the number of dimensions. If bucket B contains less than n records, the global pivot factor O can be used as a fallback solution to transform the set L in the corresponding bucket B.

[0091] FIG. 4 visualizes (a) a representation of a globally mapped bucket and (b) a representation of a locally mapped bucket according to one embodiment of the present invention.

[0092] The buckets depicted in Figure 4 represent a set length of 20 and are selected from the process mining dataset.

[0093] Panel (a) of Figure 4 visualizes a representation Φ of a linked set L assigned to a bucket B transformed into a two-dimensional vector space V with a global pivot element O represented by a large circle on the two axes. The distance function D in this example is the Jaccard distance, and two pivot elements O are randomly selected from the collection of sets. Note that the global pivot element O does not occur within the selected bucket.

[0094] Panel (b) of Figure 4 shows the representation Φ of the linked set L of the same bucket B, but computed using a local pivot element O, which is randomly selected from the set L assigned to bucket B and is marked with a double circle.

[0095] The local pivot element O better distributes the representation Φ in the two-dimensional vector space V of bucket B, thus substantially increasing the pruning power. Although this example was chosen purely for demonstration purposes, the inventors have found that the effect of increasing the pruning power by selecting local pivot elements over global pivot elements is not limited to this particular dataset, as it translates to experiments on further datasets.

[0096] Since a local pivot element O needs to be selected for each bucket B individually, it is preferable to select a local pivot element O randomly, which is very fast, instead of computing the best pivot element O separately for each bucket B. Furthermore, searching a selected query set Q against bucket B requires remapping the selected query records Q using the local pivot elements O. Remapping search records Q becomes more expensive as the set size becomes larger.

[0097] FIG. 5 sketches an exemplary low load bucket representing a length of three for a given token as part of the inverse metric index, according to one embodiment of the present invention.

[0098] As mentioned above, a low-load bucket B emerges when fewer linked sets L are assigned to bucket B, with a predefined cutoff parameter. Therefore, for low-load buckets, creating and populating a spatial index is not effective. Instead, the dimension j is drawn uniformly at random from [1,n], and the records stored in the "records" attribute are sorted by the jth dimension in their vector space representation. Based on this sorting, one-dimensional range queries can be performed that only require ordering by j.

[0099] To search the selected query set Q against the low-load bucket B, first the values ​​of its representation are extracted in dimension j, where dimension j is simply the distance between the selected query set Q and the j-th pivot element O. Then the first record of the bucket whose distance is greater than or equal to the extracted value, i.e., the extracted value minus a predefined threshold T, is searched and scanned until the extracted value plus the predefined threshold T plus 1 is reached. For example, a binary search can be utilized. It has been found that in most cases, a one-dimensional range query will report fewer low-load bucket candidates than scanning the entire list of set L as done by AllPairs.

[0100] FIG. 6 contrasts the performance of methods according to two embodiments of the present invention with two prior art methods, each evaluated on two different datasets.

[0101] In experimental evaluation of the method according to the invention, we observed that the number of candidates reported by the set similarity combining algorithm correlates with the runtime behavior. In other words, the more candidates C the set similarity combining algorithm reports, the longer the runtime the set similarity combining algorithm requires for many data sets. Since the method according to the invention always reports fewer candidates C than the latest prefix interval solution, it can often outperform the latter.

[0102] In Fig. 6a the results of the different methods are shown on the process mining dataset labelled CELONIS2. In the left panel of Fig. 6a the running times of the methods for different choices of the predefined threshold T are shown. More specifically, the diamond symbols represent the running times of the MetricJoin algorithm, which represents one embodiment of the present invention. The square symbols represent the running times of the state-of-the-art AllPairs algorithm. For all choices of the predefined threshold T, the MetricJoin algorithm outperforms the AllPairs algorithm on the process mining dataset CELONIS2. The technical advantage of the MetricJoin algorithm compared to the AllPairs algorithm comes from its ability to reduce the number of candidates, as visualized in the right panel of Fig. 6a.

[0103] Qualitatively similar results are achieved for the second dataset, a bibliographic collection of academic papers in the field of computer science from the Digital Bibliography and Libraries Project (DBLP), labeled DBLP-V12, as shown in Fig. 6b. The running time of the MetricJoin algorithm clearly outperforms that of the AllPairs algorithm for each predefined threshold T, especially as the threshold T increases.

[0104] Most notably, experiments performed by the inventors have shown that even additional filters applied to modern algorithms such as AllPairs cannot simply compensate for the shortcomings of the prior art. For example, a positional filter was applied on top of the AllPairs algorithm, later called PPJoin, whose running time is represented by a triangular symbol in Figure 6.

[0105] The positional filter enhances the prefix filter in that it exploits the positional information of tokens to further reduce the number of candidates. The key idea is to find overlapping tokens in the prefixes of a search set Q and a set of indexes R, and verify whether the remaining number of tokens can still reach the minimum required amount of overlap to consider the two sets similar. If the remaining number of tokens cannot reach the minimum required amount of overlap, the corresponding set of indexes can be safely pruned.

[0106] As the positional filter further reduces the number of candidates,C,, the overall execution time also decreases for all predefined thresholds,T,for PPJoin on AllPairs, as shown in Fig. 6. However, the PPJoin algorithm is not able to close the gap against the MetricJoin algorithm in terms of performance, since the candidate reduction is not as efficient as it turns out when exploiting the metric properties of the set,R,combined with,Q,and a predefined distance function,D,.

[0107] Furthermore, the combination of the position filter and the MetricJoin algorithm, called MetricJoinPos and represented by the star symbol in Figure 6, further reduces the number of reported candidates. Thus, an additional filter applied to MetricJoin can further reduce the execution time of the set similarity join. As a result, the performance gap between MetricJoin and AllPairs is very similar to that between MetricJoinPos and PPJoin. [Explanation of symbols]

[0108] A subspace of a vector space V, in particular the query rectangle B is a group of linked sets L, specifically represented as buckets C Potentially Similar or Candidate Sets D Predefined Distance Functions E are the unique elements of the data structure M L is a set linked to the unique elements E of a data structure M M data structures, especially inverse metric indexes O Pivot element selected from a set of sets P is a set similar to the selected query set Q Selected query set R is a set of collections of sets S1 Exploration step S2 Candidate Verification Step T threshold V n-dimensional vector space

Claims

1. A computer-implemented method for finding at least one similar set (P) to a selected query set (Q) in a collection of sets, each set (R) comprising a number of tokens: Each set (R) represents a process, each token represents a process step, said process including a sequence of process steps executed on at least one source computer system; Each set (R) is transformed into a representation in a vector space (V), said representation being stored using a storage device; Each set (R) comprises a prefix, said prefix comprising at least one token of said corresponding set up to a predefined number of tokens; A data structure (M) is created and stored in the storage device, the data structure comprising a plurality of unique elements (E), each unique element (E) representing a token included in the collection of sets, each set (R) of at least a subset of the collection of sets being linked to at least one unique element (E) of the data structure representing said at least one token of its prefix, the linked sets (L) being grouped into at least one group (B) of corresponding unique elements (E) according to the number of tokens; A similar set (P) to the selected query set (Q) is identified according to the following steps: in a search step (S1), for each group (B) of said data structure (M) a spatial index is constructed from said linked sets (L), each linked set (L) is inserted exclusively into a collection of potentially similar sets (C) if its representation in said vector space (V) is located within a predefined subspace (A) of said vector space (V), the collection of potentially similar sets (C) for each group (B) being stored in said storage device such that the collection of potentially similar sets (C) comprises a subset of said linked sets (L) of each group (B); - in a candidate validation step (S2), for each potentially similar set (C) of the spatial index, a distance value to the selected query set (Q) is calculated using a predefined distance function (D), and if said distance value is less than or equal to a predefined threshold (T), the potentially similar set (C) is identified as one of the at least one similar set (P); The at least one similar set (P) represents at least one similar process to a selected process in a collection of processes.

2. The method of claim 1 , wherein the predefined subspace (A) is a query rectangle, the size of the query rectangle being related to the predefined threshold (T).

3. The method of claim 1 or 2, wherein the predefined number of tokens in the prefix is ​​calculated based on the predefined distance function (D) and the threshold (T).

4. 4. The method of claim 1, wherein the method is repeated and in each iteration a different set (R) of the collection of sets is the selected query set (Q) until all sets (R) of the collection of sets have been processed.

5. 5. The method according to claim 1, wherein the data structure (M) and the collection of potentially similar sets (C) are stored in a volatile memory of the storage device, in particular in the main memory.

6. 6. The method of claim 1, wherein a predefined number of pivot elements (O) are selected from the set of elements, the predefined number of pivot elements (O) spanning the vector space (V).

7. 7. The method of claim 1, wherein for each group (B) of the data structure (M), the predefined number of pivot elements (O) are selected from the linked set (L) of the corresponding group (B).

8. 8. The method of claim 1, wherein the spatial index is built only for groups (B) that contain at least a predefined number of linked sets (L), and for each group (B) that contains less than the predefined number of linked sets, the linked sets (L) are sorted according to a random dimension of their representation.

9. 9. The method of claim 1, wherein for the selected query set (Q), an upper bound and a lower bound are calculated, and each set that contains more tokens in the selected query set (Q) than the upper bound or fewer tokens in the selected query set (Q) than the lower bound is skipped.

10. 10. The method according to any one of claims 1 to 9, wherein in the searching step (S1), the groups (B) are processed according to increasing numbers of tokens in their respective linked sets (L), and each set (R) containing more tokens than the number of tokens in the selected query set (Q) is skipped.

11. 11. The method according to any one of claims 1 to 10, wherein in the searching step (S1), each set (R) having a token within its prefix that matches a token within the prefix of the selected query set (Q) is skipped if the matching token is located in a position within the set where there remain fewer tokens than are required for similarity between the set and the selected query set according to the predefined threshold (T).

12. 12. The method according to claim 1, wherein each group (B) of the data structure (M) comprises at least one linked set (L).

13. The predefined distance function (D) is - a bounded distance function, in particular a Jaccard distance function, where the distance values ​​are scaled to unbounded distance values; - unrestricted distance functions, in particular the Hamming distance function, A method according to any one of claims 1 to 12, which is one of the group including a non-metric distance function, in particular a Dice distance function or a Cosine distance function, where the predefined threshold is scaled by a parameter, the parameter being calculated based on the number of tokens in each set.

14. 2. The method of claim 1, wherein the collection of sets comprises a majority of sets (R) that contain a similar number of tokens.

15. 15. The method according to claim 1, wherein the number of unique elements (E) in the data structure (M) is small compared to the cardinality of the collection of sets.

Images