Methods for large-scale set similarity joining

Abstract

A computer-implemented method is provided for finding sets similar to a selected query set in a collection of sets, where each set represents a process. Each set is converted to a representation in a vector space. Furthermore, each set includes a prefix. The method includes creating a data structure representing an inverse metric index and storing it in a storage device. Similar sets to the selected query set are identified by a search step followed by a candidate validation step. In the search step, the inverse metric index is filtered by a predefined subspace of the vector space. In the candidate validation step, each set of filtered inverse metric indexes is identified as a similar set if its distance value is less than or equal to a predefined distance threshold.

Description

Technical Field

[0001] The present invention relates to a method for finding sets similar to a selected query set within a set of sets, and more generally, to the problem of set similarity join.

Background Art

[0002] In search of more information, an ever-increasing amount of data is currently being recorded. In order to access relevant information within a dataset, especially a large dataset, a search operation is essential. However, finding an exact match within a dataset is not always sufficient, for example, when considering spelling or typing errors that frequently occur in text data. Therefore, a rather practical feature for matching records within a dataset is the similarity operator. The similarity operator finds similar records based on a function that determines the distance between two records.

[0003] One approach for finding similar records within a dataset is given by set similarity join. Set similarity join reports all pairs of similar records within a set of sets and / or between multiple sets of sets. A set of sets contains sets, and each set within the set of sets contains several tokens. Sets can represent user behavior in a social media network, different types of documents (e.g., tweets or abstracts), photo metadata, or process activities related to the area of process mining. Applications that illustrate finding similar pairs within a set of sets are, for example, fraud detection, data cleaning, document clustering, process mining, and data mining in social networks.

[0004] Finding pairs of similar sets in a set of sets is a difficult task, even with moderately large datasets. Using brute force, the number of distance calculations required is quadratic with respect to the set size, i.e., the number of sets in the set. Therefore, set similarity joins quickly become computationally unfeasible.

[0005] The performance of modern solutions for set similarity joining heavily depends on the characteristics of the input data. To date, efficient set similarity joining algorithms are based on a filter validation framework, in which candidates are retrieved from a reverse list index across signatures. Candidates must be validated in order to prune pairs of ineligible results. Unfortunately, the reverse list of candidates in the index is often large for large datasets, for example, when the number of tokens is small compared to the size of the set and / or when the distribution of tokens is close to uniform. Scanning long reverse lists is computationally expensive and severely limits the performance of existing solutions. [Overview of the project]

[0006] Therefore, an object of the present invention is to provide a method for computing set similarity combinations that at least partially overcomes the aforementioned problems. [Problems that the invention aims to solve]

[0007] According to the present invention, this objective is solved by the method described in the independent claim. Preferred embodiments and further developments of the present invention are defined in the dependent claims.

[0008] Therefore, a computer implementation method is provided for finding (i.e., determining) at least one similar set for a selected queryset within a set of sets. Of course, it should be noted that this method can also report an empty set if no similar sets are found for the selected queryset within the set of sets.

[0009] Each set contains several tokens. Each set represents a process, and each token represents a process step. This process includes a series of process steps executed in at least one source computer system.

[0010] Each set is transformed into a representation in vector space, and the representation is stored in memory. Each set includes a prefix, which contains at least one token from the corresponding set up to a predefined number of tokens.

[0011] A data structure is created and stored in memory. The data structure contains multiple unique elements, each unique element representing a token contained in a set of sets. Each set in at least a subset of the set of sets is linked to at least one unique element of the data structure that represents at least one token of its prefix. The linked sets are grouped into at least one group for each unique element by the number of tokens.

[0012] Similar sets to the selected queryset are identified according to the following steps: - In the search step, for each group of data structures, a spatial index is constructed from the linked sets. Each linked set is exclusively inserted into a set of potentially similar sets if its representation in vector space lies within a predefined subspace of vector space. The set of potentially similar sets for each group is stored in memory such that the set of potentially similar sets contains a subset of the linked sets for each group.

[0013] In many cases, a subset of linked sets in each group is smaller than the set of linked sets in each group. Therefore, the total number of potentially similar sets can be reduced. However, depending on the set of sets being analyzed, a subset of linked sets can also have the size of the set of linked sets, i.e., if all linked sets are similar to the selected query set. - In the candidate validation step, for each potentially similar set of spatial indexes, a distance value to the selected query set is calculated using a predefined distance function. If the distance value is less than or equal to a predefined threshold, the potentially similar set is identified as one of at least one similar set.

[0014] At least one similar set represents at least one similar process to a selected process within the set of processes. In other words, a similar set represents similar processes, in particular process instances, within the set of processes.

[0015] The method according to the present invention has the advantage that the inverse list for each unique token can be effectively reduced in advance, i.e., before evaluating a predefined distance function for each member of the inverse list. Note that distance calculation is a computationally expensive operation. Therefore, in the candidate validation step, a much reduced number of potentially similar sets must be tested by calculating the actual distance values ​​for the selected queryset using a predefined distance function.

[0016] Reducing the number of potentially similar sets for each unique element of a data structure is achieved by leveraging the metric properties of linked sets. In general, transforming a set of sets of sets into representations in a vector space and range queries within it is less computationally expensive than calculating distance values ​​between each set in the set of sets and a selected query set using a predefined distance function. Combined with a well-predefined subspace of the vector space for filtering potentially similar set representations, the number of distance calculations required in the candidate validation step can be reduced, and therefore the overall execution time of the set similarity join can be reduced.

[0017] The inventors have found that reducing potentially similar sets using metric range query filters works particularly well in the case of sets of sets representing sets of processes. In practice, sets of processes are typically very large datasets, and the token universe, i.e., the total number of unique tokens in a set of sets, is small compared to the number of sets in the set of sets. In the field of process mining, tokens represent process steps, sets represent process instances, and process instances are executions of processes in or using a source computer system.

[0018] Preferably, the predefined subspace is a query rectangle, and the size of the query rectangle is related to a predefined threshold.

[0019] Preferably, a predefined number of tokens within a prefix are calculated based on a predefined distance function and a predefined threshold.

[0020] Depending on the selection of the pre-defined distance function, additional parameters can be included in the calculation of the number of tokens within the prefix. For example, if the distance function is not the Hamming distance function, the number of tokens within the prefix further depends on the number of tokens per set, i.e., the prefix length can vary for different set sizes.

[0021] Preferably, the method according to the invention is repeated, and in each repetition, different sets of the set of sets are selected as the query set until all sets of the set of sets are processed.

[0022] By repeating the method for each set of the set of sets, it is guaranteed to find all pairs of similar sets within the set of sets.

[0023] In one embodiment, the data structure and the set of potentially similar sets are stored in the volatile memory of the storage device, particularly the main memory.

[0024] Preferably, a pre-defined number of pivot elements are selected from the set of sets, and the pre-defined number of pivot elements spans the vector space.

[0025] In this embodiment, the vector space is the same for each group of the data structure, i.e., the vector space is global.

[0026] In a preferred embodiment, for each group of the data structure, a pre-defined number of pivot elements are selected from the corresponding linked set of the group.

[0027] In a preferred embodiment, the vector space is generated separately for each group of data structures based on a linked set of corresponding groups. It has been found that the local vector space can allow a sparser distribution of the representation of the linked sets within these local vector spaces so that a metric query range filter using a (same) predefined subspace of the local vector space functions even more effectively.

[0028] Preferably, the spatial index is constructed only for groups containing at least a predefined number of linked sets. For each group containing less than the predefined number of linked sets, the linked sets are sorted according to the random dimensions of their representations.

[0029] The creation and deployment of the spatial index can be inefficient for filtering only a small number of linked sets, but sorting the linked sets according to the random dimensions allows filtering of a small number of linked sets in an efficient way, for example, by using a one-dimensional range query.

[0030] In one embodiment, upper and lower bounds are calculated for the selected query set. Each set containing more tokens than the upper bound of the selected query set, or fewer tokens than the lower bound of the selected query set, is skipped.

[0031] Preferably, in the search step, the groups are processed according to the increasing number of tokens of each linked set. Each set containing more tokens than the number of tokens of the selected query set is skipped.

[0032] The advantage of processing the sets in ascending order is that once a set passes the lower bound criterion, it can be permanently removed from the data structure. This avoids the inspection of ineligible sets and further frees up memory.

[0033] Preferably, in the search step, each set having a token in a prefix that matches a token in the prefix of the selected queryset is skipped if the matching token is located in a position in the set where fewer tokens remain than are required for the similarity between the set and the selected queryset according to a predefined threshold.

[0034] In one embodiment, each group of data structures includes at least one linked set.

[0035] In other words, a group is added to a data structure only if at least one linked set is found for that group. Technically, a group can be represented by buckets linked to each unique element of the data structure. Therefore, avoiding the creation of empty buckets in the data structure has the advantage of occupying less memory.

[0036] Preferably, the predefined distance function is one of the following groups: - A limiting distance function, in particular a Jackard distance function, which scales to distance values ​​that are not limited. - Unrestricted distance functions, especially Hamming distance functions - Nonmetric distance functions, particularly dice distance functions or cosine distance functions, wherein a predefined threshold is scaled by a parameter, and the parameter is calculated based on the number of tokens in each set.

[0037] In one embodiment, the set of sets includes most of the sets containing a similar number of tokens.

[0038] In other words, the method according to the present invention works well when many sets within a set of sets are of similar size, which results in fewer groups in the data structure and therefore fewer range queries on spatial indexes.

[0039] In one embodiment, the number of unique elements in the data structure is small compared to the cardinality of the set of elements.

[0040] In other words, the token universe is small compared to the number of sets in the set of sets, which is typically the case with process mining datasets. The method according to the present invention has been found to work particularly well with datasets having this property, since these datasets are intended to generate long inverse lists in modern methods for set similarity joining. Thus, while other algorithms spend their time traversing those long inverse lists, a solution according to one aspect of the present invention considers only a subset of each inverse list, and the subset is often small compared to the original inverse list (in the best case, it may even be an empty set). [Brief explanation of the drawing]

[0041] Details and features of the present invention, as well as specific embodiments of the present invention, can be derived from the following description relating to the drawings. [Figure 1] Figure 1 shows an inverse metric index according to one embodiment of the present invention, based on an example from Table 2. [Figure 2] Figure 2 shows a range query according to one embodiment of the present invention in the search step. [Figure 3] Figure 3 shows an exemplary regular bucket representing a length of 3 for a given token as part of an inverse metric index, according to one embodiment of the present invention. [Figure 4] Figure 4 shows a representation of a globally mapped bucket (a) and a locally mapped bucket (b) according to one embodiment of the present invention. [Figure 5] Figure 5 shows an exemplary low-load bucket representing a length of 3 for a given token as part of an inverse metric index, according to one embodiment of the present invention. [Figure 6]Figure 6 shows the performance of the methods according to two embodiments of the present invention compared to two prior art methods, each evaluated on two different datasets. [Modes for carrying out the invention]

[0042] Set similarity joins find all pairs of similar sets P within a set of sets based on a distance function D and a threshold T. In particular, two sets R in a set of sets are considered similar if their distance values, calculated using the distance function D, are less than or equal to a predefined threshold T.

[0043] To determine the similarity between two sets R, there must exist a clear measure between all sets R within the set. This measure is given by a predefined distance function D. Distance functions D commonly used in the field of process mining include, in particular, the Jackard, cosine, dice, and Hamming distance functions. Similarly, corresponding similarity functions can be used.

[0044] A simple nested loop join to find all pairs of similar sets P within a set of sets quadratically scales the number of distance calculations. Clearly, quadratic runtime complexity quickly becomes unfeasible when the number of sets becomes sufficiently large. Therefore, a filtering framework to reduce complexity is crucial. Two filtering techniques for set similarity joins, prefix filtering and length filtering, in particular, have been known for some time.

[0045] A prefix filter is a technique for pruning ineligible sets beforehand, i.e., before calculating their distance values ​​for a selected queryset Q. Prefix filters are based on the observation that a subset of a set and the selected queryset Q must share at least one token to qualify as a candidate or potentially similar set C. A predefined subset is called a prefix and is formed by the first few tokens of set R. Q is calculated under the assumption that all sets R are in perfect order. Thus, a prefix filter allows us to find potentially similar sets P by examining only a small number of tokens rather than the entire set.

[0046] Prefix filters can be combined with length filters. For every distance function D, a lower bound and an upper bound for join partners R for the selected queryset Q are defined. The length filter then defines the criteria for pruning potentially similar sets C, i.e., the criteria for when the number of elements in a set C is not between the lower and upper bounds of the selected queryset Q. Thus, the length filter further reduces the number of potentially similar sets C beforehand.

[0047] The most advanced algorithm for accurate similarity joining based on prefix filtering is the AllPairs algorithm, which combines it with length filtering.

[0048] Essentially, the AllPairs algorithm is based on the generation of a prefix index. The prefix index is an array containing, for each token in a universe of sets, pointers to a reverse list of sets that share the corresponding token in the prefix. The sets in the reverse list are sorted by increasing the set size. Therefore, for large datasets with small token universes, the reverse list can become very long. Once the index is built, a selected queryset can be searched against the index. In doing so, for each token in the prefix of the selected queryset, a set of reverse lists pointed to by the corresponding token in the prefix index is selected as pre-candidates. Furthermore, length filters may be applied to remove duplicate sets from the pre-candidate set. For the resulting candidates, similar sets to the selected queryset can be identified by evaluating the distance value between each candidate set and the selected queryset. However, for large datasets, the linked lists become large, and the number of valid candidates remains large, leading to the validation of many, often unnecessary, candidates.

[0049] We were the first to discover that the limitations of previous filtering techniques for large datasets can be overcome by extending the list of filters using filtering methods that leverage the metric properties of a set R in combination with a predefined distance function D. While prefix filters and length filters are custom-tuned to the set, metric-based filters take advantage of the fact that, in combination with the distance function D, the set R forms a metric space.

[0050] Therefore, we can transform a set R into a vector space V and limit it by the maximum distance using a set of pivot elements O selected from the set of sets. The vector space V can be n-dimensional, where n is the number of selected pivot elements O. Once the set R of sets is mapped to the n-dimensional vector space V, we can use range queries to find potentially similar sets C. More specifically, for a selected queryset Q, the metric range query aims to find candidate sets C located within a predefined subspace A of the n-dimensional vector space V, where the subspace O is also denoted as the ε-range of the representation Φ of the selected queryset Q in the vector space.

[0051] Metric range queries are based on the triangle inequality; therefore, the predefined distance function D must be metric. However, the method according to the present invention also works for non-metric distance functions such as dice distance and cosine distance by projecting the candidate generation of dice or cosine distance onto a metric distance, e.g., Jackard distance.

[0052] In its simplest form, the projection may be as follows: The join is calculated using a Jackard distance with an adjusted threshold T so that a superset C of candidates is received. This superset of candidates C contains all similar sets P that are received by calculating the join using either dice distance or cosine distance, respectively. Finally, in the validation step, unwanted candidates C are removed.

[0053] In the following, one embodiment of the method according to the present invention is presented by a specific example drawn from the field of process mining.

[0054] The dataset used in this example assumes process steps such as "order," "payment," "delivery," and "return" in the execution of a specific customer's process. A process instance, which is an execution of a customer process, can be represented by forming a 2-gram, for example, {("order," "payment"), ("payment," "delivery"), ("delivery," "return")}. A 2-gram can be mapped to a set containing several tokens, and duplicate tokens are moved so that a process instance can be represented by an integer-based set, for example, {1, 2, 3}.

[0055] For this example, an even simpler data model is assumed. Each process step is assigned to a unique token. Table 1 summarizes the assignment of process steps to tokens. For example, the process step "Order" is assigned to token "1", the process step "Withdrawal" is assigned to token "2", and so on. Thus, the exemplary processes {"Order", "Payment", "Delivery", "Return"} can be represented by the set {1, 7, 9, 8}.

[0056] TIFF0007833055000001.tif38142

[0057] Table 2 shows a set of records or sets that form a subset of process mining datasets and are used to demonstrate one embodiment of the method according to the present invention. In particular, it is shown below how a similar set P for a selected query record Q is found from the set of sets in Table 2.

[0058] TIFF0007833055000002.tif68170

[0059] A predefined distance function D determines the similarity between two records R, i.e., two sets R of sets of sets. Intuitively, the larger the distance value calculated by the distance function D, the less similar the two records R are to each other.

[0060] The methods according to the present invention generally require that the distance function D is metric, i.e., that additional properties such as the triangle inequality must be satisfied. However, this requirement can be overcome by using appropriate approximations so that nonmetric distance functions D can also be used.

[0061] In this example, the Hamming distance is used. For example, to calculate the Hamming distance between records R1 and R4 from Table 2, we obtain a distance value of 2.

[0062] In this example, for simplicity, we assume that all sets R are of equal size, i.e., size 6, and the selected query record Q is of size 7. From the set of sets, the set of pivot elements O is selected as {R1, R8}. The exact prefix size, i.e., the number of elements in the prefix, can be derived from the distance function D in combination with a predefined threshold T. In this example, the prefix size is 2.

[0063] Figure 1 shows an inverse metric index according to one embodiment of the present invention, based on an example from Table 2.

[0064] The inverse metric index M shown in Figure 1 is constructed using one embodiment of the method according to the present invention.

[0065] Each set R in the set of sets is transformed into a representation Φ in the vector space V. This transformation can be performed in advance during a preprocessing step or on the fly when processing through the set of sets. The representations Φ, Q of set R are stored in memory, particularly in main memory.

[0066] The vector space V is spanned by a predefined number of pivot elements O, which are selected from a set of sets. Preferably, the pivot elements O are selected very quickly and randomly. However, any existing method can also be used to find pivot elements O that are optimized with respect to pruning ability. In this example, the global pivot elements O are selected as R1 and R8.

[0067] The inverse metric index M abstracts prefix, length, and metric range filtering into a single data structure. The index structure M can be represented as a fixed-size array. The data structure M contains multiple unique elements E, each unique element E representing a token included in a set of sets.

[0068] During processing, each set R, L is linked to at least one unique element E of a data structure M that represents at least one token of its prefix. The linked sets L are grouped into at least one group B for each unique element by the number of their tokens, i.e., their set length. In one embodiment, the groups may be stored in a reverse list of buckets linked to the corresponding unique element E. Buckets B store all sets L of a particular length that share the corresponding token in their prefix. The set length or set size is given by the number of tokens in set R. Thus, each bucket B can be identified by a combination of tokens and set length. In the reverse list linked to the unique element E, it is preferable that the buckets B are sorted in ascending order by set length.

[0069] In this example, for each record R, the first two tokens form a prefix. For all tokens in the prefix, the corresponding bucket list of the inverse metric index M is accessed, and record R is inserted into bucket B, which represents a set length of 6. Since all records R1 through R8 in Table 2 are of equal size, all tokens in the inverse metric index M shown in Figure 1 point to a single bucket B representing a length of 6. The relationship between the unique element E representing a token and bucket B is represented by the arrow in Figure 1. With the records R in Table 2 distributed among the buckets B of the inverse metric index M, all records L in bucket B, i.e., all linked sets L in a group, are bulk loaded into their corresponding spatial index.

[0070] In many cases, a bucket B representing a particular combination of tokens and set length will contain only a very small number of sets R. Spatial indexes can carry additional computational overhead, such as in the case of tree structures, so it may be beneficial to avoid computing such complex structures for only a very small number of sets. For example, in the case of an R tree, although the number of records in bucket B is small, a minimum number of nodes must be pre-allocated. A bucket B allocated with a small number of linked sets L can also be defined as a low-load bucket.

[0071] If many buckets B within the inverse metric index M turn out to be low-load, the cost of building the metric index M increases, potentially outweighing its benefits. Therefore, it is preferable that the spatial index for bucket B is created and populated only if there are at least a predefined number of records to insert, and this predefined number of records to insert is a hyperparameter further called the cutoff parameter. Otherwise, low-load bucket B is created.

[0072] However, in this example, the cutoff parameter is chosen to carry a value of 0, meaning that a spatial index is created and populated for all buckets B.

[0073] Similar sets P to the selected queryset Q are identified according to the following steps: search step S1 and candidate validation step S2.

[0074] In the search step S1, the selected query record Q is searched against the inverse metric index M. For all tokens of the prefix of the selected query record Q, in this case tokens "3" and "4", the inverse metric index M is accessed at the position of the corresponding unique element E, and a linked list of buckets is retrieved. In this example, both lists for tokens "3" and "4" contain only a single entry, i.e., bucket B of length 6. For various set lengths, a length filter can be applied. Using a length filter, one only needs to consider the buckets in the linked list that represent set lengths within the lower and upper size limits, respectively, of the selected query record Q.

[0075] The linked set L of the retrieved bucket B is filtered by a range that queries the corresponding spatial index. In this example, the vector space V is two-dimensional, and the range query is performed by arranging a query rectangle A around the representation Φ of the selected query record Q. The size of query rectangle A is related to a predefined threshold T, in this case T is equal to 2. Thus, query rectangle A is given by the representation <[1,5], [1,5]> on a two-dimensional grid. All representations Φ of the linked set L of the corresponding bucket B that are not included in this query rectangle A can be pruned.

[0076] Figure 2 visualizes a range query according to one embodiment of the present invention in the search step S1.

[0077] In Figure 2, records R and Q are labeled in lowercase for readability.

[0078] As shown in panel (a) of Figure 2, in this example only record R4 is returned as a candidate for the bucket linked to the unique token "3". Alternatively, records R1, R7, and R8 can be pruned and do not need to be evaluated using the predefined distance function D.

[0079] The same query rectangle A is used for the bucket list at position "4" in the inverse metric index M, corresponding to the second token in the prefix of the selected query record Q, and is placed on the grid of the spatial index of bucket B extracted from index M. As shown in panel (b) of Figure 2, only a single record occupies rectangle A, and record R5 is added to the set of potentially similar sets C. Records R2, R3, and R6 of bucket B linked to the unique token "4" in the inverse metric index M are pruned instead.

[0080] In this example, the prefix size is 2, so the search step ends, and finally, the generated candidates are validated, for example, by calculating the actual distance value to the selected query record Q.

[0081] In the candidate validation step S2, for each potentially similar set C of the spatial index, the distance value to the selected queryset Q is calculated using a predefined distance function D. If the distance value is less than or equal to a predefined threshold T, the potentially similar set C is identified as a similar set P. In this example, the Hamming distance between the selected query record Q and record R4 is equal to 5, and therefore greater than the predefined threshold 2. The Hamming distance between the selected query record Q and record R5, which is from the second candidate set, is equal to 1, which satisfies the criterion for R5 to be considered a similar set P to the selected queryset Q. Therefore, the pair (Q, R5) is reported as the result pair.

[0082] This simple example already demonstrates the technical advantage of the method according to the present invention over the AllPairs algorithm. In contrast to AllPairs, which reports eight candidates for this particular example—namely, all linked sets in the lists of tokens "3" and "4" in the corresponding prefix indices—the method according to the present invention reports only two candidates, R4 and R5. Instead of calculating eight distance values ​​for the eight candidates found using the AllPairs algorithm, the method according to the present invention calculates only two distance values ​​using a predefined distance function D. This reduction in the candidate set is achieved by transforming the generic metric space of sets R into a two-dimensional vector space V in combination with the predefined distance function D by using two pivot elements O from the set of sets.

[0083] Naturally, pivot-based approaches generate a certain computational overhead because intermediate distances based on the pivot element O must be pre-calculated. Once intermediate distances are pre-calculated, they can be used to determine whether two sets R, Q are similar before calculating the expensive distance values ​​between them. Doing so effectively reduces the number of candidate sets C, which gives an overall advantage over the AllPairs algorithm, especially when the token universe is small compared to the number of sets in the set.

[0084] To a certain extent, the pivot-based approach continues to improve as more pivot elements O are used, but more pivot elements O also increase the amount of memory space required. The inventors have found that a suitable index structure, such as an inverse metric index M, is essential to efficiently obtain a reduction in the set of candidate C by leveraging pre-calculated distances using pivot elements O.

[0085] In this example, only a single selected query record Q was searched against the inverse metric index M. To obtain the results of a set similarity join, all sets R and Q must be searched against index M, and each of the resulting candidate C must be validated.

[0086] Figure 3 sketches an exemplary regular bucket representing a length of 3 for a given token as part of an inverse metric index, according to one embodiment of the present invention.

[0087] A regular bucket, as shown in Figure 3, includes the attribute B "spatial_index", and linked sets L assigned to bucket B are stored in such a way that n-dimensional range queries can be performed. In this particular example in Figure 3, linked sets L are stored in a two-dimensional R tree. In one embodiment, when a spatial index in bucket B representing a particular combination of tokens and set lengths is queried, only potentially similar sets C are returned that (1) share the corresponding token with the selected query record Q in the prefix, (2) pass the length filter, and (3) are located within the query rectangle of the vector space. Ideally, bucket B representing a particular combination of tokens and set lengths is created only in an inverse metric index M if there is at least one set R for a particular token of a particular size.

[0088] In one preferred embodiment, the selected query record Q is searched against the inverse metric index M in ascending order of set size so that bucket B can be removed from the inverse metric index M when the length represented by bucket B is no longer needed.

[0089] In the example shown above, the global pivot element O was initially selected once from the set of sets. Preferably, the pivot element O is not initially selected globally, but is used to transform the entire dataset based on the assumption of all records R. The set of sets Q resides in a single vector space V. Instead, the inverse metric index M reveals that linked sets L, grouped into individual buckets B, form their own local vector space V. Thus, the pivot element O is preferably selected locally, i.e., separately for each bucket B, from the linked records L assigned to bucket B. As shown in the example in Figure 3, the local pivot element O(R1, R4) can be stored in the "local_pivot" attribute of bucket B. In doing so, all sets L within each bucket are mapped to their vector space V representation using the corresponding local pivot element O.

[0090] Nevertheless, this procedure requires that bucket B contains at least n records, where n represents the number of dimensions. If bucket B contains fewer than n records, the global pivot element O can be used as a fallback solution to transform the corresponding set L in bucket B.

[0091] Figure 4 visualizes the representation of a globally mapped bucket (a) and a locally mapped bucket (b) according to one embodiment of the present invention.

[0092] The buckets shown in Figure 4 represent 20 set lengths and are selected from the process mining dataset.

[0093] Panel (a) of Figure 4 visualizes the representation Φ of linked sets L assigned to bucket B, which has been transformed into a two-dimensional vector space V using a global pivot element O, where the global pivot element O is represented by a large circle on two axes. The distance function D in this example is the Jackard distance, and the two pivot elements O are randomly selected from a set of sets. Note that the global pivot element O does not occur within the selected bucket.

[0094] Panel (b) of Figure 4 shows the representation Φ of the linked set L in the same bucket B, but the representation Φ was calculated using the local pivot element O. The local pivot element O is randomly selected from the set L assigned to bucket B and is marked with a double circle.

[0095] The local pivot element O distributes the representation Φ better in the two-dimensional vector space V of bucket B, and therefore substantially increases the pruning power. Although this example was chosen merely for demonstration purposes, the inventors found that the effect of increasing pruning power by selecting local pivot elements over global pivot elements is not limited to this particular dataset, as it can be translated into experiments on further datasets.

[0096] Since the local pivot element O needs to be selected individually for each bucket B, it is preferable to randomly select the local pivot element O, which is very fast, rather than separately calculating the best pivot element O for each bucket B. Furthermore, exploring the selected queryset Q for bucket B requires remapping the selected query records Q using the local pivot element O. Remapping the explored records Q becomes more costly as the set size increases.

[0097] Figure 5 sketches an exemplary low-load bucket representing a length of 3 for a given token as part of an inverse metric index, according to one embodiment of the present invention.

[0098] As mentioned above, when fewer linked sets L are allocated to bucket B, a low-load bucket B emerges, and a predefined cutoff parameter appears. Therefore, creating and populating a spatial index is not effective for low-load buckets. Instead, dimension j is randomly and uniformly drawn from [1,n], and records stored in the "records" attribute are sorted by the j-th dimension in their vector space representation. Based on this sort, one-dimensional range queries that only require ordering by j can be executed.

[0099] To search for a selected queryset Q against low-load buckets B, first, the values ​​of its representation are extracted in dimension j, which is simply the distance between the selected queryset Q and the j-th pivot element O. Then, the first record in buckets is searched for, where the distance is greater than or equal to the extracted value, i.e., the extracted value minus a predefined threshold T, in a one-dimensional query rectangle A, and this is scanned until a value equal to the extracted value plus the predefined threshold T plus 1 is reached. For example, binary search can be used. In most cases, one-dimensional range queries have been found to report fewer low-load bucket candidates than scanning the entire list of set L, as done by AllPairs.

[0100] Figure 6 compares the performance of the methods according to two embodiments of the present invention with two prior art methods, each of which is evaluated on two different datasets.

[0101] In experimental evaluations of the method according to the present invention, the inventors observed that the number of candidates reported by the set similarity join algorithm correlates with runtime behavior. In other words, the more candidates C reported by the set similarity join algorithm, the longer the execution time required by the set similarity join algorithm for a large dataset. The method according to the present invention always reports fewer candidates C than the latest prefix interval solution, and therefore may often outperform the latter.

[0102] Figure 6a shows the results of various methods on a process mining dataset labeled CELONIS2. The left panel of Figure 6a shows the execution times of the methods for different selections of a predefined threshold T. More specifically, the diamond symbols represent the execution time of the MetricJoin algorithm, which represents one embodiment of the present invention. The square symbols represent the execution time of the modern AllPairs algorithm. For all selected predefined thresholds T, the MetricJoin algorithm performs better than the AllPairs algorithm on the process mining dataset CELONIS2. The technical advantage of the MetricJoin algorithm compared to the AllPairs algorithm stems from its ability to reduce the number of candidates, as visualized in the right panel of Figure 6a.

[0103] Qualitatively similar results are achieved for a second dataset, labeled DBLP-V12, which is a bibliographic collection of academic papers in the field of computer science from the Digital Bibliography and Library Project (DBLP). The execution time of the MetricJoin algorithm is clearly superior to that of the AllPairs algorithm for each predefined threshold T, and especially as the threshold T increases.

[0104] Most notably, experiments conducted by the inventors showed that even additional filters applied to modern algorithms such as AllPairs could not simply compensate for the shortcomings of the prior art. For example, a position filter was applied on top of the AllPairs algorithm, later called PPJoin, and its execution time is represented by a triangle in Figure 6.

[0105] Positional filters enhance prefix filters by utilizing token positional information to further reduce the number of candidates. The key idea is to find duplicate tokens in the prefixes of the search set Q and the index set R, and then verify whether the remaining number of tokens can still reach the minimum amount of duplication required for the two sets to be considered similar. If the remaining number of tokens cannot reach the minimum required amount of duplication, the corresponding set of indices can be safely pruned.

[0106] As the position filter further reduces the number of candidates C, the overall execution time also decreases for all predefined thresholds T for PPJoin on AllPairs, as shown in Figure 6. However, the PPJoin algorithm cannot close the gap with the MetricJoin algorithm in terms of performance because candidate reduction is not as efficient as it would be when utilizing the metric properties of the set R, Q, and a predefined distance function D.

[0107] Furthermore, the combination of the position filter and the MetricJoin algorithm, called MetricJoinPos and represented by the star symbol in Figure 6, further reduces the number of candidates reported. Therefore, additional filters applied to MetricJoin can further reduce the execution time of set similarity joins. As a result, the performance gap between MetricJoin and AllPairs is very similar to the performance gap between MetricJoinPos and PPJoin. [Explanation of Symbols]

[0108] A subspace of vector space V, particularly query rectangles. B A group of linked sets L, specifically represented as buckets. C Potentially similar sets or candidate sets D Predefined distance function E. Unique elements of data structure M. L is a set linked to a unique element E of data structure M. M data structures, especially inverse metric indexes. Pivot elements selected from the set of O P Sets similar to the selected queryset Q Selected queryset R sets of sets S1 Search Step S2 Candidate Verification Step T threshold V n-dimensional vector space

Claims

1. A computer implementation for finding at least one similar set (P) for a selected query set (Q) in a set of sets, wherein each set (R) contains some tokens, Each set (R) represents a process, each token represents a process step, and the process includes a series of process steps executed in at least one source computer system. Each set (R) is converted into a representation in a vector space (V), and the representation is stored using a memory device. Each set (R) includes a prefix, the prefix includes at least one token of the corresponding set up to a predefined number of tokens. A data structure (M) is created and stored in the storage device, the data structure comprises a plurality of unique elements (E), each unique element (E) representing a token included in the set, each set (R) of at least a subset of the set is linked to at least one unique element (E) of the data structure representing at least one token of its prefix, the linked sets (L) are grouped by the number of tokens into at least one group (B) of corresponding unique elements (E), Similar sets (P) to the selected query set (Q) are identified according to the following steps: - In the search step (S1), for each group (B) of the data structure (M), a spatial index is constructed from the linked sets (L), and each linked set (L) is exclusively inserted into a set of potentially similar sets (C) if its representation in the vector space (V) lies within a predefined subspace (A) of the vector space (V), and the set of potentially similar sets (C) for each group (B) is stored in the memory device such that the set of potentially similar sets (C) includes a subset of the linked sets (L) of each group (B). - In the candidate validation step (S2), for each potentially similar set (C) of the spatial index, a distance value to the selected query set (Q) is calculated using a predefined distance function (D), and if the distance value is less than or equal to a predefined threshold (T), the potentially similar set (C) is identified as one of the at least one similar set (P). A computer implementation method wherein the at least one similar set (P) represents at least one similar process to a selected process within a set of processes.

2. The method according to claim 1, wherein the predefined subspace (A) is a query rectangle, and the size of the query rectangle is related to the predefined threshold (T).

3. The method according to claim 1 or 2, wherein the predefined number of tokens in the prefix are calculated based on the predefined distance function (D) and the threshold (T).

4. The method according to claim 1, wherein the method is repeated, and in each iteration, a different set (R) of the set of sets is the selected query set (Q) until all sets (R) of the set of sets have been processed.

5. The method according to claim 1, wherein the data structure (M) and the set of potentially similar sets (C) are stored in the volatile memory of the storage device.

6. The method according to claim 1, wherein a predefined number of pivot elements (O) are selected from the set of the set, and the predefined number of pivot elements (O) extend to the vector space (V).

7. The method according to claim 6, wherein for each group (B) of the data structure (M), the predefined number of pivot elements (O) are selected from the linked set (L) of the corresponding group (B).

8. The method according to claim 1, wherein the spatial index is constructed only for groups (B) that contain at least a predefined number of linked sets (L), and for each group (B) that contains fewer than the predefined number of linked sets, the linked sets (L) are sorted according to the random dimension of their representation.

9. The method according to claim 1, wherein an upper and lower limit is calculated for the selected query set (Q), and each set containing more tokens than the upper limit of the selected query set (Q) or fewer tokens than the lower limit of the selected query set (Q) is skipped.

10. The method according to claim 1, wherein in the search step (S1), the groups (B) are processed according to an increasing number of tokens in each of their linked sets (L), and each set (R) containing more tokens than the number of tokens in the selected query set (Q) is skipped.

11. The method according to claim 1, wherein in the search step (S1), each set (R) having tokens in prefixes that match tokens in prefixes of the selected query set (Q) is skipped if the matching token is located in a position in the set where fewer tokens remain than required for similarity between the set and the selected query set, according to the predefined threshold (T).

12. The method according to claim 1, wherein each group (B) of the data structure (M) includes at least one linked set (L).

13. The aforementioned predefined distance function (D) is: - A marginal distance function, which scales to distance values ​​that are not limited, - Unrestricted distance function, The method according to claim 1, wherein the group comprises nonmetric distance functions, the predefined threshold being scaled by a parameter, the parameter being calculated based on the number of tokens in each set.

14. The method according to claim 1, wherein the set of sets comprises a majority of sets (R) that contain a similar number of tokens.

15. The method according to claim 1, wherein the number of unique elements (E) in the data structure (M) is smaller than the cardinality of the set.

Images