How to create an index for reporting large-scale variant clustering

Abstract

A computer-implemented method is provided for computing an index for a first density-based clustering of a set of records, the index being stored using a storage device. The index is stored within any selected threshold distance ε less than or equal to a predefined threshold distance ε. * and a given number of records MinPts that form the input parameter pair on which the index is computed.

Description

[Technical field]

[0001] The present invention relates to a computer-implemented method for computing an index for density-based clustering of a set of records, and more particularly, the index enables efficient exploration of multiple parameter settings for accurate density-based clustering. [Background technology]

[0002] An important task in data analysis projects in general, and in the domain of process mining in particular, is the identification of groups of similar objects in a dataset. Methods that address this problem are summarized under the term clustering.

[0003] Among clustering methods, one approach is called density-based clustering, which was introduced by Ester et al. (A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise, Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (1996), AAAI Press, 226-231) along with the DBSCAN algorithm for computing density-based clusters for a given dataset. The idea of ​​density-based clustering is to form clusters from regions of high density in the space occupied by the dataset, also called the data space. The clusters are separated by regions of low density, and objects far away from the high density regions are labeled as noise.

[0004] The DBSCAN algorithm can be applied to a dataset without needing to know in advance the number of clusters present in the dataset. Furthermore, no assumptions about the statistical distribution of the data in the dataset are required. The DBSCAN algorithm only requires input of the minimum density that objects must meet to form a cluster. This minimum density is determined by two input parameters, commonly referred to as a predefined threshold distance ε and the minimum number of neighboring records MinPts. Thus, the DBSCAN algorithm requires minimal background knowledge to analyze and explore patterns in the dataset.

[0005] In many scenarios, the exact parameter settings are not known in advance and depend on the actual data being studied. In fact, in many cases, there is no unique density setting that captures all meaningful clusters in a dataset. Therefore, it is necessary to try several different settings until a good / satisfying clustering is found.

[0006] However, the DBSCAN algorithm is O(n 2 Since it involves distance calculations of 100,0 ...

[0007] In practice, DBSCAN must be applied several times for different input parameter settings until a satisfactory result is achieved. The OPTICS algorithm by Ankerst et al. (OPTICS: Ordering Points to Identify the Clustering Structure, Proceedings of ACM SIGMOD Int. Conf. on Management of Data (1999), Vol. 28 ACM New York, NY, USA, 49-60) attempts to overcome this problem.

[0008] Similar to the DBSCAN algorithm, the OPTICS algorithm requires two input parameters: a predefined distance value ε and a minimum number of neighbor records MinPts. The OPTICS algorithm determines the threshold ε that does not exceed the input threshold ε for the same number of neighbor records MinPts. * The idea is that a single computation of OPTICS will iterate over several runs of DBSCAN, with a threshold ε below the input threshold ε of the OPTICS algorithm. * The OPTICS algorithm computes a cluster ordering, which is a permutation of the input data set that includes a specific additional measurement for each record in the data set.

[0009] However, the cluster ordering computed by OPTICS can formally only yield approximate DBSCAN clusters. While DBSCAN produces exact clusterings, OPTICS-based clusterings, although density-based, are not exact. The clustering results are approximate in that objects located on the boundaries of clusters are often erroneously identified as noise. Thus, the cluster ordering computed by OPTICS is also non-deterministic. In practice, the degree of approximation achieved by the OPTICS algorithm cannot be estimated intrinsically, i.e. without the need for further computations. Summary of the Invention

[0010] It is therefore an object of the present invention to provide a method for the efficient computation of improved density-based clustering, thereby allowing a user to interactively explore a parameter space bounded by input parameters, while guaranteeing accurate clustering results for at least the input parameters. [Means for solving the problem]

[0011] According to the invention, this object is solved by the methods defined in the independent claims. Preferred embodiments and further developments of the invention are defined in the dependent claims.

[0012] Thus, there is provided a computer-implemented method for computing an index for a first density-based clustering of a set of records, where the index is stored using a storage device and each record is -First marked as unprocessed, -having a first attribute and a second attribute; - forming part of a neighborhood within a given threshold distance, A record is a core record if its neighborhood contains at least a predetermined number of records, and a non-core record if its neighborhood contains less than a predetermined number of records, and the index is calculated according to the following steps: a) selecting an unprocessed record from the set of records and calculating a core distance of the selected unprocessed record, the core distance being the minimum distance from the selected unprocessed record whose neighborhood still contains at least a predetermined number of records; b) if the selected unprocessed record is a core record, assigning a core distance to a first attribute and labeling the selected unprocessed record as a core record, otherwise assigning a predetermined value to the first attribute and labeling the selected unprocessed record as a non-core record; c) adding the selected unprocessed records to an index and marking the selected unprocessed records as processed; and d) if the selected processed record is a core record, processing each record in the neighborhood of the selected processed record, calculating a reachability distance for each record, the reachability distance being the minimum distance at which each record is still directly density reachable from the selected processed record; adding each record marked as unprocessed to a priority queue in ascending order of its reachability distance, the reachability distance being assigned to a second attribute; removing from the index each record previously labeled as a non-core record and outputting a reachability distance less than that previously assigned to its second attribute, each removed record being marked as unprocessed and each removed record being inserted into a priority queue according to ascending order of its reachability distance; e) adding each record of the priority queue to an index, where each record is marked as processed, and if the added record is a core record, its neighboring records are processed according to step d); Steps a) through e) are repeated until all records in the set of records have been processed and inserted into the index.

[0013] The method according to the invention has the advantage of allowing the threshold distance of the density-based clustering to be varied, resulting in an exact clustering for a given threshold distance and at least an approximation of the exact clustering for any smaller threshold distance. This approximation is generally better than the corresponding approximation obtained from the OPTICS algorithm. In particular, the inventors have found that the density-based clustering resulting from the index according to the invention is very close or even equal to the exact clustering resulting from the DBSCAN algorithm. In fact, the inventors have found that the OPTICS algorithm can only reach the quality of the approximation resulting from the index according to the invention at the same time. The standard cluster ordering resulting from OPTICS is technically limited to report a better approximation than the index according to the invention reports, since the index according to the invention collects more information from the set of records to be clustered.

[0014] It should be noted that the method according to the present invention is designed for situations where the neighborhood is unknown in advance. To calculate the neighborhood, a computationally expensive range query is required to calculate the neighborhood of a given record against a certain threshold distance, and thus the core distance of a given record. When the neighborhood is known in advance, the range query scales at O(1) in runtime so that traditional methods can be easily used. However, in many situations, especially for new datasets, the neighborhood is unknown in advance.

[0015] Exact clustering differs from density-based clustering in that it assigns each record, which represents a fuzzy boundary object, to exactly one cluster. More formally, exact clustering of a set of records for a given threshold and a given number of neighboring records is defined as dividing the set of records into at least one partition, such that: (1) every partition is identical to or a subset of a respective density-based cluster of the set of records; (2) all core records of a density-based cluster of the set of records form part of each partition; (3) Each boundary object forms part of exactly one partition, corresponding to the density-based cluster to which the boundary object belongs.

[0016] With respect to DBSCAN, our index has the advantage that it provides an improved cluster ordering that supports index queries for exact clustering for any threshold distance less than or equal to the input threshold distance, without computing the exact clustering from scratch. We have found that our index-query framework based on our index consistently results in better clustering runtimes than those achieved by the standard DBSCAN algorithm, especially for large data sets. Furthermore, our index has exactly the same requirements for the data model as DBSCAN does, i.e., it can be applied to any domain.

[0017] The improvement of the method according to the invention to the OPTICS algorithm is based on an improved cluster ordering that picks up more information from the objects in the dataset than the standard cluster ordering of OPTICS. The additional information is generated in particular by adaptation of the updates of the priority queue in step d). Whereas in the OPTICS algorithm the neighbouring records of a core record are processed only once, the method according to the invention reprocesses neighbouring records if they turn out to be non-core records and their reachability distance is smaller than previously assigned.

[0018] Furthermore, when inserting a record of the priority queue into the index according to step e), neighboring records of the inserted record that are core records are also processed according to the improved update of the priority queue in step d), in particular, if a neighboring record is already in the priority queue, its reachability distance and position are updated as necessary.

[0019] Although the method according to the invention involves reprocessing of records, an empirical analysis of the OPTICS algorithm and index according to the invention carried out by the inventors has revealed that both algorithms share the same asymptotic computational complexity, in other words, the small overhead caused by the improved update of the priority queue according to the invention is negligible for large data sets.

[0020] Preferably, between steps a) and b), it is determined whether the selected unprocessed record is a core record by counting records in the vicinity of the selected unprocessed record.

[0021] Preferably, during processing of each record of the priority queue in step d), a core distance is calculated and assigned to the first attribute only if the first attribute has not been assigned.

[0022] Preferably, each record of the index includes a third attribute in which the sort order of the record is stored.

[0023] According to one aspect of the present invention, an index-query framework is provided, namely, the method includes extracting a first accurate clustering of a set of records for a given threshold distance based on a combined evaluation of reachability distance and core distance of the records in the index following a linear scan through the index, where each record is assigned either a cluster identifier or a noise identifier.

[0024] To extract clusters from the clustering results, the index calculated by the method according to the invention is queried. A standard query of the improved index, for example known from OPTICS, was found by the inventors to already retrieve at least one correct clustering when the threshold distance is equal to a given input threshold distance. This property does not apply to standard cluster orderings, thus emphasizing that the index built by the method according to the invention contains more information than standard orderings.

[0025] Preferably, a first accurate clustering of the set of records is extracted for a selected threshold distance, the selected threshold distance being less than or equal to a predetermined threshold distance, and the method further comprises a candidate validation step, where for the extracted clusters of the first accurate clustering, each record is - located at the index immediately preceding the extracted cluster, a first attribute is assigned to the calculated core distance; - A noise identifier is assigned, This is tested against records of the extracted cluster that are labeled as core records, also called candidate records, and whose assigned calculated core distance is less than or equal to a selected threshold distance.

[0026] The preferred index query has the advantage that accurate clusterings can be reported within an efficient runtime even for threshold distances smaller than a given threshold distance used during index creation. This advantage is achieved through targeted searches on records representing missing boundary objects that were previously assigned as core records. These boundary objects are also referred to hereafter as pre-core objects. A particular object is considered a pre-core object for a threshold distance smaller than an input threshold distance if the number of neighbors of the object meets the core criterion only for the input threshold distance but not for the smaller threshold.

[0027] The targeted search for the previous core object may be limited to records labeled as noise records located to the left of the cluster under consideration in the index, i.e., located in the index immediately preceding the extracted cluster. In other words, the search space of the targeted search may be limited to noise records having a lower permutation order number assigned to their third attribute than the records of the respective extracted cluster. Furthermore, the permutation order of a particular candidate record must be equal to or greater than the permutation order of the closest record having a predefined value (infinity) assigned to its second attribute and located to the left of the cluster under consideration. This restriction of the search space is based on the observation that the previous core object cannot occur to the left of the first object in the cluster with respect to a predefined threshold distance surrounding the cluster under consideration with respect to the smaller selected threshold distance.

[0028] Compared to calculating exact clustering by DBSCAN from scratch, the index query according to one aspect of the present invention is more efficient, especially in terms of the number and complexity of range queries required to calculate neighborhoods for any threshold distance less than or equal to a given threshold distance. The DBSCAN algorithm requires a range query over the entire dataset for each object. In contrast, the index query requires a range query only for candidate records over the set of core records (and vice versa).

[0029] Preferably, a candidate validation step is performed for each extracted cluster of the first exact clustering respectively after extraction of each extracted cluster.

[0030] Preferably, a second precise clustering of the set of records is calculated based on a predetermined threshold distance and a number of selected records, the number of selected records being greater than the predetermined number of records, and the set of records is reduced to those records that are assigned cluster identifiers according to the first precise clustering.

[0031] The second exact clustering may be computed using any method that provides an exact clustering, in particular using the index-query framework and / or the DBSCAN algorithm according to an aspect of the present invention.

[0032] In one embodiment, a second accurate clustering is calculated based on a selected threshold distance, and the set of records is reduced to records that are assigned cluster identifiers according to the first accurate clustering based on the predetermined threshold distance.

[0033] Preferably, the set of records is separated into at least one subset, each subset of the at least one subset corresponding to an extracted cluster of the first accurate clustering with respect to a predetermined threshold distance, and a second accurate clustering is calculated separately for each subset.

[0034] Preferably, in step d), the cluster identifiers are incremented and each record in the vicinity of the selected processed record is inserted into a boundary record set of the respective cluster identifier if the respective record is labelled as a non-core record, and each boundary record set is merged with the subset corresponding to the respective cluster identifier before computing a second precise clustering for each subset.

[0035] Therefore, each subset has exactly one corresponding set of boundary records.

[0036] A non-core record for a given threshold and required number of neighboring records is defined as a record that has fewer records in its neighborhood than the required number of neighboring records.

[0037] Preferably, each record includes a fourth attribute, and for each record, the number of its neighbouring records is assigned to the fourth attribute during steps b) and d).

[0038] Preferably, for each core record only, the number of records contained in its neighborhood is assigned to a fourth attribute during steps b) and d).

[0039] Preferably, the second exact clustering is calculated using a density-based spatial clustering with noise-DBSCAN-algorithm.

[0040] Preferably, each record in the set of records represents a process instance of a process, the process being executed at or with the aid of a source computer system. [Brief description of the drawings]

[0041] Further details and features of the invention as well as specific embodiments of the invention can be derived from the following description in conjunction with the drawings. [Figure 1] FIG. 1 shows a flow chart of step d) of creating an index using an embodiment of the present invention. [Figure 2a] FIG. 2a is a diagram illustrating the concept of core distance C. [Figure 2b] FIG. 2b illustrates the concept of reachability distance R. [Diagram 3] FIG. 3 shows reachability plots of standard queries for an index built using OPTICS (a) and an index built by an embodiment of the invention (b) for the example dataset of Table 1 with a threshold of 1 / 3. [Figure 4] FIG. 4 shows reachability plots of standard queries for an index built using OPTICS (a) and an index built by an embodiment of a method according to the invention (b) for the example dataset of Table 1 with a threshold of 1 / 4. [Diagram 5] FIG. 5 is a diagram summarizing the index-query framework according to one embodiment of the present invention in comparison with the prior art. [Figure 6]FIG. 6 is a diagram visualizing the cumulative runtime of the DBSCAN algorithm and index-query framework according to an embodiment of the present invention for the selection of a given number of records MinPts measured on the CELONIS-1 dataset. [Figure 7] FIG. 7 is a diagram illustrating density filtering according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] The index for density-based clustering of a set of records computed by the method according to the invention represents an improved cluster ordering of the records, which is in particular an improvement over standard cluster orderings such as those constructed by the OPTICS algorithm.

[0043] The index is stored by the storage device, particularly in the main memory of the storage device.

[0044] Each record in the set of records is initially marked as unprocessed. When inserted into the index, each record includes a first attribute and a second attribute. Furthermore, each record forms part of a neighborhood within a predefined threshold distance ε. The predefined threshold distance ε is a first input parameter to the method according to the invention.

[0045] A record is a core record if its neighborhood contains at least a predefined number of records MinPts, which is a second input parameter to the method according to the invention. Thus, a record is a non-core record if its neighborhood contains less than a predefined number of records MinPts.

[0046] The index is calculated according to the following steps:

[0047] In step a), an unprocessed record is selected from the set of records. For the selected unprocessed record, a core distance is calculated. The core distance of a record is the minimum distance such that the record is still a core object. Non-core records, such as records representing noise objects and boundary objects, do not have a valid core distance value.

[0048] In step b), it is tested whether the selected unprocessed record is a core record, for example by counting the number of records in the neighborhood of the selected unprocessed record. If yes, assign the core distance calculated in step a) to the first attribute of the selected unprocessed record and label the selected unprocessed record as a core record. If not, assign a predefined value, such as infinity, ∞, to the first attribute of the selected unprocessed record and label the selected unprocessed record as a non-core record.

[0049] Therefore, steps a) and b) essentially require the computation of neighborhoods of the selected unprocessed records, which is based on range queries that are computationally expensive.

[0050] In step c), the selected unprocessed records are added to the index, thus marking the records as processed.

[0051] In step d), if the selected processed record is a core record, each of its neighboring records is processed according to the refined updates of the priority queue.

[0052] In step e), the records of the priority queue are inserted into the index, and step d) is repeated for the neighboring records that are core records. In doing so, for each record of the priority queue to be inserted into the index, a core distance is calculated and assigned to the first attribute. Further, if the corresponding record is a core record, it is labeled as a core record, otherwise it is labeled as a non-core record.

[0053] FIG. 1 shows a flow chart of step d) according to one embodiment of the present invention, which visualizes the processing of the neighborhood records, as will be explained in more detail below.

[0054] For each record in the neighborhood of the selected processed record, a reachability distance is calculated. The reachability distance is defined in terms of the concept of density reachability. The reachability distance of a neighborhood record with respect to a core record is the minimum distance such that the neighborhood record is still directly density reachable from the core record. At any smaller distance, either the neighborhood record is removed from the neighborhood of the core record or the core record loses its core property, which means that there are no more records that can be directly density reached from it.

[0055] Each neighbor record marked as unprocessed is inserted into the priority queue in ascending order of its reachability distance, which is assigned to a second attribute of the record. If the neighbor record under consideration is already in the priority queue and the newly calculated reachability distance is smaller than the currently assigned reachability distance, its position is updated according to the ascending order of its reachability queue, specifically decreasing.

[0056] Some of the records in the neighborhood of the selected processed record may have already been processed before. According to the present invention, each neighboring record marked as processed is removed from the index if it was previously labeled as a non-core record, and outputs a reachability distance for the selected processed record that is smaller than the one previously assigned to its second attribute. The neighboring records removed from the index are marked as unprocessed.

[0057] Neighboring records that are marked as processed and were previously labeled as core records are skipped.

[0058] Step d) of the method according to the invention therefore adapts the update function of the priority queue for OPTICS by a branch containing the decisions enclosed in double lines in Fig. 1. There are several conditions for the adaptation of the update function of the priority queue to be performed. Firstly, the record under consideration must have already been processed before. Furthermore, the record under consideration must be a non-core record, in particular a boundary object with respect to a given threshold distance ε. Finally, the newly calculated reachability distance for the object under consideration must be smaller than the reachability distance currently assigned to its second attribute. If all these conditions apply, the record under consideration is again marked as unprocessed and removed from the index. Finally, the record under consideration is inserted into the priority queue at the position corresponding to the updated value assigned to its second attribute.

[0059] In step e), each record of the priority queue is added to the index and thus marked as processed. If a record of the priority queue is a further core record, its corresponding neighbouring records are also processed according to step d).

[0060] In standard cluster ordering, the reachability distance of an object is only minimized with respect to previously processed objects. In contrast, step d) of the method according to the invention ensures that the reachability distance of each neighbouring record with respect to a given threshold distance ε is minimized with respect to all other records in the dataset.

[0061] Finally, steps a) through e) are repeated until all records in the set of records have been processed and inserted into the index.

[0062] Thus, the resulting data structure representing an index according to the invention contains the following attributes for each record: 1) The first attribute where the core distance of the record is stored. 2) A second attribute in which the reachability distance of the records is stored, the reachability distance being globally minimized if the record under consideration is a neighboring record with respect to a given threshold distance ε. 3) A third attribute in which the permutation result, i.e. a number between 1 and the number of records in the set of records, is stored.

[0063] Compared to a standard cluster ordering as obtained from OPTICS, the values ​​of the second and third attributes are different in our index due to the fact that nearby objects are rearranged as their reachability distance decreases further along the processing of the set of records.

[0064] FIG. 2a illustrates the concept of core distance C.

[0065] Shown in Fig. 2a is the neighborhood of a core record (located at the center) for a given threshold distance ε, where the given number of records MinPts is equal to 5. The neighborhood of a core record has 6 neighboring records, some of which may be non-core records and others may be further core records. Note that each record counts as its own neighboring record. The core distance C to the core record under consideration is visualized as a circle with a solid radius. In this example, the core distance C is smaller than the given threshold distance ε.

[0066] FIG. 2b illustrates the concept of reachability distance R.

[0067] In Fig. 2b, the same neighborhood as in Fig. 2a is shown. For neighboring records located further away from the core record under consideration than the core distance C of the core record, the reachability distance R is equal to the distance between the respective non-core record and the core record, as indicated by the dashed line. The distance can be calculated using a predefined distance function. For neighboring records located closer to the core record than the core distance C of the core record, the reachability distance R is equal to the core distance C.

[0068] In the following, an embodiment of the method according to the invention is presented by means of a concrete example drawn from the domain of process mining.

[0069] In the field of process mining, the aim is to analyze data describing processes, such as technological processes (e.g. soldering processes or manufacturing processes) or business processes (e.g. ordering processes). The processes are executed in and / or with the help of a source computer system. An execution of a process is called a case or process instance. A process instance comprises a sequence of process steps or events that were executed during the record of the process instance. A process instance is therefore described as a sequence of time-stamped events and is technically represented by a process protocol or event log.

[0070] An exemplary process instance may be given by the sequence: "Order", "Package", "Issue receipt", "Payment", and "Collect". In practice, a process instance can be modeled by forming a 2-gram, for example, {("Order", "Package"), ("Package", "Issue receipt"), ("Issue receipt", "Payment"), ("Payment", "Collect"). The 2-gram can be mapped to a set containing several tokens, and duplicate tokens are removed so that a process instance can be represented by an integer-based set, for example, {1, 2, 3, 4}. This modeling approach has the advantage of preserving order information between individual events. Furthermore, a process instance can be efficiently stored in a database as a set of its tokens.

[0071] For simplicity, however, the following examples assume a simpler data model: each process step is directly mapped to a unique integer token. For example, the process step "order" is assigned to token "1", the process step "package" is assigned to token "2", and so on. Thus, an example process can be represented by the set {1, 2, 3, 4, 5}. The resulting assignment of process steps to tokens is summarized in Table 1, with additional process steps added for the sake of the example.

[0072] TIFF2025508602000002.tif47142

[0073] Density-based clustering assigns each record in a set of records to either some of the detected clusters or to a noise group. The records assigned to the detected clusters are further subdivided into core objects and boundary objects. The core objects (core records) have at least a predefined number of records MinPts of neighboring records and hold the cluster together. The boundary objects (boundary records) have less than a predefined number of records MinPts as neighboring records but are located close to some of the core objects.

[0074] On the other hand, the DBSCAN algorithm can be used to compute an exact clustering of a set of records for one particular pair of input parameters ε, and the index, MinPts, computed by an embodiment of the method according to the invention can be calculated by using a combination of multiple input parameters ε for a given number of records MinPts without having to recalculate DBSCAN from scratch. * It also supports accurate cluster extraction for .ltoreq..epsilon..

[0075] The creation of the index and the extraction (query) of the detected clusters from the index is demonstrated on the set of records presented in Table 2.

[0076] TIFF2025508602000003.tif64142

[0077] In this example, the predefined distance function is selected to be the Jaccard distance function. For Jaccard distance, a distance value close to 0 reflects two similar records, and a distance value between two records close to 1 reflects two dissimilar records. As an example, the Jaccard distance between records r7 and r11 is equal to 1 / 6. Thus, for density-based clustering, records r7 and r11 are considered to be neighbors for an arbitrary threshold distance ε of at least 1 / 6.

[0078] First, the exact clustering of a set of records for a particular parameter pair is calculated. These calculations serve as a benchmark for the results obtained from the index according to the invention and from the standard cluster ordering generated by OPTICS. In the following, a given number of records MinPts is fixed equal to 4, and two given threshold distances ε * , i.e., 1 / 3 and 1 / 4 are used. Exact clusterings can be computed using DBSCAN, but for reproducibility, pairs of neighborhoods and corresponding threshold distances are provided for all relevant core objects in the set of records in Table 3. The computed exact clusterings are listed in Tables 4 and 5 for each given threshold distance.

[0079] TIFF2025508602000004.tif36170

[0080] TIFF2025508602000005.tif32170

[0081] TIFF2025508602000006.tif32170

[0082] For correct clustering from Table 4, the threshold distance ε *is equal to 1 / 3. Cluster 1 consists of records r1-r6. One of the common characteristics of these records is their short length, which ranges from 2 to 4 tokens. Furthermore, all of these records contain the token "1" and are generally characterized by a low token count, with the maximum token count being "4". From Table 1, we interpret that the low tokens represent process steps that are typically executed at the beginning of a process instance (e.g., "order" or "payment"). Therefore, we can conclude that cluster 1 in this example contains process instances from a set of records that are incomplete and not yet continued.

[0083] Cluster 2, on the other hand, consists of records r7-r12. These records are of relatively large size with set lengths ranging from 5-8 tokens. Furthermore, these records feature events that do not occur in any of the records from cluster 1. For example, the process step "withdraw" occurs in all records from cluster 2 but is absent in the records of cluster 1. Note that the process step "withdraw" frequently co-occurs with process steps such as "recover", "allocate" or "refund". Cluster 2 can therefore be considered as a group of process instances that are associated with a withdrawal and, more generally, that are advanced further than the process instances assigned to cluster 1.

[0084] Decreasing the threshold distance prevents some of the records from remaining in their corresponding neighborhood. As shown in Table 5, the threshold distance ε * If ε is equal to 1 / 4, then these records are dropped from their cluster and assigned to the noise group. In this example, records r1 and r2 have ε * The noise records for which the threshold distance is equal to 1 / 4 are the records that remain in the cluster for the lower threshold distance. In contrast, the records that remain in the cluster for the lower threshold distance are closer to each other, thereby forming a denser region in the data space.

[0085] So far, this is the exact cluster semantics of Tables 4 and 5. In what follows, the clustering is computed using both the OPTICS algorithm and the index according to one embodiment of the present invention.

[0086] In both cases, the generating distance, a predefined threshold distance, is set to the maximum distance of interest, i.e., ε is equal to 1 / 3. According to one embodiment of the present invention, the OPTICS algorithm results in a standard cluster ordering and the index construction results in an index. Standard cluster ordering and indexes are different extended permutations of a set of records, where permutation refers to the measures of core distance C and reachability distance R obtained for each record in the set of records. The resulting permutations can be visualized in a reachability plot, where the permuted records are plotted according to their resulting permutation order on the x-axis against their reachability distance on the y-axis.

[0087] FIG. 3 visualizes the reachability distances of the cluster orderings obtained from OPTICS (a) and from the index (b) computed by an embodiment of the method according to the invention.

[0088] A reachability plot is a bar plot with one bar for each record in the dataset. The height of the bar corresponds to the reachability distance assigned during construction of the index or cluster ordering, respectively. Records are plotted in processing order (of extraction) from left to right along the x-axis. The clustering structure of the dataset is therefore made visible in the form of a succession of valleys and peaks in the reachability distances. The valleys correspond to areas of high density and the peaks are sparse records. A record's bar height indicates the minimum distance this record is density reachable from records to its left (smaller values ​​of x), so a record with a large bar separates the cluster from other clusters and / or noise records.

[0089] As shown in the reachability plot in Fig. 3, the index constructed for a given threshold distance ε of 1 / 3 results in the correct clustering in Table 4. In contrast, OPTICS fails to obtain the correct clustering since there are several records that are incorrectly assigned to the noise group, namely records r1, r2, r7, r8, and r10.

[0090] This simple example already demonstrates that the clustering results obtained from our index for generating the threshold distance ε are exact clusterings like those obtained from DBSCAN, which can also be formally proven. However, the results obtained from OPTICS are not exact clustering results.

[0091] Note that in this example, the detected clusters are extracted using the standard query algorithm for both techniques, which is essentially a linear scan through permutations and therefore has linear time complexity.

[0092] Figure 4 shows the threshold distance ε * , we visualize the reachability distances obtained from the OPTICS algorithm (a) and from an index according to one embodiment of the invention (b).

[0093] As shown in Figure 4, the clustering results deviate from the exact clustering of Table 5 for both methods. OPTICS manages to come close to cluster 1 with one record (r3) missing. However, OPTICS can only find two out of six records from cluster 2. On the other hand, the complete cluster 2 can be extracted based on the index according to an embodiment of the present invention. For cluster 1, both methods find the same number of records.

[0094] Threshold distance ε *If ,is smaller than a predefined threshold distance ε, then both OPTICS and the method according to an embodiment of the present invention yield an accurate clustering approximation. However, the approximation reported from the index is better in the sense that it loses fewer neighboring records to the noise group.

[0095] Indeed, the set of records that may be misassigned to a noise group by an index according to an embodiment of the present invention may be controlled and therefore tightly limited. The inventors have found that, among the core records for a given threshold distance, only those that are neighboring records for a smaller threshold distance may be misassigned. This restriction is imposed by the smaller threshold distance ε * Using this targeted search, we are able to report accurate clustering from an index according to an embodiment of the present invention, and can report clustering results for any threshold distance ε smaller than a predefined threshold distance ε. * This improves on the standard query in that it can also report on

[0096] FIG. 5 summarizes the index-query framework according to one embodiment of the present invention in comparison with the prior art.

[0097] The index-query framework includes a first phase 10 where the cluster order is computed, and a second phase 20 where the cluster order is queried to extract / report clusters. The index-query framework has the general advantage that the potentially time-consuming computation of the cluster ordering only needs to be performed once for a maximum threshold distance ε of interest and a given number of records MinPts. Then, in the second phase 20, the index-query framework queries various threshold distances ε up to the given threshold distance ε. * Clusters can be extracted from the clustering results for . Cluster extraction is usually fast, and exploration of different parameter settings can be performed interactively.

[0098] As outlined above, the OPTICS algorithm provides a state-of-the-art algorithm for the index-query framework. In a first phase 10, a standard cluster ordering is computed, which can be queried in a second phase 20 using a standard query. A standard query is essentially a linear scan through the permutations obtained from the first phase 10. However, all clusters extracted from the OPTICS cluster ordering are approximate clusters.

[0099] Instead, the method according to the invention creates an index in a first phase 10, which supports the extraction of accurate clusterings using standard queries with a given threshold distance ε. * For ≦ε, the standard query also searches for approximate clusters from the index, where the approximate clusters can of course correspond to exact clusters at the same time.

[0100] Preferably, the information contained in the index constructed by the method according to the invention has a threshold distance ε smaller than a predetermined threshold distance ε * The index is exploited in that during cluster extraction for , a targeted search for prior core objects is performed, where a set of candidate prior core objects is verified against the records of the extracted cluster that are labeled as core records using a range query. This targeted search is only possible due to the additional information contained in the index compared to the standard cluster ordering. As a result, once the index is established, according to one aspect of the invention, a search for any threshold distance ε less than or equal to a predefined threshold distance ε is performed. * We can interactively obtain an exact clustering for

[0101] In general, accurate clustering contains more cluster objects, i.e., records assigned to clusters, than approximate clustering as calculated by OPTICS. To quantify the difference between accurate and approximate clustering, the relative ratio of the number of (deduplicated) cluster objects obtained from the index according to the invention to the number of (deduplicated) cluster objects obtained from OPTICS can be calculated. In experiments on large process mining datasets, the inventors found that the median value of this relative ratio obtained for various generative input parameter pairs is at least 1%. Missed cluster objects in the clustering obtained from OPTICS are erroneously assigned to noise groups. Thus, for large datasets of the order of millions of records, a misassignment of cluster objects to noise groups of the order of 1% can effectively result in a large absolute number of missing cluster objects.

[0102] FIG. 6 compares the cumulative runtimes of correct clustering obtained from DBSCAN and the index-query framework, according to one embodiment of the present invention.

[0103] The performance of the accurate clustering computation is evaluated on a large-scale dataset from the process mining domain, CELONIS-1. Accurate clustering is achieved for 10 threshold distances ε equally spaced between 0.07 and 0.25. * A predetermined number of records MinPts are set to 16, 128, and 1024.

[0104] In the case of the DBSCAN algorithm, DBSCAN is * Since we compute the exact clustering for all pairs of , and a given number of points MinPts, only parameter pairs that contain the given number of points MinPts are evaluated. For the 10 pairs in this example, the overall cumulative runtime for DBSCAN was more than 1 day.

[0105] Using the index-query framework according to one aspect of the present invention, the index only needed to be computed once, which took approximately 9 hours for a given number of records MinPts 16. Subsequent extraction of 10 accurate clusterings from the precomputed index in a second phase 20 took only 1 hour.

[0106] While the cumulative runtime for computing the index in the first phase 10 is approximately constant for further evaluated values ​​of the predefined number of records MinPts, the index query 20 is more efficient for higher values ​​of the predefined number of records MinPts. At lower values ​​of the predefined number of records MinPts, the number of core records in the dataset is large. Thus, a large number of candidate validations must be performed during the targeted search, which explains why the cumulative runtime for the index query 20 increases as the value of the predefined number of records MinPts decreases.

[0107] As a result, the cumulative runtime of this experiment shows that the index-query framework according to an embodiment of the present invention is up to 3 times faster than the DBSCAN algorithm on the CELONIS-1 dataset. The OPTICS algorithm is not part of this comparison because it cannot produce accurate clustering.

[0108] In one embodiment, the neighborhood computation can be separated from the actual clustering algorithm; that is, the neighborhood is precomputed and materialized in advance. The materialization cost is O(n 2) memory, where n denotes the collection size. However, the advantage of this approach is the improved runtime due to the fact that the neighborhoods can be computed asymmetrically. For example, a set similarity join can be used, according to which the filter validation framework avoids part of the distance calculation. In particular, the AllPairs or MetricJoin algorithms can be used to compute the set similarity joins. In step b) of the method according to the invention, the precomputed neighborhoods can be simply looked up from the storage (instead of being computed), which reduces the runtime for index construction.

[0109] In one embodiment, the neighborhood computation and candidate validation during index query to extract the exact clusters can both be performed in parallel, thus further improving the runtime shown in FIG.

[0110] In practice, deduplication of records in a set of records is important. In one embodiment, duplicate data is exploited by deduplication of the set while reading the set of records into memory, and a duplicate count is stored for each deduplication set. These duplicate counts are finally used to determine the neighborhood size. This procedure ensures that an accurate clustering is obtained while actually only a portion of the data is processed in memory. In the case of neighborhood materialization, deduplication is a necessary prerequisite to prevent the neighborhood from becoming too large and exceeding the available memory on the computer.

[0111] FIG. 7 illustrates density filtering according to one embodiment of the present invention.

[0112] In one aspect of the present invention, a density filter 30 is introduced, which allows the user to increase the density of the clustering by increasing a given number of records MinPts. At a threshold distance ε, ε *Note that ε is considered constant during density filtering 30. In combination with the index-query framework introduced above, density filter 30 filters any pair (ε * ,MinPts * ), where ε * <= ε and MinPts * >=MinPts. The basic idea of ​​density filtering is to exploit the fact that dense clusters are a subset of spare clusters.

[0113] Therefore, once the clustering for a given number of records MinPts has been calculated, the clustering can be used to * It is possible to reduce the set of records by noise groups for any subsequent calculations using ≧MinPts. Of course, the initial clustering should be an accurate clustering to avoid filtering out incorrectly assigned noise records. Therefore, density filtering 30 is ideally applied to the clustering resulting from the index calculated by the method according to the invention.

[0114] According to one aspect of the invention, the index is calculated based on a selected number of records MinPts that is greater than the predetermined number of records MinPts. * There are three approaches to computing an exact clustering for a chosen threshold distance ε * If ε is equal to a predefined threshold distance ε, the first approach 1 is recommended. * is smaller than a predefined threshold distance ε, the second approach 2 and the third approach 3 can be applied. * It can also be applied when ∈ is equal to a predetermined threshold distance ε.

[0115] The first approach 1 and the third approach 3 are based on the computation of an extended index provided according to an aspect of the present invention. The extended index includes a fourth attribute in which the size of each neighborhood, the neighborhood count, is stored. Furthermore, for each (exact) cluster for a given threshold distance ε and a given number of records MinPts, the set of all boundary objects (boundary record set) of the respective cluster, i.e. non-core records, is recorded.

[0116] In the first approach 1, the following steps are performed to obtain an accurate clustering 2A for a given threshold distance and a number of selected records MinPts greater than (or equal to) a given number of records MinPts. * and are obtained. The exact clusters 1A-1,...1A-n are obtained from the query of the extended index, where n denotes the number of clusters detected. By extracting clusters from the extended index, noise records are automatically discarded. Merge 31: Merging each extracted cluster 1A-1,...1A-n with its respective boundary record set, such that all ambiguous boundary objects are also included in the resulting n subsets of the set of records. Compute 32: A selected number of records MinPts are computed by invoking the DBSCAN algorithm on each subset separately, specifically in parallel. * 2. Compute an exact clustering 2A for x, y ...

[0117] In the second approach 2, the following steps are performed to find a selected threshold distance ε that is smaller than (or equal to) a predefined threshold distance ε (used to generate the index): * and the number of selected records MinPts is greater than (or equal to) the given number of records MinPts * From the index query, the selected threshold distance ε * 3: Obtain an exact clustering 1B for x, where the exact clustering 1B can also be obtained by querying the extended index. Discard 31: Discard noise groups from the exact clustering 1B to reduce the set of records. Call (DBSCAN algorithm) 32: Number of selected records MinPts * Invoke the DBSCAN algorithm on the reduced set of records to compute an exact clustering 2B for

[0118] In the third approach 3, the following steps are performed to find a selected threshold distance ε that is smaller than (or equal to) a predefined threshold distance ε (used to generate the index): * , and a selected record number MinPts greater than (or equal to) a given record number MinPts * 1A-n for a given threshold distance ε is obtained from querying the extended index with the given threshold distance ε, where n denotes the number of detected clusters. By extracting clusters from the extended index, noise records are automatically discarded. Merge 31: Merge each extracted cluster 1A-1,...1A-n with its respective boundary record set such that all ambiguous boundary objects are also included in the resulting n subsets of the set of records. Calculate 32: Obtain the correct clusters 1A-1,...1A-n for a given threshold distance ε from querying the extended index with the given threshold distance ε, where n denotes the number of detected clusters. Merge 33: Merge each extracted cluster 1A-1,...1A-n with its respective boundary record set such that all ambiguous boundary objects are also included in the resulting n subsets of the set of records. Calculate 34: Merge each extracted cluster 1A-1,...1A-n with its respective boundary record set such that all ambiguous boundary objects are also included in the resulting n subsets of the set of records. * , Number of selected records MinPts *Compute an accurate clustering for x, y, by invoking the DBSCAN algorithm separately, in particular in parallel, on each subset, and post-process the resulting clusterings to assign each ambiguous boundary object to its correct detected cluster.

[0119] Selected number of records MinPts * Using density filters according to any of the three approaches 1;2;3 for the computation of exact clustering for >= MinPts. The dataset (set of records) is reduced by the number of noise records, and range queries (neighborhood evaluation) performed on this reduced dataset are accelerated. This is particularly useful since range queries are the main cause of runtime in density-based clustering. As a result, an (extended) index according to one embodiment of the present invention can be used to compute clustering for any denser number of records MinPts. * It can be used directly as a filter 30 for

[0120] In any case, storing the size of each neighborhood during (extended) index construction according to one aspect of the present invention is a method for determining the size of each neighborhood based on a predefined threshold distance ε and the number of selected records MinPts * It has the further advantage that the core records for can be determined quickly, thus reducing the number of range queries required.

[0121] As a result, the expansion index calculated according to one aspect of the present invention is ε * ≦ε and for a fixed MinPts, or *For ε≧MinPts and fixed ε, an exact clustering can be provided. That is, the user can interactively increase the density of the exact clustering via one of two input parameters, while the other of the two input parameters remains fixed. In the second phase 20, index queries are used to vary the threshold distance ε. Density filtering 30 is used to vary the number of records MinPts. In combination, any pair of threshold distance and number of records (ε * ≤ ε,MinPts * Exact clusterings for all points in a set (>= MinPts) can be computed and explored interactively.

Claims

1. 1. A computer-implemented method for computing an index for a first density-based clustering of a set of records, the index being stored using a storage device; Here, each record is - first marked as unprocessed, - comprising a first attribute and a second attribute, - forms part of a neighborhood within a given threshold distance, A record is a core record if its neighborhood contains at least a predetermined number of records and a non-core record if its neighborhood contains less than a predetermined number of records, and the index is calculated according to the following steps: a) selecting an unprocessed record from the set of records and calculating a core distance of the selected unprocessed record, the core distance being the minimum distance from the selected unprocessed record whose neighborhood still contains at least a predetermined number of records; b) if the selected unprocessed record is a core record, assigning the core distance to the first attribute and labeling the selected unprocessed record as a core record, otherwise assigning a predetermined value to the first attribute and labeling the selected unprocessed record as a non-core record; c) adding the selected unprocessed records to an index and marking the selected unprocessed records as processed; d) if the selected processed record is a core record, processing each record in a neighborhood of the selected processed record, calculating a reachability distance for each record, the reachability distance being the minimum distance at which each record is still directly density reachable from the selected processed record; placing each record marked as unprocessed in a priority queue in ascending order of its reachability distance, the reachability distance being assigned to the second attribute; removing from the index each record previously labeled as a non-core record and outputting a reachability distance less than that previously assigned to its second attribute, each removed record being marked as unprocessed and each removed record being inserted into a priority queue according to ascending order of its reachability distance; e) adding each record of the priority queue to the index, where each record is marked as processed, and if the added record is a core record, its neighboring records are processed according to step d); A method wherein steps a) through e) are repeated until all records of the set of records have been processed and inserted into the index.

2. 2. The method of claim 1, comprising between steps a) and b) determining whether the selected unprocessed record is a core record by counting the records in a neighborhood of the selected unprocessed record.

3. 3. The method of claim 1, wherein during processing of each record of the priority queue in step d), the core distance is calculated and assigned to the first attribute only if the first attribute is not assigned.

4. The method of any one of claims 1 to 3, wherein each record of the index includes a third attribute in which the permutation order of the record is stored.

5. 5. The method of claim 1, further comprising: extracting a first accurate clustering of the set of records for the predetermined threshold distance based on a combined evaluation of the reachability distance and the core distance of the records in the index according to a linear scan through the index, wherein each record is assigned either a cluster identifier or a noise identifier.

6. The first accurate clustering of the set of records is extracted for a selected threshold distance, the selected threshold distance being less than or equal to the predetermined threshold distance, and the method further comprises a candidate validation step, where for the extracted clusters of the first accurate clustering, each record is - located at the index immediately preceding the extracted cluster, - a first attribute is assigned to the calculated core distance; - a noise identifier is assigned, The method of claim 5 , wherein records labeled as core records and assigned with a calculated core distance less than or equal to the selected threshold distance are verified against the records of the extracted cluster.

7. The method of claim 6 , wherein the candidate validation step is performed for each extracted cluster of the first exact clustering respectively after the extraction of each extracted cluster.

8. 8. The method of claim 5, 6 or 7, wherein a second exact clustering of the set of records is calculated based on the predetermined threshold distance and a selected number of records, the selected number of records being greater than the predetermined number of records, and the set of records is reduced to the records that are assigned the cluster identifiers according to the first exact clustering.

9. 9. The method of claim 8, wherein the second accurate clustering is calculated based on the selected threshold distance, and the set of records is reduced to those records that are assigned the cluster identifiers according to the first accurate clustering based on the predetermined threshold distance.

10. 10. The method of claim 9, wherein the set of records is separated into at least one subset, each subset of the at least one subset corresponding to an extracted cluster of the first accurate clustering with respect to the predetermined threshold distance, and the second accurate clustering is calculated separately for each subset.

11. 11. The method of claim 10, wherein in step d), the cluster identifier is incremented and each record in the neighborhood of the selected processed record is inserted into a boundary record set of the respective cluster identifier if the respective record is labeled as a non-core record, and each boundary record set is merged with the subset corresponding to the respective cluster identifier before computing the second precise clustering for each subset.

12. 12. The method according to claim 1, wherein each record comprises a fourth attribute, and for each record, the number of said records in its neighbourhood is assigned to said fourth attribute during said steps b) and d).

13. 13. The method of claim 12, wherein for each core record only, the number of said records contained in its neighborhood is assigned to the fourth attribute during steps b) and d).

14. The method according to any one of claims 8 to 13, wherein the second accurate clustering is calculated using a density-based spatial clustering with noise-DBSCAN-algorithm.

15. A method according to any preceding claim, wherein each record in the set of records represents a process instance of a process, the process being executed at or with the aid of a source computer system.

Images