How to create an index to report large-scale variant clustering

Abstract

A computer-implemented method is provided for computing an index for a first density-based clustering of a set of records, the index being stored using a storage device. The index is stored within any selected threshold distance ε less than or equal to a predefined threshold distance ε. * and a given number of records MinPts that form the input parameter pair on which the index is computed.

Description

[Technical Field]

[0001] The present invention relates to a computer implementation method for calculating an index for density-based clustering from a set of records. More specifically, the index enables the efficient exploration of multiple parameter settings for accurate density-based clustering. [Background technology]

[0002] In general, a key task in data analysis projects, particularly in the domain of process mining, is identifying groups of similar objects within a dataset. The methods used to address this problem are summarized under the term clustering.

[0003] Among clustering methods, one approach is called density-based clustering, which was introduced by Ester et al. (A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise, Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (1996), AAAI Press, 226-231) along with the DBSCAN algorithm for calculating density-based clusters for a given dataset. The idea behind density-based clustering is to form clusters from high-density regions within the space occupied by the dataset, also known as the data space. Clusters are separated by low-density regions, and objects far from high-density regions are labeled as noise.

[0004] The DBSCAN algorithm can be applied to a dataset without needing to know the number of clusters present in the dataset beforehand. Furthermore, it does not require any assumptions about the statistical distribution of the data within the dataset. The DBSCAN algorithm requires only the minimum density that objects must satisfy to form a cluster as input. This minimum density is determined by two input parameters commonly called a given threshold distance ε and the minimum number of neighboring records MinPts. Therefore, the DBSCAN algorithm requires minimal background knowledge to analyze and explore patterns within a dataset.

[0005] In many scenarios, precise parameter settings are not known in advance and depend on the actual data being investigated. Indeed, often there is no inherent density setting that captures all meaningful clusters in the dataset. Therefore, it is necessary to try several different settings until good / satisfactory clustering is found.

[0006] However, the DBSCAN algorithm takes O(n 2 Because it involves calculating the distance between individuals, it is computationally expensive. This complexity poses a serious problem for the application of DBSCAN in the age of big data. In fact, even highly optimized, state-of-the-art implementations of DBSCAN, when applied to large real-world datasets, especially those in the domain of process mining, have very long runtimes and limit the parameter space that can be explored to optimize the clustering results.

[0007] In practice, DBSCAN must be applied several times with different input parameter settings until satisfactory results are achieved. The OPTICS algorithm by Ankerst et al. (OPTICS: Ordering Points to Identify the Clustering Structure, Proceedings of ACM SIGMOD Int.Conf. on Management of Data (1999), Vol.28 ACM New York, NY, USA, 49-60) attempts to overcome this problem.

[0008] Similar to the DBSCAN algorithm, the OPTICS algorithm requires two input parameters: a predetermined distance value ε and a minimum number of neighboring records MinPts. For the same number of neighboring records MinPts, the OPTICS algorithm sets a threshold ε that does not exceed the input threshold ε. * Compute a (standard) cluster ordering that contains information about arbitrary density-based clustering. The idea is that one calculation of OPTICS is equivalent to several runs of DBSCAN, with an input threshold ε less than or equal to the input threshold ε of the OPTICS algorithm. * This involves replacing it with the OPTICS algorithm, which computes cluster ordering, which is a permutation of the input datasets that includes a specific additional measurement for each record in the dataset.

[0009] However, the cluster ordering calculated by OPTICS can only formally yield approximate DBSCAN clustering. While DBSCAN produces accurate clustering, OPTICS-based clustering is density-based but not accurate. The clustering results are approximate in that objects located at the cluster boundaries are often misidentified as noise. Therefore, the cluster ordering calculated by OPTICS is also non-deterministic. In practice, the degree of approximation achieved by the OPTICS algorithm cannot be estimated in essence, i.e., without requiring further computation. [Overview of the project]

[0010] Thus, an object of the present invention is to provide a method for efficient calculation of improved density-based clustering, thereby enabling a user to interactively explore a parameter space bounded by input parameters while ensuring accurate clustering results at least for the input parameters.

Means for Solving the Problems

[0011] According to the present invention, this object is solved by the method described in the independent claims. Preferred embodiments and further developments of the present invention are defined in the dependent claims.

[0012] Thus, a computer-implemented method is provided for calculating an index for the first density-based clustering of a set of records, where the index is stored using a storage device and each record - is first marked as unprocessed, - has a first attribute and a second attribute, - forms part of a neighborhood within a predetermined threshold distance, a record is a core record if its neighborhood contains at least a predetermined number of records and a non-core record if its neighborhood contains less than a predetermined number of records, and the index is calculated according to the following steps, a) selecting an unprocessed record from the set of records and calculating the core distance of the selected unprocessed record, where the core distance is the minimum distance from the selected unprocessed record such that its neighborhood still contains at least a predetermined number of records, b) if the selected unprocessed record is a core record, assigning the core distance to the first attribute and labeling the selected unprocessed record as a core record, otherwise assigning a predetermined value to the first attribute and labeling the selected unprocessed record as a non-core record, c) Add the selected unprocessed record to the index and mark the selected unprocessed record as processed, and d) If the selected processed record is a core record, process each record in the vicinity of the selected processed record, calculate the reachability distance for each record, where the reachability distance is the minimum distance at which each record is still directly density reachable from the selected processed record, add each record marked as unprocessed in ascending order of its reachability distance to the priority queue, where the reachability distance is assigned to the second attribute, remove from the index each record that was previously labeled as a non-core record and outputs a reachability distance smaller than that previously assigned to its second attribute, where each removed record is marked as unprocessed and each removed record is inserted into the priority queue according to the ascending order of its reachability distance, e) Add each record in the priority queue to the index, where each record is marked as processed, and if the added record is a core record, the records in its vicinity are processed according to step d), Steps a) to e) are repeated until all records in the set of records are processed and inserted into the index.

[0013] The method according to the present invention has the advantage of allowing the threshold distance of density-based clustering to be varied, resulting in accurate clustering for a given threshold distance and at least an approximation of accurate clustering for any smaller threshold distance. This approximation is generally superior to the corresponding approximation obtained from the OPTICS algorithm. In particular, the inventors have found that the density-based clustering resulting from the index according to the present invention is very close to, or even equal to, the accurate clustering obtained from the DBSCAN algorithm. In fact, the inventors have found that the OPTICS algorithm can only simultaneously achieve the quality of the approximation resulting from the index according to the present invention. Standard cluster ordering resulting from OPTICS is technically limited to reporting a better approximation than that reported by the index according to the present invention because the index according to the present invention collects more information from the set of records being clustered.

[0014] It should be noted that the method according to the present invention is designed for situations where the neighborhood is unknown beforehand. Calculating the neighborhood requires computationally expensive range queries to calculate the neighborhood of a given record relative to a given threshold distance, and therefore the core distance of the given record. When the neighborhood is known beforehand, the range queries scale to O(1) at runtime, making conventional methods readily usable. However, in many situations, especially for new datasets, the neighborhood is unknown beforehand.

[0015] Exact clustering differs from density-based clustering in that it assigns each record representing an ambiguous boundary object to exactly one cluster. More formally, exact clustering of a set of records for a given threshold and a given number of neighboring records is defined as dividing the set of records into at least one partition, as follows: (1) All partitions are identical to, or subsets of, each density-based cluster in the set of records. (2) All core records of a density-based cluster in the set of records form part of their respective partitions. (3) Each boundary object forms part of exactly one partition corresponding to the density-based cluster to which the boundary object belongs.

[0016] With respect to DBSCAN, the index according to the present invention has the advantage of providing improved cluster ordering, supporting index queries for accurate clustering for any threshold distance less than or equal to the input threshold distance, without having to compute accurate clustering from scratch. The inventors have found that the index-query framework based on the index according to the present invention consistently yields a better clustering runtime than that achieved by the standard DBSCAN algorithm, especially for large datasets. Furthermore, the index according to the present invention has exactly the same requirements for the data model as DBSCAN, i.e., it can be applied to any domain.

[0017] The improvement of the method according to the present invention over the OPTICS algorithm is based on an improved cluster ordering that picks up more information from objects in the dataset than the standard cluster ordering of OPTICS. The additional information is generated, in particular, by the adaptation of the priority queue update in step d). In the OPTICS algorithm, neighbor records of a core record are processed only once, but the method according to the present invention reprocesses neighbor records if they are found to be non-core records and their reachability distance is smaller than previously assigned.

[0018] Furthermore, when inserting records from the priority queue into the index according to step e), neighboring records among the inserted records, which are core records, are also processed according to the improved update of the priority queue in step d). In particular, if neighboring records are already in the priority queue, their reachability distance and location are updated as necessary.

[0019] Although the method according to the present invention involves reprocessing records, empirical analysis of the OPTICS algorithm and index according to the present invention conducted by the inventors revealed that both algorithms share the same asymptotic computational complexity. In other words, the small overhead resulting from the improved prioritization queue update according to the present invention is negligible for large datasets.

[0020] Preferably, between step a) and step b), it is determined whether the selected unprocessed record is a core record by counting the records in its vicinity.

[0021] Preferably, during the processing of each record in the prioritized queue in step d), the core distance is calculated and assigned to the first attribute only if the first attribute has not been assigned.

[0022] Preferably, each record in the index includes a third attribute that stores the permutation order of the records.

[0023] According to one aspect of the present invention, an index-query framework is provided, namely, the method includes extracting a first accurate clustering of a set of records for a given threshold distance based on a combined evaluation of the reachability distance and core distance of records in the index, following a linear scan through the index, wherein each record is assigned either a cluster identifier or a noise identifier.

[0024] To extract clusters from the clustering results, an index calculated by the method according to the present invention is queried. A standard query for the improved index, for example, is well known from OPTICS, and the inventors have found that it already retrieves at least one accurate clustering when the threshold distance is equal to a given input threshold distance. This characteristic does not apply to standard cluster ordering, and therefore highlights that the index constructed by the method according to the present invention contains more information than standard ordering.

[0025] Preferably, a first accurate clustering of the set of records is extracted for a selected threshold distance, the selected threshold distance being less than or equal to a predetermined threshold distance, and the method further includes a candidate validation step, where for the extracted clusters of the first accurate clustering, each record is -Located in the index immediately preceding the extracted cluster, -The first attribute is assigned to the calculated core distance. - A noise identifier is assigned, These are also called candidate records, but are labeled as core records and are validated against records in extracted clusters whose assigned calculated core distance is less than or equal to a selected threshold distance.

[0026] A preferred index query has the advantage of being able to report accurate clustering within an efficient runtime, even for threshold distances smaller than the predetermined threshold distance used during index creation. This advantage is achieved through a targeted search for records representing missing boundary objects that were previously assigned as core records. These boundary objects will hereafter be referred to as pre-core objects. A particular object is considered a pre-core object for threshold distances smaller than the input threshold distance if the number of neighboring objects of the object meets the core criterion only for the input threshold distance, but not for smaller thresholds.

[0027] A targeted search for a previous core object may be limited to records labeled as noise records located to the left of the cluster under consideration in the index, i.e., located in the index immediately preceding the extracted cluster. In other words, the search space for a targeted search may be limited to noise records having a lower permutation order number assigned to their third attribute than the records of each extracted cluster. Furthermore, the permutation order of a particular candidate record must be greater than or equal to the permutation order of the nearest record located to the left of the cluster under consideration, having a predetermined value (infinity) assigned to its second attribute. This limitation of the search space is based on the observation that a previous core object cannot occur to the left of a first object in a cluster with respect to a predetermined threshold distance surrounding the cluster under consideration with respect to a smaller selected threshold distance.

[0028] Compared to calculating accurate clustering from scratch using DBSCAN, an index query according to one aspect of the present invention is more efficient, particularly in terms of the number and complexity of range queries required to calculate neighborhoods for any threshold distance below a given threshold distance. The DBSCAN algorithm requires range queries across the entire dataset for each object. In contrast, an index query requires range queries only for candidate records across a set of core records (and vice versa).

[0029] Preferably, the candidate validation step is performed for each extracted cluster of the first accurate clustering after each extracted cluster has been extracted.

[0030] Preferably, a second accurate clustering of the set of records is calculated based on a predetermined threshold distance and a number of selected records, where the number of selected records is greater than the predetermined number of records, and the set of records is reduced to records to which cluster identifiers are assigned according to the first accurate clustering.

[0031] The second accurate clustering can be computed using any method that provides accurate clustering, in particular using an index-query framework and / or DBSCAN algorithm according to one aspect of the present invention.

[0032] In one embodiment, a second accurate clustering is calculated based on a selected threshold distance, and the set of records is reduced to records to which cluster identifiers are assigned according to a first accurate clustering based on a predetermined threshold distance.

[0033] Preferably, the set of records is separated into at least one subset, each subset of which corresponds to extracted clusters of a first exact clustering with respect to a predetermined threshold distance, and a second exact clustering is calculated separately for each subset.

[0034] Preferably, in step d), the cluster identifiers are incremented, and each neighboring record of the selected processed record is inserted into the boundary record set of the respective cluster identifier, if each record is labeled as a non-core record, and each boundary record set is merged with the subset corresponding to the respective cluster identifier before a second exact clustering is computed for each subset.

[0035] Therefore, each subset corresponds precisely to one set of boundary records.

[0036] A noncore record for a given threshold and required number of neighboring records is defined as a record that has fewer neighboring records than the required number of neighboring records.

[0037] Preferably, each record includes a fourth attribute, and for each record, the number of neighboring records is assigned to the fourth attribute during steps b) and d).

[0038] Preferably, for each core record only, the number of records contained in its neighborhood is assigned to the fourth attribute during steps b) and d).

[0039] Preferably, the second accurate clustering is calculated using density-based spatial clustering with a noise-DBSCAN algorithm.

[0040] Preferably, each record in the set of records represents a process instance of a process, and the process was executed in or with the help of the source computer system. [Brief explanation of the drawing]

[0041] Details and features of the present invention, as well as specific embodiments of the present invention, can be derived from the following description relating to the drawings. [Figure 1] Figure 1 is a flowchart showing step d) of creating an index using one embodiment of the present invention. [Figure 2a] Figure 2a shows the concept of core distance C. [Figure 2b] Figure 2b illustrates the concept of reachable distance R. [Figure 3] Figure 3 shows the reachability plots of standard queries for the exemplary dataset in Table 1 with a 1 / 3 threshold, for indexes constructed using OPTICS(a) and indexes constructed according to the embodiment of the present invention (b). [Figure 4] Figure 4 shows reachability plots of standard queries for an exemplary dataset from Table 1 with a 1 / 4 threshold, for an index constructed using OPTICS(a) and an index constructed by one embodiment of the method according to the present invention (b). [Figure 5] Figure 5 is a diagram summarizing an index-query framework according to one aspect of the present invention in comparison with the prior art. [Figure 6]Figure 6 visualizes the cumulative runtime of the DBSCAN algorithm and index-query framework according to one aspect of the present invention for selecting a predetermined number of records MinPts measured for the CELONIS-1 dataset. [Figure 7] Figure 7 shows a density filtering method according to one aspect of the present invention. [Modes for carrying out the invention]

[0042] The index for density-based clustering of the set of records calculated by the method according to the present invention represents an improved cluster ordering of the records. This cluster ordering is particularly improved over standard cluster ordering, such as that constructed by the OPTICS algorithm.

[0043] The index is stored by the storage device, specifically in the main memory of the storage device.

[0044] Each record in the set of records is initially marked as unprocessed. When inserted into the index, each record contains a first attribute and a second attribute. Furthermore, each record forms part of a neighborhood within a predetermined threshold distance ε. The predetermined threshold distance ε is a first input parameter to the method according to the present invention.

[0045] A record is a core record if its neighborhood contains at least a predetermined number of records, MinPts. The predetermined number of records, MinPts, is a second input parameter to the method according to the present invention. Therefore, a record is a non-core record if its neighborhood contains less than the predetermined number of records, MinPts.

[0046] The index is calculated according to the following steps:

[0047] In step a), unprocessed records are selected from the set of records. The core distance is calculated for the selected unprocessed records. The core distance of a record is the minimum distance at which the record remains a core object. Non-core records, such as records representing noise objects and boundary objects, do not have a valid core distance value.

[0048] In step b), whether the selected raw record is a core record is tested, for example, by counting the number of records in its neighborhood. If yes, the core distance calculated in step a) is assigned to the first attribute of the selected raw record, and the selected raw record is labeled as a core record. Otherwise, a predetermined value such as infinity or ∞ is assigned to the first attribute of the selected raw record, and the selected raw record is labeled as a non-core record.

[0049] Therefore, steps a) and b) essentially require calculations of the neighborhood of the selected unprocessed records, which are based on computationally expensive range queries.

[0050] In step c), the selected unprocessed records are added to the index. Therefore, the records are marked as processed.

[0051] In step d), if the selected processed record is a core record, each of its neighboring records is processed according to the improved update of the priority queue.

[0052] In step e), the records of the priority queue are inserted into the index, and step d) is repeated for the neighboring records that are core records. In doing so, for each record of the priority queue that should be inserted into the index, the core distance is calculated and assigned to the first attribute. Furthermore, if the corresponding record is a core record, it is labeled as a core record; otherwise, it is labeled as a non-core record.

[0053] Figure 1 shows a flowchart of step d) according to one embodiment of the present invention, which visualizes the processing of neighboring records, as will be described in more detail below.

[0054] For each neighboring record of a selected processed record, the reachability distance is calculated. The reachability distance is defined in terms of the concept of density reachability. The reachability distance of a neighboring record with respect to the core record is the minimum distance at which the neighboring record is still directly density reachable from the core record. At any distance smaller than this, the neighboring record is either removed from the neighborhood of the core record or the core record loses its core properties, meaning that there are no longer any records that can be directly density reachable from it.

[0055] Each neighboring record marked as pending is inserted into a priority queue in ascending order of its reachability distance. Reachability distance is assigned to the record's second attribute. If a neighboring record under consideration is already in the priority queue and the newly calculated reachability distance is less than the currently assigned reachability distance, its position is updated according to the ascending order of the reachability queue, specifically decreasing.

[0056] Some of the records in the vicinity of the selected processed record may have already been processed previously. According to this invention, each neighboring record marked as processed is removed from the index if it was previously labeled as a non-core record, and for the selected processed record, a reachability distance smaller than that previously assigned to its second attribute is output. Neighboring records removed from the index are marked as unprocessed.

[0057] Neighboring records that have been marked as processed and previously labeled as core records are skipped.

[0058] Therefore, step d) of the method according to the present invention adapts the priority queue update function for OPTICS by branching, which includes the decision enclosed by the double line in Figure 1. There are several conditions for the adaptation of the priority queue update function to be performed. First, the record under consideration must have already been processed previously. Furthermore, the record under consideration must be a non-core record, in particular a boundary object with respect to a given threshold distance ε. Finally, the newly calculated reachability distance for the object under consideration must be less than the reachability distance currently assigned to its second attribute. If all of these conditions are met, the record under consideration is again marked as unprocessed and removed from the index. Lastly, the record under consideration is inserted into the priority queue at a position corresponding to the updated value assigned to its second attribute.

[0059] In step e), each record in the priority queue is added to the index and therefore marked as processed. If a record in the priority queue is a further core record, its corresponding neighboring records are also processed according to step d).

[0060] In standard cluster ordering, the reachability distance of an object is minimized only with respect to previously processed objects. In contrast, step d) of the method according to the present invention ensures that the reachability distance of each neighboring record with respect to a given threshold distance ε is minimized with respect to all other records in the dataset.

[0061] Ultimately, steps a) through e) are repeated until all records in the set of records have been processed and inserted into the index.

[0062] Therefore, the resulting data structure representing the index according to the present invention includes the following attributes for each record. 1) The first attribute where the core distance of the record is stored. 2) A second attribute that stores the reachability distance of a record, wherein if the record under consideration is a neighboring record with respect to a predetermined threshold distance ε, the reachability distance is minimized overall. 3) A third attribute that stores the permutation result, i.e., the number between 1 and the number of records in the set of records.

[0063] Compared to standard cluster orderings such as those obtained from OPTICS, the values ​​of the second and third attributes differ in the index according to the present invention. This is due to the fact that neighboring objects are rearranged as their reachability distance further decreases along the processing of the set of records.

[0064] Figure 2a illustrates the concept of core distance C.

[0065] Figure 2a shows the neighborhood of a core record (located at the center) for a given threshold distance ε, where the given number of records, MinPts, is equal to 5. The neighborhood of a core record has 6 neighboring records, some of which may be non-core records, and other neighboring records may be further core records. Note that each record is counted as a neighboring record of itself. The core distance C to the core record under consideration is visualized as a circle with a solid radius. In this example, the core distance C is smaller than the given threshold distance ε.

[0066] Figure 2b illustrates the concept of reachability distance R.

[0067] Figure 2b shows the same neighborhood as in Figure 2a. For neighboring records located further away from the core record under consideration than the core distance C of the core record, the reachability distance R is equal to the distance between each non-core record and the core record, as shown by the dashed line. The distance can be calculated using a given distance function. For neighboring records located closer to the core record than the core distance C of the core record, the reachability distance R is equal to the core distance C.

[0068] In the following, one embodiment of the method according to the present invention is presented by a specific example drawn from the domain of process mining.

[0069] The field of process mining aims to analyze data describing processes, such as technical processes (e.g., soldering processes or manufacturing processes) or business processes (e.g., order processes). Processes are executed in and / or with the help of a source computer system. The execution of a process is called a case or process instance. A process instance contains a series of process steps or events executed during the process instance's record. Therefore, a process instance is described as a sequence of timestamped events and is technically represented by a process protocol or event log.

[0070] An exemplary process instance may be given by the sequence: "Order", "Package", "Issue Receipt", "Payment", and "Collection". In practice, a process instance can be modeled by forming a 2-gram, for example, {("Order", "Package"), ("Package", "Issue Receipt"), ("Issue Receipt", "Payment"), ("Payment", "Collection"). A 2-gram can be mapped to a set containing several tokens, and duplicate tokens are removed so that a process instance can be represented by an integer-based set, for example, {1, 2, 3, 4}. This modeling approach has the advantage of preserving order information between individual events. Furthermore, process instances can be efficiently stored in a database as their set of tokens.

[0071] However, for simplicity, the following example assumes an even simpler data model. Each process step is directly mapped to a non-repeating integer token. For example, the process step "Order" is assigned to token "1", the process step "Packaging" is assigned to token "2", and so on. Thus, the exemplary process can be represented by the set {1, 2, 3, 4, 5}. The resulting assignment of process steps to tokens is summarized in Table 1, with additional process steps added for the example.

[0072] TIFF0007843375000001.tif47142

[0073] Density-based clustering assigns each record in a set of records to one of several detected clusters or one of a noise group. Furthermore, records assigned to detected clusters are subdivided into core objects and boundary objects. Core objects (core records) have at least a predetermined number of neighboring records (MinPts) and hold the cluster together. Boundary objects (boundary records) have fewer than a predetermined number of neighboring records (MinPts) but are located near several core objects.

[0074] On the other hand, the DBSCAN algorithm can be used to compute the exact clustering of a set of records with respect to one particular pair of input parameters ε, and the index, MinPts computed by one embodiment of the method according to the present invention can be calculated for a given number of records MinPts with respect to multiple input parameters ε without the need to recalculate DBSCAN from scratch. * It also supports accurate cluster extraction for ≤ε.

[0075] The creation of the index and the extraction (querying) of clusters found from the index are demonstrated on the set of records presented in Table 2.

[0076] TIFF0007843375000002.tif64142

[0077] In this example, the given distance function is chosen to be the Jaccard distance function. For the Jaccard distance, distance values ​​close to 0 reflect two similar records, and distance values ​​close to 1 reflect two dissimilar records. As an example, the Jaccard distance between records r7 and r11 is equal to 1 / 6. Therefore, with respect to density-based clustering, records r7 and r11 are considered neighbors for any threshold distance ε of at least 1 / 6.

[0078] First, the precise clustering of the set of records for a specific parameter pair is calculated. These calculations serve as a benchmark for the results obtained from the index according to the present invention and from standard cluster ordering generated by OPTICS. Below, a predetermined number of records MinPts is fixed at 4, and two predetermined threshold distances ε * That is, 1 / 3 and 1 / 4 are used. While accurate clustering can be calculated using DBSCAN, for reproducibility, pairs of neighbors and their corresponding threshold distances are provided for all relevant core objects in the set of records in Table 3. The calculated accurate clusterings are listed in Tables 4 and 5 for each given threshold distance.

[0079] TIFF0007843375000003.tif36170

[0080] TIFF0007843375000004.tif32170

[0081] TIFF0007843375000005.tif32170

[0082] For accurate clustering from Table 4, the threshold distance ε *This is equal to 1 / 3. Cluster 1 consists of records r1 to r6. One common characteristic of these records is their short length, which is in the range of 2 to 4 tokens. Furthermore, all of these records contain the token "1", are generally characterized by a low number of tokens, and have a maximum number of tokens of "4". From Table 1, we interpret that the low number of tokens represents a process step that is typically performed at the start of a process instance (e.g., "order" or "payment"). Therefore, we can conclude that cluster 1 in this example contains process instances from a collection of incomplete and not yet continued records.

[0083] On the other hand, cluster 2 consists of records r7 to r12. These records are relatively large in size, with set lengths ranging from 5 to 8 tokens. Furthermore, these records feature events that do not occur in any of the records from cluster 1. For example, the process step "withdrawal" occurs in all records from cluster 2, but not in any records from cluster 1. Note that the process step "withdrawal" frequently occurs concurrently with process steps such as "recovery," "allocation," or "refund." Therefore, cluster 2 can be considered a group of process instances associated with withdrawal, and more generally, processes that progress further than the process instances allocated to cluster 1.

[0084] Reducing the threshold distance prevents some of the records from remaining in their corresponding neighborhoods. As shown in Table 5, the threshold distance ε * If equal to 1 / 4, these records are removed from their cluster and assigned to the noise group. In this example, records r1 and r2 are ε * A record becomes noisy when the threshold distance is equal to 1 / 4. In contrast, records that remain within those clusters for lower threshold distances are closer to each other, thereby forming denser regions in the data space.

[0085] Up to this point, we have the precise cluster semantics shown in Tables 4 and 5. Below, clustering is calculated using both the OPTICS algorithm and the index according to one embodiment of the present invention.

[0086] In both cases, the generated distance, which is a predetermined threshold distance, is set to the maximum distance of interest, i.e., ε is equal to 1 / 3. According to one embodiment of the present invention, the OPTICS algorithm yields a standard cluster ordering, and index construction yields an index. The standard cluster ordering and index are different extended permutations of a set of records, where a permutation refers to a measure of the core distance C and reachability distance R obtained for each record in the set of records. The resulting permutations can be visualized in a reachability plot, where the permuted records are plotted on the x-axis according to their resulting permutation order with respect to their reachability distance on the y-axis.

[0087] Figure 3 visualizes the reachability distance of the cluster ordering obtained from OPTICS(a) and index(b) calculated by one embodiment of the method according to the present invention.

[0088] A reachability plot is a bar plot with one bar for each record in the dataset. The height of each bar corresponds to the reachability distance assigned during the construction of the index or cluster ordering. The records are plotted from left to right along the x-axis in the order of processing (extraction). Thus, the clustering structure of the dataset appears as a continuous series of valleys and peaks in reachability distance. Valleys correspond to dense areas, and peaks to sparse records. The height of a record's bar indicates the minimum density reachable distance from the record to its left (smaller x-values), so records with large bars separate their cluster from other clusters and / or noise records.

[0089] As shown in the reachability plot of FIG. 3, the index constructed for 1 / 3 of the predetermined threshold distance ε results in the exact clustering of Table 4. In contrast, OPTICS cannot obtain an exact clustering because there are several records, namely records r1, r2, r7, r8, and r10, that are erroneously assigned to the noise group.

[0090] This simple example has already demonstrated that the clustering result obtained from the index according to the present invention for generating the threshold distance ε is an exact clustering as obtained from DBSCAN, and this can also be formally proven. However, the result obtained from OPTICS is not an exact clustering result.

[0091] Note that in this example, the detected clusters are extracted using a standard query algorithm for both techniques. The standard query algorithm is basically a linear scan by permutation and thus has a linear time complexity.

[0092] FIG. 4 visualizes the reachability distances obtained from the OPTICS algorithm (a) and the reachability distances obtained from the index according to an embodiment of the present invention (b) for 1 / 4 of the threshold distance ε * As shown in FIG. 4, the clustering results deviate from the exact clustering of Table 5 for both methods. OPTICS manages to get somewhat closer to cluster 1 with one record (r3) missing. However, OPTICS only finds 2 out of 6 records from cluster 2. On the other hand, based on the index according to an embodiment of the present invention, the complete cluster 2 can be extracted. For cluster 1, both methods detect the same number of records.

[0093]

[0094] Threshold distance ε *When the distance is smaller than a predetermined threshold distance ε, both the OPTICS and the method according to one embodiment of the present invention yield an accurate clustering approximation. However, the approximation reported from the index is better in the sense that it loses fewer neighbor records for noise groups.

[0095] In fact, the set of records that can be misassigned to a noise group by an index according to one embodiment of the present invention can be controlled and therefore strictly limited. The inventors have found that among core records with respect to a given threshold distance, only core records that become neighbor records with respect to a smaller threshold distance can be misassigned. This limitation is due to the smaller threshold distance ε * This enables targeted searching for missing neighboring records. By using this targeted searching, accurate clustering from an index can be reported according to one embodiment of the present invention, for any threshold distance ε smaller than a given threshold distance ε * The standard query is improved in that it can also report on this.

[0096] Figure 5 summarizes an index-query framework according to one aspect of the present invention in comparison to the prior art.

[0097] The index-query framework includes a first phase 10 in which the cluster order is calculated, and a second phase 20 in which the cluster order is queried to extract / report clusters. The index-query framework has the general advantage that the potentially time-consuming calculation of cluster ordering only needs to be performed once for the maximum threshold distance ε and a predetermined number of records MinPts. Subsequently, in the second phase 20, various threshold distances ε less than or equal to a predetermined threshold distance ε are queried. * Clusters can be extracted from the clustering results. Cluster extraction is usually fast, and various parameter settings can be explored interactively.

[0098] As outlined above, the OPTICS algorithm provides a state-of-the-art algorithm for index-query frameworks. In the first phase 10, a standard cluster ordering is computed, which can then be queried using the standard query in the second phase 20. The standard query is essentially a linear scan of the permutations obtained from the first phase 10. However, all clusters extracted from the OPTICS cluster ordering are approximate clusters.

[0099] Instead, the method according to the present invention creates an index in the first phase 10, and the index supports the extraction of accurate clusterings using a standard query having a predetermined threshold distance ε. * For ≤ε, the standard query also searches for approximate clusters from the index, where approximate clusters can, of course, simultaneously correspond to exact clusters.

[0100] Preferably, the information contained in the index constructed by the method according to the present invention is a threshold distance ε smaller than a predetermined threshold distance ε. * This is utilized in that a targeted search is performed against previous core objects during cluster extraction. Here, a set of candidate previous core objects is validated against the extracted cluster records labeled as core records using range queries. This targeted search is only possible with the additional information contained in the index compared to standard cluster ordering. As a result, once the index is established, according to one aspect of the present invention, any threshold distance ε less than or equal to a predetermined threshold distance ε * Regarding this, accurate clustering can be obtained interactively.

[0101] Generally, accurate clustering includes more cluster objects, i.e., records assigned to clusters, than approximate clustering, such as that calculated by OPTICS. To quantify the difference between accurate and approximate clustering, the relative ratio of the number of (deduplicated) cluster objects obtained from the index according to the present invention to the number of (deduplicated) cluster objects obtained from OPTICS can be calculated. In experiments on large process mining datasets, the inventors found that the median of this relative ratio obtained for various generative input parameter pairs was at least 1%. Missing cluster objects in clustering obtained from OPTICS are incorrectly assigned to noise groups. Therefore, for large datasets of several million records, an incorrect assignment of cluster objects to noise groups of about 1% can effectively result in a large absolute number of lost cluster objects.

[0102] Figure 6 compares the cumulative runtime of accurate clustering obtained from DBSCAN and the index-query framework according to one aspect of the present invention.

[0103] The performance of accurate clustering calculations is evaluated on a large dataset from the process mining domain, CELONIS-1. Accurate clustering involves 10 threshold distances ε placed at equal intervals between 0.07 and 0.25. * The calculation is performed for a predetermined number of records. MinPts are set to 16, 128, and 1024.

[0104] In the DBSCAN algorithm, DBSCAN uses a selected threshold distance ε * Since accurate clustering is calculated from all pairs of points MinPts, only parameter pairs containing the predetermined number of points MinPts16 are evaluated. For 10 pairs in this example, the total cumulative runtime for DBSCAN was longer than one day.

[0105] Using an index-query framework according to one aspect of the present invention, the index needs to be computed only once, which took approximately 9 hours for a given number of records (MinPts16). The subsequent extraction of 10 accurate clusters from the pre-computed index in a second phase 20 took only 1 hour.

[0106] In the first phase 10, the cumulative runtime for calculating the index remains nearly constant for a given number of records with a further evaluated MinPts value, but the index query 20 is more efficient for a given number of records with a higher MinPts value. For a given number of records with a lower MinPts value, the number of core records in the dataset is large. Therefore, a large number of candidate validations must be performed during the targeted search, which explains why the cumulative runtime of the index query 20 increases as the MinPts value for a given number of records decreases.

[0107] As a result, the cumulative runtime of this experiment shows that the index-query framework according to one aspect of the present invention is up to three times faster than the DBSCAN algorithm on the CELONIS-1 dataset. The OPTICS algorithm is not included in this comparison because it cannot provide accurate clustering.

[0108] In one embodiment, neighborhood calculation can be separated from the actual clustering algorithm. That is, neighborhoods are pre-calculated and materialized in advance. Materialization cost O(n 2) memory, where n represents the collection size. However, the advantage of this approach is the improved runtime resulting from the fact that neighborhoods can be computed asymmetrically. For example, a set similarity join can be used, and accordingly the filter validation framework avoids part of the distance calculation. In particular, the AllPairs or MetricJoin algorithm can be used to compute set similarity joins. In step b) of the method according to the present invention, pre-computed neighborhoods can be simply looked up from memory (instead of being computed), which reduces the runtime for index construction.

[0109] In one embodiment, the neighbor calculation and candidate validation during the index query for extracting accurate clusters can both be performed in parallel. Therefore, the runtime shown in Figure 6 can be further improved.

[0110] In practice, deduplication of records in a set of records is important. In one embodiment, duplicate data is utilized by deduplication of the set while loading the set of records into memory, and a duplicate count is stored for each deduplication set. These duplicate counts are ultimately used to determine the neighborhood size. This procedure ensures that accurate clustering is obtained while only a portion of the data is actually processed in memory. In the case of neighborhood embodiment, deduplication is a necessary prerequisite to prevent neighborhoods from becoming too large and exceeding the memory available on the computer.

[0111] Figure 7 shows density filtering according to one aspect of the present invention.

[0112] In one aspect of the present invention, a density filter 30 is introduced, which allows the user to increase the clustering density by increasing a predetermined number of records MinPts. At a threshold distance ε, ε *It should be noted that this is considered to be constant during density filtering 30. Combined with the index-query framework introduced above, density filtering 30 is performed on any pair (ε) of records with a selected threshold distance and a selected number of records. * MinPts * We extend the parameter space for an essentially interactive search for accurate clustering of ), where ε * <=ε and MinPts * >= MinPts. The basic idea of ​​density filtering is to take advantage of the fact that high-density clusters are a subset of spare clusters.

[0113] Therefore, once clustering is calculated for a predetermined number of records MinPts, that clustering is used to calculate the clustering for a predetermined number of records MinPts. * The set of records by noise groups can be reduced for any subsequent calculations using ≥MinPts. Needless to say, the initial clustering should be accurate to avoid filtering out misassigned noise records. Thus, density filtering 30 is ideally applied to the clustering resulting from the index calculated by the method according to the present invention.

[0114] According to one aspect of the present invention, the index is calculated for a selected number of records MinPts that is greater than a predetermined number of records MinPts. * There are three approaches to calculating accurate clustering for a selected threshold distance ε. * If the selected threshold distance ε is equal to a predetermined threshold distance ε, then the first approach 1 is recommended. * If the selected threshold distance ε is smaller than a predetermined threshold distance ε, the second approach 2 and the third approach 3 can be applied. The second approach 2 and the third approach 3 are applied to the selected threshold distance ε * This can also be applied when the distance is equal to a predetermined threshold distance ε.

[0115] The first approach 1 and the third approach 3 are based on the calculation of an extended index provided according to one aspect of the present invention. The extended index includes a fourth attribute in which the size and neighborhood count of each neighborhood are stored. Furthermore, for each (exact) cluster with respect to a predetermined threshold distance ε and a predetermined number of records MinPts, a set of all boundary objects (boundary record set), i.e., noncore records, of the respective cluster is recorded.

[0116] In the first approach 1, the following steps are performed to obtain accurate clustering 2A for a given threshold distance and a selected number of records MinPts that is greater than (or equal to) a given number of records MinPts. * The following is obtained. The exact clusters 1A-1,...1A-n are obtained from the extended index query, where n is the number of clusters found. Noisy records are automatically discarded by extracting clusters from the extended index. Merge 31: Merge each extracted cluster 1A-1,...1A-n with its respective boundary record set so that all ambiguous boundary objects are also included in the n subsets obtained as a result of the set of records. Calculate 32: The number of MinPts of a selected number of records are calculated by calling the DBSCAN algorithm separately, especially in parallel, on each subset. * The process involves calculating accurate clustering 2A for the object, using the neighbor count of the extended index to directly obtain the set of core objects and thus reduce the number of range queries required. Furthermore, the resulting clustering is post-processed to assign ambiguous boundary objects to the correct discovered clusters, respectively. By merging the clusterings obtained from separate DBSCAN runs, ambiguous boundary objects are assigned to multiple discovered clusters. The post-processing step achieves reducing multiple assignments to a single assignment, i.e., each to exactly one cluster. Note that the post-processing step has linear complexity.

[0117] In the second approach, the following steps are performed to select a threshold distance ε that is smaller than (or equal to) a given threshold distance ε (used to generate the index). * and the number of selected records MinPts that is greater than (or equal to) the predetermined number of records MinPts. * Accurate clustering 2B is obtained for this. By querying the index, the selected threshold distance ε * The objective is to obtain the accurate clustering 1B, which can also be obtained by querying the extended index. Discard 31: Discard noise groups from the accurate clustering 1B to reduce the set of records. Call (DBSCAN algorithm) 32: Number of selected records MinPts * To compute accurate clustering 2B for this, call the DBSCAN algorithm on the reduced set of records.

[0118] In the third approach, the following steps are performed to select a threshold distance ε that is smaller than (or equal to) a given threshold distance ε (used to generate the index). * , and a selected number of records MinPts that is greater than (or equal to) a predetermined number of records MinPts. * Accurate clustering 2B is obtained for the given threshold distance ε. The extended index is queried using a predetermined threshold distance ε to obtain the accurate clusters 1A-1,...1A-n for the given threshold distance ε, where n is the number of clusters detected. Noisy records are automatically discarded by extracting clusters from the extended index. Merge 31: Merge each extracted cluster 1A-1,...1A-n with its respective boundary record set so that all ambiguous boundary objects are also included in the n subsets obtained as a result of the set of records. Calculation 32: Selected threshold distance ε * , Number of selected records MinPts *The accurate clustering 2C for is calculated by calling the DBSCAN algorithm separately, and especially in parallel, for each subset, and the resulting clustering is post-processed to assign ambiguous boundary objects to the correct detected clusters, respectively.

[0119] MinPts for the number of records selected * For accurate clustering calculations for ≥MinPts, use density filtering by one of three approaches 1; 2; 3. The dataset (set of records) is reduced by the number of noisy records, and range queries (neighborhood evaluations) performed on this reduced dataset are accelerated. This is particularly useful because range queries are the main cause of runtime in density-based clustering. As a result, an (extended) index by one embodiment of the present invention can be used for any number of records with higher density MinPts * It can be used directly as filter 30 for this purpose.

[0120] In either case, storing the size of each neighborhood during (extended) index construction according to one aspect of the present invention is based on a predetermined threshold distance ε and the number of selected records MinPts. * This has the added advantage of allowing for the rapid determination of core records related to the topic, and thus reducing the number of range queries required.

[0121] As a result, the extended index calculated according to one aspect of the present invention is ε * With respect to ≤ε and fixed MinPts, or MinPts *For ≥MinPts and a fixed ε, accurate clustering can be provided. That is, the user can interactively increase the density of accurate clustering via one of two input parameters, while the other of the two input parameters remains fixed. In the second phase 20, an index query is used to vary the threshold distance ε. Density filtering 30 is used to vary the number of records MinPts. In combination, any pair of threshold distance and number of records (ε) can be used. * ≤ε, MinPts * It is possible to calculate and interactively explore accurate clustering for values ​​(≥MinPts).

Claims

1. A computer implementation method for calculating an index for a first density-based clustering of a set of records, wherein the index is stored using a memory device. Here, each record is, - Initially marked as unprocessed, - Having a first attribute and a second attribute, - Forms a portion of the neighborhood within a predetermined threshold distance, A record is a core record if its neighborhood contains at least a predetermined number of records, and a non-core record if its neighborhood contains fewer than a predetermined number of records. The index is calculated according to the following steps: a) Selecting unprocessed records from a set of records and calculating the core distance of the selected unprocessed records, wherein the core distance is the minimum distance from the selected unprocessed record such that its neighborhood still contains at least a predetermined number of records. b) If the selected unprocessed record is a core record, assign the core distance to the first attribute and label the selected unprocessed record as a core record; otherwise, assign a predetermined value to the first attribute and label the selected unprocessed record as a non-core record. c) Add the selected unprocessed records to the index and mark the selected unprocessed records as processed, d) If the selected processed record is a core record, processing each record in the vicinity of the selected processed record, The reachability distance is calculated for each record, where the reachability distance is the minimum distance that each record is still directly and densely reachable from the selected processed record. The process involves placing each record marked as pending in ascending order of reachability distance into a priority queue, wherein the reachability distance is assigned to the second attribute, The process involves removing from the index each record that was previously labeled as a noncore record and outputs a reachability distance smaller than previously assigned to its second attribute, each removed record being marked as unprocessed, and each removed record being placed in a priority queue in ascending order of its reachability distance. e) Adding each record of the priority queue to the index, wherein each record is marked as processed, and if the added record is a core record, the neighboring records are processed according to step d), Steps a) through e) are repeated until all records in the set of records have been processed and inserted into the index.

2. The method according to claim 1, further comprising determining whether the selected unprocessed record is a core record by counting the records in the vicinity of the selected unprocessed record between step a) and step b).

3. The method according to claim 1 or 2, wherein, during the processing of each record in the prioritized queue in step d), the core distance is calculated and assigned to the first attribute only if the first attribute has not been assigned.

4. The method according to claim 1, wherein each record of the index includes a third attribute that stores the permutation order of the records.

5. The method according to claim 1, comprising extracting a first accurate clustering of the set of records for a predetermined threshold distance based on a combined evaluation of the reachability distance and the core distance of the records in the index, according to a linear scan through the index, wherein each record is assigned either a cluster identifier or a noise identifier.

6. The first accurate clustering from the set of records is extracted for a selected threshold distance, the selected threshold distance being less than or equal to the predetermined threshold distance, and the method further includes a candidate validation step, for the extracted clusters of the first accurate clustering, each record is, - Located in the index immediately preceding the extracted cluster, - The first attribute is assigned to the calculated core distance. - A noise identifier is assigned, The method according to claim 5, wherein the records of the extracted cluster are labeled as core records and validated for which the assigned calculated core distance is less than or equal to the selected threshold distance.

7. The method according to claim 6, wherein the candidate verification step is performed for each extracted cluster of the first accurate clustering after the extraction of each extracted cluster.

8. The method according to claim 5, 6, or 7, wherein a second accurate clustering of the set of records is calculated based on a predetermined threshold distance and a number of selected records, the number of selected records being greater than a predetermined number of records, and the set of records is reduced to the records to which the cluster identifier is assigned according to the first accurate clustering.

9. The method according to claim 8, wherein the second accurate clustering is calculated based on the selected threshold distance, and the set of records is reduced to the records to which the cluster identifier is assigned according to the first accurate clustering based on the predetermined threshold distance.

10. The method according to claim 9, wherein the set of records is separated into at least one subset, each subset of the at least one subset corresponds to an extracted cluster of the first accurate clustering with respect to a predetermined threshold distance, and the second accurate clustering is calculated separately for each subset.

11. The method of claim 10, wherein in step d), the cluster identifier is incremented, and each neighboring record of the selected processed record is inserted into the boundary record set of the respective cluster identifier if the respective record is labeled as a noncore record, and each boundary record set is merged with the subset corresponding to the respective cluster identifier before the second exact clustering is calculated for each subset.

12. The method according to claim 1, wherein each record includes a fourth attribute, and for each record, the number of neighboring records is assigned to the fourth attribute during steps b) and d).

13. The method according to claim 12, wherein, for each core record only, the number of records contained in its neighborhood is assigned to the fourth attribute during steps b) and d).

14. The method according to claim 8, wherein the second accurate clustering is calculated using density-based spatial clustering with a noise-DBSCAN algorithm.

15. The method according to claim 1, wherein each record in the set of records represents a process instance of a process, and the process was executed in or with the help of the source computer system.

Images