Distributed tracking data analysis method, device and system based on interval event pattern mining algorithm and computer storage medium
By introducing target constraints and pruning strategies into distributed tracking data analysis, and constructing a linked list structure for pattern mining, the problem of ignoring time interval information in traditional methods is solved, achieving efficient pattern mining and system optimization.
Patent Information
- Application Number
- CN202510775720.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-11
- Publication Date
- 2025-11-07
AI Technical Summary
In existing distributed tracking data analysis methods, traditional pattern mining methods ignore the time intervals and durations between events, resulting in the discovery of a large number of irrelevant patterns, which affects user analysis and system optimization.
The concept of target constraint is introduced to preprocess distributed tracking data, construct position linked lists, target linked lists, and suffix linked lists, and target pruning and suffix pruning strategies are used to prune candidate patterns. The interval event pattern mining algorithm based on target constraint is then used for pattern mining.
It effectively reduces the search space of the algorithm, improves the efficiency of pattern mining, uncovers patterns that users are interested in, and helps analyze the performance and failure issues of distributed systems.
Smart Images
Figure CN120910122A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the fields of data mining technology and distributed tracing, and particularly relates to a distributed tracing data analysis method, device, system and computer storage medium based on an interval event pattern mining algorithm. BACKGROUND
[0002] In the wave of digital era, data analysis has become the core task in various fields. Especially under the promotion of Internet, Internet of Things, cloud computing and big data technology, the scale and complexity of data show exponential growth. A large number of event sequence data generated globally every day covers various information such as user behavior, device status and system log. These event sequence data are not only the basis for enterprise decision-making, but also the important basis for optimizing system performance and improving user experience. Event sequence data usually exist in the form of time series, recording the occurrence time, type and related attributes of events. The analysis and mining of these data can not only reveal potential laws, but also help organizations identify system bottlenecks, predict failures and optimize resource allocation. Therefore, how to efficiently mine and utilize these event sequence data has become an important problem to be solved in the field of data science and engineering. Traditional pattern mining methods mainly focus on the occurrence order of events, often ignoring important information such as time interval and duration between events.
[0003] With the development of technology, researchers gradually realize that it is not enough to rely on the order information of events only, and the time interval and duration play a crucial role in event sequences, which can reflect the mutual relationship between events. Therefore, interval event pattern mining emerges as the times require, becoming a new research direction, aiming to extract valuable patterns from rich event sequence data. This pattern not only focuses on the occurrence order of events, but also considers multi-dimensional information such as time interval and duration between events, which can more comprehensively reflect the relationship between events and help to discover potential laws and abnormal phenomena. Especially in distributed systems, the complexity and diversity of event sequence data make the demand for interval event pattern mining more urgent. In the field of distributed tracing, with the popularity of microservice architecture and the wide application of cloud computing, the complexity and dynamics of systems increase significantly. Distributed tracing technology can help developers and operation and maintenance personnel to monitor each component of the system in real time, identify performance bottlenecks and failures. However, with the explosive growth of events, how to effectively analyze and mine these event sequence data becomes the key to improving system performance and optimizing user experience. Distributed tracing data is essentially interval events, because each service call has a start time and an end time. By applying interval event pattern mining algorithm, the running state of distributed system can be understood more deeply, thereby providing data support for system optimization.
[0004] In the field of distributed tracking, the usual analysis method is to input a condition, and then the algorithm obtains all the distributed tracking data that meets the condition. However, in the field of interval event pattern mining algorithm, there is no relevant research at present. This leads to the algorithm mining a large number of irrelevant patterns, which is not conducive to the analysis and use of users. Therefore, it is of great significance to study the interval event pattern mining algorithm and its application in the field of distributed tracking.
[0005] The present application provides an interval event pattern mining algorithm-based distributed tracking data analysis method to solve the problems in the prior art, comprising the following steps:
[0006] Step S1: improving the interval event pattern mining algorithm and introducing the concept of target constraint;
[0007] Step S2: preprocessing the distributed tracking data;
[0008] Step S3: based on the preprocessed data, constructing a position chain table, a target chain table and a suffix chain table of a specified length;
[0009] Step S4: using a target pruning strategy and a suffix pruning strategy to prune the candidate patterns in the pattern mining process;
[0010] Step S5: interval event pattern mining based on target constraint.
[0011] As a further improvement of the present application, in step S1, the user specifies a target pattern as a constraint, and the patterns mined by the algorithm must contain the target pattern. As a further improvement of the present application, in step S1, the definition of the interval event pattern mining algorithm based on target constraint is as follows:
[0012] Given a sequence interval event database, a minimum support threshold and a target constraint, the interval event pattern mining based on target constraint is to find all the complete sets of patterns that meet the support greater than or equal to the minimum support threshold and are super patterns of the target constraint from the database.
[0013] As a further improvement of the present application, in step S2, the distributed tracking data needs to be converted into an interval event sequence database, which contains the features of the start time of the event, the end time of the event and the symbol of the event. After preprocessing, the data needs to be arranged in ascending order according to the start time, end time and symbol. As a further improvement of the present application, in step S3,
[0014] The definition of the position chain table is as follows:
[0015] Given database D and pattern P, the position list records all the position information of pattern P in database D, the position list contains the specific information of pattern P and a hash table, the hash table is used to record the sid of all the sequences containing pattern P in database and the position of all the occurrences of pattern P in the sequences;
[0016] The definition of target list is as follows:
[0017] Given database D, target constraint q and pattern P, the target list records all the position information of the target constraint matching pattern P in database, the position list contains the specific information of pattern P, prefix and hash table, the prefix records the position of the next symbol of pattern P matching target constraint q, the hash table records the sid of all the sequences containing pattern P in database and the position of all the occurrences of the target constraint matching pattern P in the sequences;
[0018] The definition of suffix list is as follows:
[0019] Given database D and target constraint q, the suffix list is a three-layer list, the first layer records all the suffixes of target constraint q, the second layer records all the sids, and the third layer records the position of the last occurrence of the suffix in the sequence. As a further improvement of the present application, in the step S4, the target pruning strategy is defined as follows:
[0020] Given target constraint q, pattern P and sequence S, the pattern P is extended on sequence S, assuming that the symbol of the event being extended is a, for each position pi of symbol a in sequence S, the extension is performed in the order, if the pattern P does not appear in the sequence after position pi for the suffix of the target constraint, the scanning of sequence S is stopped;
[0021] The suffix pruning strategy is defined as follows:
[0022] Given database D and target constraint q, for each sequence in database, the index of the position of the last occurrence of target constraint q in the sequence is recorded as LP, in the process of extending the pattern with length of 1 to the pattern with length of 2, assuming that the symbol of the event being extended is e, the first position of e in the sequence is FP, if FP> LP, it means that the pattern starting with e cannot satisfy the target constraint q. As a further improvement of the present application, in the step S5, the specific process of pattern mining is as follows:
[0023] In step S50, the position list with length of 1, target list and suffix list are used to generate candidate patterns, specifically, the pattern with length of 1 is spliced to obtain the pattern with length of 2, and then the extension is recursively performed;
[0024] Step S51, pruning the candidate mode by using suffix pruning strategy and target pruning strategy;
[0025] Step S52, calculating the support degree of the candidate mode, if the support degree of the candidate mode is greater than the threshold, outputting the candidate mode and ending, if the support degree is less than the threshold, continuing to execute step S52, making subsequent judgment, and then recursively performing mode mining until the candidate mode with the support degree greater than the threshold is mined.
[0026] The data preprocessing module is used for converting the distributed tracking data into an interval event sequence database, and constructing a position chain table, a target chain table and a suffix chain table with a length of 1.
[0027] The candidate mode generation module is used for generating a candidate mode by using the position chain table and the target chain table of the mode.
[0028] The mode mining module is used for pruning the candidate mode according to the suffix pruning strategy and the target pruning strategy, and outputting a frequent mode by calculating the support degree of the candidate mode.
[0029] The application further discloses a distributed tracking data analysis system based on an interval event mode mining algorithm, which comprises a memory, a processor and a computer program stored in the memory.
[0030] The application further discloses a computer readable storage medium, which stores a computer program configured to realize the steps of the method of the application when called by a processor.
[0031] The application has the following beneficial effects: 1. The interval event mode mining algorithm is combined with the distributed tracking data, and the concept of target constraint is introduced into the interval event mode mining algorithm to design an interval event mode mining algorithm based on target constraint; 2. The interval event mode mining algorithm based on target constraint can mine the mode interested by users, and then the developers can analyze the performance or fault problem of the distributed system by using the mode. BRIEF DESCRIPTION OF DRAWINGS
[0032] Figure 1 is a distributed tracking data analysis flowchart based on the interval event mode mining algorithm of the application;
[0033] Figure 2 is a target chain table data structure diagram in the embodiment of the application;
[0034] Figure 3 is a suffix chain table data structure diagram in the embodiment of the application;
[0035] Figure 4 Figure is a device structure diagram of the target constraint-based interval event pattern mining in the embodiment of the application;
[0036] Figure 5 Figure is a structure diagram of the computer device of the embodiment of the application;
[0037] Figure 6 Figure is a running time comparison diagram of the TPMiner algorithm and Z-Miner and TIRPClo in the embodiment of the application;
[0038] Figure 7 Figure is a memory consumption comparison diagram of the TPMiner algorithm and Z-Miner and TIRPClo in the embodiment of the application. DETAILED DESCRIPTION
[0039] The application discloses a distributed tracking data analysis method, system and device based on an interval event pattern mining algorithm and a computer storage medium, and mainly applies to the following technical fields:
[0040] 1. Cloud computing and micro-service architecture: in a cloud environment, a micro-service architecture enables an application to be composed of multiple services, and distributed tracking can help developers monitor the call relationship between services, identify performance bottlenecks and fault points.
[0041] 2. Big data processing: in a big data platform (such as Hadoop, Spark, etc.), distributed tracking can be used to monitor the execution of data processing tasks, optimize resource use and task scheduling.
[0042] 3. Network performance monitoring: in network applications, distributed tracking can help analyze the delay of requests between different network nodes, identify network bottlenecks and optimize data transmission.
[0043] The application discloses a distributed tracking data analysis method based on an interval event pattern mining algorithm, which preprocesses distributed tracking data to remove irrelevant information, constructs a position linked list with a length of 1, a target linked list and a suffix linked list for subsequent pattern expansion, and adopts a target pruning strategy and a suffix pruning strategy to prune candidate patterns in advance, which can effectively reduce the search space of the algorithm and improve the performance of the algorithm.
[0044] As shown in Figure 1 The application discloses a distributed tracking data analysis method based on an interval event pattern mining algorithm, which includes the following steps:
[0045] Step S1: improving the interval event pattern mining algorithm and introducing the concept of target constraint;
[0046] The definition of the target-constrained interval event pattern mining algorithm is as follows: given an interval event database, a minimum support threshold and a target constraint. The target-constrained interval event pattern mining is to find all the complete set of patterns that satisfy the support greater than or equal to the minimum support threshold and are super patterns of the target constraint from the database.
[0047] In a specific implementation, users can set the pattern they are interested in as a target constraint, and then the algorithm will mine all the frequent patterns containing the target constraint.
[0048] Step S2: Preprocessing of distributed trace data;
[0049] In a specific embodiment, preprocessing includes removing incomplete data, ensuring that each record contains complete TraceID, operation name, start time and end time, and uniformly converting timestamps into a computable format; since the operation name can be long and complex, it needs to be mapped to a simple event identifier, for example: different service names are mapped to event ID. Similarly, the TraceID is mapped to the sequence ID for subsequent processing; for each operation in each piece of distributed trace data, a triple is obtained, including the operation name, start time and end time; according to the sequence ID, all triples belonging to the same piece of distributed trace data are aggregated together, and all triples belonging to the same sequence ID are sorted in ascending order according to the start time, end time and event ID. The database after preprocessing is shown in Table 1:
[0050] Table 1 Interval event database
[0051]
[0052] Step S3: Constructing the position list, target list and suffix list with a length of 1 from the preprocessed data; in step 3, the algorithm scans the preprocessed data to construct the three data structures for pattern expansion.
[0053] In a specific implementation, the definition of the position list is as follows: given a database D and a pattern P. The position list records all the position information of the pattern P in the database D. The position list contains two items, the first item is the specific information of the pattern P, and the second item is a hash table, which records the sid of all sequences containing the pattern P in the database and the position of all occurrences of the pattern P in the sequence; the definition of the target list is as follows: given a database D, a target constraint q, and a pattern P. The target list records all the position information of the target constraint matching the pattern in the database. The position list contains three items, the first item is the specific information of the pattern P; the second item is prefix, which records the position of the next symbol matching the target constraint q; the third item is a hash table, which records the sid of all sequences containing the pattern P in the database and the position of all occurrences of the target constraint matching the pattern in the sequence; the definition of the suffix list is as follows: given a database D and a target constraint q. The suffix list is a three-layer list, the first layer records all the suffixes of the target constraint q, the second layer records all the sids, and the third layer records the last occurrence position of the suffix in the sequence. Figure 2 and Figure 3 are the target list and the suffix list.
[0054] Step S4: using the target pruning strategy and the suffix pruning strategy to prune the candidate patterns in the pattern mining process; the target pruning strategy is used to reduce the search space of the algorithm in the pattern mining process, and the suffix pruning strategy is used to reduce the search space of the algorithm before the pattern mining starts;
[0055] In a specific embodiment, the definitions of the target pruning strategy and the suffix pruning strategy are as follows. Target pruning strategy: given a target constraint q, a pattern P and a sequence S, we want to extend the pattern P on the sequence S, assuming that the symbol of the extended event is α. For each position pi of the symbol α in the sequence S, we perform the extension in order, and if the suffix of the pattern P for the target constraint does not appear in the sequence after the position pi, we stop scanning the sequence S; suffix pruning strategy: given a database D and a target constraint q, for each sequence in the database, we record the index of the last occurrence position of q in the sequence as LP. In the process of extending the pattern of length 1 to the pattern of length 2 by the algorithm, assuming that the symbol of the current extended event is e, and the first position of e in the sequence is FP, if FP> LP, then it is indicated that the pattern starting with e cannot satisfy the target constraint q.
[0056] Step S5: target constraint-based interval event pattern mining.
[0057] In a specific embodiment, the target constraint-based interval event pattern mining is a depth-first search process, which is as follows:
[0058] Step S50: First, the algorithm uses the position linked list of length 1, the target linked list, and the suffix linked list created in step S1 to generate candidate patterns;
[0059] Step S51: Then scan the database to calculate the support of candidate patterns. During the scanning process, suffix pruning and target pruning strategies are used to prune candidate patterns.
[0060] Step S52: Finally, calculate the support of the candidate pattern. If the support of the candidate pattern is greater than the threshold, output the candidate pattern and end. If the support is less than the threshold, return and continue to execute step S52 for subsequent judgment. Then the pattern mining algorithm recursively performs pattern mining until a candidate pattern with a support greater than the threshold is mined, that is, no new candidate patterns are generated.
[0061] The following is for reference Figure 4 To describe the relationship with embodiments of this disclosure Figure 1 The apparatus corresponding to the method shown is an apparatus 100 for mining interval event patterns based on target constraints. Figure 4 This is a schematic diagram of the structure of a distributed tracking data analysis device based on an interval event pattern mining algorithm in an embodiment of the present invention. Figure 4 As shown, the device 100 includes:
[0062] Data preprocessing module 101: used to transform distributed tracking data into an interval event sequence database, and to construct a position linked list, a target linked list and a suffix linked list of length 1;
[0063] Candidate pattern generation module 102: used to generate candidate patterns using the position list and target list of patterns;
[0064] Pattern mining module 103: This module prunes candidate patterns according to suffix pruning and target pruning strategies, and outputs frequent patterns by calculating the support of candidate patterns. In addition to these three modules, the device 100 may also include other components; however, since these components are not relevant to the embodiments of this disclosure, their illustrations and descriptions are omitted here.
[0065] The specific working process of a distributed tracking data analysis device 100 based on an interval event pattern mining algorithm is described in the same way as the distributed tracking data analysis method based on the interval event pattern mining algorithm, and will not be repeated here.
[0066] Furthermore, the apparatus according to embodiments of the present invention can also be used by means of Figure 5 The architecture of the computing device shown is used to implement this. Figure 5 The architecture of the computing device is shown. For example... Figure 5As shown, the computer system 201 includes a system bus 203, one or more CPUs 204, input / output components 202, and a memory 205. The memory 205 can store various data or files used by the computer for processing and / or communication, as well as program instructions executed by the CPU. Figure 5 The architecture shown is merely exemplary and should be adjusted according to actual needs when implementing different devices. Figure 5 One or more components in.
[0067] The present invention also discloses a computer-readable storage medium storing a computer program configured to implement the steps of the method described in the present invention when invoked by a processor.
[0068] This invention provides embodiments of the method, apparatus, and computer storage medium for implementing target-constrained interval event pattern mining in large-scale distributed tracking data. The results of these three embodiments are compared with the current best implementations of Z-Miner and TIRPClo in terms of running speed and memory consumption. The embodiments are performed on four real datasets: ASL_BU, BLOCKS, AUSLAN2, and SKATING. In these four datasets, each clickstream sequence's itemset contains only one item. The running time and memory consumption of the TPMiner algorithm, Z-Miner, and TIRPClo on the four datasets are as follows: Figure 6 and Figure 7 As shown.
[0069] Depend on Figure 6 TPMiner's running speed is significantly faster than Z-Miner and TIRPClo algorithms, especially on the AUSLAN2 dataset, where TPMiner's speed advantage is even more pronounced. Figure 7 TPMiner consumes less memory than Z-Miner and TIRPClo on the ASL_BU, AUSLAN2, and SKATING datasets. However, on the BLOCKS dataset, TPMiner consumes slightly more memory than TIRPClo, but still significantly less than the Z-Miner algorithm.
[0070] The distributed tracking data analysis method, apparatus, system, and computer storage medium based on interval time pattern mining provided in the above embodiments can perform pattern mining on distributed tracking data based on user interests, and the performance of the TPMiner algorithm is higher than that of existing algorithms.
[0071] The above is further detailed description of the present application in combination with specific preferred embodiments, and cannot be deemed as limitation of the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, and all should be deemed as falling within the protection scope of the present application.
Claims
1. A distributed trace data analysis method based on interval event pattern mining algorithm, characterized in that , comprising the following steps: Step S1: improving the interval event pattern mining algorithm, and introducing the concept of target constraint; Step S2: preprocessing the distributed tracking data; Step S3: based on the preprocessed data, constructing a position chain table, a target chain table and a suffix chain table of a specified length; Step S4: pruning the candidate patterns in the pattern mining process by using a target pruning strategy and a suffix pruning strategy; Step S5: interval event pattern mining based on target constraint.
2. The method of claim 1, wherein, In the step S1, the user specifies a target pattern as a constraint, and the pattern mined by the algorithm must contain the target pattern.
3. The method of claim 2, wherein, In the step S1, the definition of the interval event pattern mining algorithm based on target constraint is as follows: Given a sequence interval event database, a minimum support threshold and a target constraint, the interval event pattern mining based on target constraint is to find all the complete sets of patterns that meet the support greater than or equal to the minimum support threshold and are super patterns of the target constraint from the database.
4. The method of claim 1, wherein, In the step S2, the distributed tracking data needs to be converted into an interval event sequence database, and the features include: the start time of the event, the end time of the event, the symbol of the event, and the data needs to be arranged in ascending order according to the start time, the end time and the symbol after preprocessing.
5. The method of claim 1, wherein, In the step S3, The definition of the position chain table is as follows: Given a database D and a pattern P, the position chain table records all the position information of the pattern P in the database D, and the position chain table contains the specific information of the pattern P and a hash table, which is used to record the sid of all sequences containing the pattern P in the database and all the positions of the pattern P in the sequence; The definition of the target chain table is as follows: Given a database D, a target constraint q and a pattern P, the target chain table records all the position information of the target constraint matching the pattern P in the database, and the position chain table contains the specific information of the pattern P, prefix and hash table, prefix records the position of the next symbol matching the target constraint q, and hash table records the sid of all sequences containing the pattern P in the database and all the positions of the target constraint matching the pattern P in the sequence; The definition of the suffix chain table is as follows: Given a database D and a target constraint q, the suffix chain table is a three-layer chain table, the first layer records all the suffixes of the target constraint q, the second layer records all the sids, and the third layer records the last position of the suffix in the sequence.
6. The method of claim 1, wherein, In the step S4, the definition of the target pruning strategy is as follows: Given a target constraint q, a pattern P and a sequence S, extend the pattern P on the sequence S, assume that the symbol of the extended event is α, for each position pi of the symbol α in the sequence S, extend in order, and if the pattern P does not appear in the sequence after the position pi for the suffix of the target constraint, stop scanning the sequence S; The definition of the suffix pruning strategy is as follows: Given a database D and a target constraint q, for each sequence in the database, record the index of the last position of the target constraint q in the sequence as LP. In the process of extending the pattern of length 1 to the pattern of length 2, assume that the symbol of the current extended event is e, and the first position of e in the sequence is FP. If FP>LP, then the pattern starting with e cannot satisfy the target constraint q.
7. The method of claim 1, wherein, In the step S5, the specific process of the pattern mining is as follows: In step S50, the position chain table of length 1, the target chain table and the suffix chain table are used to generate the candidate pattern. Specifically, the length-1 pattern is spliced to obtain the length-2 pattern, and then the extension is recursively performed. In step S51, the suffix pruning strategy and the target pruning strategy are used to prune the candidate pattern. In step S52, the support degree of the candidate pattern is calculated. If the support degree of the candidate pattern is greater than the threshold, the candidate pattern is output and the process is ended. If the support degree is less than the threshold, the subsequent judgment is continued, and then the pattern mining is recursively performed until the candidate pattern with the support degree greater than the threshold is mined.
8. A distributed trace data analysis apparatus based on an interval event pattern mining algorithm, characterized by, The method comprises: A data preprocessing module is configured to convert the distributed tracking data into an interval event sequence database, and construct a position chain table of length 1, a target chain table and a suffix chain table. A candidate pattern generation module is configured to generate a candidate pattern using the position chain table of the pattern, the target chain table and the suffix chain table.
9. A distributed trace data analysis system based on interval event pattern mining algorithm, characterized in that, A pattern mining module is configured to prune the candidate pattern according to the suffix pruning strategy and the target pruning strategy, and output the frequent pattern by calculating the support degree of the candidate pattern. The computer readable storage medium stores a computer program configured to realize the steps of the method of any one of claims 1-7 when called by the processor.
10. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program configured to realize the steps of the method of any one of claims 1-7 when called by the processor.