Tail sampling method for distributed tracking system in RPA workflow

Abstract

The invention discloses a tail sampling method for a distributed tracking system in RPA workflow, which relates to the field of RPA workflow, and comprises the following steps: carrying out unsupervised representation learning on tracking data by utilizing a variational graph automatic encoder model, and storing a trained model; obtaining a feature vector based on a pre-trained variational graph automatic encoder model; based on the reconstruction loss of the variational graph automatic encoder model, detecting and retaining all abnormal tracking; carrying out classification and preference sampling on normal tracking by adopting a clustering method; and combining sampling results of abnormal tracking and normal tracking to form a final sampling data set. According to the method, biased sampling is carried out on normal tracking data through clustering and feature preference sampling strategies, the storage cost of mass tracking data is reduced, meanwhile, the sensitivity of a sampler to abnormal tracking is improved, and the diversity and observability of sampling results are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of RPA workflow, and in particular to a tail sampling method for a distributed tracking system in an RPA workflow. Background Art

[0002] With the deepening of digital transformation of enterprises, Robotic Process Automation (RPA) technology has been widely used. RPA technology can significantly improve the automation and execution efficiency of business processes by simulating human operations such as mouse clicks, keyboard input and data processing to execute business processes. RPA technology has a wide range of application scenarios, covering data entry, document processing, information extraction, customer service, financial report generation and other fields. It can free employees from repetitive and tedious work, allowing them to devote more time and energy to creative and strategic work, and promote the innovation and development of enterprises.

[0003] In complex RPA systems, the distributed tracing system enables operation and maintenance personnel to monitor and analyze requests in the system, and better understand and diagnose performance bottlenecks and fault points in the system. However, as the scale of the RPA system expands, the distributed tracing system will generate a large amount of tracing data, which brings high storage costs. In addition, this data may also contain a large number of duplicate and redundant tracing. How to effectively reduce the storage overhead of tracing data has become an important research topic.

[0004] In production environments, it is common to sample trace data and only retain a small subset of all trace data to ease storage pressure. However, due to the obvious long-tail distribution of trace data, some common types of traces account for the vast majority of them, while some rare or abnormal traces appear very rarely. However, for developers and operation and maintenance personnel, these rare traces are often more valuable and can help them better understand edge cases in system operation, and can play an important role in downstream tasks such as anomaly detection and root cause location. If the trace data is only randomly sampled (i.e., head sampling), it is difficult to cover low-frequency traces, which may cause rare abnormal traces to be missed in the sampling results, while retaining a large number of repeated and redundant traces, resulting in reduced observability of the system.

[0005] Therefore, it is very important to design a tail sampling method for distributed tracing systems in RPA workflows. Such a method can perform tail sampling decisions in real time based on the call structure, latency, status code and other characteristics of the tracing data, retain important and valuable tracing in a targeted manner, reduce the number of redundant tracing, and thus optimize the collection quality of tracing data.

[0006] Although existing methods can reduce the number of redundant traces in sampling results and improve the diversity of traces to a certain extent, the characterization of traces is usually inaccurate. They only consider the call relationship and ignore attributes such as latency and status code, and omit the use of trace feature information. In addition, exception tracking can collect request information that produces erroneous results or has performance bottlenecks, which is important for operation and maintenance personnel to locate the root cause and detect anomalies. However, existing methods are not sensitive to abnormal tracking, and abnormal tracking is often missed. Summary of the invention

[0007] In order to solve the problems of huge number of workflow logs, high storage cost and low quality in RPA system, the present invention proposes a tail sampling method for distributed tracking system in RPA workflow. The present invention provides a tail sampling method for a distributed tracking system in an RPA workflow, which mainly includes: S1: Collect the traces generated by the distributed tracing system, use the variational graph autoencoder model to perform unsupervised representation learning on the trace data, and save the pre-trained variational graph autoencoder model; each piece of data represents the execution status of the request in the RPA workflow system, consisting of node information, parent node information, and operation execution time; S2: Encode the tracking data into graph structure data in the form of adjacency matrix, and obtain the feature vector based on the encoder of variational graph autoencoder model; S3: Decoder based on variational graph autoencoder model, obtains reconstruction loss, performs anomaly detection on tracking data, and samples all abnormal tracking; S4: clustering method is used to classify normal tracking and preferentially sample; S5: Combine the sampling results of abnormal tracking and normal tracking to form the final sampling data set.

[0008] A storage device stores instructions and data for implementing a tail sampling method for a distributed tracing system in an RPA workflow.

[0009] A tail sampling device for a distributed tracking system in an RPA workflow includes: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement a tail sampling method for a distributed tracking system in an RPA workflow.

[0010] The beneficial effects of the technical solution provided by the present invention are as follows: the present invention extracts features of tracking data through representation learning based on a variational graph autoencoder model, uses reconstruction loss to perform anomaly detection to ensure that all abnormal tracking is retained, and performs biased sampling of normal tracking data through clustering and feature preference sampling strategies, thereby reducing the storage cost of massive tracking data while improving the sampler's sensitivity to abnormal tracking and ensuring the diversity and observability of the sampling results. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which: Figure 1 It is a flowchart of a tail sampling method for a distributed tracking system in an RPA workflow in an embodiment of the present invention; Figure 2 is a structural diagram of a variational graph autoencoder model in an embodiment of the present invention; Figure 3 It is a schematic diagram of the operation of the hardware device in the embodiment of the present invention. DETAILED DESCRIPTION

[0012] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0013] like Figure 1 As shown, the present invention proposes a tail sampling method for a distributed tracking system in an RPA workflow, comprising: S1: Collect the traces generated by the distributed tracing system, use the variational graph autoencoder model to perform unsupervised representation learning on the trace data, and save the pre-trained variational graph autoencoder model; each piece of data represents the execution status of the request in the RPA workflow system, consisting of node information, parent node information, and operation execution time; specifically: S1.1: Continuously collect the trace data generated by the distributed tracing system, including service call relationships, latency, and status codes, and use the trace batches for variational graph autoencoder model training.

[0014] S1.2: Encode the tracking data into a graph data structure in the form of an adjacency matrix:

[0015] Where n is the number of spans in the trace, f is the number of attributes in each span, and the adjacency matrix Indicates the call relationship of the trace. It is the node attribute matrix of the tracking, which records the attribute characteristics of each Span other than tracking (such as service name, interface name, etc.). The latency corresponding to each Span is recorded. Span represents the nodes that make up the tracking data.

[0016] S1.3: Unsupervised representation learning of tracking data based on variational graph autoencoder model GVAE. The model structure of GVAE is as follows Figure 2 As shown, it includes two parts: structural VAE and delayed VAE. GVAE uses graph structure in structural VAE In and To learn the structural features of tracking, the feature vector is obtained by encoding through the GNN encoder , and then reconstruct the tracking structure through the GNN decoder to obtain the reconstructed output , , In delayed VAE, both structural features and temporal features are input into the neural network, but the model weights of structural VAE do not participate in gradient descent.

[0017] GVAE uses the variational lower bound (ELBO) as the loss function:

[0018] in, is the reconstruction loss of the model, which evaluates the similarity between the reconstructed output and the original input through the log likelihood. The model expects the reconstructed output to be as close to the original input as possible; is the KL divergence, which is used to measure the approximate posterior distribution and the true posterior distribution The model expects the approximate distribution to be as similar as possible to the original distribution. represents the approximate posterior distribution expectations, Represents the reconstruction of data given the feature vector z and model parameters theta The log-likelihood of is used to measure the ability of the model to reconstruct data from latent variables. Represents the prior probability distribution of the feature vector z.

[0019] S1.4: After collecting a certain amount of traces, retrain the variational graph autoencoder model and save the model parameters. S2: Encode the trace data into graph structure data in the form of an adjacency matrix, and obtain the feature vector based on the encoder of the pre-trained variational graph autoencoder model. Specifically: S2.1: Cache a certain amount of tracking data, build its graph structure, input it into the encoder of the variational graph autoencoder model for encoding, and extract the feature vector:

[0020] in, represents the feature vector, and are the hidden layer vectors of structural VAE and delayed VAE respectively.

[0021] S3: Based on the reconstruction loss of the variational graph autoencoder model, perform anomaly detection on the trace and sample all abnormal traces. Specifically: S3.1: The decoder of the variational autoencoder model reconstructs the graph structure and obtains the negative log-likelihood value of the reconstructed output relative to the original input ( ), that is, the reconstruction loss:

[0022] in, and Sampled from the hidden layer vector in the GVAE model and , is the graph structure, is the number of spans being tracked, Represents the three matrices in the graph structure G, represents the probability distribution learned by the encoder, represents the probability distribution learned by the decoder, and from and It is obtained by sampling, and L represents the number of samples. It reflects the reconstruction error of the model. For a few tracks with structural or delay anomalies, since they do not conform to the data distribution learned by the model, the reconstructed output is usually quite different from the original track, resulting in a high .therefore, It can be used as an evaluation indicator for anomaly detection.

[0023] S3.2: Output the model Compared with the pre-set abnormal threshold, if If the value is lower than the threshold, the trace is considered normal; otherwise, the trace is considered abnormal and is retained.

[0024] S4: Use clustering method to classify normal tracking and preferentially sample. Specifically: S4.1: For the normal tracks selected in step S3, the tracks are divided into categories using the K-means clustering algorithm, and different categories are sorted from small to large according to the tracked data in the category.

[0025] S4.2: Use the max-min fairness algorithm to allocate the remaining sampling budget to different categories in turn (part of the sampling budget is also used in model training, normal tracking and abnormal tracking detection). When the number of categories is less than the remaining average budget, all the tracking of the category is sampled; otherwise, the remaining sampling budget is evenly distributed to all the remaining categories, and random sampling is performed in each class.

[0026] The preference sampling based on the maximum-minimum fairness algorithm can balance the amount of trace data of different categories being sampled and improve the diversity and observability of the sampling results.

[0027] S5: Combine the sampling results of abnormal tracking and normal tracking to form a final sampling data set. Combine all the sampling results of abnormal tracking data and normal tracking data to form a final data set.

[0028] See also Figure 3 , Figure 3 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, and the hardware device specifically includes: a tail sampling device 401, a processor 402 and a storage device 403 for a distributed tracking system in an RPA workflow.

[0029] A tail sampling device 401 for a distributed tracing system in an RPA workflow: The tail sampling device 401 for a distributed tracing system in an RPA workflow implements the tail sampling method for a distributed tracing system in an RPA workflow.

[0030] Processor 402: The processor 402 loads and executes instructions and data in the storage device 403 to implement the tail sampling method for a distributed tracking system in an RPA workflow.

[0031] Storage device 403: The storage device 403 stores instructions and data; the storage device 403 is used to implement the tail sampling method for a distributed tracking system in an RPA workflow.

[0032] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A tail sampling method for a distributed tracing system in an RPA workflow, characterized in that: include: S1: Collect the traces generated by the distributed tracing system, use the variational graph autoencoder model to perform unsupervised representation learning on the trace data, and save the trained variational graph autoencoder model; each piece of data represents the execution status of the request in the RPA workflow system, consisting of node information, parent node information, and operation execution time; S2: Encode the tracking data into graph structure data in the form of adjacency matrix, and obtain the feature vector based on the encoder of variational graph autoencoder model; S3: Decoder based on variational graph autoencoder model, obtains reconstruction loss, performs anomaly detection on tracking data, and samples all abnormal tracking; S4: clustering method is used to classify normal tracking and preferentially sample; S5: Combine the sampling results of abnormal tracking and normal tracking to form the final sampling data set.

2. The tail sampling method for a distributed tracking system in an RPA workflow according to claim 1, characterized in that: Step S1 is specifically as follows: S1.1: Continuously collect the tracing data generated by the distributed tracing system, including service call relationships, latency, and status codes, and use the tracing data in batches for variational graph autoencoder model training; S1.2: Encode the tracking data into a graph data structure in the form of an adjacency matrix: Where n is the number of spans in the trace, spans are the nodes that make up the trace data, and f is the number of attributes in each span. Indicates the call relationship of the trace. represents the node attribute matrix to be tracked, Indicates the delay corresponding to each Span; S1.3: Unsupervised representation learning of tracking data based on the variational graph autoencoder model; the variational graph autoencoder model includes structural VAE and delayed VAE, and the graph structure is used in the structural VAE In and To learn the structural features of tracking, the feature vector is obtained by encoding through the encoder , and then reconstructed by the decoder to obtain the reconstructed output of the tracking structure; in the time-delay VAE, both the structural features and the time features are input into the neural network; S1.4: After collecting a certain amount of tracking data, retrain the variational graph autoencoder model and save the model parameters.

3. The tail sampling method for a distributed tracking system in an RPA workflow according to claim 2, characterized in that: The variational graph autoencoder model uses a variational lower bound as the loss function: in, is the reconstruction loss of the model, is the KL divergence, which is used to measure the approximate posterior distribution and the true posterior distribution The difference between represents the approximate posterior distribution expectations, Reconstructed data The log-likelihood of Represents the prior probability distribution of the feature vector z.

4. The tail sampling method for a distributed tracking system in an RPA workflow according to claim 1, characterized in that: Step S2 is specifically as follows: Cache the tracking data, build its graph structure, input it into the encoder of the variational graph autoencoder model for encoding, and extract the feature vector: in, represents the feature vector, and are the hidden layer vectors of structural VAE and delayed VAE respectively.

5. The tail sampling method for a distributed tracking system in an RPA workflow according to claim 4, characterized in that: Step S3 is specifically as follows: S3.1: The decoder of the variational autoencoder model reconstructs the graph structure and obtains the negative log-likelihood value of the reconstructed output relative to the original input, i.e., the reconstruction loss: in, and is the hidden layer vector of structure VAE and delay VAE, is the graph structure, is the number of spans in the trace. A span represents the nodes that make up the trace data. Represents the three matrices in the graph structure G, represents the probability distribution learned by the encoder, represents the probability distribution learned by the decoder, and from and The sampling is obtained, Indicates the number of samples; S3.2: Compare the negative log-likelihood value with a preset abnormal threshold. If the negative log-likelihood value is lower than the abnormal threshold, the tracking is considered normal; otherwise, the tracking is considered abnormal and is retained.

6. The tail sampling method for a distributed tracking system in an RPA workflow according to claim 5, characterized in that: Step S4 is specifically as follows: S4.1: For the normal traces screened out in step S3, the traces are divided into categories using the K-means clustering algorithm, and different categories are sorted from small to large according to the trace data in the category; S4.2: Use the maximum-min fairness algorithm to allocate the remaining sampling budget to different categories in turn; when the number of categories is less than the remaining average budget, all the tracking of the category will be sampled; otherwise, the remaining sampling budget will be evenly distributed to all remaining categories, and random sampling will be performed in each class.

7. A storage device, characterized in that: The storage device stores instructions and data for implementing the tail sampling method for a distributed tracking system in an RPA workflow as described in any one of claims 1 to 6.

8. A tail sampling device for a distributed tracing system in an RPA workflow, characterized in that: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the tail sampling method for a distributed tracking system in an RPA workflow as described in any one of claims 1 to 6.

Images