Distributed trajectory anomaly detection based on self-attention deep learning

Abstract

Apparatuses, methods, and computer programs for detecting anomalies in distributed systems by distributed tracking data and self-attention based deep learning are disclosed. Distributed traces are received from a monitoring entity that monitors execution of functions in a distributed system. The distributed traces are analyzed for anomaly detection. Results of the anomaly detection are provided to a control entity for evaluation. The anomaly detection is performed by applying a self-attention based deep learning process to judge whether the distributed traces involve anomalies. Since the self-attention based deep learning only uses information from the traces of system monitoring, unsupervised anomaly detection can be performed by it. Since the self-attention based deep learning has a memory of the whole trace, end-to-end trace analysis can also be performed by it.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of distributed systems, and in particular to related devices, methods and computer programs for detecting anomalies in distributed systems through distributed tracking data and self-attention based deep learning. BACKGROUND

[0002] Distributed systems comprise multiple subsystems that communicate over a network, and are therefore often large and complex. Any of the subsystems or the network can fail.

[0003] Reliability is an important aspect that often arises in large-scale distributed systems. Information technology platforms such as cloud platforms must be able to cope with scalability, transient failures, demand peaks, upgrades, interference between tenants, etc. An effective general framework for achieving reliability, given a reference sequence of symbols (representing a trace), typically includes a mechanism for detecting anomalies or detecting irregular sequences of symbols (representing events or spans).

[0004] For example, if a distributed system processes a user request by generating a sequence of events / symbols S1 = [A, B, C, X, E], this sequence can be compared to a set of known sequences S i that represent valid behavior of the distributed system. If S i does not include the sequence S1, it can be assumed that a failure or anomaly occurred during processing of the user request. On the other hand, if S i includes the sequence S1, it can be assumed that the request processing was successful. In distributed systems, sequences of events / symbols are often referred to as traces.

[0005] Complex large-scale distributed systems often generate very long traces that contain a high level of noise. Therefore, the state space to be modeled is very large, which makes it very difficult to represent and process this state space.

[0006] For example, in some prior art solutions, finite state machines (FSM) are used to model the correct behavior of a distributed system. These prior art solutions work well when the traces do not contain noise. Noise is created due to the complex system relying on software patterns such as caching and load balancing to improve efficiency and reliability. The presence of noise is an important problem for trace anomaly detection, as it requires that all traces that have never been seen before are classified as normal. This is counter-intuitive. For example, using a cache suppresses a symbol in a trace because some instructions are not executed when caching the result. For example, a behavioral model of a system can contain the following observed traces: [A, B, C, D], [A, C, D] and [A, B, D]. When a new trace [A, D] needs to be tested, it is intuitively considered abnormal because it has never been seen before. Nevertheless, [A, D] can be an invalid trace or a valid trace containing noise if the system under analysis can generate traces containing noise. It is not feasible to model noise using FSM-based methods because it can only model noise that has been observed.

[0007] Another challenge is related to the length of the traces. For example, in some prior art solutions relying on long short-term memory (LSTM), only traces of a certain length k can be processed. These methods are called autoregressive because they use the symbols before in the trace to predict the following symbols. For example, in this technique, it is possible to predict which symbol comes after C according to the behavioral model [A, B, C, D, E], thus testing the trace [A, B, C, D, E, F]. If the prediction is correct (i.e. D), the trace is classified as normal. Otherwise, the trace is abnormal. However, the problem here is that as the trace gets longer, the LSTM is not able to establish a correlation between the head symbols (A, B) and the tail symbols (E, F). The first symbols of the trace in the behavioral model are forgotten. This means that the predicted symbol does not depend on all the symbols before itself. SUMMARY

[0008] This summary introduces a selection of concepts that are further described below in the detailed description. This summary is not intended to determine key or essential features of the claimed subject matter or limit the scope of the claimed subject matter.

[0009] It is an object of the present disclosure to allow the detection of anomalies in distributed systems using distributed trace data and self-attention based deep learning. The above and other objects are achieved by the features of the independent claims. Other implementation forms are evident from the dependent claims, the description and the figures.

[0010] According to a first aspect of the present disclosure, there is provided an apparatus for evaluating a distributed trace originating from a distributed system. The apparatus for evaluating the distributed trace is configured to receive monitoring data from a monitoring entity monitoring execution of functions in the distributed system. The monitoring data comprises a distributed trace comprising a sequence of span records of an executing function. The apparatus for evaluating the distributed trace is further configured to analyze the received monitoring data to detect one or more anomalies in the distributed trace. The one or more anomalies refer to patterns associated with abnormal execution of the executing function. The apparatus for evaluating the distributed trace is further configured to provide one or more results of detecting the one or more anomalies to a control entity for evaluation. The detecting the one or more anomalies is performed by applying a self-attention based deep learning process to judge whether the distributed trace is involved in an anomaly. The present disclosure allows using distributed trace data and self-attention based deep learning to detect anomalies in a distributed system. The present disclosure also allows exploiting the full information of a trace, which in turn allows preserving global properties of a trace, e.g. the length of a trace. The use of self-attention based deep learning allows unsupervised anomaly detection, since the self-attention based deep learning only uses information in the traces of the system monitoring. No labels are needed to label traces as normal or abnormal, which is usually a very costly task to label traces as normal or abnormal. The traces can be collected directly from the running system under observation. The use of self-attention based deep learning further allows end-to-end trace analysis, since implementations according to the present disclosure have a memory of the whole trace. Thus, arbitrary relationships between distant locations in the sequence can be modeled.

[0011] In an implementation form of the first aspect, before the analyzing, the apparatus for evaluating the distributed trace is further configured to preprocess the distributed trace by encoding the distributed trace into a plurality of groups of sequences specific to spans. Each group of sequences comprises a timestamp of a span, an identification of a trace the span belongs to, and a function description of the span. Preprocessing the distributed trace allows inferring the spans as a precise set of actions that do not introduce additional noise and do not require access to large amounts of learning data.

[0012] In an implementation form of the first aspect, the preprocessing of the distributed trace further comprises grouping the spans of the distributed trace based on the function description of the spans using a Drain method. This implementation form allows grouping spans according to span functions, such that span groups representing the same function are grouped in the same group. This in turn allows a concise representation of the distributed trace.

[0013] In an implementation of the first aspect, the preprocessing of the distributed trajectory further includes adding a start span at the beginning of the distributed trajectory and a stop span at the end of the distributed trajectory to indicate the length of the distributed trajectory. This implementation allows self-attention-based deep learning to preserve global properties of the span (e.g., trajectory length).

[0014] In a first implementation, the preprocessing of the distributed trajectory further includes replacing randomly selected spans in the distributed trajectory with masked spans. This implementation allows for the creation of training samples for masked span prediction in anomaly detection.

[0015] In an implementation of the first aspect, the preprocessing of the distributed trajectories further includes padding the distributed trajectories to a predefined maximum span number using one or more padding spans. This implementation allows for the uniformity of size of each trajectory by padding it to the maximum span number, thereby facilitating their use in self-attention-based deep learning.

[0016] In the implementation of the first aspect, the device for evaluating the distributed trajectory is further configured to generate a model by applying the self-attention-based deep learning process, the model utilizing all other spans as contextual indicators of the spans most likely to appear within the trajectory. The utilized self-attention mechanism allows for efficient modeling of multiple workloads using a single model. This, in turn, allows for learning arbitrary dependencies within the trajectory. It also allows for focusing on the normal locations of spans, which are unique and specific to the workload being modeled. Furthermore, using a single model for a large number of workloads provides less overhead compared to using multiple models for each workload.

[0017] In an implementation of the first aspect, the device for evaluating the distributed trajectory is further configured to generate an anomaly score for the distributed trajectory by applying the self-attention-based deep learning process, indicating a confidence level that the distributed trajectory is correctly classified as an anomaly. This implementation allows for anomaly detection from noisy trajectories using an error margin. This margin can be used as an indicator of the amount of error (i.e., the anomaly score) allowed in the prediction of a span sequence before declaring the sequence an anomaly.

[0018] In a first implementation, the device for evaluating the distributed trajectory is further configured to determine that the distributed trajectory involves an anomaly in response to the anomaly score exceeding a scoring threshold. This implementation allows for quantification because the uncertainty of the determination made can be quantified.

[0019] According to a second aspect of the present disclosure, a method for evaluating a distributed trace originating from a distributed system is provided. The method for evaluating the distributed trace comprises a device for evaluating the distributed trace receiving monitoring data from a monitoring entity monitoring execution of functions in a distributed system. The monitoring data comprises a distributed trace comprising a sequence of span records of a function being executed. The method for evaluating the distributed trace further comprises the device analyzing the received monitoring data to detect one or more anomalies in the distributed trace. The one or more anomalies refer to patterns associated with abnormal execution of the function being executed. The method for evaluating the distributed trace further comprises the device providing one or more results of detecting the one or more anomalies to a control entity for evaluation. The detecting the one or more anomalies is performed by applying a self-attention based deep learning process to judge whether the distributed trace is involved in an anomaly. The present disclosure allows using distributed tracing data and self-attention based deep learning to detect anomalies in a distributed system. The present disclosure also allows exploiting the full information of a trace, which in turn allows preserving global properties of a trace, e.g. the length of a trace. The use of self-attention based deep learning allows unsupervised anomaly detection, as the self-attention based deep learning only uses information in the trace that is monitored by the system. No labels are needed to label traces as normal or abnormal, which is usually a very costly task to label traces as normal or abnormal. The traces can be collected directly from the running system that is being observed. The use of self-attention based deep learning further allows end-to-end trace analysis, as implementations according to the present disclosure have a memory of the whole trace. Thus, arbitrary relationships between distant positions in the sequence can be modeled.

[0020] In an implementation form of the second aspect, before the analyzing, the method for evaluating the distributed trace further comprises the device preprocessing the distributed trace by encoding the distributed trace into a plurality of groups of sequences specific to spans. Each group of sequences comprises a timestamp of a span, an identification of a trace the span belongs to, and a function description of the span. Preprocessing the distributed trace allows inferring the spans as a precise set of actions that do not introduce additional noise and do not require access to large amounts of learning data.

[0021] In an implementation form of the second aspect, the preprocessing of the distributed trace further comprises the device grouping the spans of the distributed trace based on the function description of the spans using a Drain method. This implementation form allows grouping spans according to span functions, such that span groups representing the same function are grouped in the same group. This in turn allows a concise representation of the distributed trace.

[0022] In an implementation form of the second aspect, the preprocessing of the distributed traces further comprises the device adding a start span at the beginning of the distributed trace and a stop span at the end of the distributed trace to indicate the length of the distributed trace. This implementation form allows self-attention based deep learning to preserve global properties of spans, e.g. trace length.

[0023] In an implementation form of the second aspect, the preprocessing of the distributed traces further comprises the device replacing randomly selected spans in the distributed traces with masked spans. This implementation form allows creating learning samples for masked span prediction for anomaly detection.

[0024] In an implementation form of the second aspect, the preprocessing of the distributed traces further comprises the device padding the distributed traces with one or more padding spans to a predefined maximum number of spans. This implementation form allows making the traces of uniform size by padding each trace to the maximum number of spans, thereby facilitating their use in self-attention based deep learning.

[0025] In an implementation form of the second aspect, the method for evaluating the distributed traces further comprises the device applying the self-attention based deep learning process to generate a model that utilizes all other spans as context to indicate the span that is most likely to occur at a position within a trace. Utilizing a self-attention mechanism allows efficient modeling of multiple workloads with a single model. This in turn allows learning arbitrary dependencies in the traces. This in turn allows focusing on normal positions of spans that are unique and specific to the workload being modeled. Furthermore, using a single model for a large number of workloads provides less overhead compared to using multiple models for each workload.

[0026] In an implementation form of the second aspect, the method for evaluating the distributed traces further comprises the device applying the self-attention based deep learning process to generate an anomaly score for the distributed traces indicating a confidence that the distributed trace is correctly classified as anomalous. This implementation form allows anomaly detection from noisy traces by using an error margin. This margin can be used as an indicator of the amount of error allowed in the prediction of the sequence of spans before declaring the sequence as anomalous (i.e. the anomaly score).

[0027] In an implementation form of the second aspect, the method for evaluating the distributed traces further comprises the device determining that the distributed trace involves an anomaly in response to the anomaly score exceeding a score threshold. This implementation form allows quantification, as the uncertainty of the made decision can be quantified.

[0028] According to a third aspect of the present disclosure, a computer program product is provided. The computer program product comprises program code for performing the method according to the second aspect when the program code is executed on a computer. The present disclosure allows for detecting anomalies in distributed systems using distributed trace data and self-attention based deep learning. The present disclosure also allows for utilizing the full information of a trace, which in turn allows for preserving global properties of a trace, e.g. the length of a trace. The use of self-attention based deep learning allows for unsupervised anomaly detection, as the self-attention based deep learning only uses information in the traces that are monitored by the system. No labels are needed to label traces as normal or anomalous, which is typically a very costly task to label traces as normal or anomalous with labels. The traces can be collected directly from the running system that is being observed. The use of self-attention based deep learning further allows for end-to-end trace analysis, as the implementation according to the present disclosure has a memory of the entire trace. Thus, arbitrary relationships between distant locations in the sequence can be modeled.

[0029] According to a fourth aspect of the present disclosure, a monitoring entity for monitoring execution of a function in a distributed system is provided. The monitoring entity is configured to, in response to receiving a monitoring request from the distributed system, record monitoring data related to the execution of the function in the distributed system. The monitoring data comprises a distributed trace comprising a sequence of span records of the executing function. The monitoring entity is further configured to provide the recorded monitoring data to a device for evaluating the distributed trace for anomaly detection. This aspect allows for a mechanism to record events in a trace in response to a user request and to provide the recorded monitoring data to the device of the first aspect.

[0030] In an implementation form of the fourth aspect, the monitoring data further comprises at least one of: one or more logs for the executing function, or one or more metrics for the executing function. This implementation form allows the monitoring entity to provide the monitoring data in the form of logs, traces and / or metrics to the device of the first aspect.

[0031] According to a fifth aspect of the present disclosure, a method for monitoring execution of a function in a distributed system is provided. The method for monitoring the execution of the function comprises a monitoring entity recording, in response to receiving a monitoring request from the distributed system, monitoring data related to the execution of the function in the distributed system. The monitoring data comprises a distributed trace comprising a sequence of span records of the executing function. The method for monitoring the execution of the function further comprises the monitoring entity providing the recorded monitoring data to a device for evaluating the distributed trace for anomaly detection. This aspect allows for a method to record events in a trace in response to a user request and to provide the recorded monitoring data to the device of the first aspect.

[0032] In an implementation form of the fifth aspect, the monitoring data further comprises at least one of: one or more logs for the executing function, or one or more metrics for the executing function. This implementation form allows the monitoring method to provide the device of the first aspect with monitoring data in the form of logs, traces and / or metrics.

[0033] According to a sixth aspect of the present disclosure, a computer program product is provided. The computer program product comprises program code configured to perform the method according to the fifth aspect when the program code is executed on a computer. This aspect allows a mechanism to record events in a trace and provide the recorded monitoring data to the device of the first aspect in response to a user request.

[0034] According to a seventh aspect of the present disclosure, a control entity for evaluating anomaly detection results is provided. The control entity is configured to receive, from a device for evaluating a distributed trace originating from a distributed system, one or more results of detecting one or more anomalies in the distributed trace. The one or more anomalies refer to patterns associated with abnormal execution of an executing function in the distributed system. The control entity is further configured to evaluate the received one or more results of detecting the one or more anomalies in the distributed trace. This aspect improves the overall reliability of the distributed system by allowing to scan anomaly detection results for detecting potential harmful behavior.

[0035] In an implementation form of the seventh aspect, evaluating the received one or more results of detecting the one or more anomalies in the distributed trace comprises classifying the received one or more results of detecting the one or more anomalies in the distributed trace. The evaluation of the anomaly detection results improves the overall reliability of the distributed system by allowing to detect potential harmful behavior.

[0036] In an implementation form of the seventh aspect, the control entity is further configured to provide one or more control actions to the distributed system based on the evaluated one or more results of detecting the one or more anomalies in the distributed trace. This implementation form allows to decide on a corresponding control action as a response to a type of abnormal behavior.

[0037] According to an eighth aspect of the present disclosure, a method for evaluating anomaly detection results is provided. The method for evaluating the anomaly detection results comprises a control entity receiving, from a device for evaluating distributed traces originating from a distributed system, one or more results of detecting one or more anomalies in the distributed traces. The one or more anomalies refer to patterns associated with abnormal execution of functions being executed in the distributed system. The method for evaluating the anomaly detection results further comprises the control entity evaluating the received one or more results of detecting the one or more anomalies in the distributed traces. This aspect improves the overall reliability of the distributed system by allowing scanning of anomaly detection results to detect potential harmful behaviors.

[0038] In an implementation form of the eighth aspect, evaluating the received one or more results of detecting the one or more anomalies in the distributed traces comprises the control entity classifying the received one or more results of detecting the one or more anomalies in the distributed traces. The evaluation of the anomaly detection results improves the overall reliability of the distributed system by allowing detection of potential harmful behaviors.

[0039] In an implementation form of the eighth aspect, the method for evaluating the anomaly detection results further comprises the control entity providing one or more control actions to the distributed system based on the evaluated one or more results of detecting the one or more anomalies in the distributed traces. This implementation form allows deciding on a corresponding control action as a response to a type of abnormal behavior.

[0040] According to a ninth aspect of the present disclosure, a computer program product is provided. The computer program product comprises program code which, when executed on a computer, is configured to perform a method according to the eighth aspect. This aspect improves the overall reliability of the distributed system by allowing scanning of anomaly detection results to detect potential harmful behaviors.

[0041] Many features will be better understood with reference to the following detailed description when considered in connection with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0042] In the following, example embodiments are described in more detail with reference to the enclosed drawings, in which:

[0043] Figure 1 is a diagram illustrating an overview of a system architecture according to embodiments of the present disclosure;

[0044] Figure 2A is a block diagram illustrating a device for evaluating distributed traces originating from a distributed system according to embodiments of the present disclosure;

[0045] Figure 2B is a block diagram illustrating a monitoring entity according to embodiments of the present disclosure;

[0046] Figure 2C is a block diagram illustrating a control entity according to embodiments of the present disclosure;

[0047] Figure 2D is a diagram illustrating a span grouping and trajectory preprocessing workflow according to embodiments of the present disclosure;

[0048] Figure 3 is a diagram illustrating an example of a trajectory according to embodiments of the present disclosure;

[0049] Figure 4 is a diagram illustrating an example of augmenting, resolving, masking, and padding trajectories according to embodiments of the present disclosure;

[0050] Figure 5 is a diagram illustrating an example of a self-attention mechanism as a pooling layer with bias alignment according to embodiments of the present disclosure;

[0051] Figure 6 is a diagram illustrating an example of a self-attention mechanism architecture implementation according to embodiments of the present disclosure;

[0052] Figure 7 is a diagram illustrating an example of trajectory prediction using masked span prediction according to embodiments of the present disclosure;

[0053] Figure 8 is a diagram illustrating an example overview of embodiments of the present disclosure;

[0054] Figure 9 is a flowchart illustrating a method according to embodiments of the present disclosure;

[0055] Figure 10 is a flowchart illustrating another method according to embodiments of the present disclosure;

[0056] Figure 11 is a flowchart illustrating yet another method according to embodiments of the present disclosure.

[0057] Hereinafter, like reference numerals are used to refer to like or at least functionally equivalent features. DETAILED DESCRIPTION

[0058] The following detailed description is made with reference to the accompanying drawings, of which: the detailed description is made with reference to the accompanying drawings, of which:

[0059] For example, it should be understood that disclosure in connection with a described method can also hold true for a corresponding device or system configured to perform the described method unless the context clearly dictates otherwise. For example, if a specific method step is described, a corresponding device can include a unit to perform the described method step, even if the unit is not explicitly described or illustrated in the figures. On the other hand, for example, if a specific apparatus or device is described, a corresponding method can include a step to perform the described function, even if the step is not explicitly described or illustrated in the figures. Further, it will be understood that the features of the various exemplary aspects described herein can be combined with each other, unless the context clearly dictates otherwise.

[0060] As will be discussed in greater detail below, the present disclosure provides a technique for detecting anomalies in a distributed system architecture, such as a microservice distributed system architecture, using distributed tracing data. More specifically, a device receives distributed trace information that monitors execution of a workload in response to a user request. A distributed trace includes a sequence of event records for functions being executed. The device analyzes the distributed traces to detect patterns associated with anomalous execution of the workload. The device utilizes a self-attention mechanism that allows for efficient modeling of multiple workloads with a single model. An advantage of using this architecture is that it allows for learning arbitrary dependencies in the traces. This allows for focusing on normal locations of spans that are unique and specific to the workload being modeled. Determination of anomalies is done based on setting a margin on the error rate for each trace produced by the self-attention mechanism.

[0061] In other words, the present disclosure allows for solving the anomaly detection problem from distributed traces. The present disclosure allows for efficiently leveraging the overall rich information available from the trace data while still providing small overhead by using a single model for a large number of workloads instead of multiple models for each workload.

[0062] The present disclosure solves the problem of noisy traces by utilizing an error margin. This error margin serves as an indicator of the amount of error in span sequence predictions allowed before declaring a sequence as anomalous. In a simplified example, the sequence [A, B, C, E, D] represents a trace. Due to noise caused, for example, by a load balancer or a cache, the normal sequence [A, B, D] is observed. If the error margin threshold is 0.5 and [A, B, D] is predicted correctly, then the trace is considered normal even though C and E are missing. In this case, the amount of error is 0.4, which is lower than the error margin threshold of 0.5. Thus, this feature allows for handling traces with noise.

[0063] The use of the full information of the trajectory allows to preserve global properties of the trajectory, such as its length. This is advantageous because a correspondence is established between any span within the trajectory. For example, given the sequence [A, B, C, D, E], the disclosure establishes a long-term dependency (in the relevant case) between spans [A, B] and [D, E]. To this end, it uses an encoder-decoder structure with a new adaptive and learnable neural network architecture that implements a self-attention mechanism. This is particularly suitable for analyzing complex distributed systems because they generate long trajectories. The error quantity can also be used as an anomaly score, allowing the integration of the disclosure into full-size general-purpose modules to ensure a higher level of reliability of the system.

[0064] At least some embodiments of the disclosure can use an encoder-decoder structure as a building block. The encoder is a self-attention mechanism, a learnable averaging technique that belongs to the family of deep learning methods. This type of encoder offers, for example, the following advantages:

[0065] - No supervision required: the self-attention mechanism is a self-supervised technique because it only uses information from the trajectories of the system under surveillance. It is not necessary to label the trajectories as normal or abnormal, which is usually a very expensive task. The trajectories can be collected directly from the running system under observation;

[0066] - End-to-end trajectory analysis: it has memory of the entire trajectory. Therefore, this encoder can model arbitrary relationships between distant locations within the sequence. This is advantageous because the global span distribution within a normal trajectory can differ between executions. Focusing on a smaller number of spans makes the process less sensitive to small local changes in the trajectory.

[0067] To detect anomalies, a post-processing procedure can be applied to the predictions made by the decoder. This procedure counts the number of erroneous predictions in the encoder-decoder network along the trajectory. This number is scaled using the length of the trajectory, which allows to define a bounded anomaly score for a given input trajectory. This value represents the confidence of the process in its judgment of the information published to the outside world. If the anomaly score is greater than a fixed threshold, which is a parameter in the process, anomaly detection can be completed with an appropriate trajectory labeling. For example, this procedure offers the following advantages:

[0068] - Quantifiability: since the anomaly detection system can usually cooperate with other troubleshooting systems, this procedure provides a mechanism to quantify the uncertainty of the judgments made.

[0069] Figure 1 Figure 1 is a diagram showing an overview of a system architecture 100 according to an embodiment of the disclosure. Figure 1A distributed system (e.g., a microservice-based distributed system) 110, a monitoring system or entity 220, a control system or entity 230, and an anomaly detection system or device 210 for evaluating distributed traces are shown in FIG. 1. While embodiments can be described in terms of a microservice-based distributed system, it is by way of example and not limitation.

[0070] The microservice-based distributed system 110 is a complex distributed system, when a user (or Figure 1 observes the behavior of the system, e.g., by requesting operations through an application programming interface, API, gateway. The requests trigger the execution and interaction of multiple microservices Figure 1 (e.g., MS 11, MS 12, MS 21, MS 22, MS 3) in the example. Each microservice can implement a specific set of software functionality and utilize, e.g., remote procedure calls Figure 1 (e.g., RPC1, RPC2, RPC3, RPC4) and / or database (DB) calls.

[0071] The microservice-based distributed system 110 can be related or associated with, but not limited to, e.g., healthcare and / or health, communication and / or cyber security, transportation and / or smart mobility, digital industry, industry 4.0, Internet of Things, cloud continuum, etc.

[0072] The present disclosure can be applied, e.g., to debugging, deployment analysis, fault correlation and analysis, latency analysis and correlation, resource monitoring, root cause analysis, security audit, etc. However, the present disclosure is not limited to these examples.

[0073] The monitoring system or entity 220 monitors the distributed system 110. It provides a mechanism to record events in a trace in response to a user request. As a result, the monitoring system or entity 220 generates monitoring data, including traces 2201 and, optionally, logs 2202 and / or metrics 2203.

[0074] Figure 3 FIG. 300 is an example of a trace showing the creation and deletion of a network workload from OpenStack, according to an embodiment of the present disclosure. More specifically, FIG. 300 is an example of a trace for a network workload created and deleted from OpenStack. Figure 3 The trace of FIG. 300 includes multiple spans 301-305, building up the workload as a response to a user request. Each of the spans 301-305 records information, e.g., the operation being performed (shown in FIG. 300 as hypertext transfer protocol (HTTP) operations “GET”, “POST”, “DELETE”), the name and IP of the microservice, the endpoint of the span, the response time, etc. Figure 3

[0075] ​The control system or entity 230 allows to guarantee the reliability of the overall microservice system 110. To this end, the control system or entity 230 receives as input the data generated by the anomaly detection system or device 210. The control system or entity 230 comprises two functions: anomaly classification 2302 and control action decision 2301. The anomaly classification 2302 scans the monitoring data to detect potential harmful behavior, while the control action decision 2301 decides on a corresponding control action as a response to the type of abnormal behavior.

[0076] The anomaly detection system or device 210 receives and processes monitoring data in the form of logs, traces and / or metrics from the monitoring system or entity 220. The purpose of the anomaly detection system or device 210 is to detect potentially unstable system behavior due to a specific anomaly. To this end, the anomaly detection system or device 210 can first process raw traces into a suitable form, e.g. by functional span grouping 2101 and / or trace pre-processing 2102. Then, a corresponding model based on a set of input traces can be built (e.g. by functional mask span model training 2103), which is applied to anomaly detection, e.g. with newly presented traces at the input. As output, the anomaly detection system or device 210 can judge for a trace whether the trace is abnormal or not, as well as a corresponding confidence score which can be used as a confidence of the judgment. The output can be forwarded to the control system or entity 230, which can then evaluate the type of anomaly and a corresponding control action.

[0077] Next, based on Figure 2A An example embodiment of a device 210 for evaluating distributed traces originating from a distributed system 110 is described. Some features of the described device are optional features that provide more advantages.

[0078] Figure 2A is a block diagram illustrating a device 210 according to an embodiment of the present disclosure. In an embodiment, the device 210 can comprise a suitable computing device, e.g. a server device or the like. In at least some embodiments of the present disclosure, the device 210 can be integrated with the monitoring entity 220 and / or the control entity 230.

[0079] The device 210 can comprise at least one processor or processing unit 212 and at least one memory 214 coupled to the at least one processor 212, which can be used to implement the functions described in more detail later.

[0080] The at least one processor 212 can include, for example, one or more of various processing devices, such as a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), processing circuitry including or not including a DSP, or various other processing devices including integrated circuits such as, for example, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a microcontroller unit (MCU), a hardware accelerator, a special-purpose computer chip, or the like.

[0081] The memory 214 can be used for storing, for example, a computer program, and the like. The memory can include one or more volatile memory devices, one or more non-volatile memory devices, and / or a combination of one or more volatile memory devices and non-volatile memory devices. For example, the memory can be implemented as a magnetic storage device (e.g., hard disk drive, floppy drive, tape drive, etc.), an optical storage device, and / or a semiconductor memory (e.g., mask ROM, programmable ROM (PROM), erasable PROM (EPROM), flash ROM, random access memory (RAM), etc.).

[0082] The device 210 for evaluating distributed traces is configured to receive monitoring data from a monitoring entity 220 monitoring execution of a function in a distributed system 110. The monitoring data comprises a distributed trace (e.g., trace 2201) comprising a sequence of spans for the function being executed. The monitoring data can further comprise one or more logs (e.g., log 2202) for the function being executed, and / or one or more metrics (e.g., metric 2203) for the function being executed. Herein, the terms “span” and “event” are used interchangeably. Furthermore, the term “trace” denotes a sequence of events / spans. Events / spans can be denoted symbolically.

[0083] The device 210 can further be configured to pre-process the distributed trace prior to the analysis.

[0084] In other words, as part of anomaly detection, an appropriate representation of spans can be provided as part of the trajectories. By design, a span represents a limited set of actions that are instantiated during the execution of a distributed system 110. Since the instances of the operations are being recorded by the monitoring entity 220, many different looking trajectories can be produced even for the same operation. To mitigate this, spans can be first derived as a set of precise actions that do not introduce additional noise and do not require access to large amounts of learning data. This can be achieved, for example, by the span grouping 2011 function of Figure 2D

[0085] Figure 2D is a diagram illustrating span grouping 2011 and trajectory pre-processing 2102 functions according to embodiments of the present disclosure. The span grouping 2011 function can include a span encoding function 210 11 and / or a span clustering 210 12 function. The trajectory pre-processing 2102 function can include a trajectory augmentation function 210 21 , a masking function 210 22 and / or a padding function 210 23 . Additional functions can include a masked span model training 2103 and / or a masked span prediction 2104. These functions will be described in more detail below.

[0086] In embodiments, the pre-processing of the distributed trajectories can include encoding the distributed trajectories into span-specific multi-sets of sequences. For example, at least some of the multi-sets can be triplets. Each multi-set can include a timestamp of a span, an identification of a trajectory to which the span belongs, and a function description of the span.

[0087] In other words, the pre-processing of the distributed trajectories can include a span encoding function 210 11 . In examples, the spans generated by the distributed system 110 can be in a structured format. For the representation of the trajectories, triplets (timestamp, traceID, and span function) can be chosen, for example. An example of a triplet span representation of a sequence of trajectories with traceIDs is shown in Table 1 below.

[0088]

[0089] Table 1

[0090] ​The first element in the triple is used to access the order of spans in a trace. The second element in the triple identifies the trace to which the span belongs. The third element in the triple represents the functional difference between spans. Since a trace representation characterizes a sequence of events with a specific function, the span function is used to provide information about the type of function related to the span. The information can be in text format. An http operation can use the http method and the http path as indicators of the span function. The http method can be an operation defined by the http protocol (e.g., GET, POST, DELETE, PUT), while the http path can be a uniform resource locator (URL) of a component that the system can access through the http protocol. For RPC calls, a function descriptor can be used to represent the span function. These functions can be different remote procedure calls that components within the distributed system 110 access. Each span from a database query can be represented with this procedure.

[0091] The preprocessing of the distributed trace can also include grouping spans of the distributed trace based on the function description of the spans using a Drain method. The Drain method is a log parsing method that can parse logs in a streaming and timely manner. To speed up the parsing process, Drain uses a fixed-depth parsing tree, which encodes specially designed parsing rules. The Drain method is described in more detail in, for example, Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R. Lyu, Drain: An Online Log Parsing Approach with Fixed Depth Tree, IEEE International Conference on Web Services (ICWS), 2017.

[0092] In other words, the preprocessing of the distributed trace can include span clustering 210 12 The function. At least in some embodiments, the representation of the trace generated by the monitoring entity 220 can lack a concise representation. More specifically, many span functions as span descriptors can have variable parts or variable parameters. For example, GET / v3 / schema / d303dd45-66ab-4d18-bea1-2d5e4b76e0cc / images / and GET / v3 / schema / cfe7b11c884c455cbe2df3ec43521be / images / both represent the same structure of GET / v3 / schema / * / images / where * represents the part that changes. In other words, span representations can represent the same action even though their structure parts are different.

[0093] Accordingly, in at least some embodiments, it can be desirable to group spans based on their span function, such that spans representing the same function are grouped in the same group. The indices of these groups can be denoted, for example, as SpanClusterID(X), and they form a database of span clusters. To group spans by their function, the Drain method can be applied, for example, due to its consistently high performance on a large number of log parsing problems. The Drain method constructs a tree from the frequency count of words occurring at positions in the text. The trace augmentation function 210 21 The output generated is fed through the Drain. The Drain generates a set of span clusters as output, which form a database of spans. Each element in this set can be denoted by SpanClusterID(X), which has an injective mapping to natural numbers.

[0094] The number of span templates generated is typically small, as not many different actions can occur during the execution of one particular workload. In this way the traces are represented as sequences of integer values (representing SpanClusterID), which is a form that learning methods typically accept. As a final step, to formally represent the traces, a dictionary can be created. The keys of the dictionary can be traceID. These values represent the sequence of SpanClusterID that make up the trace.

[0095] As described above, Figure 2D The trace pre-processing 2102 function is shown, which can include a trace augmentation function 210 21 , a masking function 210 22 , and / or a padding function 210 23 . The goal of these functions is to extract information from the traces for use in anomaly detection. The spans generated by the distributed system 110 are stored in a database of spans. First, a query is performed on the set of spans generated by the distributed system 110 in response to a particular workload. The output of this function can be a dictionary object of normal traces and anomaly data. The keys of the dictionary can be the unique identifier of the trace (traceID). The values of the dictionary elements can be a sequence of integers, where each integer can be a unique numerical representation of a span template (SpanClusterID) extracted from the database of original spans.

[0096] The trajectory preprocessing 2102 function can receive as input a set of observed trajectories until a certain time period. The trajectories can consist of spans. Since there can be many unique spans reflecting a limited set of actions performed with different parameters, grouping the spans can help to infer their number. To this end, the spans can be grouped based on, for example, the static part of their representation (e.g., the function name in case of RPC calls or http methods and the corresponding http URL called endpoint). The number of such spans can be in the range of, for example, 20-10000. The span database can further enhance the spans with special purposes such as masking, start, end, and / or specialized appendixes.

[0097] The trajectory preprocessing 2102 function can also perform a masking function 210 22 . Given a trajectory with a certain ID, the masking function 210 22 can be used to mask the spans of the trajectory with a certain ID. The masking function 210 <mask>each span within a particular span-replacement trajectory of the span-replacement trajectory. The replaced original span can also be stored. This replaced original span creates a learning sample for the Masked Span Prediction (MSP) 2104 function for anomaly detection. The live incoming trajectories can first be augmented, processed, and masked by the trajectory pre-processing 2102 function. To make the size of each trajectory uniform, they can be padded with special appendix spans to a maximum padding length.

[0098] Figure 4 is shown illustrating a trajectory as shown in Figure 3 <mask>FIG. 400. Each trace can be reintroduced multiple times with slight modifications, where each span is replaced with a masked token. The replaced spans can also be stored. The output of the trace pre-processing 2102 function can be a set of transformed and masked traces suitable for the learning process described further below.

[0099] The pre-processing of distributed traces can also include adding a start span at the beginning of the distributed trace and a stop span at the end of the distributed trace to indicate the length of the distributed trace.

[0100] In other words, the pre-processing of distributed traces can include a trace augmentation function 210 21 In at least some embodiments, the spans can be grouped into a time-ordered sequence using their traceID and timestamp. The traces can be augmented by, for example, two special spans: <start>and <stop>These artificial spans can be used to enable attention-based algorithms to retain global properties of the spans (e.g. the length of a track). They can also be represented as triples. <start>The span can have the same timestamp as the first span of the trace. The traceID can be inherited. For the span function, the token START can be used. Similarly, <stop>A span can have the same timestamp as the previous span. It can also inherit the traceID. A span function can have the token STOP. The trace augmentation function 210 21 The output can be a table with, for example, three columns: the span timestamp, the traceID to which the span belongs, and the span function representing span-specific information.

[0101] The preprocessing of the distributed traces can also include replacing randomly selected spans in the distributed traces with masked spans.

[0102] In other words, the preprocessing of the distributed traces can include a masking function 210 22 At least in some embodiments, the masking can prepare the input for the MSP task. To this end, the masking function 210 22 can take as input the augmented traces, where each trace can be represented as a sequence SpanClusterID. The masking function 210 22 can replace, for example, the token STOP with a special token, e.g., <mask>a randomly selected span in the special span-replacement trajectory of the. The random span can be used as the target in the MSP task. In this way, each augmented and processed trajectory in the previous step can be re-introduced multiple times, each time with a different random span <mask>Span replacement replaces different spans. The output of the masking function 210 22 may be, for example, a set of |T| tuples of two values: 1) masked traces and 2) replaced single spans, where |T| represents the length of the traces. Each processed output trace can share the same traceID, as they are derived from the original traces given as input.

[0103] The pre-processing of the distributed traces can also include padding the distributed traces with one or more padding spans to a pre-defined maximum number of spans.

[0104] In other words, the pre-processing of the distributed traces can include a padding function 210 23 . At least in some embodiments, the padding is performed after the masking is completed. While the traces are generated by the same workload, their lengths can differ. However, the learning mechanism can require a fixed number of spans for all traces. To address this, each trace can be padded, for example, from the right side (after the last span) until a user-defined maximum number of spans. Traces larger than the maximum number can be cut at the maximum position. For padding, a special <spec>Span. This span (like all other spans) can have its unique SpanClusterID to which it belongs. The output can be a set of processed, masked, and padded traces represented as a sequence of integers.

[0105] The device 210 is further configured to analyze the received monitoring data to detect one or more anomalies in the distributed traces. The one or more anomalies refer to patterns associated with abnormal execution of the function being executed. The detection of the one or more anomalies is performed by applying a self-attention based deep learning process to determine whether the distributed traces involve anomalies.

[0106] For example, the device 210 can be further configured to apply a self-attention based deep learning process to generate a model that utilizes all other spans as context to indicate the span that is most likely to occur at a position within a trace.

[0107] In other words, the functionality of the device 210 can include a masked span model training 2103 functionality. At least in some embodiments, the masked span model training 2103 functionality can include a deep learning encoder-decoder architecture implementing a self-attention machine learning method. The masked span model training 2103 functionality can be used to solve a masked span prediction pseudo-task for anomaly detection. The goal of this task is to generate, given all other non-masked spans of a trace as input, the span that is most likely to occur at the masked position within the trace. The non-masked part of the input is referred to as the context of the masked span. As output, the masked span model training 2103 functionality can generate a learned model that can be called, e.g., based on newly presented monitoring data from a live running distributed system 110.

[0108] The masked span model training 2103 functionality can take as input, for example Figure 4 The illustrated traces. The masked span model training 2103 functionality predicts the span at the [MASK] position. Since in this example, these traces are derived from Figure 3 the true values of the [MASK] position are known. Thus, the accuracy of the prediction can be evaluated. For masked traces, this computation yields a binary value, true if the predicted span equals the true span, and false otherwise.

[0109] Figure 5 is a diagram 500 illustrating an example of a self-attention mechanism as a pooling layer with bias alignment, in accordance with an embodiment of the present disclosure. In Figure 5 In the example of, it is assumed that each of the inputs 5011, 5012, 5013 is initially represented with a random vector value (learned implicitly during the learning phase). For each input element 5011, 5012, 5013, there can be three real-valued vectors: a query 5023, 5033, 5043, a key 5021, 5031, 5041, and a value 5022, 5032, 5042. The key 5021, 5031, 5041 can be used as a unique identifier for the token in the sequence. The query 5023, 5033, 5043 can be used to allow other tokens to access a specific token in the sequence. The value vector 5022, 5032, 5042 can be obtained as a linear combination of each token in the sequence weighted by the relative contribution computed by the normalized dot product of each key-query pair. The output vector 505 can be constructed as a linear combination from the values of each span. To provide a more powerful representation, multiple self-attention mechanisms can be stacked, each fed with a different random projection of the data. This stacking forms a structure called multi-headed attention, while a single self-attention mechanism is called a head. Each head in the multi-headed mechanism can be fed with a different part of the input vector 5011, 5012, 5013. The output 505 can be obtained by stacking all the outputs from the heads.

[0110] Figure 6 is a diagram 600 illustrating an example of a self-attention mechanism architecture implementation according to an embodiment of the present disclosure. Figure 6 The example of includes two parts: a position encoder 610 and an encoder-decoder structure 621, 622 using self-attention mechanisms as building blocks.

[0111] All elements of the model require numerical input. To represent this, the integer representation of a span can be initially converted to a randomly initialized numerical vector x e R d Herein, these vector embeddings are called span vector embeddings, or simply span embeddings. In this way, each span can be mapped to a random vector that is updated during the learning process. The purpose of the update is to learn a unique d-dimensional span vector representation of the given span context. The dimension d of the chosen vector space is a parameter of the learning method. It is fixed to, for example, 256. This length can vary depending on the number of available spans and the complexity of the trajectory.

[0112] The span embeddings can be passed to a position encoding block 610 provided by the input trajectory 601. The position encoding block 610 can be used to preserve the position of the spans in the trajectory. This can be done by a periodic function given in equation 1 below. This block computes a vector p e R d .

[0113]

[0114] Here, k = {0, 1,..., d - 1} is the index of each element in p and j = {0, 1,..., max - 1} is the position index of each span in the trajectory. The parameter k describes the position of the value in the vector p. The sine and cosine functions are applied interchangeably to allow distinguishing the respective values in a particular span vector of p. As output, the positional encoding block or layer 610 produces the sum x' = x + p of the original vector and its positional encoding.

[0115] The encoder-decoder comprises two parts: an encoder 621 and a decoder 622. All layers can be feed-forward neural networks.

[0116] The encoder or encoder layer 621 comprises two elements: a multi-head self-attention element 6211 and a feed-forward element 6212. The encoder 621 starts with the multi-head attention element 6211. The number of heads in the multi-head attention element 6211 can be set to, for example, 8, but it can be optimized. Each head can comprise four layers, representing the query 5023, 5033, 5043, the key 5021, 5031, 5041, the value 5022, 5032, 5042 and the output 505. The size of these vectors can be determined as a fraction of the model size and the number of heads. All token embedding vectors can be summarized as rows of a matrix X'. For efficient computation, all queries 5023, 5033, 5043, keys 5021, 5031, 5041 and values 5022, 5032, 5042 can be organized in matrices labeled Q, K and V, respectively. Such a structural organization allows the following mathematical formula of the architecture:

[0117]

[0118] wherein are the learnable weights of the neural network;

[0119]

[0120] wherein H is the number of heads. Next, a softmax function 626 can be applied and the result can be used to scale each value of the span. The span values can be concatenated to form a matrix X" as output of the multi-head element 6211.

[0121] The residual connection 6231, 6232 provides a way to combine the original information and the information processed by the multi-head attention element 6211. This method addresses the small gradient problem that can occur when training. It computes X' + X".

[0122] Layer normalization layers 6241, 6242 can be used to address the problem of covariate shift that can occur during training. It can apply a simple column-wise normalization using the mean and standard deviation of the corresponding input norm(X' + X"). At the end of each layer, a regularization technique can be applied aimed at improving the performance of the learning method.

[0123] The feed-forward element or layer 6212 is the last element of the encoder 621 and can comprise two feed-forward linear layers with a ReLU activation in between. It can be applied individually to each row of the input of the previous step. The residual connections 6231, 6232 and normalization 6241, 6242 layers can also be applied.

[0124] The decoder element or layer 622 can be a simple feed-forward linear layer. As input size, it can take the size of the output from the encoder layer 621. The output of the linear layer 622 can correspond to the number of unique spans that exist on the input. The decoder element or layer 622 can generate a score estimate of the likelihood that a span occurs on the masked position of the given trace of the input.

[0125] A suitable cost function for this task can be the categorical cross-entropy loss. It can be computed, for example, between the output 627 of the masked span generated at 625 and the true span that occurs at this position. Taking into account the context of the span, this objective can maximize the probability of the span that is most likely to occur on the masked position. To optimize this objective, the Adam (Adaptive Moment Estimation) solver or optimizer can be used, with a common value of 0.0001 for the learning rate, and values of 0.90 and 0.98 for β1 and β2, respectively. Adam is an optimization algorithm that can be used instead of the classical stochastic gradient descent procedure to update the network weights based on iterations of training data.

[0126] The device 210 can also be used to apply a self-attention based deep learning process to generate an anomaly score for the distributed trace, indicating a confidence that the distributed trace is correctly classified as anomalous. The device 210 can also be used to determine that the distributed trace involves an anomaly in response to the anomaly score exceeding a score threshold.

[0127] In other words, the functionality of the device 210 can include a masked span prediction 2104 functionality. The masked span prediction 2104 functionality can be used to determine whether an input trace is anomalous, at least in some embodiments.

[0128] Figure 7 is a flowchart 700 illustrating an example of trajectory prediction using shield span prediction according to embodiments of the present disclosure. The flowchart 700 starts from operation 701, which provides all augmented, shielded, and padded trajectories with the same traceID to the input of the shield span prediction 2104 function. At operation 702, for each of the shielded trajectories, the shield span prediction 2104 function generates a ranked list of the most relevant spans that would have occurred on the shielded locations within each trajectory. At operation 703, the parameter top_k is used to decide whether the prediction on the augmented, shielded, and padded trajectories is correct. If the prediction of the shield span prediction 2104 function is within the first top_k, it is considered a correct prediction. Otherwise, it is considered an incorrect prediction. The value of this parameter can be between 5-10% of the total number of spans in the span group database.

[0129] For a trajectory with traceID, for example, anomalies can be detected by using the following steps. First, all the processed, augmented, and shielded trajectories with the same traceID as the original trajectory can be presented as input to the shield span prediction 2104 function. Second, the number of incorrect predictions made on the shielded trajectories can be counted. Finally, the error rate for each (original) trajectory can be calculated as the ratio between the number of incorrect predictions and the total trajectory length.

[0130] The error rate can be used as an anomaly score to represent the confidence of the prediction. The higher the score, the higher the confidence that the trajectory is correctly classified as an anomaly. The final decision of the anomaly can be done using, for example, a score threshold. The score threshold can be an external parameter that can be bounded, for example, in the range between 0 and 1. A smaller threshold indicates that the shield span prediction 2104 function has a good understanding of the normal execution of the trajectory and can be used as a quality indicator. This allows to provide a confidence score estimate for the prediction, so the corresponding entity can be informed about the reliability of the decision.

[0131] The device 210 is further configured to provide one or more results of detecting one or more anomalies to the control entity 230 for evaluation.

[0132] Figure 8 is a diagram 800 illustrating an example overview of a workflow of embodiments of the present disclosure. The workflow starts from operation 801. At operation 802, the input is the input trajectory represented with spans and threshold parameters. The trajectories during the training process can be used to learn the parameters of the model. During the prediction phase 810, predictions on the trajectory states can be generated. As output, the decisions 812, 813 of the trajectory (affected by the predictions) can be reported with the appropriate confidence score 811.

[0133] The trajectories can first be pre-processed 803 by the trajectory. The trajectory pre-processing 803 enhances, masks the trajectories and adds padding to the trajectories. If retraining is required, a set of available trajectories (operation 805) can be used to perform training at operation 806 and induce a model stored in memory at operation 807. For initial training, it is assumed that there is a set of available trajectories that describe the available workloads. If one is interested in the prediction of a particular trajectory, the corresponding flag indicating the training period (operation 804) can be set to false. The first stage of the prediction part checks if the latest trained model is loaded at operation 808. If it is loaded, the model uses it for the pre-processed test trajectory. Otherwise, the model is loaded from memory at operation 809 and then proceeds to the evaluation.

[0134] The input threshold parameter is then applied to the confidence score generated by the loaded model's prediction. If the confidence score exceeds the threshold at operation 811, the trajectory is declared abnormal at operation 812, otherwise the trajectory is declared normal at operation 813. After obtaining the prediction, the workflow can continue by scanning the input for new trajectories.

[0135] Whether to perform retraining can be determined by an operator. After a period of time, retraining can be required due to various reasons, such as updates to the system. This can account for potential new spans and new trajectories, which can prevent false alarms from being generated.

[0136] The generated prediction can be furthered to later steps (e.g., in an autonomous reliability control system) or reported to an operator (e.g., in a semi-autonomous reliability control system) so that an appropriate control action can be selected. This can ensure that the appropriate level of microservice system reliability requirements are met.

[0137] Next, based on Figure 2B Example embodiments of a monitoring entity 220 for monitoring execution of functions in a distributed system 110 are described. Some features of the described apparatus are optional features that provide more advantages.

[0138] Figure 2B is a block diagram illustrating a monitoring entity 220 according to embodiments of the present disclosure. In embodiments, the monitoring entity 220 can comprise a suitable computing device, such as a server device or the like. In at least some embodiments of the present disclosure, the monitoring entity 220 can be integrated with the apparatus 210 and / or the control entity 230.

[0139] As discussed above in connection with Figure 1 The monitoring entity 220 monitors the distributed system 110 and provides a mechanism to record events within trajectories in response to user requests, as discussed above in connection with

[0140] The monitoring entity 220 can include at least one processor or processing unit 222, and at least one memory 224 coupled to the at least one processor 222, which can be used to implement functionalities described in more detail later.

[0141] The at least one processor 222 can include, for example, one or more of various processing devices, such as a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), processing circuitry including or not including a DSP, or various other processing devices including integrated circuits such as, for example, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a microcontroller unit (MCU), a hardware accelerator, a special-purpose computer chip, or the like.

[0142] The memory 224 can be used for storing, for example, a computer program, etc. The memory can include one or more volatile memory devices, one or more non-volatile memory devices, and / or a combination of one or more volatile memory devices and non-volatile memory devices. For example, the memory can be implemented as a magnetic storage device (e.g., hard disk drive, floppy drive, tape drive, etc.), an optical storage device, and / or a semiconductor memory device (e.g., a mask ROM, programmable ROM (PROM), erasable PROM (EPROM), flash ROM, random access memory (RAM), etc.).

[0143] The monitoring entity 220 is configured to record monitoring data related to execution of a function in the distributed system 110 in response to receiving a monitoring request from the distributed system 110. The monitoring data includes a distributed trace (e.g., trace 2201) including a sequence of span records for the function being executed. The monitoring data can also include one or more logs (e.g., logs 2202) for the function being executed, and / or one or more metrics (e.g., metrics 2203) for the function being executed. The monitoring entity 220 can also be configured to provide the recorded monitoring data to the device 210 for evaluating the distributed trace for anomaly detection.

[0144] Next, based on the evaluation of the anomaly detection result, the control entity 230 is configured to perform one or more of the following operations. Figure 2C Example embodiments of the control entity 230 for evaluating the anomaly detection result are described. Some features of the described devices are optional features that provide more advantages.

[0145] Figure 2C is a block diagram illustrating a control entity 230 according to an embodiment of the present disclosure. In an embodiment, the control entity 230 can comprise a suitable computing device, such as a server device or the like. In at least some embodiments of the present disclosure, the control entity 230 can be integrated with the device 210 and / or the monitoring entity 220.

[0146] The control entity 230 can comprise at least one processor or processing unit 232 and at least one memory 234 coupled to the at least one processor 232, which can be used to implement the functionalities described in more detail later.

[0147] The at least one processor 232 can comprise, for example, one or more of various processing devices, such as a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), processing circuitry including or not including a DSP, or various other processing devices including integrated circuits such as, for example, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a microcontroller unit (MCU), a hardware accelerator, a special-purpose computer chip, or the like.

[0148] The memory 234 can be used to store, for example, a computer program and the like. The memory can include one or more volatile memory devices, one or more non-volatile memory devices, and / or a combination of one or more volatile memory devices and non-volatile memory devices. For example, the memory can be implemented as a magnetic storage device (e.g., hard disk drive, floppy disk, magnetic tape, etc.), an optical storage device, and a semiconductor memory (e.g., mask ROM, programmable ROM (PROM), erasable PROM (EPROM), flash ROM, random access memory (RAM), etc.).

[0149] The control entity 230 is configured to receive, from the device 210 for evaluating distributed traces originating from the distributed system 110, one or more results of detecting one or more anomalies in the distributed traces. The one or more anomalies refer to patterns associated with abnormal execution of functions being executed in the distributed system 110. In other words, the control entity 230 can receive data generated by the device 210 as input.

[0150] The control entity 230 is also configured to evaluate the received one or more results of detecting one or more anomalies in the distributed trace. The evaluation of the received one or more results of detecting one or more anomalies in the distributed trace can include classifying the received one or more results of detecting one or more anomalies in the distributed trace. In other words, the control entity 230 can include an anomaly classification 2302 function that scans the monitoring data to detect potential harmful behavior.

[0151] The control entity 230 can also be configured to provide one or more control actions to the distributed system 110 based on the evaluated one or more results of detecting one or more anomalies in the distributed trace. In other words, the control entity 230 can include a control action decision 2301 function that decides a corresponding control action as a response to the type of anomalous behavior.

[0152] Figure 9 is a flowchart of a method 900 for evaluating a distributed trace originating from a distributed system according to an embodiment of the disclosure.

[0153] At operation 901, a device for evaluating a distributed trace receives monitoring data from a monitoring entity that monitors execution of functions in a distributed system. The monitoring data includes a distributed trace that includes a sequence of span records of functions being executed.

[0154] At optional operation 902, the device can pre-process the distributed trace by encoding the distributed trace into a span-specific sequence of groups. Each group includes a timestamp of a span, an identification of a trace to which the span belongs, and a function description of the span.

[0155] At optional operation 903, the device can group the spans of the distributed trace based on the function description of the spans using a Drain method.

[0156] At optional operation 904, the device can add a start span at the beginning of the distributed trace, and the device can add a stop span at the end of the distributed trace to indicate the length of the distributed trace.

[0157] At optional operation 905, the device can replace a randomly selected span in the distributed trace with a masking span

[0158] At optional operation 906, the device can pad the distributed trace with one or more padding spans to a predefined maximum number of spans.

[0159] At operation 907, the device analyzes the received monitoring data to detect one or more anomalies in the distributed traces. The one or more anomalies refer to patterns associated with abnormal execution of the functions being executed. The detecting the one or more anomalies is performed by applying a self-attention based deep learning process to determine whether the distributed traces involve anomalies.

[0160] At operation 908, the device provides one or more results of detecting the one or more anomalies to a control entity for evaluation.

[0161] The method 900 can be performed by the device 210 for evaluating distributed traces originated from the distributed system 110. For example, the operations 901-908 can be performed by the at least one processor 212 and the memory 214. Other features of the method 900 are directly from the functions and parameters of the device 210, and are not repeated here. The method 900 can be performed by a computer program.

[0162] Figure 10 is a flowchart of a method 1000 for monitoring execution of functions in a distributed system according to an embodiment of the disclosure.

[0163] At operation 1001, a monitoring entity receives a monitoring request from a distributed system.

[0164] At operation 1002, in response, the monitoring entity records monitoring data related to execution of functions in the distributed system. The monitoring data includes distributed traces including sequences of span records of functions being executed.

[0165] At operation 1003, the monitoring entity provides the recorded monitoring data to a device for evaluating the distributed traces for anomaly detection.

[0166] The method 1000 can be performed by the monitoring entity 220. For example, the operations 1001-1003 can be performed by the at least one processor 222 and the memory 224. Other features of the method 1000 are directly from the functions and parameters of the monitoring entity 220, and are not repeated here. The method 1000 can be performed by a computer program.

[0167] Figure 11 is a flowchart of a method 1100 for evaluating anomaly detection results according to an embodiment of the disclosure.

[0168] At operation 1101, a control entity receives one or more results of detecting one or more anomalies in distributed traces originated from a distributed system from a device for evaluating the distributed traces. The one or more anomalies refer to patterns associated with abnormal execution of functions being executed in the distributed system.

[0169] At operation 1102, the control entity evaluates the received one or more results of detecting one or more anomalies in the distributed trajectory. The evaluation of the received one or more results of detecting one or more anomalies in the distributed trajectory can include the control entity classifying the received one or more results of detecting one or more anomalies in the distributed trajectory.

[0170] At optional operation 1103, the control entity can provide one or more control actions to the distributed system based on the evaluated one or more results of detecting one or more anomalies in the distributed trajectory.

[0171] The method 1100 can be performed by the control entity 230. The operations 1101-1103 can be performed, for example, by the at least one processor 232 and the memory 234. Other features of the method 1100 stem directly from the function and parameters of the control entity 230, and are thus not repeated here. The method 1100 can be performed by a computer program.

[0172] The functions described herein can be executed by one or more computer program product components (e.g., software components). Alternatively, or additionally, the functions described herein can be executed by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), program-specific integrated circuits (ASICs), program-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), and graphics processing units (GPUs).

[0173] Any ranges or device values given herein can be extended or modified without losing the intended effect. Also, any embodiment can be combined with another embodiment, unless explicitly not allowed.

[0174] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example implementations of implementing the claims, and other equivalent features and acts are intended to be encompassed by the scope of the claims.

[0175] It should be understood that the advantages and benefits described above can relate to one embodiment or can be realized with multiple embodiments. The embodiments are not limited to resolving any or all of the problems described and are not limited to having any or all of the advantages described. It should also be understood that the reference to 'an' item can mean one or more of those items.

[0176] The steps of the methods described herein can be performed in any suitable order, or simultaneously where appropriate. Furthermore, individual blocks can be deleted from any of the methods without departing from the spirit and scope of the subject matter described herein. Aspects of any of the embodiments described above can be combined with aspects of any of the other embodiments described to form further embodiments without affecting the intended effect.

[0177] The term 'comprising', as used herein, is intended to include the related method, block or element, but such block or element does not include exclusive lists, and the method or apparatus can include additional blocks or elements.

[0178] It should be understood that the above description is provided only as an example and that various modifications can be made by those skilled in the art. The above specification, examples and data provide complete description of the structure and use of exemplary embodiments. Although various embodiments have been described above with some degree of particularity, those skilled in the art could make various changes to the embodiments disclosed, without departing from the scope of the present description.< / spec> < / mask> < / mask> < / stop> < / start> < / stop> < / start> < / mask> ​< / mask>

Claims

1. A device (210) for evaluating distributed trajectories originating from a distributed system (110), characterized in that, Used for: Monitoring data is received from a monitoring entity (220) that monitors the execution of functions in the distributed system (110), the monitoring data including the distributed trajectory, the distributed trajectory including a sequence of span records for the function being executed; The distributed trajectory is preprocessed, which includes adding a start span at the beginning of the distributed trajectory and a stop span at the end of the distributed trajectory to indicate the length of the distributed trajectory; replacing randomly selected spans in the distributed trajectory with masked spans; encoding the distributed trajectory into multiple span-specific sequences and grouping the spans of the distributed trajectory based on the function description of the spans; and filling the distributed trajectory with one or more filler spans to a predefined maximum number of spans. Each sequence includes a timestamp of the span, an identifier of the trajectory to which the span belongs, and a function description of the span. The received monitoring data is analyzed to detect one or more anomalies in the distributed trajectory, the one or more anomalies being related to patterns associated with abnormal execution of the function being executed; Provide the control entity (230) with one or more results of detecting the one or more anomalies for evaluation, wherein The detection of the one or more anomalies is performed by applying a deep learning process based on stacked self-attention to generate a model to determine whether the distributed trajectory involves anomalies. Each self-attention mechanism uses a different random projection feed of data. The model uses all other spans as context indicators of the spans most likely to appear at the shielded position within the trajectory, and the other spans are unshielded spans.

2. The device (210) according to claim 1, characterized in that, The grouping of the span of the distributed trajectory based on the function description of the span includes: grouping the span of the distributed trajectory based on the function description of the span using the Drain method.

3. The device (210) according to claim 1 or 2, characterized in that, It is also used to generate anomaly scores for the distributed trajectories by applying a self-attention-based deep learning process, indicating the confidence level that the distributed trajectories are correctly classified as anomalous.

4. The device (210) according to claim 3, characterized in that, It is also used to determine that the distributed trajectory involves an anomaly in response to the anomaly score exceeding a score threshold.

5. A method (900) for evaluating distributed trajectories originating from a distributed system, characterized in that, The method (900) include: The device for evaluating the distributed trajectory receives (901) monitoring data from a monitoring entity that monitors the execution of functions in a distributed system, the monitoring data including the distributed trajectory, the distributed trajectory including a sequence of span records for the function being executed; The device preprocesses the distributed trajectory, the preprocessing including adding a start span at the beginning of the distributed trajectory and a stop span at the end of the distributed trajectory to indicate the length of the distributed trajectory, replacing randomly selected spans in the distributed trajectory with masked spans, encoding the distributed trajectory into multiple span-specific sequences and grouping the spans of the distributed trajectory based on the function description of the spans, and filling the distributed trajectory with one or more filler spans to a predefined maximum number of spans. Each sequence includes a timestamp of the span, an identifier of the trajectory to which the span belongs, and a function description of the span. The device analyzes (907) the received monitoring data to detect one or more anomalies in the distributed trajectory, the one or more anomalies relating to patterns associated with abnormal execution of the function being executed; The device provides (908) one or more results of detecting the one or more anomalies to the control entity for evaluation, wherein The detection of the one or more anomalies is performed by applying a deep learning process based on stacked self-attention to generate a model to determine whether the distributed trajectory involves anomalies. Each self-attention mechanism uses a different random projection feed of data. The model uses all other spans as context indicators of the spans most likely to appear at the shielded position within the trajectory, and the other spans are unshielded spans.

6. A computer program product, characterized in that, Includes program code, which, when executed on a computer, is used to perform the method according to claim 5.

7. A monitoring entity (220) for monitoring the execution of functions in a distributed system (110), characterized in that, Used for: In response to receiving a monitoring request from the distributed system (110), monitoring data related to the execution of the function in the distributed system (110) is recorded, the monitoring data including a distributed trajectory, the distributed trajectory including a span recording sequence for the function being executed; The device (210) is provided with the recorded monitoring data to evaluate the distributed trajectory after preprocessing by the device, thereby performing anomaly detection; the preprocessing includes adding a start span at the beginning of the distributed trajectory and adding a stop span at the end of the distributed trajectory to indicate the length of the distributed trajectory, replacing randomly selected spans in the distributed trajectory with masked spans, encoding the distributed trajectory into multiple span-specific sequences and grouping the spans of the distributed trajectory based on the function description of the spans, and filling the distributed trajectory with one or more filler spans to a predefined maximum number of spans, each sequence including a timestamp of the span, an identifier of the trajectory to which the span belongs, and a function description of the span.

8. The monitoring entity (220) according to claim 7, characterized in that, The monitoring data also includes at least one of the following: one or more logs for the function being executed, or one or more metrics for the function being executed.

9. A method (1000) for monitoring the execution of functions in a distributed system, characterized in that, The method (1000) include: In response to receiving a monitoring request (1001) from the distributed system, the monitoring entity records (1002) monitoring data related to the execution of the function in the distributed system, the monitoring data including a distributed trajectory, the distributed trajectory including a span record sequence for the function being executed; The monitoring entity provides the device with the recorded monitoring data (1003) to evaluate the distributed trajectory after preprocessing by the device, thereby performing anomaly detection; the preprocessing includes adding a start span at the beginning of the distributed trajectory and adding a stop span at the end of the distributed trajectory to indicate the length of the distributed trajectory, replacing randomly selected spans in the distributed trajectory with masked spans, encoding the distributed trajectory into multiple span-specific sequences and grouping the spans of the distributed trajectory based on the function description of the spans, and filling the distributed trajectory with one or more fill spans to a predefined maximum number of spans, each sequence including a timestamp of the span, an identifier of the trajectory to which the span belongs, and a function description of the span.

10. A control entity (230) for evaluating anomaly detection results, characterized in that, Used for: Receive from a device (210) for evaluating distributed trajectories originating from a distributed system (110) one or more results of detecting one or more anomalies in the distributed trajectory after preprocessing by the device, the one or more anomalies relating to patterns associated with abnormal execution of a function being executed in the distributed system (110); the preprocessing includes adding a start span at the beginning of the distributed trajectory and adding a stop span at the end of the distributed trajectory to indicate the length of the distributed trajectory, replacing randomly selected spans in the distributed trajectory with masked spans, encoding the distributed trajectory into multiple span-specific sequences and grouping the spans of the distributed trajectory based on the function description of the spans, and filling the distributed trajectory with one or more filler spans to a predefined maximum number of spans, each sequence including a timestamp of the span, an identifier of the trajectory to which the span belongs, and a function description of the span; Evaluate one or more results of the received detection of one or more anomalies in the distributed trajectory.

11. The control entity (230) according to claim 10, characterized in that, Evaluating one or more results of the received detection of one or more anomalies in the distributed trajectory includes classifying the one or more results of the received detection of one or more anomalies in the distributed trajectory.

12. The control entity (230) according to claim 10 or 11, characterized in that, It is also used to provide one or more control actions to the distributed system (110) based on one or more results of an evaluation of the detection of one or more anomalies in the distributed trajectory.

13. A method (1100) for evaluating anomaly detection results, characterized in that, The method (1100) includes: A control entity receives (1101) one or more results from a device used to evaluate distributed trajectories originating from a distributed system, detecting one or more anomalies in the distributed trajectory after preprocessing by the device, the one or more anomalies relating to patterns associated with abnormal execution of functions being executed in the distributed system; the preprocessing includes adding a start span at the beginning of the distributed trajectory and a stop span at the end of the distributed trajectory to indicate the length of the distributed trajectory, replacing randomly selected spans in the distributed trajectory with masked spans, encoding the distributed trajectory into multiple span-specific sequences and grouping the spans of the distributed trajectory based on the function description of the spans, and filling the distributed trajectory with one or more filler spans to a predefined maximum number of spans, each sequence including a timestamp of the span, an identifier of the trajectory to which the span belongs, and a function description of the span; The control entity evaluates (1102) one or more results of the received detection of one or more anomalies in the distributed trajectory.

Images