Adaptive Event Aggregation

Abstract

The invention discloses an adaptive aggregation. An application network is monitored using multiple agents. Adaptive event aggregation is performed to determine the retention value of the aggregation dimension. A report of the application network is generated based on the aggregation dimension.

Description

[0001] This application is a divisional application of Chinese patent application No. 201811276066.8 filed on October 30, 2018.

[0002] Cross-references to other applications

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 579,045, filed on October 30, 2017, entitled “ADAPTIVE EVENT AGGREGATION,” which is incorporated herein by reference for all purposes. Background Art

[0004] An application network includes at least one server computer coupled to multiple client computers, typically communicating via at least one API (application programming interface). The popularity of application networks may result in tens of thousands of API calls, which represent client-to-server calls. For monitoring, troubleshooting, and / or analysis purposes, the number of API calls may be very large. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0006] Figure 1 is a functional diagram illustrating a programmed computer / server system for adaptive event aggregation according to some embodiments.

[0007] Figure 2A is a diagram that visualizes the application network.

[0008] Figure 2B A sample shape of an application network is shown.

[0009] Figure 3 The proxy and topology servers are shown.

[0010] Figure 4 is an illustration of a sequence diagram for adaptive event aggregation.

[0011] Figure 5 is a flow chart illustrating an embodiment of a process for adaptive event aggregation.

[0012] Figure 6 is a flow chart illustrating an embodiment of a process for aggregation processing. DETAILED DESCRIPTION

[0013] The present invention may be implemented in a variety of ways, including as a process; an apparatus; a system; a composition of matter; a computer program product contained on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on and / or provided by a memory coupled to the processor. In this specification, these implementations or any other form that the present invention may take may be referred to as techniques. In general, the order of the steps of the disclosed processes may be changed within the scope of the present invention. Unless otherwise stated, a component (such as a processor or memory described as being configured to perform a task) may be implemented as a general component temporarily configured to perform a task at a given time or as a specific component manufactured to perform a task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data (such as computer program instructions).

[0014] A detailed description of one or more embodiments of the present invention and accompanying drawings illustrating the principles of the present invention are provided below. The present invention is described in conjunction with these embodiments, but the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications and equivalents. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. These details are provided for illustrative purposes, and the present invention can be practiced according to the claims without some or all of these specific details. For the sake of clarity, technical materials known in the technical field related to the present invention are not described in detail, so as not to unnecessarily obscure the present invention.

[0015] Adaptive event aggregation is disclosed. In one embodiment, a lightweight agent monitors network / API communications that only requires small memory / processor resources on the host to capture relevant metrics. No prior knowledge is required in advance of the data flow, for example, whether the metric to be captured is between a server and a server with relatively few API calls, or whether the metric to be captured is between a server and a client with a large number of API calls. In either case, the relevant metrics can be captured through completely lossless event reporting without statistical sampling. Inline, real-time learning of different computing environments determines which dimensions are relevant for such lossless reporting, and which dimensions may be lossy and / or aggregated / collapsed in the case of a large number of API calls.

[0016] In contrast, traditionally, there is no a priori knowledge about whether there are a large number of API calls, and sampling is used for event monitoring, troubleshooting and / or analysis. Another traditional approach is to capture all API calls for offline and / or post-processing, which can be expensive in terms of computing / memory / time / network / financial resources at large scale.

[0017] A runtime application network visualizer is disclosed. The visualizer displays inter-application API communications between software application instances. In the runtime application network visualization, each application instance can be represented as a graph node. Graph edges can be generated by deriving unique combinations of source information and destination information from API call events captured from one application instance to another application instance. The number of API call events can be a single digit to tens of thousands per minute per application instance. Deriving edges using a large number of API call events per application instance may consume a large amount of computing resources.

[0018] A system for aggregating API call events to reduce the amount of computing resources used in edge generation is disclosed. The system adaptively learns and saves important information. In one embodiment, lightweight agents monitor network / API communications and / or monitor events. These lightweight agents have sufficient resources in terms of memory, processor and / or network capacity. Compared to traditional statistical sampling, these agents capture relevant metrics using lossless event reporting. These agents perform inline, real-time learning for different computing environments to determine which dimensions are relevant to such lossless reporting and which dimensions may be lossy and / or aggregated / degraded.

[0019] Figure 1 is a functional diagram illustrating a computer / server system programmed for adaptive event aggregation according to some embodiments. As shown in the figure, Figure 1 A functional diagram of a general purpose computer system programmed to provide adaptive event aggregation according to some embodiments is provided. It will be apparent that other computer system architectures and configurations may be used for adaptive event aggregation.

[0020] The computer system 100, including various subsystems as described below, includes at least one microprocessor subsystem, also referred to as a processor or central processing unit ("CPU") 102. For example, the processor 102 may be implemented by a single-chip processor or multiple cores and / or processors. In some embodiments, the processor 102 is a general-purpose digital processor that controls the operation of the computer system 100. Using instructions retrieved from the memory 110, the processor 102 controls the receipt and manipulation of input data, as well as the output and display of data on output devices, such as a display and a graphics processing unit (GPU) 118.

[0021] The processor 102 is bidirectionally coupled to a memory 110, which may include a first main memory, typically a random access memory ("RAM"), and a second main storage area, typically a read-only memory ("ROM"). As is known in the art, the main memory may be used as a general storage area and a high-speed scratch pad memory, and may also be used to store input data and processed data. In addition to other data and instructions for processes operating on the processor 102, the main memory may also store programming instructions and data in the form of data objects and text objects. Also as is known in the art, the main memory typically includes basic operating instructions, program code, data, and objects, such as programming instructions, used by the processor 102 to perform its functions. For example, the main storage device 110 may include any suitable computer-readable storage medium described below, depending on, for example, whether data access needs to be bidirectional or unidirectional. For example, the processor 102 may also directly and very quickly retrieve frequently needed data and store it in a cache memory (not shown). The processor 102 may also include a coprocessor (not shown) as an additional processing component to assist the processor and / or the memory 110.

[0022] The removable mass storage device 112 provides additional data storage capacity for the computer system 100 and can be coupled to the processor 102 bidirectionally (read / write) or unidirectionally (read-only). For example, the memory 112 may also include computer-readable media such as flash memory, portable mass storage devices, holographic storage devices, magnetic devices, magneto-optical devices, optical devices, and other storage devices. The fixed mass storage 120 can also provide additional data storage capacity, for example. An example of the mass storage 120 is an eMMC or microSD device. In one embodiment, the mass storage 120 is a solid-state drive connected via the bus 114. The mass storage 112, 120 typically stores additional programming instructions, data, etc. that are not typically actively used by the processor 102. It should be understood that if desired, the information retained in the mass storage 112, 120 can be incorporated in a standard manner as part of the main memory 110 (e.g., RAM), as virtual memory.

[0023] In addition to providing access to the storage subsystem to the processor 102, the bus 114 can also be used to provide access to other subsystems and devices. As shown, these can include a display monitor 118, a communication interface 116, a touch (or physical) keyboard 104, and one or more auxiliary input / output devices 106 (including an audio interface, a sound card, a microphone, an audio port, an audio recording device, an audio card, a speaker, a touch (or click) device, and / or other subsystems as needed). In addition to a touch screen and / or a capacitive touch interface, the auxiliary device 106 can be a mouse, a stylus, a trackball, or a tablet computer and can be used to interact with a graphical user interface.

[0024] As shown, communication interface 116 allows processor 102 to be coupled to another computer, computer network or telecommunication network using network connection. For example, through communication interface 116, processor 102 can receive information, such as data object or program instruction, from another network, or output information to another network in the process of executing method / processing step. Information is usually represented as a series of instructions to be executed on processor, which can be received from another network and output to another network. Interface card or similar device and by suitable software such as executed / implemented on processor 102 can be used to connect computer system 100 to external network and transmit data according to standard protocol. For example, various processing embodiments disclosed herein can be executed on processor 102, or can be executed on a network (such as the Internet, intranet or local area network) in conjunction with a remote processor of a part of shared processing. Throughout the specification, "network" refers to any interconnection between computer components, including the Internet, Bluetooth, WiFi, 3G, 4G, 4GLTE, GSM, Ethernet, intranet, local area network ("LAN"), home area network ("HAN"), serial connection, parallel connection, wide area network ("WAN"), Fibre Channel, PCI / PCI-X, AGP, VLbus, PCI Express, Expresscard, Infiniband, ACCESS.bus, wireless LAN, home PNA, optical fiber, G.hn, infrared network, satellite network, microwave network, cellular network, virtual private network ("VPN"), universal serial bus ("USB"), FireWire, serial ATA, 1-Wire, UNI / O, or any form of connecting homogeneous, heterogeneous systems and / or groups of systems together. Additional mass storage devices (not shown) may also be connected to processor 102 via communication interface 116.

[0025] An auxiliary I / O device interface (not shown) may be used in conjunction with the computer system 100. The auxiliary I / O device interface may include common and custom interfaces that allow the processor 102 to send data and, more typically, receive data from other devices such as microphones, touch-sensitive displays, sensor readers, tape readers, voice or handwriting recognizers, biometric readers, cameras, portable mass storage devices, and other computers.

[0026] In addition, various embodiments disclosed herein also relate to a computer storage product with a computer-readable medium, which includes program codes for performing various computer-implemented operations. A computer-readable medium is any data storage device that can store data that can be read by a computer system thereafter. Examples of computer-readable media include, but are not limited to, all of the above media: flash media, such as NAND flash, eMMC, SD, compact flash; magnetic media, such as hard disks, floppy disks, and tapes; optical media, such as CD-ROM disks; optical magnetic media, such as optical disks; and specially configured hardware devices, such as application-specific integrated circuits ("ASICs"), programmable logic devices ("PLDs"), and ROM and RAM devices. Examples of program codes include, for example, machine code generated by a compiler, or files containing higher-level code, such as scripts that can be executed using an interpreter.

[0027] Figure 1 The computer / server system shown in is only an example of a computer system suitable for use with various embodiments disclosed herein. Other computer systems suitable for such use may include additional or fewer subsystems. In addition, bus 114 illustrates any interconnection scheme for linking subsystems. Other computer architectures with different subsystem configurations may also be used.

[0028] Figure 2A It is a diagram that visualizes an application network. The runtime application network described in this article is a set of software application instances that communicate with each other via a computer network through their APIs. An application instance can be any executed software program, such as an IOT (Internet of Things) sensor, a software service, a database server, a desktop client browser, or an embedded software instance in a mobile application.

[0029] In one embodiment, for monitoring, troubleshooting, and analysis purposes, communications between application instances are visualized through their APIs, such as Figure 2A This visualization can be in the form of a directed graph. To visualize the runtime application network, each application instance can be represented as a graph node, e.g. Figure 2AThere are 11 nodes in the graph labeled "Mule" (202), "Twitter" (204), "Workday" (206), etc. The nodes can be found in Figure 2A are named the same in order to indicate instances of the same application.

[0030] Graph edges can be derived by analyzing API call events between application instances. The direction of the edge can indicate the direction of the API call from one application instance to another application instance, for example, Figure 2A Graph edge (208) pointing from “Mule” to “Twitter” in Figure 2. The number of API events observed by each node can be highly variable. In the case of server-to-server, tens of thousands of API events may produce only a few edges. On the other hand, in the case of client-to-server calls, tens of thousands of API calls may produce thousands of edges.

[0031] Figure 2B shows a sample shape of an application network. Figure 2B As shown, "Service G" (252) can see thousands of clients making API calls to servers (254a...254z). At the same time and in the same terrain, there may be a small number of servers that communicate frequently, such as between "Service F" (256) and "Service A" (258), in which case the API calls to each server can be carefully tracked.

[0032] Tracking a large number of API call events (eg, between (252) and (254a...254z)) may be computationally expensive. Techniques are disclosed for inline aggregating API call events to reduce and limit the amount of data used to generate graph edges.

[0033] Figure 3 The agent and topology server are shown. The runtime application network can be composed of application instances. In one embodiment, a lightweight network visualizer agent is installed and run in each application instance, such as Figure 3 302e), such as the agent (302b) in "Service C". The agent can collect incoming API calls to the application instance and outgoing API calls from the application instance. Typically, the agent can only run on a subset of the application instances because not all application instances may allow the agent to be installed.

[0034] Agents may run independently of each other. An agent may not know a priori if other agents are running elsewhere in the application network. To generate a complete picture of the entire application network, the communication information collected by each agent may be sent to a network topology server (304). The topology server combines the information from each instance to form a global view of the runtime application network.

[0035] In one embodiment, the agent can collect information about various service nodes in the network, such as agents (302a...302e), but not about database nodes (306a, 306b) or client browser nodes (308a...308z). The collected information is sent to the topology server (304).

[0036] In a runtime environment, the agent (302a...302e) can continuously send data to the topology server (304) because the observed API call information changes over time. For example, if the agent sends the collected communication information to the topology server (304) every minute, in the first minute, there may be five different clients (308a...308z) making one API call to "Server G". In the next minute, there may be only one client (308a) making many API calls to "Server G". As a result, the API call event data sent by the agent to the topology server (304) is highly variable in quantity and in the uniqueness of the event content.

[0037] The invention discloses deriving edges using time series events. An edge in a runtime application network graph represents an API interaction from one application instance to another application instance. An edge may contain information about the application instance that initiated the API call, the application instance that received the API call, and response information of the API call (such as start time, call duration, response code, response size, etc.). An edge may represent a single API call or multiple API calls with the same characteristics (such as the same source, destination, method, response code, etc.).

[0038] A sample capture of the API call data is given in this table:

[0039]

[0040] At 12:32:42.003 on March 2, 2018, a call was made from IP address: port 10.0.23.123:8374 to IP address: port 10.2.34.212:80, with a response, in this case an HTTP response, with code 200 (OK), size 2314 bytes, and duration 34ms. Since API calls in runtime occur only within a time interval, as represented by "start time" and "duration" as described above, each API call can be represented as a time series event. As a time series event, the various information contained in the API call can be grouped into at least one of the following: timestamp, context dimension, and event data.

[0041] API calls can be represented as time series events as given in the following table:

[0042]

[0043]

[0044] In this time series event table:

[0045] Timestamp - is the timestamp when the API call starts;

[0046] Contextual data - a set of label-value pairs describing the source and destination information about the API call. These data points are non-additive; and

[0047] Event data - a set of label-value pairs. There are two types of event data, additive metric data and non-additive data.

[0048] Additional metric data is a tagged value, where the value is a numeric metric characterized by the fact that the numeric metric can be combined from many events to form a sum. These sums can then be used to derive statistical measures, such as averages. Non-additive data is a tagged value, where the value can be categorical, ordered, free-form text, or a number that does not add value or sense to sum across events.

[0049] The content of time series events includes:

[0050] Event timestamp: event timestamp;

[0051] Dimensions: one or more dimension values ​​of non-additional contextual data and non-additional event data; and

[0052] Metrics (optional): Zero or more additional metrics for the event data.

[0053] Using the above structure, you can use structured objects to represent API calls:

[0054]

[0055]

[0056] Aggregating time series events is disclosed. There are improvements over traditional techniques in aggregating time series events to reduce the amount of storage and processing required. In order to aggregate time series events, it must be determined which events are "similar". Event similarity is defined in this article as events that have the same dimension values ​​for all dimensions or a subset of dimensions in the event. Using the example above, a user can declare that two events are similar when their "source.ip" is the same. Or two events may be similar when their "source.ip", "destination.ip", and "client-id" are the same. Or two events may be similar when all dimensions have the same values.

[0057] After the user defines the event similarity, an aggregation time window can also be specified to degenerate similar events within the time window into a single aggregate event. For example, the aggregation time window can be 1 minute, 5 minutes, 1 hour, 1 day, etc.

[0058] When multiple events are degraded into a single aggregate event, the metrics from the individual events are combined to form the aggregated metric. Any metric can be chosen for aggregation, for example, response size or duration can be combined for multiple events. Thus, an aggregate event consists of:

[0059] Aggregation time interval: the time starting point and duration of the aggregation event;

[0060] Aggregate dimensions: all or a subset of event dimension values ​​for non-additional contextual data and non-additional event data; and

[0061] Aggregated metrics: Metric values ​​that are combined and aggregated from individual event data.

[0062] Aggregate dimensions as defined can be a subset of event dimensions, event dimensions that are not part of aggregate dimensions are called degenerate dimensions. An example of a degenerate dimension is a response code. Aggregate events do not contain any information about degenerate dimensions.

[0063] The result of time series event aggregation produces one aggregated event for each unique combination of time interval and aggregation dimension value. The number of aggregated events may be smaller than the number of original events.

[0064] The use of aggregate events for graph edges is disclosed. In a runtime application network, a time series event may represent a single API call between two applications. These API calls may be used to derive graph edges representing interactions between applications in the network. Each edge may represent a unique combination of a source and a destination of an API call. Repeated calls from the same source to the same destination generate the same edge in the graph.

[0065] A set of examples of API call events between application instances are given in the table:

[0066]

[0067] Using the five events above, an application network with three different application instances at 10.0.23.123, 10.0.4.211, and 10.2.34.212 can be derived. Node 10.0.23.123 is communicating with 10.2.34.212. Node 10.0.4.211 is communicating with 10.2.34.212, but node 10.0.23.123 is not communicating with 10.0.4.211.

[0068] Instead of using raw events to derive edges, you can use aggregated events to generate graph edges. You can first aggregate the five events using a one-minute time window with the aggregation dimensions of source IP and destination IP. The aggregation process will then produce the following results:

[0069]

[0070] As shown above, five events are reduced to three aggregate events. From the three aggregate events, the same application network visualization can be exported. This example shows that if the system first aggregates the raw events into aggregate events, it may be possible to reduce the memory and computation used to export the application network. This is especially effective for server-to-server communication, such as in Figure 3 Between "Service A" and "Service F" in .

[0071] This article describes high cardinality dimensions. During aggregation of time series events, the aggregate dimension can contain many unique values. The number of unique values ​​for each dimension is the cardinality of the dimension. When a dimension has high cardinality, the dimension is likely to have a large number of unique values. Therefore, the cardinality of the aggregate dimension determines the extent of data reduction during aggregation.

[0072] For example, if the original event is represented by the following table:

[0073]

[0074] Using aggregation based on source IP and destination IP, the aggregation results are represented by the following table:

[0075] Aggregate event timestamp Source IP Destination IP Event Count Response duration average Response size average 2018-03-02 12:32:00.000 10.0.23.123 10.2.34.212 1 34 2314 2018-03-02 12:33:00.000 10.0.23.21 10.2.34.212 1 45 1734 2018-03-02 12:33:00.000 10.0.17.100 10.2.34.212 1 39 2314 2018-03-02 12:33:00.000 10.0.4.211 10.2.34.212 1 49 1934 2018-03-02 12:33:00.000 10.0.4.123 10.2.34.212 1 21 42

[0076] The above example is an example of high cardinality within the source IP dimension, which represents an almost worst-case scenario where no reduction in aggregation is done. This can occur in client-to-server communications, which is also an instance where a large number of API calls can overwhelm traditional monitoring, troubleshooting, and / or analysis.

[0077] Handling high cardinality dimensions is disclosed. High cardinality dimensions in runtime application networks may occur on several event dimensions of API events, such as "source IP", "API URI", "client ID", etc.

[0078] For example, an application network may include an application server ( 308a . . . 308z ) that receives requests from desktop or mobile clients ( 308a . . . 308z ). Figure 3 302e in the example above). From the application server's perspective, there are a large number of different client IP addresses. This could mean many edges in a client-to-server application, perhaps on the order of 10,000 edges.

[0079] However, if an application server ("Service F" associated with 302d) is designed for other application servers ("Service A" associated with 302a, "Service C" associated with 302b, and "Service G" associated with 302e), then the application server may only see a few different server IP addresses. This may mean that there are very few edges in the server-to-server application, perhaps about ten edges.

[0080] The results are for the same set of dimensions, such as source IP, which can be high cardinality for client-to-server type applications, or low cardinality for server-to-server type applications.

[0081] One challenge in building a runtime application network through runtime API call events is that there may be no a priori and / or upfront information about the type of application server usage: serving clients or serving other servers. Alternatively, the usage of application servers may vary over time. Therefore, it may be difficult to automatically handle high-cardinality dimensions during event aggregation using traditional techniques, such as the following common strategies for handling potentially high-cardinality dimensions:

[0082]

[0083]

[0084] A new technique is disclosed that outperforms these traditional approaches. The technique is referred to herein as "Adaptive Event Aggregation" and:

[0085] 1. Provides low resource utilization: it does not retain all raw events to produce aggregations;

[0086] 2. Automatically select values ​​from high-cardinality dimensions to preserve without prior knowledge; and

[0087] 3. Generate accurate metrics for retained values.

[0088] Figure 4 is an illustration of a sequence diagram of adaptive event aggregation. In one embodiment, adaptive event aggregation sends the original API call event to the aggregation system. After a specified amount of time, the caller can trigger the aggregation system to return all aggregate events at that point. The aggregation system uses a set of aggregation processors. As described herein, an "aggregation processor" is a software and / or software-hardware element dedicated to processing data, which can be, for example, object / binary code. In one embodiment, the aggregation processor is part of each lightweight agent (302a...302e). Each aggregation processor works through the following internal stages.

[0089] 1. Phase 1: Sample data collection;

[0090] 2. Stage 2: Learning;

[0091] 3. Stage 3: Inline aggregation; and

[0092] 4. Stage 4: Extract summary results and reset to stage 1.

[0093] Configuration may occur at each aggregation processor (302a...302e) and / or within the topology server (304). The following configurable parameters are provided to the adaptive aggregation system:

[0094] 1. Lossless, lossy, and degraded aggregation dimensions;

[0095] 2. Aggregated and unused metrics;

[0096] 3. Learning buffer size, learning algorithm, retaining value count; and

[0097] 4. Output size.

[0098] In one embodiment, configurable parameters control the behavior of aggregation. Each event dimension can be configured as one of the following types:

[0099] 1. Lossless aggregation dimensions - all or a subset of event dimensions, retaining their unique dimension values ​​during aggregation;

[0100] 2. Lossy aggregation dimensions - all or a subset of event dimensions whose unique dimension values ​​are partially preserved during aggregation. The system can preserve as much information as possible about these dimension values ​​using a limited / bounded amount of memory and computational resources; or

[0101] 3. Degenerate dimensions - all or a subset of event dimensions whose dimension values ​​are not preserved during aggregation.

[0102] Event metrics can be specified as one of the following types:

[0103] 1. Aggregate metrics - event metrics that are aggregated and output; or

[0104] 2. Unused metrics - event metrics that are ignored during aggregation and that may not appear in the output.

[0105] Additionally, users of the system can specify:

[0106] 1. Learning buffer size - the number of unique keywords in the learning buffer, as further described in Phase 1 below;

[0107] 2. Learning algorithms - plug-in / pluggable algorithms for performing (adaptive) learning;

[0108] 3. Output size - size guidelines that specify the maximum number of outputs produced for an aggregation regardless of the number of input events; and / or

[0109] 4. Retained value count - the number of significant values ​​retained for each lossy aggregation dimension.

[0110] Examples of sample code for configuring a data aggregation system include:

[0111]

[0112]

[0113] Lossless dimensions. When the user configures one or more lossless dimensions, multiple independent adaptive aggregation processors (302a...302e) can be used. If the user does not specify a lossless dimension, a single adaptive aggregation processor can be used. Given an event, the system can first create a lookup key using the value from the lossless dimension. The lookup key can be used to locate an existing aggregation processor in the processor map. If such a processor is not found, a new aggregation processor can be created / instantiated and added to the processor map for the lookup key. The event can then be sent to the aggregation processor. All events in the aggregation processor have the same value for its lossless dimension. Each aggregation processor independently performs 4-stage adaptive aggregation on the lossy dimension.

[0114] For example, the API event sequence is represented in this table:

[0115] Event timestamp Source IP Destination IP Response Codes Response duration Response size 2018-03-02 12:32:42.003 10.0.23.123 10.2.34.212 200-OK 34 2314 2018-03-02 12:33:13.243 10.0.23.123 10.2.34.212 400 - Bad Request 22 42 2018-03-02 12:33:38.624 10.0.23.123 10.2.34.212 200-OK 39 2314 2018-03-02 12:33:38.523 10.0.4.211 10.2.34.13 200-OK 49 1934 2018-03-02 12:33:51.028 10.0.23.123 10.2.34.212 400 - Bad Request 21 42

[0116] If the lossless dimensions are "Destination IP" and "Response Code", then three separate aggregation processors are created / instantiated, each processing events with the same lossless dimension values, as represented in each of these three tables:

[0117] Aggregation Processor 1

[0118]

[0119] Aggregation Processor 2

[0120]

[0121]

[0122] Aggregate Processor 3

[0123]

[0124] Phase 1 - Sample data collection. Each aggregation processor goes through up to four internal phases, such as Figure 4 As shown. During stage 1 (402), each event received is stored into a keyword-value structure in memory. The keyword-value structure is referred to as a "learning buffer" in this article. Each keyword is a unique combination of aggregation dimension values, and the associated value is an aggregated event that holds a set of aggregated metrics for events with the same keyword. For the example in the aggregation processor 1 table above, the lookup keyword is "Destination IP = 10.2.34.212, Response Code = 200-OK." The size of the learning buffer can not exceed the learning buffer size configured by the user.

[0125] In one embodiment, a pre-processing phase is used before each event is received, prior to phase 1. For example, one pre-processing phase may be to take the IP address and reduce it to subdomains, such as 24.1.23.237 is reduced to 24.1.23.0, and 24.1.23.123 is also reduced to 24.1.23.0.

[0126] When a new event comes in, if the key from the unique combination of dimension values ​​already exists in the key-value structure, the value is updated to include the metric value of the new event. If the key does not exist, the system checks to see if the current size of the key-value is less than the learning buffer size. If so, a new entry is added to the key-value of the new event.

[0127] If the current size of the keyword-value structure is the learning buffer size, so that the learning buffer is full, the new event is not inserted into the keyword-value structure. Instead, phase 2 is triggered and the new event can be processed in phase 3, as described in detail below.

[0128] For example, the learning buffer collects events where “Source IP” and “Destination IP” are lossy dimensions for an aggregator process, with response code as the lossless dimension. Since all events processed by this processor may have the same lossless dimension values, the source IP and destination IP values ​​may vary. But in phase 1, the learning phase, before the learning buffer is full, all unique combinations of source IP and destination IP are maintained despite the lossiness. The timestamp of each aggregated event in the learning buffer shows the arrival time of each first event that has that unique combination of source IP and destination IP. The learning buffer in this example is represented by this table:

[0129] Learning Buffer Example

[0130]

[0131] Phase 2 - Learning. As described above, Phase 2 (404) is triggered when the learning buffer is at the learning buffer size limit, and new events arrive that do not fit within the existing set of unique keywords. Phase 2 analyzes the aggregated events in the learning buffer using a specified learning algorithm. Many algorithms can be provided, from traditional sampling to adaptive algorithms. In this section, the adaptive algorithm will be described in more detail.

[0132] In adaptive learning, the system attempts to identify the most important dimension values ​​for each lossy dimension. Once these dimension values ​​are identified, the metrics around these values ​​are tracked. To identify the most important dimension values, the learning step iterates over all the aggregated events in the learning buffer and generates frequency counts for each unique value of the lossy dimension. If more than one lossy dimension is defined, dimension value counts are generated for all lossy dimensions.

[0133] Using the learned buffer example table from stage 1, with lossy dimensions of source IP and destination IP, the following value counts produce the following histogram:

[0134] Source IP histogram:

[0135] Source IP Event Count 10.0.23.123 3 10.0.23.122 2 10.0.6.56 1

[0136] Destination IP Histogram:

[0137] Destination IP Event Count 10.2.34.212 5 10.2.34.210 1

[0138] As shown above, the value count for each lossy aggregation dimension is a frequency histogram of that dimension. Using this histogram, the system picks the value to keep. The selected value is referred to as the "retained value" in this article. The system uses multiple factors to determine the retained value from the total number of unique values, the frequency of event counts, the deviation of the frequency, and the shape of the histogram.

[0139] For example, if the total number of unique values ​​is less than the holdout count, then all values ​​can be added as holdouts. If the histogram shows a high deviation between high-frequency values ​​and other values, then only the high-frequency values ​​are held. If the histogram shows many low-frequency values, then no values ​​may be held. The result of the analysis can be the selection of zero or more holdout values. However, the number of holdout values ​​cannot exceed the holdout count configuration property specified by the user.

[0140] After learning the analysis, the processor enters stage 3.

[0141] Phase 3 - Inline Aggregation. After the learning phase and at the start of Phase 3 (406), the aggregation processor now has the following data points:

[0142] 1. Maximum learning buffer; and

[0143] 2. For each lossy aggregation dimension, a set of zero or more retained values.

[0144] The processor creates an auxiliary buffer referred to herein as an "overflow buffer". The overflow buffer is similar in structure to the learning buffer. The overflow buffer is used to store additional aggregate events with their aggregated metrics.

[0145] At this point, the processor resumes processing incoming events. The first event to be processed in phase 3 is the event that triggered phase 2. For each event processed in phase 3, the following will be applied:

[0146] 1. Create a keyword using a combination of values ​​from the lossy dimension of the event;

[0147] 2. Check if the keyword is in the learning buffer. If found, aggregate the metric into the existing learning buffer;

[0148] 3. If the keyword is not found in the learning buffer, then for each lossy dimension, check if the value of that lossy dimension is one of the retained values. If the value is one of the retained values, the value remains unchanged. If the value is not one of the retained values, the value in the event is updated with a special tag " <other>”.

[0149] 4. Create the overflow key from the lossy dimension using the updated value from step 3 above.

[0150] 5. Use the overflow keyword to find the corresponding aggregate event in the overflow buffer. If an entry is found in the aggregate buffer, the event's metrics are aggregated into the found entry. If no entry is found, a new aggregate event entry is created for the overflow keyword and the event metrics are added to the new aggregate event.

[0151] In stage 3 (406), the size of the overflow buffer has a known upper limit. The maximum number of entries in the overflow buffer is the product of

[0152]

[0153] Where m i is the dimension of each retained value. For example, if the processor chooses to retain the following values ​​for the source IP and destination IP dimensions, using the learning buffer example shown above for stage 1 (402) and stage 2 (404):

[0154] Source IP Event Count reserve 10.0.23.123 3 yes 10.0.23.122 2 yes 10.0.6.56 1 no

[0155] as well as

[0156] Destination IP Event Count reserve 10.2.34.212 5 yes 10.2.34.210 1 no

[0157] Then the maximum number of entries in the overflow buffer is 3×2=6.

[0158] Continuing with the example, assume that the following additional events are received at stage 3 (406):

[0159] Event timestamp Source IP Destination IP Response Codes Response duration Response size 2018-03-02 12:33:38.103 10.0.23.123 10.2.34.212 200-OK 30 2133 2018-03-02 12:33:38.624 10.0.23.123 10.2.34.211 200-OK 39 2314 2018-03-02 12:33:42.024 10.0.6.51 10.2.34.212 200-OK 23 3111 2018-03-02 12:33:38.523 10.0.6.51 10.2.34.13 200-OK 49 1934

[0160] but:

[0161] 1. For the first event, (source=10.0.23.123, destination=10.2.34.212) is found in the learning buffer, so the learning buffer metric is updated;

[0162] 2. For the second event, (source=10.0.23.123, destination=10.2.34.211) is added to the event with (source=10.0.23.123, destination= <other>) is the overflow keyword in the overflow buffer because the original source IP and destination IP combination does not exist in the learning buffer, and the source IP value is a reserved value, but the destination IP is not;

[0163] 3. For the third event, (source=10.0.6.51, destination=10.2.34.212) is added to the event with (source= <other>, destination = 10.2.34.212) is the overflow keyword in the overflow buffer, because the original source IP and destination IP combination does not exist in the learning buffer, and the source IP value is not a reserved value, while the destination IP is a reserved value; and

[0164] 4. For the fourth event, (source=10.0.6.51, destination=10.2.34.13) is added to the event with (source= <other>,destination= <other>) in the overflow buffer of the overflow keyword because both the source IP and the destination IP are not reserved values.

[0165] The resulting learning buffer is updated as:

[0166]

[0167] And the resulting overflow buffer is updated to:

[0168]

[0169] Since both the learning buffer and the overflow buffer are bounded and / or have a maximum size, stage 3 processing saves a finite amount of memory regardless of how many additional events are processed. In one embodiment, each event processed by stage 3 is processed in a time frame that is of the order of a constant O(1) since the event is only checked against two buffers to determine how to aggregate the event.

[0170] Given a retention value, there are entries in the learning buffer and the potential overflow buffer that contain that value. Combining the metrics of all entries with the same retention value provides an accurate metric for that retention value.

[0171] Phase 4 - Extract Aggregate Results. The caller may then extract the aggregate results from the Aggregate Processor (408). When the extract is called, the Aggregate Processor may be in Phase 1 (402) or Phase 3 (406).

[0172] If the processor is still in stage 1 (402), the system checks to see if the learning buffer contains more entries than the configured output size. If the learning buffer does not contain more entries than the configured output size, the entire contents of the learning buffer are returned. If the learning buffer contains more entries than the configured output size, stage 2 learning is forced to generate a retained value, and when the aggregation processor is in stage 3 (406), the aggregated results are extracted as described below.

[0173] If the aggregation processor is in stage 3 (406) when extract is called, the processor first sorts the learning buffer by descending event count. The processor then selects the first N entries that reach the configured output size. If any entries remain in the learning buffer, each of the remaining entries is processed into the overflow buffer by checking the lossy dimension value against the retained value to generate an overflow key. Using the overflow key, all remaining aggregation events in the learning buffer are placed into the overflow buffer. After this processing, the contents of the overflow buffer are appended to the output.

[0174] Continuing with the example above, if the output size is 2, here is the output from the aggregation processor:

[0175] Learning Buffer:

[0176]

[0177] Overflowing the buffer:

[0178]

[0179]

[0180] Combined output:

[0181]

[0182] Note that the output contains exact metrics for each selected retained dimension value. For source IP 10.0.23.123, the output shows a total of five events. For source IP 10.0.23.122, the output shows a total of two events. For destination IP 10.2.34.212, the output produces a total of seven events.

[0183] When the processor is in stage 3 (406), the sum of the output size and the overflow buffer size is the size of the output. In one embodiment, after extracting the results, the processor clears its learning buffer and overflow buffer and resets back to stage 1 (402).

[0184] The Converged Processor Runtime feature provides several advantages over traditional techniques when configuring converged processors:

[0185] 1. Limited memory usage - each aggregation processor uses a fixed-size learning buffer plus an overflow buffer with a known upper limit;

[0186] 2. Streaming Aggregation - Aggregates are generated on the fly as events are processed by the processor without persisting to incoming events;

[0187] 3. Constant computation time - each event is processed using a constant amount of computation via two constant-time lookups into the learning buffer and the overflow buffer; and

[0188] 4. Exact metrics for retained values ​​- Outputs the exact metrics for all retained values ​​including the lossy dimension.

[0189] Insert / insertible stage 2 learning. Figure 4 As shown, the adaptive aggregation / processing is split into several stages. As a result, different learning algorithms can be switched using insertion in stage 2 (404) to identify which lossy dimension values ​​to keep. This provides multiple possibilities for learning:

[0190] 1. Combined learning buffer. Figure 4 As described in stage 2 (404), the learning algorithm can use frequencies within a single learning buffer to determine the values ​​to keep. Since there may be multiple learning buffers, one for each unique combination of lossless dimension values, the algorithm can also use frequencies from all learning buffers to determine which values ​​to keep spatially. Using all learning buffers simultaneously can provide a global perspective to select lossy dimension values ​​to keep;

[0191] 2. Utilizing historical retained values. Another variation of learning in stage 2 (404) is to utilize previous retained values ​​in time rather than using values ​​from the current learning buffer. By combining retained values ​​from previous detections with current values, a more stable set of retained values ​​is provided. Historical retained values ​​may be associated with decay, such that if a retained value is not further observed, it is slowly, over time, removed from consideration; and / or

[0192] 3. Use a different metric. This alternative type of learning algorithm can use a different metric (such as response time or response size) to determine retention values ​​instead of using frequency. This produces different semantics for the output. Using response time or response size can produce accurate metrics for events with high or low response times and response sizes, rather than for the most frequently occurring events. Identifying the dimension values ​​for such events may be important for certain use cases.

[0193] Additional Use Cases. Using adaptive aggregation for application network visualization edge generation, as disclosed herein but not limiting, is just one use case. The adaptive aggregation approach is a general technique applicable to any event stream with dimensions and metrics. As a result, the adaptive aggregation approach is implemented as a general library that can be embedded into many host applications and services.

[0194] For example, adaptive aggregation is used in the analytics agent, which aggregates and limits the collected metrics before sending the aggregated data to the central analytics service. This reduces the network resources consumed by the metrics and protects the analytics system from being overwhelmed by an unlimited amount of data.

[0195] The same aggregation library can also be used on the service side when it is not possible to update the proxy to include adaptive aggregation. When a service receives data from the proxy, it can first pass the data through the aggregation library to reduce and limit the amount of data before passing the aggregated events to the rest of the service.

[0196] Figure 5 is a flow chart illustrating an embodiment of a process for adaptive event aggregation. In one embodiment, Figure 5 The process of Figure 3 The system is executed in .

[0197] In step 502, a plurality of agents are used to monitor an application network. In one embodiment, the plurality of agents includes a plurality of lightweight agents.

[0198] In step 504, adaptive event aggregation is performed to determine values ​​of aggregation dimensions. In one embodiment, performing adaptive event aggregation includes determining values ​​of aggregation dimensions based at least in part on aggregation of events associated with monitoring an application network using a plurality of agents. In one embodiment, performing adaptive event aggregation includes performing inline learning and collection of event data. In one embodiment, adaptive event aggregation is performed by an aggregation processor.

[0199] In step 506, a report of the application network is generated based on the aggregated dimensions. In one embodiment, the report includes at least one of: a visualization of the application network, an event monitoring report, and a network topology report.

[0200] In one embodiment, the proxy aggregate metrics are transmitted at a reduced network bandwidth. In one embodiment, a graphical visualization of the application network is generated based on a report indicating critical events and failure events based on success thresholds.

[0201] In one embodiment, a graphical visualization of the topology of the application network is generated. The graphical visualization may include graphical cues for at least one of: name-value pairs of monitored data, context dimensions, event data, timestamps, network tuples of IP addresses, network tuples of ports, network tuples of protocols, response codes, dimension counts, number of calls between entities of the application network, size of dimensions, and response size.

[0202] Figure 6 is a flow chart illustrating an embodiment of a process for aggregation processing. In one embodiment, Figure 6 The process of Figure 3 The agent (302a...302e) in Figure 5 Part of step (504) in.

[0203] In step 602, a data set is sampled into a learning buffer. In one embodiment, at least one of the following is configurable: learning buffer size, learning algorithm, output size, and retained value count.

[0204] In step 604, learning is used to identify a retained value for the aggregated dimension in the event that the learning buffer is full. In one embodiment, adaptive learning is used to identify the retained value. In one embodiment, the aggregated dimension is configurable as at least one of: a lossless dimension, a lossy dimension, and a degenerate dimension. In one embodiment, identifying the retained value includes configuring the metric as at least one of: aggregated and unused.

[0205] In one embodiment, using learning to identify includes using at least one of: a top N technique, an adapt over time technique, a combined learning buffer technique, a history retention value technique, and a different metric technique.

[0206] In step 606, the additional events are stored in the overflow buffer using inline aggregation. In one embodiment, storing the additional events includes linear computation of the additional events including lossless measurement of the preserved values.

[0207] In step 608, the aggregated results are extracted from the learning buffer and the overflow buffer.

[0208] Although the foregoing embodiments have been described in some detail for the purpose of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways to implement the invention. The disclosed embodiments are illustrative rather than restrictive.< / other> < / other> < / other> < / other> < / other>

Claims

1. A method for adaptive event aggregation, comprising: Copying event records describing API call events between applications in the application network into the learning buffer, wherein the event record includes keywords and metrics, and wherein the event records are aggregated in the learning buffer based on an aggregation dimension indicating a retention value determined using adaptive event aggregation; When the learning buffer reaches a size limit, copying additional aggregated event records with their aggregated metrics to an overflow buffer; and A visualization of the application network is generated based on aggregated results extracted from the learning buffer and the overflow buffer.

2. The method of claim 1, further comprising: Configurable parameters are received that specify the aggregation dimension and the size limit.

3. The method of claim 1, wherein: The aggregation dimension may be lossy, lossless or degenerate.

4. The method of claim 1, further comprising: The retention value of the aggregation dimension is determined using an adaptive learning algorithm.

5. The method of claim 1, copying the additional aggregate event record to the overflow buffer further comprises: When the appended aggregate event record includes a keyword not indicated by the reserved value, the event record is updated to include a special marker before copying the appended aggregate event record to the overflow buffer.

6. The method of claim 5, further comprising: Create an overflow key using the updated event record; as well as The overflow key is used to determine whether a corresponding aggregated event exists in the overflow buffer.

7. The method of claim 1, wherein: The event record in the event log includes a timestamp, a source IP address, a destination IP address, a response code, a response duration, and a response size.

8. A system for adaptive event aggregation, comprising: Memory; and at least one processor coupled to the memory and configured to: Copying event records describing API call events between applications in the application network into the learning buffer, wherein the event record includes keywords and metrics, and wherein the event records are aggregated in the learning buffer based on an aggregation dimension indicating a retention value determined using adaptive event aggregation; When the learning buffer reaches a size limit, copying additional aggregated event records with their aggregated metrics to an overflow buffer; and A visualization of the application network is generated based on aggregated results extracted from the learning buffer and the overflow buffer.

9. The system of claim 8, wherein the at least one processor is further configured to: Configurable parameters are received that specify the aggregation dimension and the size limit.

10. The system according to claim 8, characterized in that The aggregation dimension may be lossy, lossless or degenerate.

11. The system of claim 8, wherein the at least one processor is further configured to: The retained values ​​of the aggregation dimensions are determined using an adaptive learning algorithm.

12. The system of claim 8, wherein to copy the additional aggregate event record into the overflow buffer, the at least one processor is further configured to: When the additional aggregate event record includes a keyword not indicated by the reserved value, the event record is updated to include a special marker before copying the additional aggregate event record to the overflow buffer.

13. The system of claim 12, wherein the at least one processor is further configured to: creating an overflow key using the updated event record; and The overflow key is used to determine whether a corresponding aggregated event exists in the overflow buffer.

14. The system of claim 8, wherein: The event record in the event log includes a timestamp, a source IP address, a destination IP address, a response code, a response duration, and a response size.

15. A non-transitory computer-readable device having instructions stored thereon, which, when executed by at least one computing device, cause the at least one computing device to perform operations comprising: Copying event records describing API call events between applications in the application network into the learning buffer, wherein the event record includes keywords and metrics, and in, aggregating the event records in the learning buffer based on an aggregation dimension indicative of a retention value determined using adaptive event aggregation; When the learning buffer reaches a size limit, copying additional aggregated event records with their aggregated metrics to an overflow buffer; and A visualization of the application network is generated based on aggregated results extracted from the learning buffer and the overflow buffer.

16. The non-transitory computer-readable device of claim 15, the operations further comprising: Configurable parameters are received that specify the aggregation dimension and the size limit.

17. The non-transitory computer-readable device of claim 15, wherein the aggregation dimension can be lossy, lossless, or degenerate.

18. The non-transitory computer-readable device of claim 15, the operations further comprising: The retention value of the aggregation dimension is determined using an adaptive learning algorithm.

19. The non-transitory computer-readable device of claim 15, the operations further comprising: When the appended aggregate event record includes a keyword not indicated by the reserved value, the event record is updated to include a special marker before copying the appended aggregate event record to the overflow buffer.

20. The non-transitory computer readable device of claim 19, the operations further comprising: Create an overflow key using the updated event record; as well as The overflow key is used to determine whether a corresponding aggregated event exists in the overflow buffer.

Images