Method, system and storage medium for processing data generated by instrumentation software

Abstract

An analysis system receives a data stream generated by an instance of instrumented software executing on an external system. The analysis system evaluates an expression using data values ​​of the data stream in multiple time intervals. For example, the analysis system may aggregate the data values ​​of the data stream for each time interval. The analysis system determines whether to consider a data stream in a time interval based on when the data values ​​arrive in the time interval. The analysis system determines a maximum expected latency value for each data stream being processed. The analysis system evaluates the expression using data values ​​that arrive before the maximum expected latency value. The analysis system also determines a failure threshold for the data stream. If the data value of the data stream fails to arrive before the failure threshold, the analysis system marks the data stream as dead.

Description

[0001] This application is a divisional application of the invention patent application with international application number PCT / US2016 / 014957, application date January 26, 2016, application number 201680016597.6, and invention name “Real-time processing of data streams received from instrumented software”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 109,308, filed on January 29, 2015, which is incorporated herein by reference in its entirety. Technical Field

[0004] The present disclosure relates generally to analysis of instrumented software and, more particularly, to real-time processing of data streams received from instrumented software. Background Art

[0005] Software developers monitor different aspects of the software they develop by instrumenting the code. These aspects include the performance of the software, errors encountered during the execution of the software, major events encountered during the execution of the software, the parts of the code that are being executed and the parts that are not being executed, etc. Traditional techniques for instrumenting the code include statements in the code for recording information to a record file or printing information on the screen. This type of instrumentation is suitable for simple applications, such as applications with a simple flow that executes on a single machine. However, these techniques for instrumenting software are not enough for complex applications with complex execution flows (e.g., applications distributed on multiple systems, each of which executes multiple execution processes or threads).

[0006] Processing data generated by instrumented software for distributed systems requires assimilation of the data for analysis. Assimilating and processing data sent by instrumented software executing on distributed systems is complicated by the fact that data values ​​sent by different systems at the same time may encounter different network delays and therefore even take different amounts of time to reach the system assimilating the data. Moreover, one or more systems executing the instrumented software may fail. Therefore, the system assimilating the data needs to determine whether the data value is late due to network delays or cannot arrive due to a failure of the data source. Making these determinations results in delays in processing the data value and / or inaccuracies in the results presented. As a result, traditional systems for generating reports based on software-based instrumentation are often insufficient for analyzing highly distributed systems running instrumented software. Summary of the invention

[0007] The described embodiments process data generated by instrumented software. Software developers often include code snippets for instrumented code in the software being developed. An analysis system receives a data stream generated by instrumented software executing on an external system. The analysis system performs analysis on the received data stream. The analysis system evaluates expressions using data values ​​of the data stream at multiple time intervals. For example, the analysis system can aggregate the data values ​​of the data stream for each time interval and send the results for presentation via a chart that is updated in real time.

[0008] The analysis system determines whether a data stream is considered for evaluation of an expression in the time interval based on the time of arrival of the data values ​​during the time interval. The analysis system excludes data streams whose data values ​​arrive late during the time interval. The analysis system determines a maximum expected delay value for each data stream being processed. The analysis system excludes data values ​​that fail to arrive before the maximum expected delay value during the time interval. Therefore, the analysis system evaluates the expression for the time interval without considering the data streams. The analysis system sends the evaluation results of the expression for each time interval for presentation.

[0009] In some embodiments, the analysis system determines the maximum expected delay value based on the delay of past data values ​​of the data stream. For example, the analysis system can determine the maximum expected delay value of the data stream based on a moving average of a large number of data values ​​of the data stream. The analysis system periodically (e.g., every time interval) updates the maximum expected delay value.

[0010] In some embodiments, the analysis system further determines a fault threshold for the data stream. If the data value of the data stream cannot be reached before the fault threshold of the data stream, the analysis system marks the data stream as dead. Therefore, for the expression evaluation in subsequent time intervals, the analysis system does not consider the data stream. When the next data value of the data stream is received, the analysis system marks the data stream as alive. Once the data stream is marked as alive, the analysis system begins to consider the data stream for the evaluation of the expression.

[0011] The features and advantages described in the specification are not all inclusive, and in particular, many additional features and advantages will be apparent to those of ordinary skill in the art based on the drawings, description and claims. Moreover, it should be noted that the language used in the specification is primarily selected for readability and teaching purposes, and is not selected to delineate the boundaries of the disclosed subject matter or to limit the disclosed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The disclosed embodiments have other advantages and features that will become apparent from the detailed description, the appended claims, and the accompanying drawings (or illustrations).The following is a brief description of the drawings.

[0013] Figure 1 An overall system environment for generating real-time reports based on data streams received from instrumentation software according to an embodiment is shown;

[0014] Figure 2 shows the architecture of a system for generating real-time reports based on data streams received from instrumentation software according to an embodiment;

[0015] Figure 3 shows a screenshot of a user interface displaying a real-time updated chart of data based on a data stream received by an instrumented analysis system according to an embodiment;

[0016] Figure 4 shows a screenshot of a user interface displaying a graph updating in real time and showing an expression determining a sum of data flows grouped by data center according to an embodiment;

[0017] Figure 5 shows the effect of arrival delay of data values ​​on processing data streams by an instrumented analysis system according to an embodiment;

[0018] Figure 6 The overall process of processing data stream by the instrumented analysis system according to the embodiment is shown;

[0019] Figure 7 The overall process of determining the value of an expression based on the data value of a data stream received by an instrumented analysis system according to an embodiment is shown; and

[0020] Figure 8 A process of identifying a failure of a data source sending a data stream is shown according to an embodiment.

[0021] Reference will now be made in detail to several embodiments shown in the example drawings thereof. Note that similar or like reference numerals may be used in the drawings whenever feasible, and similar or like reference numerals may indicate similar or like functions. The drawings depict embodiments of the disclosed systems (or methods) for illustrative purposes only. Those skilled in the art will readily appreciate from the following description that alternative embodiments of the structures and methods set forth herein may be employed without departing from the principles described herein. DETAILED DESCRIPTION

[0022] Overall system environment

[0023] Figure 11 shows an overall system environment for generating real-time reports based on data streams received from instrumentation software according to an embodiment. The overall system environment includes an instrumentation analysis system 100, one or more development systems 120, a management system 160, and a reporting system 150. In other embodiments, the system may be used in conjunction with Figure 1 For example, development system 120, management system 160, and reporting system 150 may be connected to each other via a network ( Figure 1 ) interacts with the instrumented analysis system 100. In addition, there may be Figure 1 There may be more or fewer instances of each system shown, for example, there may be multiple reporting systems 150 .

[0024] Figure 1 Similar reference numerals are used to identify similar elements in the drawings and other figures. A letter following a reference numeral, such as "130a," indicates that the text refers specifically to the element having that particular reference numeral. A reference numeral, such as "130," in the text without a following letter refers to any or all elements bearing that reference numeral in the drawings (e.g., "130" in the text refers to reference numerals "130a" and / or "130b" in the drawings).

[0025] Instrumented analysis system 100 receives data including values ​​of metrics sent by an external system, such as development system 120 (instrumented analysis system 100 may also be referred to herein as an analysis system or data analysis system). Development system 120 executes software that has been instrumented, such as application 130. Figure 1 The application 130 is shown as an example of instrumented software, but the techniques disclosed herein are not limited to application software, but are applicable to other types of software, such as server software, software executed on a client device, a website, etc.

[0026] The software executed on the development system 120 is configured to send information generated as a result of instrumenting the software to the instrumented analysis system 100. For example, the application 130 can periodically send data to the instrumented analysis system 100. Different applications 130 can send the same metric or different metrics at different rates. The same application can send different metrics at different rates. The application 130 sends data to the instrumented analysis system 100 by calling an application programming interface (API) supported by the instrumented analysis system 100.

[0027] Application 130 (or any other software) can be instrumented to add counters or meters to the application. Counters include instructions in the software that store values ​​that are incremented when specific events occur. Counters can be used to determine the number of times a specific portion of code (e.g., a function or method, a specific branch of conditional code, an exception, a loop, etc.) is executed. Application 130 is an example of a data source for a data stream.

[0028] Typically, the counter value changes monotonically, i.e., the counter value may increase / decrease monotonically. The values ​​of the counters may be compared to determine the change in a particular counter value at two different points in time. For example, the number of times a particular event occurred in a time interval between time t1 and t2 may be determined by calculating the change in the corresponding counter value from t1 to t2. The API of the instrumented analysis system 100 may be called by the application 130 to send the current value of the counter to the instrumented analysis system 100.

[0029] The following is an example of instrumented code for application 130. The following instructions included in the instrumented code create a counter object for tracking a count of actions or entities.

[0030] counter1=createCounter(source="web1", metric="metric1");

[0031] The above instructions create a counter object and assign it to the variable counter1. The counter object is associated with the source "web1" and the metric "metric1". In an embodiment, the source value and the metric value uniquely identify the data stream associated with the counter (or meter). In other embodiments, more or fewer key value pairs may be used to uniquely identify the data stream.

[0032] When data corresponding to a counter is sent to the instrumented analysis system 100 by the instrumentation code, one or more of the values ​​specified during counter creation are received. For example, a source value and a metric value are typically received with each tuple of values ​​received in the data stream and the data value being reported. Optionally, the tuple of values ​​may include a timestamp, such as the timestamp when the data value being reported was captured by the instrumentation software.

[0033] The instrumented code of application 130 may include instructions for updating counter values ​​at various locations in the code. For example, counter counter1 may be incremented by executing the instruction "counter1.increment()". Counters may be incremented to track various actions or entities associated with the code. For example, a counter may be incremented whenever a specific function or method is called, a counter may be incremented whenever a specific branch of a conditional expression is executed, or a counter may be incremented whenever an object of a specific type is created (e.g., by incrementing the counter in the constructor of the object). The counter increment instruction may be called conditionally (e.g., when a function is called using a specific combination of parameters). Application 130 transmits the counter value to instrumented analysis system 100 by calling the API of instrumented analysis system 100.

[0034] Counters defined in the instrumented code can be reset periodically. For example, a counter can be reset after a specific, configurable time interval. In this case, the received counter value may not be monotonically increasing (or decreasing) because the value may be reset at the end of the interval. Counters can be cumulative, i.e., the counter is not reset unless an explicit instruction is provided to reset it. In this case, the value of the cumulative counter changes monotonically, i.e., increases (or decreases) monotonically, unless explicitly reset by the user.

[0035] The meter includes instructions for measuring certain runtime characteristics of the application 130 (e.g., heap size, number of cache misses or hits, active memory used, CPU (central processing unit) utilization, total time spent responding to requests, time spent connecting to services, etc.). The meter can also be used to track certain application specific parameters or business related values, such as number of transactions, number of users, etc. The meter can be called periodically at a configurable interval. The value of the meter is sent periodically to the instrumentation analysis system 100.

[0036] The management system 160 allows a privileged user (e.g., a system administrator) to associate data flows with metadata. The management system 160 includes a management application 170 that provides a user interface for specifying metadata to the system administrator. The metadata includes a plurality of attributes, such as name-value pairs (attributes are also referred to herein as metadata tags or tags). The instrumented analysis system 100 receives metadata describing the data flows and stores the metadata. The ability to specify metadata describing the data flows independently of the data received from each data flow provides several benefits for generating reports based on the data flows.

[0037] As an example, the instrumented analysis system 100 may receive modifications to the metadata describing each data flow without requiring any modifications to the instrumentation software of the application 130. Thus, the instrumented analysis system 100 receives specifications for new reports and modifications to existing reports, and generates results based on the new / modified reports without requiring the developer to modify the application 130.

[0038] The instrumented analysis system 100 generates the results of the report and sends them for real-time presentation as the instrumented analysis system 100 receives data streams from the instrumented software. The instrumented analysis system 100 allows existing reports to be modified without any modification to the instrumented code of the application 130. In addition, new metadata can be defined for previously received data streams. Thus, new reports can be generated, where the report is based on the data received as a data stream and the data stored previously (before the metadata is associated with the data stream). For example, a report that provides a moving average over a large time interval can be generated, which calculates a moving average based on the data currently being received and the data received previously (before the metadata used in the report is associated with the data). Moreover, these new reports can be defined without modifying the instrumented software (by re-instrumenting the software) and without redeploying the instrumented software.

[0039] In addition, the instrumented analysis system 100 provides a separation of metadata describing the data flow from the data of the data flow. Therefore, the amount of data that needs to be sent from the development system 120 to the instrumented analysis system 100 is reduced. Each application 130 only sends the data value of the metric and information identifying the metric. The metadata information is received separately from a source that is independent of the data source of the data flow. Therefore, any amount of metadata can be introduced without increasing the amount of data for each data flow.

[0040] The reporting system 150 may be a client device. The reporting system 150 includes a client application 140 that allows a user to interact with the instrumented analysis system 100. In an embodiment, the client application 140 is an Internet browser that may include client-side code (e.g., Java Script) for accessing the instrumented analysis system 100. In other embodiments, the client application 140 is a proprietary application developed to interact with the instrumented analysis system 100.

[0041] The reporting system 150 can be a conventional computer system (e.g., a desktop or laptop computer), a tablet computer, or a device with computer functionality (e.g., a personal digital assistant (PDA), a mobile phone, a smart phone, or another suitable device). The reporting system 150 interacts with the instrumented analysis system 100 via a network. The network can include any combination of local area networks and / or wide area networks using wired and / or wireless communication systems. In one embodiment, the network uses standard communication technologies and / or protocols.

[0042] The instrumented analysis system 100 may be hosted on a computing system that includes one or more processors, memory, auxiliary storage, and input / output controllers. In contrast to a typical computing system used, for example, as a reporting system 150, the computing system used to host the instrumented analysis system 100 is typically a server-class system that uses a powerful processor, large memory, and fast input / output system.

[0043] In an embodiment, data from several development systems 120 can be combined, for example, by a server, and the combined data is sent to the instrumented analysis system 100. For example, an enterprise can install a server that receives data streams from different development systems 120 and periodically sends the combined data in batches to the instrumented analysis system 100. This supports the efficiency of external communications from the enterprise. However, this configuration may result in delays in transmitting information to the instrumented analysis system 100, and corresponding delays in reporting data by the reporting system 150.

[0044] System architecture of instrumented analysis system

[0045] Figure 2 The architecture of a system for generating real-time reports based on data streams received from instrumentation software according to an embodiment is shown. The instrumented analysis system 100 includes an interface module 210, a data stream processor 280, a quantification module 240, a metadata module 220, a metadata storage 230, a data point routing module 250, an analysis engine 270, a user interface manager 290, and a time series data storage 260. In other embodiments, the instrumented analysis system 100 may include other modules not described herein. The functions indicated as provided by specific modules may be implemented by other modules instead.

[0046] The interface module 210 receives requests from external systems (e.g., a development system 120 that sends a data stream to the instrumented analysis system 100). The interface module 210 supports various application programming interfaces (APIs) that can be called by the external system. The interface module 210 receives and processes data provided by the application 130. The interface module 210 can receive and process data provided by the application 130 that is instrumented using functions provided by different vendors, as long as the data conforms to the format specified by the API supported by the interface module 210.

[0047] The interface module 210 receives data from an external system (e.g., development system 120) in the form of a data stream. In an embodiment, the interface module 210 represents data as a tuple. The data tuple received by the interface module includes various elements (including a metric identifier and a value of the metric). The metric identifier can be the name of the metric. The data tuple can include other elements, such as a timestamp corresponding to the time when the data source (e.g., the application 130 that sends the data) generates the data and attributes associated with the data. In an embodiment, the timestamp associated with the tuple represents the time when the instrumented analysis system 100 receives the data value. Attributes associated with the data can be provided in the form of name-value pairs. These attributes can provide additional information describing the received data, such as information describing the data source (e.g., a host name, server name, device name, or service name associated with the source), a method or function name associated with the data, an application instance identifier, etc.

[0048] In an embodiment, the interface module 210 generates an identifier and assigns the identifier to the record received by the interface module 210. The identifier is referred to as a time series identifier (also referred to as tsid herein) in this article. A unique time series identifier is assigned to all tuples matching the metric name and the attribute set received with the tuple. Therefore, the tuple (metric name, attribute, metric value, timestamp) is mapped to the tuple (tsid, metric value, timestamp). For example, if the tuple provides a metric name m1 and a host name h1, all tuples with a metric name of m1 and a host name of h1 are assigned the same time series identifier. Therefore, tsid uniquely identifies all tuples of the data stream received by the instrumented analysis system 100. The interface module 210 provides the data values ​​of the data stream to the data stream processor 280 for further processing.

[0049] The data stream processor 280 processes the data of the different data streams to prepare the data for analysis by the analysis engine 270. The data stream processor 280 determines the data values ​​to be processed in the time interval and the data values ​​to be ignored by the search engine in the time interval based on the real-time report. The data stream processor 280 determines whether the data value is processed or ignored based on the time when the data value arrives (or fails to arrive) in the time interval. Generally, the data stream processor 280 processes the data value that arrives early in the time interval and ignores the data value that arrives late in the time interval.

[0050] The data stream processor 280 determines the maximum expected delay for the data stream and compares the arrival time of the data value with the maximum expected delay value to determine whether to consider the data value within the time interval. The considered data value is provided as an input to an expression (e.g., an expression that aggregates the data values ​​of the data stream) for real-time reporting.

[0051] The data stream processor 280 also determines a fault threshold for the data stream. If the fault threshold is exceeded and no data value is received from the data stream, the data stream processor 280 marks the data stream as dead. The data stream processor 280 stores the state of each data stream in the metadata storage 230. The data stream processor 280 stops considering the dead data stream at subsequent time intervals. In other words, the data stream processor 280 does not wait for the maximum expected delay value of the dead data stream. If a data value is received from the data stream, the data stream processor 280 marks the data stream as alive again. Once the data stream is marked as alive, the data stream processor 280 restarts considering the data stream again at subsequent time intervals.

[0052] The data stream processor 280 stores the past data values ​​for each data stream in a memory and determines the value of the maximum expected delay using the aggregated value based on the past data values. For example, the maximum expected delay of the data stream can be obtained based on a moving average of N data values ​​(e.g., N=5 or N=4). The data stream processor 280 can multiply the moving average by a factor (e.g., 150%) to account for fluctuations in the delay value.

[0053] In an embodiment, the data stream processor 280 recalculates the maximum expected delay value at the beginning of each time interval or at the end of the previous time interval. In another embodiment, the data stream processor 280 periodically recalculates the maximum expected delay value at a time interval greater than the time interval at which the analysis engine 270 evaluates the expression. For example, the data stream processor 280 can recalculate the maximum expected delay value once every five time intervals at which the analysis engine 270 evaluates the expression. In an embodiment, the data stream processor 280 receives a user-configurable fixed value for the data stream as the maximum expected delay value or the fault threshold. The data stream processor 280 identifies the data values ​​that should be considered at this time interval and provides them to the quantization module 240 for further processing.

[0054] Quantization module 240 processes received data values ​​to transform input data time series into a time series in which data is available at regular time intervals, wherein data is available at any time interval in the input time series. For example, data values ​​received with input time series may appear at irregular intervals, however, quantization module 240 processes the data of time series to generate a data stream with data that appears periodically (e.g., every second or every 5 seconds or every 15 seconds, etc.). This process is referred to herein as quantization of time series. In an embodiment, interface module 210 creates multiple threads or processes, each of which is configured to receive data corresponding to a data stream. Each thread or process calls quantization module 240 to perform quantization on the data received for each data stream at each time interval.

[0055] The metadata module 220 receives and stores metadata information describing various data flows received from the development system 120. In an embodiment, the metadata stored in the metadata module 220 is received from a user (e.g., a system administrator). The metadata may be represented as name-value pairs. In an embodiment, the metadata is represented as a metadata object, and each object definition may be represented as a set of properties of name-value pairs. A set of data flows may be associated with a metadata object. Thus, all properties represented by a metadata object are associated with each data flow associated with the metadata object.

[0056] The metadata data store 230 stores metadata objects and their associations with data streams. The metadata data store 230 stores an identifier (ID) for each metadata object and the properties represented by the metadata object. In an embodiment, each data stream is associated with a time series identifier that uniquely identifies the data stream. The metadata data store 230 stores an index that maps each metadata object to a set of time series identifier values. The metadata data store 230 stores an index that maps various properties (or name-value pairs or tags) to a set of time series identifier values.

[0057] The metadata storage 230 may modify the metadata object based on the received instructions. For example, the metadata storage 230 may modify, add, or delete some attributes represented by the metadata object. Alternatively, the metadata storage 230 may modify the mapping from the metadata object to the data stream based on the received instructions. For example, the metadata storage 230 may associate the data stream with the metadata object, or delete the association between the metadata object and the data stream.

[0058] In an embodiment, metadata store 230 is represented as a relational database, but may be represented as any other type of database or data store. Metadata store 230 may be a relational database that stores a table mapping metadata object IDs to time series IDs that identify data streams. Other database tables may store properties associated with each metadata object as a mapping from a metadata object ID to each property represented as a name-value pair.

[0059] The analysis engine 270 evaluates reports of specified expressions based on metadata. Expressions can be based on various operations, such as aggregation and transformation. Expressions can be obtained by composing various functions including aggregation and transformation in various ways, and by composing other previously defined expressions. In an embodiment, the analysis engine 270 parses the expression, generates an executable representation of the program, and executes the generated representation.

[0060] The time series data store 260 stores data streams received from various sources (e.g., the development system 120). In an embodiment, the time series data store 260 also stores the time series data after the data is quantized. The time series data store 260 may also store the accumulated data for each time series. The time series data store 260 also stores the results of various analysis requests, such as the results of various reports requested by the user. The analysis engine 270 calculates the results for certain reports (e.g., a moving average over a time interval) by combining the data stored in the time series data store 260 with new data obtained from various sources as data streams.

[0061] The user interface manager 290 presents the report requested by the user via a user interface (e.g., a user interface of a client application 140 of the reporting system 150). In an embodiment, the client application 140 is an Internet browser application, and the user interface manager 290 generates a web page for display using the client application 140. In other embodiments, the client application 140 communicates with the user interface manager 290 using a suitable protocol. The user interface manager provides the report data to the client application 140 for presentation (e.g., as a chart).

[0062] In an embodiment, user interface manager 290 continuously updates a chart corresponding to a report displayed via client application 140 based on data from a data stream arriving at instrumented analysis system 100. Instrumented analysis system 100 is configured by a system administrator via management system 160 to generate data for a report based on data from a data stream. The instrumented analysis system updates a displayed report at a particular rate.

[0063] The instrumented analysis system 100 also receives the definition of the report that needs to be displayed via the reporting system 150. The report definition specifies the expression corresponding to the report to be displayed. For example, the expression may specify that the aggregate value of all data streams grouped by certain metadata attributes needs to be displayed and updated every T seconds (e.g., T=1 second). The instrumented analysis system 100 presents a real-time chart via the reporting system. A real-time chart refers to a chart that is updated as data values ​​are received. In contrast, traditional reports are generated based on queries executed against data stored in a permanent storage device of a database. In practice, as long as the data is generated due to transmission delays of the generated data from the development system to the instrumented analysis system 100 via a network, processing delays of the data, etc., the real-time chart is not updated immediately. However, an embodiment of the instrumented analysis system minimizes the delay between the time when the data source (i.e., the instrumented software executed on the external system) generates the data value and the time when the expression is presented on the user interface of the reporting system 150 based on the evaluation results of the generated data value.

[0064] Real-time reporting based on instrumentation software

[0065] The user interface manager 290 of the instrumented analysis system 100 presents the data generated by the report in real time via the user interface. The development system 120 executing the instrumentation software provides the data values ​​via the network. Once the data values ​​are generated and sent by the external system, the network will cause the data values ​​to arrive at the instrumented analysis system 100 after a delay. Figure 3 and Figure 4 An example of a report presented on a user interface by the instrumented analysis system 100 is shown.

[0066] Figure 3 A screenshot of a user interface showing a chart that updates in real time based on data from a data stream received by an instrumented analysis system according to an embodiment is shown. The screenshot shows several charts 310 showing a data stream representing a metric 320 (service.cache.hit). The metric represents a cache hit value received from instrumented software executing on the development system 120. The values ​​are accumulated in 1 second time intervals. Thus, the cache hit values ​​received in each 1 second time interval are added together. There may be a large number of services reporting a metric (service.cache.hit), so a large number of charts 310 are displayed. Figure 3Various widgets are shown that allow a user to take actions (eg, select a metric to be reported by the user interface, perform an aggregation).

[0067] Large enterprises may have a very large number of development systems 120. Each development system 120 may execute multiple services, each reporting metrics. Figure 3 The number of graphs displayed in a chart can be very large. Figure 4 As shown, users can gain deeper insight into the data reported by a data stream by grouping the data streams.

[0068] Figure 4 A screenshot of a user interface showing a graph updating in real time and showing an expression determining a sum of data flows grouped by data center is shown in accordance with an embodiment. Figure 4 A widget 420 is shown that allows the properties by which data streams are grouped to be specified and aggregation operations to be performed for each group. Figure 4 As shown, the chart 410 shows that the data center groups the data streams and sums them for each group. Assuming that there are only two data centers, the number of charts is reduced to 2. Each chart 410 shows the sum of the data values ​​of the data streams received from a specific data center.

[0069] The instrumented analysis system 100 collects data values ​​from various data streams and calculates the value of an expression to be displayed as a report. For example, the instrumented analysis system 100 determines the grouping of data values ​​based on the data stream and calculates the sum of the data values ​​for each data center to present the following: Figure 4 The instrumented analysis system 100 performs the above calculations at subsequent time intervals. Therefore, for each time interval, the instrumented analysis system 100 waits for data values ​​of the data stream to arrive. Once the instrumented analysis system 100 determines that all expected data values ​​for the time interval have arrived, the instrumented analysis system 100 performs the required calculations and sends the results for display.

[0070] However, various data values ​​from different data sources may arrive at different points in time within the time interval. Some data values ​​may not even arrive within the time interval (e.g., they may arrive in the next time interval or even later). Moreover, some data sources may fail (e.g., due to a system crash) and may not even send data in the time interval or several subsequent time intervals until the data source is restarted. Due to network delays, delays in calculation results, and system failures, the instrumented analysis system 100 is only able to provide result values ​​for display after a certain delay since the data was generated by the data source. However, the earlier the instrumented analysis system 100 can present the results within the time interval, the closer the report is to real-time reporting. Embodiments of the present invention allow the instrumented analysis system to present the evaluation results of expressions based on the earlier data streams in each time interval while maximizing the accuracy of the results.

[0071] Figure 5 The effect of arrival delay of data values ​​on a data stream processed by an instrumented analysis system according to an embodiment is shown. Figure 5 Four data sources are shown, namely applications 130a, 130b, 130c and 130d that send data streams d1, d2, d3 and d4, respectively, to the instrumented analysis system 100. The data values ​​of data stream dx are dx1, dx2, dx3, etc., each of which is sent at a specific time interval. For example, the data values ​​of data stream d1 are d11, d12, etc. It is assumed that all data values ​​are generated by their data sources at the beginning of the time interval. Therefore, Figure 5 The difference in time of a data value shown in at a point in time over the time interval compared to the start of the time interval represents the delay of the data value from reaching the instrumented analysis system 100 after being generated by the data source.

[0072] Figure 5 The timeline shown shows two time intervals, I1 (from time t0 to t2) and I2 (from time t2 to t4). The instrumented analysis system 100 receives data value d11 from data stream d1 in time interval I1 and data value d12 from data stream d1 in time interval I2; receives data value d21 from data stream d2 in time interval I1 and data value d22 from data stream d2 in time interval I2; receives data value d31 from data stream d3 in time interval I1 and data value d32 from data stream d3 in time interval I2; and receives data value d41 from data stream d4 in time interval I1 and data value d42 from data stream d4 in time interval I2.

[0073] like Figure 5As shown, all data values ​​d11, d12, d13, and d14 arrive before time t1 in interval I1. Time point t1 is relatively early in time interval I1, for example, all four data values ​​arrive before a small half of time interval I1 is completed. Therefore, instrumented analysis system 100 can calculate the required expressions and display them as soon as possible after actual t1. In contrast, in interval I2, even though data values ​​d12, d22, and d32 arrive earlier in time interval I2, data value d42 arrives at t3, which is almost the end of time interval I2. Therefore, instrumented analysis system 100 can only calculate any required expressions and present results after time t3, which is almost the end of time interval I2.

[0074] Therefore, the long delay in receiving data in the time interval leads to an undesirable user experience. These results are presented much later than the time point when the data source generates the data value. In addition, since the result of the time interval I1 is presented earlier in the time interval, and the result of the time interval I2 is presented later in the time interval, there is a gap in the real-time chart where no data is presented to the user, that is, a gap between the time when the result of the time interval I1 is presented and the time when the result of the time interval I2 is presented. This gap is longer than the typical gap between the times when the results are presented. In a chart that is expected to be updated in real time, the existence of a long gap without data presentation provides an undesirable user experience. Embodiments of the present invention allow the instrumented analysis system 100 to present the results to the user earlier in the time interval and reduce the gap between the presentation of the results between two consecutive time intervals.

[0075] Overall process

[0076] Figure 6 , Figure 7 and Figure 8 The various processes performed by the instrumented analysis system for processing data received from the instrumented software are shown. The steps shown in the flowchart in each figure can be performed in a different order than the order shown in the figure. For example, certain steps can be performed simultaneously with other steps. In addition, the steps indicated to be performed by certain modules can be performed by other modules.

[0077] Figure 6 The overall process of processing a data stream by an instrumented analysis system according to an embodiment is shown. In an embodiment, the instrumented analysis system 100 determines the value of an expression at multiple time intervals. For example, the expression can aggregate the data values ​​received as part of the data stream in the time interval. In these embodiments, Figure 6 The process shown is performed by instrumented analysis system 100 to determine whether a data value should be used as input to evaluate an expression for that time interval.

[0078] If the instrumented analysis system 100 determines that a data value from the data stream arrives earlier in the time interval, the instrumented analysis system 100 uses the data value as input for determining the value of the expression for the time interval. If the instrumented analysis system 100 determines that a data value from the data stream arrives later or is unlikely to arrive in the actual interval, the instrumented analysis system 100 evaluates the expression for the time interval without considering the later arriving data value. In other words, the instrumented analysis system 100 excludes the later arriving data value from the input for evaluating the expression for the time interval.

[0079] The interface module 210 receives (610) information describing a plurality of data flows from one or more external systems. The information describing the data flows may be provided by the external system by calling an API of the instrumented analysis system. For example, the external system may call an API of the instrumented analysis system, wherein the API allows the external system to register the data flows with the instrumented analysis system 100 by providing information describing the data flows. The information describing the data flows includes metrics associated with the data flows (e.g., cache hits, cache misses, CPU load, memory usage, etc.), attributes describing the data sources (e.g., service names), etc.

[0080] The data stream processor 280 processes the data values ​​received from the plurality of data streams at subsequent time intervals. The data stream processor 280 performs the following steps (620, 630 and U40) for each time interval and each data stream.

[0081] The data stream processor 280 determines the maximum expected delay for each data stream. The data stream processor 280 can use a fixed threshold associated with the data stream as the maximum expected delay. For example, the instrumented analysis system 100 can receive the maximum expected delay from the system administrator as a configuration parameter for the data stream. In some embodiments, the data stream processor 280 determines the maximum expected delay of the data stream based on the previous data value received for the data stream. For example, the data stream processor 280 can determine the maximum expected delay of the data value based on a moving aggregate value, wherein the moving aggregate value is based on a set of received values ​​(a fixed number of past values ​​received within a moving time window or all past values ​​received within a moving time window). As another example, the data stream processor 280 can determine the maximum expected delay for the data stream based on the last data value received from the data stream.

[0082] In an embodiment, the data stream processor 280 determines the maximum expected delay value by increasing the aggregate value based on the past data values ​​by a certain factor (e.g., a fixed percentage or a fixed offset). For example, the data stream processor 280 can determine the maximum expected delay value of the data stream by amplifying the average of the past 4 data values ​​by 150%. As another example, the data stream processor 280 can determine the maximum expected delay value of the data stream by increasing the last data value by a fixed value (e.g., 5) (the selection of the fixed value depends on the type of data received in the data stream).

[0083] The data stream processor 280 provides the data values ​​received from the data stream to a software module of the instrumented analysis system 100 for further processing of the data values ​​(e.g., for presentation as a real-time chart via a user interface). For example, in an embodiment, the data stream processor 280 provides 630 the data values ​​to the quantization module 240 to perform quantization using the data values. In another embodiment, the data stream processor 280 provides 630 the data values ​​to the analysis engine 270 to evaluate an expression (e.g., an expression that determines an aggregate value using the data values ​​received in a time interval) using the data values.

[0084] If the data value of the data stream arrives at the instrumented analysis system 100 before the maximum expected delay, the data stream processor 280 provides (630) the data value of the data stream to the quantization module (or any other module) for further processing. If the data value of the data stream fails to arrive at the instrumented analysis system 100 before the maximum expected delay, the data stream processor 280 provides (630) information indicating that the data value failed to arrive. In an embodiment, the data stream processor 280 provides the information indicating that the data value failed to arrive by providing a special data value (e.g., a null data value) to the module that performs subsequent processing.

[0085] The data stream processor 280 stores (640) the data value of the data stream in a permanent storage (e.g., the time series data store 260) regardless of whether the data value arrives before or after the maximum expected delay for the data stream. In other words, the data stream processor 280 stores (640) the data value regardless of when the data value arrives at the instrumented analysis system 100. The data stream processor 280 does not provide the data value of the data stream that arrives after the maximum expected delay to the quantization module 240 or the analysis engine 270 for performing real-time processing of the data (e.g., rendering a real-time chart). The reason for this is that the data stream processor 280 is designed not to slow down the rendering of the data in the real-time chart due to late arriving data values. However, even if the data value arrives after the maximum expected delay, the data stream processor 280 stores (640) the data value in the time series data store 260 so that subsequent queries processing the data of the data stream for that time interval use the data value regardless of when the data value arrived.

[0086] Figure 7 The overall process of determining the value of an expression based on the data value of a data stream received by an instrumented analysis system according to an embodiment is shown. Figure 7 The illustrated process shows the Figure 6 Similar steps as shown, but Figure 7 The illustrated process is in the context of evaluating expressions and rendering data as real-time charts.

[0087] Similar to Figure 6 In step 610, the interface module 210 receives (710) information describing a plurality of data streams from one or more external systems. The analysis engine 270 receives (720) an expression based on the data of the data stream for a time interval. The instrumented analysis system 100 calculates the value of the expression for each of the plurality of time intervals (e.g., every second, every 2 seconds, or every 5 seconds). The expression may calculate an aggregate value based on the data values ​​associated with the time interval. The data values ​​associated with the time interval correspond to the data values ​​sent by the external system (i.e., the data source of the data stream) for processing during the time interval. The data values ​​may arrive at the instrumented analysis system 100 in the same time interval as the external system sends the data values, or may not arrive at the instrumented analysis system 100. As an example, the expression may calculate a count, a sum, an average, a median, a percentile, or any other aggregate value for all data values ​​associated with the time interval. As another example, the expression may calculate the above aggregate for data values ​​grouped by a particular attribute. For example, the expression may calculate the sum of the data values ​​grouped by a data center attribute, thereby determining the sum of the data values ​​arriving from each data center.

[0088] Similar to Figure 6 At step 620, the data stream processor 280 determines (730) the maximum expected delay for each data stream considered for evaluating the expression. The set of data streams associated with evaluating the expression can be specified as part of an expression using metadata describing the data streams. An expression can identify a set of data streams by specifying the values ​​of one or more metadata attributes describing the data streams. For example, an expression can evaluate an aggregate value based on all data streams from a specific data center, where the specific data center is identified by a specific value of a data center attribute. Another expression can specify all data streams that provide a specific metric (e.g., cache hits). The data stream processor 280 monitors (740) the set of data streams associated with the expression. In an embodiment, the data stream processor 280 monitors all data streams received by the instrumented analysis system, thereby also monitoring the set associated with the expression. In an embodiment, the data stream processor 280 monitors the data stream by creating a process or thread that waits for data values ​​of the data streams to arrive.

[0089] If a data value of a data stream arrives late, i.e., fails to arrive before the maximum expected delay of the data stream, the data stream processor 280 excludes (750) the data stream from the set considered for the expression in the evaluation time interval. In other words, for the expression in the evaluation time interval, the data stream processor 280 only considers data values ​​that arrive before the maximum expected delay of the corresponding data stream. The data stream processor 280 collects (760) all data values ​​of the set of data values ​​(obtained by excluding the data values ​​that arrive later) and provides the data values ​​for further processing (e.g., to the quantization module 240 or the analysis engine 270).

[0090] The analysis engine 270 determines 770 the value of the expression based on the collected set of values ​​provided by the data stream processor 280. The analysis engine 270 provides the evaluation result of the expression to the user interface manager 290. The user interface manager 290 sends 780 the evaluation result of the expression for presentation via the user interface as, for example, a real-time chart. The instrumented analysis system 100 repeats steps 730, 740, 750, 760, 770, and 780 for each subsequent time interval. These steps may be repeated indefinitely, for example, as long as the user wants to view the real-time chart.

[0091] The data stream processor 280 further maintains a fault threshold to determine whether the data source providing the stream fails, for example, due to a crash or failure of the instrumented software sending the data stream or the external system providing the data stream. The fault threshold used to determine whether the data stream fails is usually longer than the maximum expected delay value of the data stream. For example, the fault threshold can be several time intervals long. On the contrary, the maximum expected delay value of the data stream is less than the length of the time interval. In an embodiment, the instrumented analysis system 100 receives the fault threshold of the data stream from the system administrator via the management system 160. The instrumented analysis system 100 can use the fault threshold for a collection of data streams, for example, for all data streams arriving from a data center or all data streams arriving from a class of external systems, a class of operating systems running on external systems, or a class of instrumented software providing data streams.

[0092] If the data stream processor 280 determines that the data value of the data stream fails to arrive before the failure threshold, the data stream processor 280 marks the data stream as dead. In an embodiment, the instrumented analysis system 100 stores the state (dead or alive) of each data stream in the metadata storage 230 or the time series data storage 260. Therefore, the data stream processor 280 excludes the dead data stream from all calculations of expressions in subsequent time intervals until the state of the data stream changes back to alive.

[0093] Therefore, the data stream processor 280 does not wait for the maximum expected delay associated with the data stream for the data stream. This prevents the data stream processor 280 from having to wait for additional time (of the maximum expected delay of the data stream) at subsequent time intervals. The data stream can remain dead for a long period of time. If the data stream processor 280 does not exclude the dead data stream from consideration at subsequent time intervals, the data stream processor 280 will continue to wait for an indeterminate amount of time for the maximum expected delay of the data stream.

[0094] Whenever the state of a data stream changes from dead to live, the data stream processor 280 begins including the data stream in the set of data streams considered in evaluating the expression for subsequent time intervals. If a data value for a data stream arrives at the instrumentation system after the data stream state is determined to be dead, the data stream processor 280 changes the state of the dead data stream back to live.

[0095] Figure 8 A process for identifying a failure of a data source that sends a data stream according to an embodiment of the invention is shown. The data stream processor 280 determines (810) a failure threshold for the data stream. The data stream processor 280 monitors a data stream whose data value arrives later (e.g., after a maximum expected delay value) to check whether the data value arrives before the failure threshold. If the data value of the data stream fails to arrive before the failure threshold, the data stream processor 280 marks the data stream as dead. The data stream processor 280 may store a tag indicating that the data stream is dead and a timestamp indicating the time when the data stream is determined to be dead in the metadata storage 230. The data stream processor 280 excludes (830) the data stream from the set of data streams considered in a subsequent time interval (e.g., to evaluate an expression based on the data stream). The data stream processor 280 excludes (830) the data stream until the data value of the data stream arrives.

[0096] Alternative Embodiments

[0097] It will be understood that the drawings and description of the present invention have been simplified to show elements relevant to a clear understanding of the present invention, while many other elements found in typical IT management systems have been excluded for clarity. It will be appreciated by those of ordinary skill in the art that other elements and / or steps may be desired and / or required in implementing the present invention. However, because these elements and steps are well known in the art, and because they do not contribute to a better understanding of the present invention, no discussion of these elements and steps is provided herein. The disclosure herein relates to all such variations and modifications to such elements and methods known to those skilled in the art.

[0098] Some parts of the above description describe embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are often used by technicians in the field of data processing to effectively convey the essence of their work to those skilled in the art. These operations described functionally, computationally, or logically are understood to be implemented by computer programs or equivalent circuits, microcodes, etc. Moreover, these operational arrangements may also be conveniently referred to as modules at times without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0099] As used herein, any reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment. The appearance of the phrase "in one embodiment" in various places in the specification does not necessarily refer to the same embodiment.

[0100] The expressions "coupled" and "connected" and their derivatives may be used to describe some embodiments. It should be understood that these terms are not synonymous with each other. For example, the term "connected" may be used to describe some embodiments to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, the term "coupled" may be used to describe some embodiments to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. Embodiments are not limited to this context.

[0101] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to those processes, methods, articles, or apparatuses. In addition, unless expressly stated to the contrary, "or" refers to an inclusive or, not an exclusive or. For example, the condition A or B is satisfied in any of the following cases: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exist).

[0102] In addition, "a" or "an" is used to describe elements and components of the embodiments herein. This is just for convenience and to give a general sense of the invention. The description should be understood to include one or at least one, and the singular also includes the plural unless it is obvious that the singular is intended.

[0103] After reading this disclosure, those skilled in the art will understand additional, optional structural and functional designs of systems and processes through the principles disclosed herein. Therefore, although specific embodiments and applications are illustrated and described, it will be understood that the disclosed embodiments are not limited to the precise structures and components disclosed herein. Various modifications, changes and variations that are obvious to those skilled in the art may be made to the arrangement, operation and details of the methods and devices disclosed herein without departing from the spirit and scope defined in the appended claims.

Claims

1. A computer-implemented method for processing data generated by instrumentation software, the method comprising: determining, for a time interval among a plurality of time intervals, a maximum expected delay value for a data flow, the maximum expected delay value comprising a limit on an arrival delay of data values ​​of the data flow from instrumentation software to an analysis system, the maximum expected delay value being determined based on a moving average of historical arrival delays of data values ​​of the data flow multiplied by a numerical factor, wherein the maximum expected delay value is periodically recalculated at the beginning of each time interval or at the end of a previous time interval; determining, for the time interval of the plurality of time intervals, a failure threshold for the data flow, the failure threshold indicating a failure of the instrumented software sending the data flow; modifying the set of data flows by excluding a data flow when the data value of the data flow fails to arrive within the maximum expected delay value for the data flow during the time interval; monitoring the data stream for another data value for the data stream received at the time interval; tagging data streams in real time based on said monitoring; If the data value of the data flow fails to arrive before the failure threshold, marking the data flow as dead, and excluding the data flow marked as dead from the data flow set; if a data value for the data stream is subsequently received, marking the data stream as live; and including the data stream in a set of data streams that are monitored for arrival of data values ​​for one or more subsequent time intervals; receiving a specification of an expression for processing data values ​​of a plurality of data streams associated with the time interval; evaluating the expression using the modified data values ​​of the data stream set; as well as The result of evaluating the expression is sent for presentation via a user interface.

2. The method according to claim 1, further comprising: Data values ​​of the data stream that arrive after the maximum expected delay value are stored in a persistent data store.

3. The method according to claim 1, further comprising: The evaluation result of the expression is configured for presentation by a real-time chart that is updated for each time interval.

4. The method according to claim 1, further comprising: Information identifying the plurality of data flows is received by the analysis system from one or more external systems, each data flow being generated by an instance of instrumentation software executing on an external system. The method of claim 1 , wherein the expression aggregates data values ​​of a collection of data streams.

6. A computer-implemented method for processing data generated by instrumentation software, the method comprising: determining, for a time interval of a plurality of time intervals, a maximum expected delay value for a data flow, the maximum expected delay value comprising a limit on an arrival delay of data values ​​of the data flow from instrumentation software to an analysis system, the maximum expected delay value for a time interval being determined based on arrival delays of the data values ​​of the data flow for a previous time interval, wherein the maximum expected delay value is periodically recalculated at the beginning of each time interval or at the end of a previous time interval; determining, for the time interval of the plurality of time intervals, a failure threshold for the data flow, the failure threshold indicating a failure of the instrumented software sending the data flow; modifying the set of data flows by excluding a data flow when the data value of the data flow fails to arrive within the maximum expected delay value for the data flow during the time interval; If the data value of the data flow fails to arrive before the failure threshold, marking the data flow as dead, and excluding the data flow marked as dead from the data flow set; if a data value for the data stream is subsequently received, marking the data stream as live; and including the data stream in a set of data streams that are monitored for arrival of data values ​​for one or more subsequent time intervals; receiving a specification of an expression for processing data values ​​of a plurality of data streams associated with the time interval; evaluating the expression using the modified data values ​​of the data stream set; as well as The result of evaluating the expression is sent for presentation via a user interface.

7. A system for processing data generated by instrumentation software, the system comprising: at least one memory having instructions stored thereon; as well as at least one processor configured to execute the instructions, wherein the at least one processor is configured to: determining, for a time interval among a plurality of time intervals, a maximum expected delay value for a data flow, the maximum expected delay value comprising a limit on an arrival delay of data values ​​of the data flow from instrumentation software to an analysis system, the maximum expected delay value being determined based on a moving average of historical arrival delays of data values ​​of the data flow multiplied by a numerical factor, wherein the maximum expected delay value is periodically recalculated at the beginning of each time interval or at the end of a previous time interval; determining, for the time interval of the plurality of time intervals, a failure threshold for the data flow, the failure threshold indicating a failure of the instrumented software sending the data flow; modifying the set of data flows by excluding a data flow when the data value of the data flow fails to arrive within the maximum expected delay value for the data flow during the time interval; If the data value of the data flow fails to arrive before the failure threshold, marking the data flow as dead, and excluding the data flow marked as dead from the data flow set; if a data value for the data stream is subsequently received, marking the data stream as live; and including the data stream in a set of data streams that are monitored for arrival of data values ​​for one or more subsequent time intervals; receiving a specification of an expression for processing data values ​​of a plurality of data streams associated with the time interval; evaluating the expression using the modified data values ​​of the data stream set; and The result of evaluating the expression is sent for presentation via a user interface.

8. The system according to claim 7, further configured to: Data values ​​of the data stream that arrive after the maximum expected delay value are stored in a persistent data store.

9. The system according to claim 7, further configured to: The evaluation result of the expression is configured for presentation by a real-time chart that is updated for each time interval.

10. The system according to claim 7, further configured to: Information identifying the plurality of data flows is received by the analysis system from one or more external systems, each data flow being generated by an instance of instrumentation software executing on an external system.

11. The system of claim 7, wherein the expression aggregates data values ​​of a collection of data streams.

12. A system for processing data generated by instrumentation software, the system comprising: at least one memory having instructions stored thereon; as well as at least one processor configured to execute the instructions, wherein the at least one processor is configured to: determining, for a time interval of a plurality of time intervals, a maximum expected delay value for a data flow, the maximum expected delay value comprising a limit on an arrival delay of data values ​​of the data flow from instrumentation software to an analysis system, the maximum expected delay value for a time interval being determined based on arrival delays of the data values ​​of the data flow for a previous time interval, wherein the maximum expected delay value is periodically recalculated at the beginning of each time interval or at the end of a previous time interval; determining, for the time interval of the plurality of time intervals, a failure threshold for the data flow, the failure threshold indicating a failure of the instrumented software sending the data flow; modifying the set of data flows by excluding a data flow when the data value of the data flow fails to arrive within the maximum expected delay value for the data flow during the time interval; If the data value of the data flow fails to arrive before the failure threshold, marking the data flow as dead, and excluding the data flow marked as dead from the data flow set; if a data value for the data stream is subsequently received, marking the data stream as live; and including the data stream in a set of data streams that are monitored for arrival of data values ​​for one or more subsequent time intervals; receiving a specification of an expression for processing data values ​​of a plurality of data streams associated with the time interval; evaluating the expression using the modified data values ​​of the data stream set; and The result of evaluating the expression is sent for presentation via a user interface.

13. A non-transitory computer-readable storage medium comprising instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform operations for processing data generated by instrumentation software, the operations comprising: determining, for a time interval among a plurality of time intervals, a maximum expected delay value for a data flow, the maximum expected delay value comprising a limit on an arrival delay of data values ​​of the data flow from instrumentation software to an analysis system, the maximum expected delay value being determined based on a moving average of historical arrival delays of data values ​​of the data flow multiplied by a numerical factor, wherein the maximum expected delay value is periodically recalculated at the beginning of each time interval or at the end of a previous time interval; determining, for the time interval of the plurality of time intervals, a failure threshold for the data flow, the failure threshold indicating a failure of the instrumented software sending the data flow; modifying the set of data flows by excluding a data flow when the data value of the data flow fails to arrive within the maximum expected delay value for the data flow during the time interval; If the data value of the data flow fails to arrive before the failure threshold, marking the data flow as dead, and excluding the data flow marked as dead from the data flow set; if a data value for the data stream is subsequently received, marking the data stream as live; and including the data stream in a set of data streams that are monitored for arrival of data values ​​for one or more subsequent time intervals; receiving a specification of an expression for processing data values ​​of a plurality of data streams associated with the time interval; evaluating the expression using the modified data values ​​of the data stream set; and The result of evaluating the expression is sent for presentation via a user interface.

14. The non-transitory computer-readable storage medium of claim 13, further configured to: Data values ​​of the data stream that arrive after the maximum expected delay value are stored in a persistent data store.

15. The non-transitory computer-readable storage medium of claim 13, further configured to: The evaluation result of the expression is configured for presentation by a real-time chart that is updated for each time interval.

16. The non-transitory computer-readable storage medium of claim 13, further configured to: Information identifying the plurality of data flows is received by the analysis system from one or more external systems, each data flow being generated by an instance of instrumentation software executing on an external system.

17. The non-transitory computer-readable storage medium of claim 13, wherein the expression aggregates data values ​​of a collection of data streams.

18. A non-transitory computer-readable storage medium comprising instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform operations for processing data generated by instrumentation software, the operations comprising: determining, for a time interval of a plurality of time intervals, a maximum expected delay value for a data flow, the maximum expected delay value comprising a limit on an arrival delay of data values ​​of the data flow from instrumentation software to an analysis system, the maximum expected delay value for a time interval being determined based on arrival delays of the data values ​​of the data flow for a previous time interval, wherein the maximum expected delay value is periodically recalculated at the beginning of each time interval or at the end of a previous time interval; determining, for the time interval of the plurality of time intervals, a failure threshold for the data flow, the failure threshold indicating a failure of the instrumented software sending the data flow; modifying the set of data flows by excluding a data flow when the data value of the data flow fails to arrive within the maximum expected delay value for the data flow during the time interval; If the data value of the data flow fails to arrive before the failure threshold, marking the data flow as dead, and excluding the data flow marked as dead from the data flow set; if a data value for the data stream is subsequently received, marking the data stream as live; and including the data stream in a set of data streams that are monitored for arrival of data values ​​for one or more subsequent time intervals; receiving a specification of an expression for processing data values ​​of a plurality of data streams associated with the time interval; evaluating the expression using the modified data values ​​of the data stream set; and The result of evaluating the expression is sent for presentation via a user interface.

Images