Data processing system, method, machine-readable hardware storage device and program product for executing a data flow graph
By introducing a dedicated audit port into the components of the data flow diagram, the recoverable output and error logging of the audit data during the data flow diagram processing are realized, and the problems of system performance degradation and error detection difficulty in the existing technology are solved. Improves the resource utilization and scalability of the system.
Patent Information
- Application Number
- CN201980077145.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-09-25
- Filing Date
- 2019-09-25
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-09-25
AI Technical Summary
The prior art lacks an effective logging mechanism when processing data flow graphs, resulting in system performance degradation and increased error detection difficulty, and the recovery output of executable logic cannot be achieved.
A method through data processing system is designed to introduce a dedicated audit port, allowing recoverable output of audit data without affecting the performance of the output port. The method includes adding an audit port to the component of the data flow graph, which is used to generate and restore audit data, and to restore component state when an error occurs, outputting an error log to rollback the data flow graph state.
It realizes efficient processing and output audit data, dynamically generates metrics, tracks system performance in real time, detects and fixes errors in a timely manner without affecting system performance, and improves resource utilization and system scalability.
Smart Images

Figure CN113168362B_ABST
Abstract
Description
Technical Field
[0001] The present description relates to computer-implemented methods, data processing systems, and machine-readable hardware storage devices for generating logs from the execution of executable logic currently applied to data items in a stream of data items, such as data items input from a networked application over a network connection.
[0002] Priority claim
[0003] This application claims priority under 35 U.S.C. §119(e) to U.S. patent application serial number 62 / 736,259, filed on September 25, 2018, the entire contents of which are incorporated herein by reference. Background Art
[0004] Logs record events that occur when an operating system or other software is running, or messages between different users of communication software. Summary of the invention
[0005] In some embodiments, a method implemented by a data processing system, the data processing system is used to execute a data flow graph, the data flow graph processes one or more data records representing a transaction, wherein the data flow graph includes multiple components and links, wherein the components represent calculations applied to one or more input records to process the one or more input records, wherein the input records have a specified relationship with the one or more data records representing the transaction, and wherein the links connect the components and represent data flows between the components, the method comprising: executing the data flow graph including the multiple components and the links, wherein a given component of the multiple components includes an input port, an audit port, and an output port; wherein the input port is used to receive one or more input records, the one or more input records have a specified relationship with the one or more data records representing the transaction; wherein the audit port is used to provide audit data, the audit data at least describing a portion of the transaction; wherein the output port is used to provide one or more output records; wherein at least one of the components is configured to be executed before executing the given component; processing one or more data records representing the transaction through a data flow graph having the components and the links, wherein the at least one of the components saves the state of one or more input records specified to be processed by the at least one of the components; when an error occurs during the processing of one or more input records by the given component, the state of the at least one of the components is restored to the saved state; and based on the restored state, at least some of the audit data is restored for the given component of the data flow graph; at least some of the restored audit data describing at least the portion of the transaction is provided on the audit port; and the one or more output records are provided on the output port. A system of one or more computers may be configured to perform these actions by means of software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform specific operations or actions in operation. One or more computer programs may be configured to perform these actions by means of instructions that, when executed by a data processing device, cause the device to perform specific operations or actions.
[0006] In some embodiments, the recovered audit data is first audit data, the audit port is a first audit port, the given component is a first component, and wherein the method further comprises: combining the first audit data with second audit data outputted on a second audit port of a second component at a specified time interval. The given component is a first component, the recovered audit data is first audit data, and wherein the method further comprises: executing the first component on a first data processing system; executing the second component of the data flow graph on a second data processing system; and receiving the recovered audit data and the second audit data outputted from the second audit port of the second component on a third data processing system. In some embodiments, the method further comprises: storing the received recovered audit data and the second audit data in a memory; and dynamically generating one or more metrics based on the stored audit data. In some embodiments, the method further comprises: aggregating data from the first audit data and the second audit data; and storing the aggregated data. In some embodiments, the method further comprises: classifying the recovered audit data by assigning the recovered audit data to a specific category; and generating audit metrics indicating user interactions with specific activities based on the audit data assigned to the specific category.
[0007] In other embodiments, in response to the occurrence of the error, the given component outputs an error log on an error log port, the error log including the state of the at least one component among the multiple components and specifying which one or which input records of the input records were last successfully processed by the at least one component among the multiple components, wherein the state specifies which component of the components last successfully completed processing. In still other embodiments, the given component outputs the error log to an error handler component, wherein the error handler component transmits a message to a rollback component based on the error log, and wherein the rollback component uses the message to return the state of the data flow graph to the following state: in this state, at least one component among the multiple components is executing, and one or more of the input records that were last successfully processed by the at least one component among the multiple components are being processed by the at least one component among the multiple components.
[0008] All or part of the foregoing may be implemented as a computer program product comprising instructions stored on one or more non-transitory machine-readable storage media and / or one or more computer-readable hardware storage devices such as hard drives, random access memory storage devices (such as dynamic random access memory, machine-readable hardware storage devices, and other types of non-transitory machine-readable storage devices) and executable on one or more processing devices. All or part of the foregoing may be implemented as an apparatus, method, or electronic system that may include one or more processing devices and a memory for storing executable instructions to implement the stated functions.
[0009] The above-described aspects may include one or more of the following advantages.
[0010] System described herein can process data with improved system performance, such as processing with less or minimum impact on system computing resources (such as memory consumption and / or processing power). In addition, system described herein processes this data (such as, input data, output data, audit data, test data or log data) with less time delay and in an extensible manner. Specifically, this system can output audit data in a recoverable manner when not affecting system performance about output port negatively. The metric of the execution of executable logic can be dynamically determined and output, so that the user can track these metrics in real time. In this context, the error state of the system detects and causes the user's concern from these metrics, preferably also provides the user with guidance (such as, instruction) of how to repair detected errors. In addition, preferably enable the user to control the log records of which executable logics will be generated, so that log records are concentrated on the part of interest, while reducing the generation of redundant data volume. Each of these helps the user to understand, track and correctly operate the underlying data processing system in a resource-efficient manner.
[0011] The details of one or more embodiments are set forth in the accompanying drawings and the description that follow.Other features, objects, and advantages of the techniques described herein will become apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 and Figure 4A Each is a schematic diagram of a database management system.
[0013] FIG. 2A to FIG. 2E Demonstrated audit port with recoverability.
[0014] Figure 3A , Figure 3B , Figure 4B , Figure 4C , Figure 5A , Figure 5B , Fig. 6E and Figure 7 Each illustrates an environment for logging the processing of structured data items using executable logic.
[0015] FIG. 6A to FIG. 6D Each is an illustration of a flow chart.
[0016] Figure 8 is a diagram of an example process that outputs audit data on an audit port with recoverability. DETAILED DESCRIPTION
[0017] The system described herein provides a dedicated audit port on a component of a data flow graph, which is also referred to as a calculation graph or a data processing graph. Generally, a data flow graph is a computer program in the form of a graph, expressing a calculation and having a first vertex representing a first process of calculation, a second vertex representing a second process of calculation, and a link connecting the first vertex and the second vertex and representing the data flow between the first process and the second process, wherein the first vertex and the second vertex each have a state associated with it, the link has a communication method associated with it, the connection between the first vertex and the link has a first access method associated with it, and the connection between the second vertex and the link has a second access method associated with it. The vertex of the graph can also be referred to as a component. A component can usually be executed by including or referring to a part of an executable logic to be applied to data, and has an input port and an output port for receiving, processing (by applying this part of the executable logic to data) and outputting data, as well as a log port and an error port and an audit port. When a component experiences an error, the data on the log port and the error port are lost because there is no recoverability. In contrast, the data on the audit port is important and represents a transaction that has occurred. Meaningful metrics are derived from the audit data, such as how many transactions have been completed or how many offers have been accepted by consumers. In this way, the audit port implements recoverability to provide recoverability of its own audit data. By having a separate audit port, the audit port is able to output the audit data in a recoverable manner without affecting the performance of the output port because the audit port provides an orthogonal data stream.
[0018] refer to Figure 1, the system 10 for logging includes a data source 12, which may include one or more data sources (such as storage devices or connections to online data streams), each of which may store data in any of a variety of storage formats (e.g., database tables, spreadsheet files, flat text files, or native formats used by the host). An execution environment 14 (e.g., a data processing system) includes an execution module 16 and a performance monitoring module 18. The execution environment 14 may be hosted on one or more computers under the control of a suitable operating system such as a UNIX operating system. For example, the execution environment 14 may include a multi-node parallel computing environment, which includes a configuration of a computer system using multiple processing units (e.g., central processing units, CPUs), which may be local (e.g., a multi-processor system such as an SMP computer), or locally distributed (e.g., multiple processors coupled as a cluster or MPP), or remote, or remotely distributed (e.g., multiple processors coupled via a LAN or WAN network), or any combination thereof.
[0019] Execution module 16 implements logging (e.g., graph logging, data flow graph logging, etc.) that provides detailed information about one or more portions of executable logic that are executed when processing data records or data items and / or the state of the data and the path the data takes through a computer program (e.g., graphs such as data flow graphs as described in U.S. Patent No. 5,966,072, entitled "Executing Computations Expressed as Graphs," or as described in U.S. Patent Publication No. 2016 / 0342396, the entire contents of each of which are incorporated herein by reference). In this application, the term "portion of executable logic" may mean a subset of the executable logic, for example, less than 100% of the executable logic. In some examples, execution module 16 implements the logging by generating an audit record that includes, for example, a data record having one or more fields, wherein each field stores a value representing a characteristic or attribute of processing a particular data item having a particular portion of the executable logic. In the following, audit records may also be referred to as log records or audit data--as described in further detail below. In some examples, execution module 16 implements log records for each unique key (e.g., identifier (ID)), and the data records for each unique key (also referred to herein as "data items") are processed by a computer program. In some examples, execution module 16 implements data log records for a computer program expressed as a graph-based calculation, as described in U.S. Patent Publication 2009 / 0327196, the entire contents of which are incorporated herein by reference. In this application, the term "data flow graph" or "chart" is used as an example or alternative to "computer program". In some examples, for the purpose of convenience, the term "data flow graph" and "chart" are used interchangeably and without distinction in this article. In some examples, a unique key represents the customer ID of each customer, and its records are processed by a chart. User interface display log record information, such as metrics (e.g., metrics indicating which parts of the graph are traversed, and summary metrics indicating how many times a set of records traverse a specific part of the graph). The metrics described throughout this specification may be determined dynamically (e.g., during execution of the executable logic). In other words, these metrics are not static but may change dynamically and may be continuously updated in real time. The metrics described throughout this specification may provide information about the computational performance attributes of one or more portions of the executable logic executed when processing data items. In this example, the graph (e.g., data flow graph) may be stored in a data storage device 26 (also referred to herein as memory 26).The storage device providing data source 12 may be local to execution environment 14, for example, stored on a storage medium connected to a computer running execution environment 14 (e.g., hard drive 17), or may be remote from execution environment 14, for example, hosted on a remote system (e.g., host 20) that communicates with the computer running execution environment 14 via a local area network or a wide area network.
[0020] The execution environment 14 communicates with a data store 26 that stores data used to display one or more user interfaces. The data store 26 may also be accessed by a development environment 28 (e.g., which may be optional) for developing user interfaces and diagrams (with logging capabilities) stored in the data store 26 that are accessed by the execution module 16 (when executing the diagram) and used to display the user interface.
[0021] In some examples, a chart or data flow graph is a computer program executed in a data flow graph execution environment that processes data from one or more data sources. The data from the data source is operated and processed according to the computer program and exported to one or more data receivers. For example, the data source and the receiver may include a file, a database, a data stream, or a queue. The data flow graph is represented as a directed graph, which includes: nodes representing data processing components, each node including a program code (e.g., executable logic) for processing data from at least one data input and providing data to at least one data output; and nodes representing data set objects for accessing data sources and / or receivers. These nodes are connected by directed links, which represent the data flow between components, starting from the data source and ending at the data receiver. The data output port of the upstream component is connected to the data input port of the downstream component. The data flow graph can be reused for different data sources and different data receivers represented by the data set object. For example, the data structure and program code used to implement the data flow graph can support a variety of different configurations by parameterization, such as to make different sources and receivers easy to replace. In addition, in some arrangements, the flow of data in the data flow graph can be changed by using parameters so that a component or a series of components can be bypassed. Typically, a parameter represents a property of a component of a dataflow graph that can be configured with a value and whose value can be changed. For example, such a property can be changed between uses of a dataflow graph, and as a result of the change, the dataflow graph can perform operations in different ways.
[0022] In this example, the data flow graph includes a graphics layer and a data processing layer. The graphics layer includes visual data (e.g., nodes, data processing components, ports, and arrows or links) in a graphical user interface. For convenience and unlimited purposes, data processing components and the underlying program code (e.g., executable logic) represented by the data processing components can be collectively referred to as "data processing components" herein. Similarly, for convenience and unlimited purposes, ports (e.g., visualization of computer ports or interfaces) and the underlying program code (e.g., executable logic) represented by the ports can be collectively referred to as "ports" herein. In some examples, a port is an interface to a system, an interface to a software program or executable logic, or a communication path to a system, software program, or executable logic. For convenience and unlimited purposes, data flow graphs and the underlying program code (e.g., executable logic) represented by the data flow graphs can be collectively referred to as "data flow graphs" herein. For convenience and unlimited purposes, charts and the underlying program code (e.g., executable logic) represented by the charts can be collectively referred to as "charts" herein. The executable logic that is executed is part of the data processing layer, while the nodes, components, and arrows that are displayed are part of the graphical layer and encapsulate information about how to execute the executable logic represented by the dataflow graph.
[0023] In some cases, the construction of a data flow graph can be highly technical in nature. Although written to achieve a specific business purpose, the underlying structure and construction of the graph is determined based on technical considerations. For example, graph components may be selected to maximize reusability, or to support parallel processing. Some of the parameters associated with such parameterized data flow graphs can be used to enable business users to customize the data flow graph without requiring the user to understand the technical complexities behind its implementation. Parameterized data flow graphs simplify customization and facilitate reuse.
[0024] In other words, Figure 1The computer system 10 shown in the figure may include: an optional development environment 28 coupled to a data storage device 26, wherein the development environment 28 is configured to build a computer program associated with a graph or data flow graph that implements a calculation, which is performed on data flowing from one or more input data sets through a graph or graph of data processing components to one or more output data sets, wherein the graph or data flow graph is specified by a data structure in the data storage device 26, and the graph or data flow graph has a plurality of nodes specified by the data structure and representing data processing components connected by one or more links, and the links are specified by the data structure and represent the flow of data between the data processing components; an execution environment 14 coupled to the data storage device 26 and hosted on one or more computers, the execution environment 14 including an execution module 16, which is configured to read a stored data structure specifying the graph or data flow graph and to allocate and configure computing resources for performing calculations on the data processing components assigned to the graph or data flow graph by the execution module 16, wherein the execution module 16 is configured to schedule and control the execution of the assigned calculations on the data processing components to enable the execution of any of the methods or processes described herein. The execution module 16 is configured to read data from the data source 12 and process the data using an executable computer program expressed in the form of a graph or a data flow diagram.
[0025] In this example, the audit record (or also referred to as a "log record") includes a data item (e.g., a structured data item) that specifies the nodes traversed (and / or the data processing components executed) when processing a data record or event and one or more attributes of the processed data record or event. In some examples, the log record is a structured data record whose first field is used to store data representing the specification state of a specific value of a key. Typically, a specification is a visual representation of executable logic (e.g., executable logic executed when processing a data item). In this way, the specification can be or include a chart, a data flow diagram, etc. As described in further detail below, the specification is part of the graphical layer. Typically, the executable logic includes source code and other computer instructions. The log record may include a second field for storing data that specifies which parts of the executable logic represented by the specification have been executed for the specific value of the key from the time the executable logic was started to the current time (and / or on which machine among multiple distributed machines the logic was executed). Typically and in this example, when processing a data record (e.g., for a particular key), the system generates an audit record (or one or more audit records) and stores data representing the state of the specification for a particular value of the key in a first field, and stores data specifying which parts of the executable logic represented by the specification have been executed for the particular value of the key from the time when the executable logic was started to the current time in a second field. In still other examples, the log record is a structured data record with such a field, which is used to store data specifying which part of the executable logic represented by the specification has been executed for the particular value of the key at the current time. In this example, when generating the log record, the system populates the fields of the log record with data specifying which part of the executable logic represented by the specification has been executed for the particular value of the key at the current time. In still other examples, the specification is a graph, and one or more log records specify a path through the graph, which path represents which parts of the executable logic are executed when processing one or more data records associated with the particular value of the key. In this example, the system generates a log record that specifies a path through the graph, the path indicating which portions of the executable logic are executed when processing one or more data records associated with a particular value of the key. In yet other examples, the system generates one or more output records based on the processing of the one or more data items. In this example, each of the one or more output records includes data that is not included in the log record.
[0026] As described in further detail below, various types of audit records are predefined using various fields, each field type specifying one or more attributes of a processing data record or event. Typically, an event includes a data record (e.g., in a predefined format) representing a specific occurrence or non-occurrence of the event.
[0027] In this example, the performance monitoring module 18 collects audit records and other performance metrics about the performance of the execution module 16. The performance monitoring module 18 includes an audit log module 24 (also referred to herein as a "summarization engine") that collects audit records from the execution module 16, which can in turn execute a plurality of different graphs that each generate audit records from each data processing component. The audit log module 24 includes a classifier 24a that classifies audit logs, such as described herein. The audit log module 24 aggregates (e.g., at specified time intervals) received audit records by combining the contents of the audit records into a "master" log (e.g., a log record that includes the contents of the audit records received within a specified time period). These aggregated log records are stored in the data storage device 26 (e.g., for subsequent access) and / or in the data storage device 40 for transmission to the metric generation module 25, which generates metrics from the aggregated audit records. Typically, the metrics include detailed data at an aggregate level, including, for example, some or all of the number of records read, the number of bytes read, the number of records written, the number of bytes written, the processor time used, and the elapsed time. Metrics may also specify calculated delays for each event (e.g., data record) and, therefore, for each event type, by the system. Other types of latency metrics generated from the aggregated audit records include latency metrics (e.g., time from the time a data record is created at the source to the current time—the “now” time), external latency metrics that specify the delay from the time a data record is created at the source to the time the data record is first seen by the system, internal latency metrics that specify the delay from the time the data record is first seen by the system to the present time, and logical delay metrics that specify the delay from the logical time to the present time. The system may also calculate many other types of metrics, such as the execution time of a single process of a computer program (e.g., a single component in a data flow graph), the total program or graph execution time, the average program or graph execution time for a particular program or graph, and the like. The metric generation module 25 outputs the generated metrics to the data storage device 42.
[0028] Performance monitoring module 18 generates an output data set (e.g., from data retrieved from one or more of data storage devices 40, 42) that includes one or more metrics generated by metric generation module 25 and / or includes raw (and / or aggregated) data from collected audit records. In this example, environment 14 (e.g., a data processing system) displays one or more portions of the output data set in a user interface.
[0029] refer to Figure 2A, the execution module 16 stores (or accesses) the data flow graph 23 at the first time point T1. In this example, the data flow graph 23 includes a data source 23a, components 23b, 23c, 23e, 23j, 23l, 23m and data receivers 23z, 23n.
[0030] In this example, data flow graph 23 is an example of a computational graph that includes multiple vertices (i.e., source 23a, components 23b, 23c, 23e, 23j, 23l, 23m, and data sinks 23z, 23n) connected by unidirectional links. Computational graph 23 processes a workflow consisting of a series of work elements (such as a single transaction processed according to a computational graph associated with a transaction processing system). A transaction can be composed of multiple work elements. Each component is associated with a portion of the computation defined by the overall computational graph. In this example, component 23a provides access to storage of an initial series of work elements associated with one or more transactions and passes the series to its output link. The process that implements the computation associated with each of the components processes the work elements and typically generates work elements on one or more of the output links of the vertex.
[0031] In this example, component 23e includes an input port 23d for receiving input records, an output port 23f for outputting output records, an error log port 23g for outputting error logs, and an audit port 23h for outputting audit data. In this example, audit port 23h is configured to output audit data to a data receiver 23n, for example, a dedicated data source for collecting audit data. Similarly, component 23j has an output port 23o (for outputting output records to a data receiver 23z), an error log port 23p (for outputting error logs to an error handler component 23l), and an audit port 23q (for outputting audit data to a data receiver 23n).
[0032] In this example, the data flow graph 23 includes an error handler component 231, which receives an error message, for example, when an error occurs during processing of a component (e.g., 23r, 23j). The data flow graph 23 also includes a rollback component 23m, which restores or rolls back the data flow graph to a state before the error. Typically, a rollback allows the processed data to be rolled back to the point where a failover occurred, thereby "rolling back" the processed data to a previous consistent point. Specifically, the input data from component 23j informs the error handler component 231 of what the state of things (e.g., data or components) was before FIG. 23 began processing the transaction and specifically what the state of FIG. 23 was before component 23e began execution. The error handler component 231 outputs this to the rollback component 23m, which uses it to restore any data modified by other components to the state of the data before the transaction was executed, as described in further detail below.
[0033] refer to Figure 2B , the data processing system executes the data flow graph at a second time point T2. In this example, source component 23a transmits input record 23r to component 23b, which processes input record 23r and output record 23s. Record 23s is then input into component 23c, which processes record 23s and outputs record 23t based on the processing. Component 23c transmits record 23t to component 23e and component 23e receives input record (record 23t) at input port 23d. Furthermore, component 23e processes input record 23t and outputs output record 23v on output port 23f. Component 23e can also implement a spot check operation to check which records (i.e., record 23t) are processed by component 23e. The spot check operation generates a checkpoint message 23u, which specifies that the current state of the data flow graph 23 is a state in which component 23e has successfully executed and processed record 23t. Component 23e also outputs a checkpoint message 23u on output port 23f. In this example, since component 23e successfully processed its input record, no information is output on error log port 23g. Component 23e outputs audit data 23w describing one or more parts of the transaction represented by input record 23r on audit port 23h.
[0034] refer to Figure 2C, at the third time T3, the execution of data flow graph 23 proceeds to component 23j, which includes output port 23o, error log port 23p and audit port 23q. In this example, component 23j receives record 23v and checkpoint message 23u as input. Component 23j encounters an error when processing input record 23v. At this moment, the audit data from processing input record 23v at component 23j is lost, because component 23j encounters an error and cannot complete the processing of input record 23v and cannot output audit data from processing. Like this, component 23j outputs error log 27a on error log port 23p, which error log includes the state of the component that was successfully executed at last and also specifies which input records (that is, record 23t) were successfully processed by this component at last. At this moment, component 23j outputs error log 27a to error handler component 23l (which is now enabled). The error handler component 231 transmits a message 27b to the rollback component 23m (which is now also enabled) that specifies the state of the last successfully executed component (component 23e) (i.e., state C3) and which records were processed by that component. Figure 2D As shown in , rollback component 23m uses message 27b to return the state of data flow graph 23 to the state where component 23e is being executed and record 23t is being processed, thereby achieving recoverability of the audit data of component 23j.
[0035] refer to Figure 2E , the state of the data flow graph 23 returns to the state where component 23e executes and processes input record 23d. In this example, component 23j is able to successfully process input record 23v and output output record 27d stored in data receiver 23z on output port 23o. In this example, no error log is output on error log port 23p, and audit port 23q outputs audit data 27c stored in data receiver 23n. In this example, audit data 27c is recovered audit data. The audit data 23w, 27c collected and stored in data receiver 23n can be summarized and combined together as described herein.
[0036] refer to Figure 3A , environment 10 includes collecting audit records from programs (eg, graphs) executed by execution module 16 (and generating metrics from those collected audit records).
[0037] The execution module 16 executes a computer program such as the data flow graph 30. In some embodiments, the data flow graph 30 may be precompiled. In this example, the execution module may be, for example, a process or a group of processes being executed by a computer system. The graph may be a computer program that can be stored in a non-transitory computer readable storage device such as the data storage device 26 ( Figure 1)) in a computer readable instruction set. The data flow graph 30 may be stored from a data storage device (e.g., Figure 1 The data storage device 26) is loaded.
[0038] In this example, data flow graph 30 includes component 32, which includes input port 32a, output port 32b and audit port 32c. Audit port may also be referred to as "log port" or "interface" in this article. Generally, audit port is a port for outputting audit records (also referred to as "log" or "log record" in this article). In some examples, audit port includes an interface for outputting audit records or log records, and audit port can be a dedicated port for outputting only audit records. In this example, audit port is different from each of input port and output port. In this example, component 32 reads data from data source 34 and receives the read data through input port 32a. In this example, component 32 (or executable logic represented by component 32) generates one or more audit records and outputs those audit records on port 32c. Component 32 is connected to another component 36 by link 38. Component 36 includes input port 36a, output port 36b and audit port 36c. Data records from output port 32b of component 32 are passed to input port 36a of component 36. In general, a port refers to any mechanism by which a component of a dataflow graph can receive or provide data. A port can be, for example, a Transmission Control Protocol (TCP) / Internet Protocol (IP) port, a network socket, or a software pipe. A port can also refer to other methods of communication between components, such as reading and writing to shared memory.
[0039] Component 36 may generate output records 51a-51b on output port 36b ( Figure 3B ) and the output record can be stored in the data storage device 38. In this example, the component 36 monitors and records the performance characteristics of the component (32 or 36) and generates an audit record including these monitoring and / or performance characteristics. For example, the component 36 can collect performance metrics (such as any one or combination of processor time used, elapsed time, number of bytes read, number of records read, number of bytes written, number of records written, number of executions, number of failed executions, total duration, average record processing rate (records / second), average byte processing rate (bytes / second), etc.) and include these collected metrics in the audit record.
[0040] Audit records may be generated on audit port 36c of component 36 (eg, by component 36 or by underlying executable logic represented by component 36). For example, audit records may be one or more records including information about the performance of component 36 and other previously described information.
[0041] In this example, the audit records output on ports 32c, 36c are collected by the audit log module 24. In this example, the audit log module 24 performs post-processing on the audit records - as described in further detail below. The audit log module 24 also combines and / or rolls up the data in the collected audit records at specified time intervals (e.g., every hour). Generally, a "combine" operation refers to combining data from two or more different files or data records. Generally, a "roll up" operation refers to summing up data from one or more files or data records and outputting the summarized data. For example, a roll up operation can calculate a sum, a minimum value, and a maximum value based on the values in the fields in one or more data records. In this example, the audit log module 24 generates a summarized audit record at a specified time interval (e.g., an audit record generated from performing a combine and / or roll up operation on a single audit record collected during the specific time interval). The audit log module 24 transmits the summarized audit record to the data storage device 40 in the performance monitoring module 18. In some embodiments, the data storage device 40 is selected to minimize the performance impact of writing the audit record. For example, it may be advantageous to reduce the latency introduced by writing audit records to the data store 40. In some embodiments, the data store 40 may be located in a shared memory 40. The shared memory 40 may be a fast memory such as a random access memory (RAM). Operations writing to a shared (e.g., semiconductor) memory typically incur less overhead and are therefore faster than similar operations writing to a slower memory, i.e., a persistent data store such as a disk (e.g., a hard disk). In this example, the audit log module 24 stores the collected audit records in batches in the data store 40, for example, by storing the collected audit records together with the summarized audit records at specified time intervals.
[0042] Metric generation module 25 intermittently (e.g., every five, ten, or thirty minutes) reads the aggregated audit logs from data storage device 40 and generates performance metrics from the aggregated audit logs. Metric generation module 25 can read the aggregated audit logs (and / or individual audit logs, when such audit logs are saved) from data storage device 40 and can further process and aggregate the data. For example, metric generation module 25 can combine performance metrics associated with multiple programs or data flow graphs based on the data included in the audit logs. Figure 1The performance monitoring module 18 may graphically present the performance metrics to the user by including those metrics in the output data set or otherwise output those metrics to the user. The metric generation module 25 writes the performance metrics to the data storage device 42. In some embodiments, the data storage device 42 may be located in a persistent data storage device. The data storage device 42 then outputs at least a portion of the generated metrics as an output data set. In this example, components 32 and 36 represent two illustrative examples of two different parts of the executable logic of a computer program.
[0043] Figure 3B is to show the data flow Figure 3A A variation of . In this example, component 32 receives (via input port 32a) data records 44a to 44d from data source 34. The component processes each of the received data records 44a to 44d. In this example, component 32 generates an audit record for each of the processed data records. In this way, component 32 generates audit records 50a to 50d for data records 44a to 44d, respectively. In this example, component 32 outputs each of audit records 50a to 50d on port 32c and transmits audit records 50a to 50d to audit log module 24 via port 32c. In this example, port 32c is a dedicated intervention for generating only audit logs and sending only those audit logs to audit log module 24.
[0044] In this example, after processing each of records 44a to 44d, component 32 outputs records 46a to 46c on output port 32b, which are transmitted to input port 36a of component 36. In this example, the processing of records 44a to 44d by component 32 produces only three output records 46a to 46c. Component 36 processes records 46a to 46c and generates audit records 48a to 48c for each of records 46a to 46c, which are output on output port 36c and transmitted to audit log module 24. In this example, audit log module 24 collects audit records 48a to 48c and 50a to 50d and performs roll-up and / or join operations on audit records 48a to 48c and 50a to 50d to produce summarized audit records. Audit log module 24 transmits the summarized audit records to data storage device 40 (e.g., shared RAM). Metric generation module 25 reads the summarized audit records and in turn generates metric data (or data records) representing the various metrics described herein. Metric generation module 25 transmits the metric data to data storage 42 for storage and output—at least a portion of the metric data—to an output data set.
[0045] refer to Figure 4A, a variation of the environment 10 is shown. In this variation, the execution module 16 includes two different systems: a data processing system 62 (also referred to herein as the first processing system 62) and a data processing system 64 (also referred to as the second processing system 64). The processing system 62 includes a memory 62a and an execution subsystem 62b. The processing system 64 includes a memory 64a and an execution system 64b. Each of the processing systems 62, 64 is configured to execute various types of executable logic of a computer program, including, for example, executable logic representing a data flow graph, a chart, etc. In this example, the audit log module 24 receives output (e.g., audit or log records) from each of the systems 62, 64. The systems 62, 64 are distributed processing systems.
[0046] refer to Figure 4B , environment 60 includes data processing systems 62, 64 (included in execution module 16) and execution environment 14, which also includes performance monitoring module 18. In this example, data processing system 62 executes executable logic representing a data flow graph 66a (which is a version of a specific data flow graph) having components 74a, 75a, 76a, 78a, 80a, 82a, 84a and data storage devices 86a, 88a. In the following, the execution of a data flow graph and / or a graph refers to the execution of executable logic representing the data flow graph or the graph, respectively. In this example, the components can represent multiple data processing entities (e.g., one or more CPUs) connected by one or more links representing data flows between the data processing entities. Data processing system 64 executes executable logic representing a data flow graph 66b, which is another version of the specific data flow graph executed by data processing system 62. In this example, data processing systems 62, 64 are distributed systems for parallel processing of incoming received data records (e.g., data records received from one or more data streams). In variations, each of the data processing systems 62 , 64 may execute a different data flow graph.
[0047] In this example, each of the data flow graphs 66a, 66b performs a process on data items included in the real-time input data stream. In this example, each of the data processing systems 62, 64 (respectively) executes the data flow graphs 66a, 66b to generate data records (e.g., a wide record of all recorded and / or received events) and publishes the generated records to the data storage devices 86a, 86b (respectively).
[0048] The data flow graph 66a includes a subscription component 75a, which subscribes to one or more source data readers (e.g., receives data from one or more source data readers). Through the subscription component 75a, the data flow graph 66a accesses the data items included in the real-time data stream in real time. In this example, the subscription component 75a receives a real-time input data stream (e.g., including thousands of records) from a data queue (e.g., some initial processing can be performed on the data stream to ensure its readability). The data flows from the subscription component 75a to the partition component 74a, which partitions or divides the data items (including events) received in the data stream by event type. In this example, the partition component 74a is configured to detect different types of data records or events and partition various types of events to other components configured to process specific types of events. In this example, each of the components 76a, 78a, 80a, 82a is configured to process specific types of events.
[0049] In this example, the data flow graph 66a includes a combination component 84a that implements a combination operation. The combination operation combines various types of data, for example, events from components 76a, 78a, 80a, 82a. In this example, data flows from components 76a, 78a, 80a, 82a to a combination component 84a that combines events together in a wide record (e.g., a data record having a field for recording values included in various processed records). In this example, each of the event components 76a, 78a, 80a, 82a sends an event associated with data uniquely identifying an entity associated with the event, such as an entity's user ID, to the combination component 84a. The output of the combination component 84a is a data record with data that specifies which specific predefined events and user IDs are associated with those events. However, in a variant, the output of the combination component 84a can be a data record with content that represents the processing of the received data record by components 76a, 78a, 80a, 82a. In this example, data (eg, data records) flows from the join component 84a to the data storage device 86a, wherein the data records output from the join component are output.
[0050] In this example, each of the components 74a, 75a, 76a, 78a, 80a, 82a, 84a (executing on the data processing system 62) has an audit port 74c, 75c, 76c, 78c, 80c, 82c, 84c that each outputs one or more audit records representing one or more attributes of processing one or more data records by the corresponding component. For example, audit port 74c outputs an audit record indicating that a partition operation was performed on a data record received from component 75a. Data storage device 88a executing on data processing system 62 receives the audit records output on audit ports 74c, 75c, 76c, 78c, 80c, 82c, 84c and transmits those audit records (e.g., at a specified time, such as every hour) to audit log module 24 for further processing as described herein. In this example, components 74a, 76a, 78a, 80a, 82a, and 84a represent illustrative examples of different portions of the executable logic of a computer program, here dataflow graph 66a.
[0051] The data flow graph 66b includes a subscription component 75b that subscribes to one or more source data readers (e.g., receives data from one or more source data readers). Through the subscription component 75b, the data flow graph 66b accesses the data items included in the real-time data stream in real time. In this example, the subscription component 75b receives a real-time input data stream (e.g., including thousands of records) from a data queue (e.g., some initial processing can be performed on the data stream to ensure its readability). The data flows from the subscription component 75b to the partition component 74b, which partitions or divides the data items (including events) received in the data stream by event type. In this example, the partition component 74b is configured to detect different types of data records or events and partition various types of events to other components configured to process specific types of events. In this example, each of the components 76b, 78b, 80b, 82b is configured to process specific types of events (e.g., records).
[0052] In this example, data flow graph 66b includes a combination component 84b that implements a combination operation. The combination operation combines various types of data, for example, events from components 76b, 78b, 80b, 82b. In this example, data flows from components 76b, 78b, 80b, 82b to combination component 84b, which combines events together in a wide record (e.g., a data record with fields for recording values included in various processed records). In this example, each of event components 76b, 78b, 80b, 82b sends an event associated with data uniquely identifying an entity associated with the event, such as an entity's user ID, to combination component 84b. The output of combination component 84b is a data record with the following data, which specifies which specific predefined events and user IDs are associated with those events. However, in a variant, the output of combination component 84b can be a data record with the following content, which represents the processing of the received data record by components 76b, 78b, 80b, 82b. In this example, data (eg, data records) flows from the join component 84b to the data storage device 86b, wherein the data records output from the join component are output.
[0053] In this example, each of components 74b, 75b, 76b, 78b, 80b, 82b, 84b (executing on data processing system 64) has an audit port 74d, 75d, 76d, 78d, 80d, 82d, 84d that each outputs one or more audit records representing one or more attributes of processing one or more data records by the corresponding component. For example, audit port 74d outputs an audit record indicating that a partition operation was performed on a data record received from component 75b. Data storage device 88b executing on data processing system 64 receives the audit records output on audit ports 74d, 75d, 76d, 78d, 80d, 82d, 84d and transmits those audit records (e.g., at a specified time, such as every hour) to audit log module 24 for further processing as described herein. In this example, components 74b, 76b, 78b, 80b, 82b, and 84b represent illustrative examples of different portions of the executable logic of a computer program, here dataflow graph 66b.
[0054] In this example, audit log module 24 receives audit records from distributed systems (e.g., data processing systems 62, 64), where each of those distributed systems in turn partitions the processing of data records received from components 75a, 75b, respectively. In effect, audit log module 24 is receiving audit records from data flow graphs that are partitioning the data records and are being executed in a distributed manner across many different systems. In this example, each of components 74a-b, 75a-b, 76a-b, 78a-b, 80a-b, 82a-b, 84a-b has an audit port. In some variants, only a subset of the components included in the data flow graph may have an audit port.
[0055] In this example, each of the distributed data processing systems 62, 64 is different from each of the execution environments 14 and 10 ( Figure 1 ). The audit log module 24 receives one or more log records from each data processing system 62, 64, based on which the distributed data processing system combines the contents of the log records received from each of the distributed data processing systems 62, 64. In this example, each of the data processing systems 62, 64 intermittently receives one or more data items of a data stream through an input device or port (e.g., components 75a, 75b, respectively). For at least one of the one or more data items received, each of the data processing systems 62, 64 processes the data item, including executing a first portion of executable logic represented by Figures 66a, 66b, respectively, on the data item. For example, the first portion may include executable logic representing a partition component (e.g., one of components 74a, 74b). In this example, each of the systems 62, 64 accesses (e.g., from Figure 1In response to the operation, each of the systems 62, 64 generates a first log record (wherein the first log record specifies a first portion of the executable logic executed when processing the data item on the distributed data processing system) and outputs the first log record to the audit log module 24 via an interface (e.g., via audit ports 74c, 74d, respectively) in the following manner: for example, by first transmitting the first log record to the data storage device 88a (e.g., shared RAM), which in turn transmits the first log record or log records (e.g., in real time, at a specified time and / or in batches) to the audit log module 24. The log record includes attribute values of a particular component processing a data record or event. There are various types of attributes, including, for example, latency attributes (specifying the amount of time that elapses from when a component begins processing a data record until the component completes processing of the data record(s), key attributes (specifying the keys included in the processed data record), content attributes (specifying the content of the data record being processed by the component), start time attributes (specifying the time when the component began processing), etc. In this example, the audit log module 24 combines the contents of the received audit log together and stores them in a memory (e.g., data storage device 40). The memory may be a RAM or a hard disk. The metric generation module 25 then accesses the combined content from the memory and generates one or more metrics from the combined content.
[0056] exist Figure 4BIn the example of , each component in the data flow diagram 66a, 66b includes an audit port to achieve more accurate and fine-grained audit log records. In addition, the audit log module 24 collects audit records from multiple systems 62, 64, wherein the records processed by each of the systems 62, 64 are partitioned during processing, for example, via partition components 74a, 74b. In this way, the audit log module can collect audit records from multiple partitioned systems. The benefit of doing so - not that each of the systems 62, 64 collects its own audit records individually - is that the audit log module 24 can detect multi-system errors (e.g., errors occurring across two systems 62, 64) and errors within the execution module itself. Means for detecting such errors have been provided and then ultimately correcting these detected errors provide means for improving system performance and reducing latency. In the example, a specific component (e.g., a component represented by components 78a, 78b in Figures 66a, 66b) is a component for reformatting data. In this example, the specific component cannot properly perform reformatting only for certain incoming data types. When the data flow graph runs across multiple systems (e.g., one hundred systems), a particular component is executed on one hundred different systems. Since the audit log module 24 is collecting audit records for components 78a, 78b (via a dedicated output port on component 78a) when components 78a, 78b are executing across multiple (such as the one hundred) systems (where each of the one hundred systems processes different types of data or different data formats), the audit log module 24 is able to identify certain data types or formats that a particular component cannot correctly perform reformatting. By centrally collecting all audit records in the audit log module 24, the audit log module 24 can identify collective error data-various different types of data types or formats that components 78a, 78b cannot reformat. In this way, the collective error data can be used by the execution environment 14 to correct system-wide problems for a particular component (e.g., by editing executable logic stored in a central library or repository of components 78a, 78b), rather than performing a single repair on each specific machine, which will ultimately take more time and may not even be able to solve the problem of the new data type. Therefore, the computing performance of the underlying data processing system is improved.
[0057] Additionally, because the audit log module 24 collects audit records centrally across all machines or multiple machines, the execution environment 14 (e.g., data processing system) generates real-time audit records and metrics on a large scale (e.g., audit records of billions of events are generated and processed every day). This is because each system (e.g., system 62, 64) that executes a computer program (such as a chart or data flow graph 66a, 66b) automatically pushes audit records to the audit log module 24 in real time (i.e., while the program / chart / data flow graph is currently being executed), so that the audit records are already accessible to the audit log module 24 during the execution of the program / chart / data flow graph and before the execution is completed, rather than storing the audit records only locally on the system and then pushing the records to the audit log module at a specified time. The audit records can be pushed directly to the log module 24 without going through any local storage of audit records on the corresponding system 62, 64 or can be stored locally in data storage devices 88a, 88b, which are already accessible to the log module 24 during the execution of the program / chart / data flow graph. The data storage devices 88a, 88b can be shared memory (e.g., RAM) that is beneficial for reducing latency even if the records are stored locally. In addition, each component has a dedicated audit port, which further enables the audit log module to receive audit records in real time when the component processes data records or events. This reduces the latency of receiving audit records. For example, if audit records are generated only when a chart or data flow graph is completed, latency will be introduced by waiting for the system to complete the execution of the chart or data flow graph, which in turn increases the latency of correcting errors (represented by audit records) and / or generating metrics. Relative to the latency of receiving audit records only when the chart is completed, the latency of receiving audit records and calculating metrics based on those audit records is reduced by having a dedicated audit port for each component and automatically pushing audit records (e.g., directly without storing the records locally or via shared memory) to the audit log module as soon as the component completes execution (e.g., as soon as the executable logic represented by the component completes execution).
[0058] In addition, the distributed nature of the graph (running on multiple machines) and the partitioning of the processing of records by each graph enable the audit records received by the audit log module to be provided in real time and / or near real time (e.g., live time) as each component in the graph processes an event or data record, which in turn provides real-time metric generation and thus real-time identification of metric-based errors. The execution environment 14 is able to process audit records on a large scale because the audit records are generated in parallel across multiple systems (as each component processes a data record or event), which in turn each partition the received records for processing. That is, as more and more data records are received or more and more queues are subscribed to and more data records are received through those queues, the execution environment 14 is able to process these data records on a large scale, for example, by adding more systems (executing appropriate graphs and / or charts, where each component of the graphs and / or charts includes a dedicated audit port) to handle the increase in data records and process them on a large scale, which in turn enables the audit records to be generated on a large scale and in real time as each component executes. The execution environment 14 is able to process these records on a large scale because the generation of audit records (and the streams resulting from the generation of audit records) is separate from the data streams that process the data streams. That is, by having a dedicated port for generating audit records, the generation of audit records is orthogonal to the data flow (obtained by data processing), and the generation of audit records does not affect the data flow. That is, the data processing system includes multiple processing systems (or subsystems), each of which includes a dedicated log port to enable log data to be output in a data flow that is different from the data flow of data items being processed by multiple subsystems of the data processing system, so that the output of log data does not introduce latency into the processing of the data. Specifically, it can be considered that the output (e.g., audit records) is asynchronous with the data flow. Data processing performed by a data flow graph is the first layer of data processing. The audit port is the second layer. In this way, the performance of the second layer does not affect the first layer. By generating audit records in this second (asynchronous) layer, the generation of records is not affected. This allows large-scale generation of audit records and metrics (generated based on audit records).
[0059] Additionally, execution environment 14 is capable of generating metrics at scale because the system is capable of processing data records at scale (e.g., by implementing additional systems that run appropriate data flow graphs and / or diagrams), as described above. Additionally, metrics are generated in real-time or near real-time (as each component across all execution systems completes execution) because metrics are generated in response to and as audit records are received and processed (which is real-time or near real-time).
[0060] By centrally collecting all metrics (e.g., by centrally collecting all audit records and generating metrics based on the centrally collected audit records), the execution environment 14 is able to generate and display more accurate, more granular, and aggregated metrics in real time. For example, the processing speed of a single component (e.g., component 78a) can be calculated in near real time as the component executes. For example, if component 78a is executing across one hundred different machines, and each audit record generated for component 78a (via a dedicated port of component 78a) includes a latency metric specifying when component 78a begins processing a data record and when component 78a completes processing the data record, the metric generation module can calculate the average latency (e.g., processing time) from when component 78a begins processing a data record to when component 78a completes processing the data record in near real time (relative to the time when component 78a executes (or the executable logic representing the execution of component 78a)). By being able to calculate metrics at the component level (as opposed to at the graph level), the metric generation module enables identification of errors and latency issues at the component level—issues that may be masked or difficult to detect in graph-level metrics. The receipt of audit records for each component (via a dedicated audit port) enables the metric generation module to generate metrics specific to these components. For example, if audit records are generated only at the level of each graph, the latency metric will only indicate the latency of the graph as a whole. In this example, a graph with a latency metric less than five milliseconds is considered acceptable. In this example, the average latency of the graph execution component 78a is four milliseconds. In this way, the system determines that the graph has acceptable performance. However, when the metric generation module calculates the average latency of the component 78a itself, the latency is fifteen milliseconds. The metric generation module is able to calculate the average latency (based on the execution of component 78a across multiple different systems) because each of the components 78a executed on each system in the system has a dedicated audit port, which generates audit records in near real time relative to the execution of component 78a, and those audit records are collected by the audit log module, which is then processed by the metric generation module to generate real-time fine-grained metrics. Because the metrics generation module is able to generate real-time metrics for individual components at an aggregate level, the system can identify latency or performance issues with individual components and resolve or otherwise remediate those latency issues (e.g., by modifying the executable logic representing the component), thereby reducing the latency of graph execution (relative to the latency of graph execution if individual component performance issues were not identified) and improving the performance of the system as a whole.
[0061] In this example, the processing system 62 includes a data storage device 88a for storing log data output. In this example, the processing system 62 includes a memory 62a (eg, a data flow graph or a diagram) for storing executable logic for processing data items. Figure 4A). A first portion (e.g., represented by component 82a) of executable logic (represented by data flow graph 66a) includes a first dedicated output port (e.g., output port 82c) for outputting log data to data storage device 88a, wherein the first dedicated output port outputs log data indicating processing of a data item by the first portion of the executable logic. In this example, a second portion (e.g., represented by component 80a) of executable logic includes a second dedicated output port (e.g., output port 80c) for outputting log data to data storage device 88a, wherein the second dedicated output port outputs log data indicating processing of a data item by the second portion of the executable logic. In this example, system 62 includes execution system 62b ( Figure 4A ), the execution system: for a first of the received data items, processes the first data item with the first portion of the executable logic. In this example, a second processing system (e.g., system 64) for processing data items is included in the environment 14. The second processing system includes a data storage device 88b and a memory (e.g., Figure 4A 64a in the second processing system). An execution system within the second processing system (e.g., Figure 4A The execution system 64b in the first processing system 62 is configured to execute the executable logic and generate log data stored in the data storage device. The monitoring system (e.g., the performance monitoring module 18) includes an input port (e.g., the input port 24a on the audit log module 24) for obtaining, retrieving or receiving the stored log data from the data storage device 88a of the first processing system 62 and for obtaining, retrieving or receiving the log data from the data storage device 88b of the second processing system 64.
[0062] For each of the first processing system 62 and the second processing system 64, a corresponding execution system (eg, Figure 4A62b, 64b) or executable logic in the processing system partitions multiple received data items so that two or more of the data items are assigned to partition components (e.g., component 74a or 74b) of the executable logic, wherein each partition component includes an output port (e.g., 74c, 74d) and an input port (e.g., 74e, 74f, respectively), and wherein the execution system performs operations including the following: for each partition component and for each data item received at the input port of the partition component, processing the data item assigned to the partition component; and outputting log data of the data item processed by the specified partition component on the output port of the partition component; wherein the data storage device of the processing system receives the log data output from the output port of the partition component of the processing system. In this example, the monitoring system 18 receives log data output from the respective data storage devices 88a, 88b of the first processing system 62 and the second processing system 64, wherein the received log data is output from the respective output ports 74c, 74d of the respective partition components 74a, 74b of the first processing system 62 and the second processing system 64.
[0063] refer to Figure 4C , environment 100 illustrates components included in audit log module 24. In this example, audit log module 24 includes a subscription component 104 that subscribes to a source data reader, multiple source data readers, and / or a database (e.g., Figure 4B The subscription component 104 includes a port 104a for receiving audit log records. The audit log module 24 also includes a subscription component 106 that subscribes to a source data reader, a plurality of source data readers, and / or a database (e.g., Figure 4B The subscription component 106 includes a port 106a for receiving audit records. In this example, the audit log module 24 also includes a combination component 112 and a classifier component 116.
[0064] In this example, the subscription component 104 (from Figure 4BSubscription component 104 may access audit records 101a to 101d in real time using data store 88a in the audit record. Subscription component 104 may perform some formatting of audit records 101a to 101d (e.g., such as to ensure that all field values are in a specified format) or may add some enhancement data (e.g., by accessing another data store (not shown) and retrieving data for a particular subscriber specified in the audit record and adding the retrieved data to the audit record as an enhancement). Subscription component 104 outputs audit records 108a to 108d to join component 112. Audit records 108a to 108d may be the same as audit records 101a to 101d or may be different (e.g., when one or more of audit records 101a to 101d are reformatted and / or enhanced).
[0065] In this example, the subscription component 106 (from Figure 4B The subscription component 106 may access the audit records 102a to 102d in real time using the data store 88b in the audit record. The subscription component 106 may perform some formatting of the audit records 102a to 102d (e.g., such as to ensure that all field values are in a specified format) or may add some enhancement data (e.g., by accessing another data store (not shown) and retrieving data for a particular subscriber specified in the audit record and adding the retrieved data to the audit record as an enhancement). The subscription component 106 outputs the audit records 110a to 110d to the join component 112. The audit records 110a to 110d may be the same as the audit records 102a to 102d or may be different (e.g., when one or more of the audit records 102a to 102d are reformatted and / or enhanced).
[0066] The combination component 112 combines the audit records 108a to 108d with the audit records 110a to 110d, for example, at specified time intervals (such as every hour). In some examples, the audit records 108a to 108d are combined with the audit records 110a to 110d by merging the contents and generating the combined audit records 114. In other examples, the audit records 108a to 108d are combined with the audit records 110a to 110d by a key. For example, if one or more audit records include a value for a specific key and a field associated with the key, the audit records for the value of the specific key (or the fields included in the audit records) are combined to generate an audit record for the key. The combination component 112 outputs the combined audit records to a classifier component 116, which classifies the audit records, for example, according to the audit record type determined according to the key. In this example, the classifier 116 identifies three different record types 116a to 116c, and then classifies these record types (e.g., time classification) through the classification component 117. The classified and categorized audit records are output to and stored in the data storage device 40 (eg, RAM).
[0067] refer to Figure 5A , shows an example of the classification of audit records. In this example, the classifier 24a (which may also be used with Figure 4C The classifiers 116 in the example (the same or different) classify various audit records according to the key value. In this example, the audit records 101a and 102c each have the same key value "Ops" and are therefore classified as a specific record type of the "operation category". The audit records 101b and 102b each have the same key value "fulfillment" and are therefore classified as a specific record type of the "fulfillment" category. The audit records 101d and 102a each have the same key value "consumer" and are therefore classified as a specific record type of the "consumer" category. Figure 5B, each of the record types 116a, 116b, 116c is classified by the classification component 117. In this example, audit records 101a, 102c are included in record type 116a. Audit records 101b, 102b are included in record type 116b. Audit records 101d, 102a are included in record type 116c. In this example, the classification component 117 is configured to classify by timestamp. Thus, audit records 102c, 101a in the operation category are classified by timestamp. Audit records 101b, 102b in the fulfillment category are classified by timestamp. Audit records 102a, 101d in the consumer category are classified by timestamp. The audit log module 24 outputs the classified audit log to the data receiver 40 for storage. In turn, the metric generation module 25 can query the data receiver to generate metrics, for example, for a specific user, within a specific time period, for a specific key, etc. In this example, the metric generation module generates the metric 103 .
[0068] In a variation, the audit log module 24 may include a single subscription component having a single input port (e.g., 104a) for receiving log records. As described in further detail below, FIG. 6A to FIG. 6D The progress is illustrated through a flowchart (as described in U.S. Publication No. US-2017-0344672, the entire contents of which are incorporated herein by reference) and shows that various nodes are executed at various states of the flowchart. FIG. 6A to FIG. 6D Shows various audit records generated in various states.
[0069] refer to Fig. 6A, diagram 129a shows a system 62 executing a flow chart 132 (specified by specification 130), wherein nodes 132a to 132g represent data processing components. In this example, each of nodes 132a, 132c, 132e has an audit port 132j, 132k, 132q, respectively, which is used to output one or more audit records including data (e.g., metadata) related to the execution of the corresponding node. In this example, the specification represents executable logic and specifies various states, wherein the state of the specification specifies one or more parts of the executable logic that can be executed in this state based on the state reached by executing the executable logic on the previous data item. In this example, one or more data processing components in the data processing component refer to a specific grouping or part of the executable logic. Typically, a diagram includes a template for processing data records. This template includes a logic diagram unit for reacting to input data records and generating output data records, for example, data records generated based on the logic included in the specification. A diagram is an example of a specification. Typically, a logic graph unit includes logic that is generated at least in part in a graphical manner, for example, by dragging and dropping various nodes from a template (not shown) into a window for building a graph. In an example, a node includes logic (not shown) that specifies how to process input data records, how to set variable values used by executable logic, which output data records are generated, for example, when conditions specified by the logic are met, etc. In an example, a node is programmable by a user who enters parameter values and / or variable values used in the node's logic.
[0070] The chart itself is executable because the logic in the node is compiled into executable logic, and each node corresponds to one or more parts of the executable logic. For example, the system transforms the specification (and / or the chart in the specification) by compiling the logic in the node into executable logic. Because the chart itself is executable, the chart itself can process data records and can be stopped, started and paused. The system also maintains the state of the flowchart 132, for example, by tracking which of the nodes 132a to 132g is currently being executed. The state of the flowchart 132 corresponds to the state of the executable logic represented by the flowchart 132. For example, each node in the flowchart 132 represents a specific state of the executable logic (wherein one or more parts of the executable logic are executable in this state). When the flowchart 132 is being executed for multiple values of a key, the system maintains the state of the flowchart 132 for each value of the key, for example, by maintaining the state of each instance as described in further detail below. Typically, an instance of a specification includes a specific implementation of the specification for a unique value of a key, for example, by executing the executable logic represented in the specification and maintaining the state of the executable logic for each unique value of the key. In this example, the flowchart 132 includes a state transition diagram in which each incoming data record drives a transition between nodes, and the data record is evaluated based on the state reached by processing the previous data record. The links between the nodes in the flowchart 132 represent the temporal flow of logic.
[0071] The specification 130 also includes a key 132h identifying a data record that the flowchart 132 processes including the key 132h or is associated with the key 132h. In this example, a custom identifier (ID) is used as the key. The key 132h may correspond to one of the fields of the data record, such as, for example, a subscriber_ID field, a customer_ID field, a session_ID field, etc. In this example, the customer_ID field is the key field. For a particular data record, the system determines the value of the key of the data record by identifying the value of the key field of the data record. In a variant, the specification may not be associated with a key and the flowchart may be executed independently of the key.
[0072] In this example, flowchart 132 subscribes to data records (e.g., data records of a specified type and specified when configuring flowchart 132). In this example, flowchart 132 subscribes to data records including key 32h. In this example, flowchart 132 and data record share a key. Typically, a flowchart subscribes to a type of data record by including logic for processing those data records including the key of the flowchart. When data record processing begins, the system starts a new flowchart instance for each new value of the key of the flowchart, for example, by maintaining the state of executable logic for each new value of the key (represented in the flowchart). Typically, a flowchart instance includes a specific implementation of the flowchart for a unique value of the key, for example, by executing executable logic represented in the specification and maintaining the state of executable logic for each unique value of the key. The system performs data record processing by configuring the flowchart instance (and therefore the underlying executable logic) to respond to data records of a specific key value. In the example, the flowchart subscribes to customer short message service (SMS) data records. The flowchart instance for a specific customer ID manages the data records for that customer. The number of flowchart instances can be as many as the number of customer IDs encountered in the incoming data records.
[0073] In this example, diagram 129a shows the start of execution of flowchart 132. In this example, node 132a represents the start of executable logic and is being executed (e.g., as indicated by the black line around node 132a). In this example, flowchart 132 is in a state of executing node 132a.
[0074] In this example, audit record 132i is generated when start node 132a is executed. In particular, start node 132a represents a data processing component that is configured to generate audit record 132i on audit port 132j, for example, when flowchart 132 is in a state where start node 132a is being executed (as shown in this example). In this example, when the audit record is generated, the system populates the fields of the audit record with data that specifies which portion of the executable logic represented by the specification has been executed for a particular value of a key at the current time. In this example, audit record 132i has three fields: a key field (having a value "32425" that specifies the value of the key for the record currently being processed by flowchart 132), a graph_id field (populated with a value "42424" that specifies an identifier of flowchart 132), and a component ("Comp") field (populated with a "start" value representing node 132a). In this example, audit records 132i are output on audit port 132j to data storage device 134, which collects audit records. At specified time intervals (e.g., every hour), data storage device 134 is configured to transmit the collected audit records to audit log module 24 ( Figure 4B ).
[0075] In still other examples, the specification is a graph, and one or more log records specify a path through the graph that indicates which portions of the executable logic are executed when processing one or more data records associated with a particular value of the key. In this example, the system generates log records that specify a path through the graph that indicates which portions of the executable logic are executed when processing one or more data records associated with a particular value of the key. In still other examples, the system generates one or more output records based on processing of the one or more data items. In this example, each of the one or more output records includes data that is not included in the log record.
[0076] Various types of audit records are predefined using various fields, and each field type specifies one or more attributes of a processing data record or event. Typically, an event includes a data record (e.g., in a predefined format) representing a specific occurrence or non-occurrence of the event. In this example, node 132a is editable, such as by clicking on node 132a, which makes it possible to display a configuration screen that can be used to configure the node and therefore can also be used to configure the data processing component represented by the node. A portion of the configuration can include turning on auditing (or also referred to as "logging") and specifying one or more types of audit records to be generated when traversing the node. In another example, the system described herein generates data for a user interface that displays visual representations of various types of audit records and provides controls for turning on / off various types of audit records. In this example, the system then generates a type of audit record (e.g., when logging of audit records of this type is enabled), which corresponds to or is associated with a specific type of node or data processing component. For example, audit record 132i is a chart start audit record for logging data for the start of a specified chart. Thus, if the user specifies to enable a graph start audit record, the system described herein identifies the correspondence between the graph start audit record and the start node (e.g., node 132a) based on the fact that the start node starts the graph and the graph start audit record logs the start of the graph, and generates a graph start audit record.
[0077] refer to Figure 6B, diagram 129f shows the state of flow chart 132 after completing node 132a. In this example, the state of flow chart 132 is converted to node 132b representing one or more other parts of executable logic. Node 132b includes a waiting node (hereinafter referred to as waiting node 132b). Waiting node 132b represents a waiting state of an input data record that a part of executable logic (corresponding to waiting node 132b) waits for one or more conditions to be satisfied. In an example, the waiting state can be a part of another state of flow chart 132, for example, the system executes the waiting node (to implement the waiting state) and then executes the state of one or more other nodes. After completing the part represented by the waiting node 132b of the executable logic, the system exits the waiting state and executes the node 132c representing the executable logic for implementing the decision. In this example, node 132c includes a decision node. Node 132c is referred to as decision node 132c hereinafter. Typically, a decision node includes a node containing logic (for example, logic for Boolean value) for executing a decision.
[0078] In this second state of flowchart 132, audit record 132r is on audit port 132k of node 132c (e.g., by Figure 1 16) is generated and output to the data storage device 134, which collects and stores the audit record 132r. In particular, the data processing component represented by the decision node 132c is configured to generate the audit record 132r, for example, when the flowchart 132 is in a state where the decision node 132c is being executed (as shown in this example). (In this example, the wait node 132b is also being executed in this state. However, the wait node 132b is not configured to generate the audit record.).
[0079] In this example, audit record 132r specifies the result of decision node 132c, for example, by specifying whether link 132l or link 132m was traversed. In this example, audit record 132 stores data specifying that link 132l was traversed in variable "Var decision". In a variant, the audit record can also specify, for example, variables that are read or set when the data processing component represented by decision node 132c is executed. In this example, audit record 132r does not need to include a key field because audit record 132r is transmitted to a processing system (e.g., an internal or external computing system that processes the audit records) at specified pulses (e.g., times or predefined time intervals), resulting in the audit records being transmitted in groups, blocks, or as part of a file.
[0080] For example, the system groups audit records together by key, wherein first audit records associated with a first key are grouped into a first group, and wherein second audit records associated with a second key are grouped into a second group. In this example, the audit records are grouped together by key in the system (e.g., Figure 1 The audit records are grouped in a memory of a system 14 in the system. To improve space efficiency, the system groups audit records in memory so that the system does not have to duplicate data on sequentially related audit data, for example by including a key in each audit record. Each audit record includes one or more fields that indicate one or more attributes of a processed data item. For at least one audit record, the field is a key field that specifies the value of a key associated with the data item. Because the audit records are grouped together by key, a subset of audit records associated with a particular key does not include the key field. In some examples, only one of the audit records included in the group for a particular key includes the key field. In this example, the system writes the audit records associated with a particular key in a particular group to a file at the end of a predefined time interval (e.g., a pulse).
[0081] For example, the system identifies the key for the audit records in the group (which may be represented as a block or file) by parsing the contents of the first audit record or the chart start audit record. Because not every audit record needs to include the key field (as well as other fields such as the chart_ID field, which other audit records included in the same pulse as audit record 132i can look up in the chart start audit record), the system described herein can process the audit records in a manner that minimizes the impact on system resources (such as memory and processing power) relative to the impact on system resources (such as memory and processing power) when every audit record needs to include the key field to enable keyed logging of events. Additionally, the system described herein reduces the latency of processing these audit records relative to the latency of processing audit records in which each audit record has the key field, because the size of the audit records is reduced because each audit record does not need to include the key field. Additionally, the system is configured to store the audit records in a partition (in memory) assigned to a particular key, thereby further reducing the need for each audit record to include the key field. At the beginning of a new pulse, the system described herein is configured to include the key field in the first audit record of the pulse.
[0082] refer to Figure 6C, diagram 129f shows the state of flowchart 132 based on the result of the decision represented by node 132c. In particular, based on the result of the decision represented by node 132c, the state of flowchart 132 transitions to node 132d, which is another wait node. (In a variant, the state of flowchart 132 can transition to node 132g (which causes the state transition back to node 132a)). After completing the portion of the executable logic represented by wait node 132d, the state of flowchart 132 transitions to node 132e including a send node. Typically, a send node includes a node representing executable logic for transmitting data to another system. After completing the portion of the executable logic represented by node 132e, the state of flowchart 132 transitions to node 132f including a completion node. Typically, a completion node indicates that the execution of the executable logic is completed.
[0083] In this third state of flow chart 132, audit record 132s is generated, for example, by executable logic represented by sending component 132e. In this example, audit record 132s specifies the action implemented by the data processing component represented by sending node 132e. Audit record 132s is output on audit port 132q of sending component 132e and collected by data storage device 134. In this example, audit record 132s includes a variable ("Var action"), the value of which specifies the action initiated by sending component 132e. In this example, sending component 132e initiates the action of sending an offer to update SMS service. Therefore, the value of "Var action" is set to "offer SMS update". Like audit records 132g and 132r, audit record 132s does not need to include a key field, because audit records 132r, 132s are transmitted as part of the same pulse including audit record 132g. Other audit records can specify the time of sending action, the value of variables read, the value of variables set, etc. In this example, the data storage device 134 captures the audit records 132i, 132r, 132s and then transmits them (eg, at a specified time) to the performance monitoring module for further processing, eg, using the techniques described herein.
[0084] refer to Fig.6D, diagram 140 shows system 64 executing graph 144 (specified by specification 141), wherein nodes 140a to 140g represent data processing components. In this example, specification 141 is independent of keys and does not merely process data records associated with a particular key. In this example, nodes 140a, 140b, 140c, 140e include audit ports 140j, 140k, 140l, 140m, respectively, for outputting audit records. In this example, upon beginning execution of graph 144, node 140a is executed, the execution of which outputs audit record 140n on audit port 140j, the audit record specifying "graph_ID" and "Var Comp" fields with values of "53422" and "start", respectively. In this example, data storage device 142 receives and stores audit record 140n. Graph 144 then proceeds to node 140b, the execution of which sends an upgrade offer (e.g., telephone service) to a user of the system. When node 140 b is executed (eg, when the executable logic represented by node 140 b is executed), the system generates an audit record 140 o , which is output on output port 140 k and transmitted to data storage device 142 .
[0085] After executing node 140b, node 140c is executed. Node 140c is a decision node. Executing decision node 140c results in traversing the "yes" link 140r, which in turn causes the system to generate an audit record 140p. In this example, audit record 140 stores the value of a variable named "Var decision". The "Var decision" variable specifies whether the "yes" link or the "no" link of node 140c is traversed. In this example, because the yes link is traversed, the value of the "Var decision" variable is set to "yes". Audit record 140p is output on audit port 140l. Data storage device 142 collects audit record 140p and stores it.
[0086] After executing node 140c, graph 144 executes nodes 140d to 140f. In this example, the execution of node 140e generates audit record 140q, which is output on portion 140m. In this example, audit record 140q stores the value of a variable (i.e., a "Var Action" variable) that specifies what action to perform or what action is initiated by the execution of the executable logic represented by node 140e. In this example, the value of the "Var Action" variable is "Offer SMS Update" to specify that an offer to update the SMS service is sent to the user. Data storage device 142 collects data record 140q. Data storage device 142 batches audit records 140n, 140o, 140p, 140q at a specified time and transmits these audit records to the audit log module.
[0087] refer to Fig. 6E, environment 150 is a variation of environment 100. In this example, subscription component 104 reads audit records 132i, 132r, 132s from data store 134 and transmits these audit records to join component 112. Subscription component 104 reads audit records 140n through 140q from data store 142 and transmits these audit records to join component 112. In this example, join component 112 joins the contents of audit records 132i, 132r, 132s and audit records 140n through 140q into audit record 152. Join component 112 outputs joined audit record 152 to rollup component 116, which in turn performs a rollup operation on one or more values of one or more corresponding fields in joined audit record 152 to generate rolled-up record 154. In this example, rolled-up record 154 includes a summary field, namely, "Count Offer SMS Updates". This field specifies the number of variables whose value is "Offer SMS Updates". In this example, joined record 152 includes two variables (both named "Var Action") with a value of "Offer SMS Updates." Thus, the aggregate field "Count Offer SMS Updates" has a value of "2." Rollup component 116 transmits rolled-up record 154 to data storage device 40 for further processing, e.g., to generate metrics as described herein.
[0088] refer to Figure 7, environment 160 shows an architecture for audit logging. Environment 160 includes a system 162 (e.g., a data processing system) having an execution module 164, which is configured to process data records (and / or other structured data items). In this example, execution module 164 includes rule set log 168, chart selection module 170, and chart 172. System 162 also includes audit log module 166 and metric generation module 167. In this example, rule set log 168 includes logs (e.g., logs indicating which rules are executed, when they are executed, parameter values and / or values set during execution, etc., or rule use case traceability information (also referred to herein as traceability information)) generated during the execution of rules in a business rule environment (BRE) as described in U.S. Patent No. 7,877,350, the entire contents of which are incorporated herein by reference. The logging information output from rule set log 168 is input into data storage device 174, as indicated by reference numeral 198. In this example, the execution module 164 collects rule use case traceability information from the business rule environment to optionally include the information in the audit record (e.g., to explain the business logic processed by the rule set rather than in the chart decision block). In the example, the BRE implements the logging data stream by writing its log to a port on the chart component and then writing the log from the port to a file. In this example, when the execution module 164 is collecting the audit record (and other audit information) of the current event, the execution module 164 intercepts the logging data stream by replacing the function call that writes the traceability information in the BRE with the function call that stores the traceability information in the internal data structure of the event engine. In some examples, the execution module 164 intercepts the logging data stream by providing an alternative implementation of the logging function that the BRE is using (e.g., write_to_log). In other examples, the BRE supports the option of using different functions.
[0089] In this example, the graph selection module 170 is configured to execute one or more data flow graphs (e.g., one of the graphs 172), or to select graphs or data flow graphs to run on various systems included in the execution module 164. In this example, the graphs 172 include the data flow graphs 66a, 66b ( Figure 4B ) and Figure 132( Fig. 6A )、144( Fig.6D In this example, the graph selection module 170 selects the data flow graphs 66a, 66b from the graph 172 ( Figure 4B ) to be executed on systems 62 and 64 respectively - such as Figure 4B At a different, previous or subsequent time, the chart selection module 170 also selects charts 132, 144 ( FIG. 6A to FIG. 6D) to be executed on systems 62 and 64 respectively - such as FIG. 6A to FIG. 6D In this example, graphs 132, 144 ( FIG. 6A to FIG. 6D ) and shows those graphs 132, 144 ( FIG. 6A to FIG. 6D ) are executed on the systems 62, 64, respectively. In this example, the output (e.g., log records) of the systems 62, 64 are transmitted to the data storage device 174, as shown by reference numerals 196, 197, respectively.
[0090] In this example, one of the graphs 172 is an event engine graph that generates cluster metrics and writes them directly to a file in the database 173. Typically, the event engine graph is a graph used to monitor the performance of the execution module 164. Typically, the cluster metrics refer to detailed information at an aggregate level of the system and / or execution module 164, such as performance details, for example, metrics indicating which parts of the data flow graph were traversed, and aggregate metrics indicating how many times a set of records traversed a particular part of the data flow graph. In some examples, this file is appended (e.g., appended to other previously stored files, or updated) on each pulse (e.g., data push), and a new file is opened each day.
[0091] In this example, another data path comes from the graph selection module 170 (e.g., based on the execution of the graph selection module 170 executing the graph). As the graph executes, the graph selection module 170 generates a flow of audit records that record the path taken through the graph (e.g., exiting wait blocks, traversing flows, and sending actions). In this example, the audit records include the activities that are logged and stored in an audit file (i.e., database 174) or in an indexed compressed flat file (ICFF) stored in database 175. Generally, the audit file includes files for storing various audit records. In this example, the audit records are written to the files at the end of each pulse. In this example, database 174 receives a new file per pulse.
[0092] Environment 160 includes an audit file processor module 176 that includes a diagram that processes raw information (i.e., audit files stored in database 174) and reformats the raw information into a format for ICFF. In this example, database 174 transfers the audit file to audit file processor module 176 for reformatting. In turn, audit file processor module 176 transfers the reformatted file to database 175, which writes the reformatted file into ICFF record 178; indexed (in index 180) by one or more key fields (e.g., hash value, subject ID, or key) and merged by time (e.g., hourly or daily). In this example, index 180 includes a "hash value" column and an "address" column. The hash value column includes a hash value of a key (or other data) in the ICFF record being indexed. The address column includes a database address of a data block referenced by the hash value. In this example, ICFF record 178 references (e.g., via a pointer or another reference data structure) data block 182. In this example, while portions of the original information are reformatted to obtain ICFF record 178 (which is then indexed in index 180), other portions of the original information are stored in data block 182, which is referenced by ICFF record 178 and is thereby identifiable and accessible via ICFF record 178.
[0093] In this example, the audit log module 166 receives audit records 132i, 132r, 132s ( Fig. 6E ) and audit records 140n to 140q ( Fig. 6E ) and produces a rolled-up record 154, which is then transmitted to the audit file processor module 176 for review of the rolled-up data record ( Fig. 6E ) is processed. Specifically, audit file processor module 176 reformats the raw information in rollup record 154 into an ICFF format, such as ICFF record 178. In this example, ICFF record 178 includes the chart_ID field (i.e., from audit records 132i, 140q) as a key field. Audit records 132i, 132r, 132s ( Fig. 6E ) and audit records 140n to 140q ( Fig. 6E) is included in data block 182. The system stores ICFF record 178 and data block 182 in database 175. In this example, database 175 stores index 180, and ICFF record 178 is indexed in index 180 by hash values of keys in ICFF record 178 (i.e., key: 42424 and key: 53422). In this example, system 162 generates hash values for key: 42424 and key: 53422 by concatenating the values "42424" and "53422" into the following string: 4242453422, which has a hash value of "0110" - as shown in row 180a of index 180. Index 180 also includes rows 180a-180b that index other ICFF records, for example, at different times and / or from different executions of the graph, for example, for other audit records. In this example, ICFF record 178 references (as conceptually shown by arrow 183) data block 182 via an address field having a value of "0x82829" (as shown by row 180a in index 180). In a variation, ICFF record 178 references data block 182 via a reference data structure (e.g., a pointer). In yet another variation, ICFF record 178 and data block 182 are combined into an indexed ICFF data block (or data blocks).
[0094] In some examples, audit records and / or audit files are reformatted by batching audit records together at a specified granularity (e.g., one event for one subscriber to one chart), computing key fields to be stored on the ICFF record (e.g., fields for event type, subscriber ID, timestamp, chart instance ID, etc.), and then grouping the remaining audit records into collections, as shown below:
[0095] In an example, execution module 164 generates audit records that are optimized for space efficiency (e.g., rather than audit records that are optimized for query efficiency). In this example, the audit records are optimized for space efficiency, for example, by not including a timestamp, a subscriber_ID, etc. with each audit record. In order to place these audit records into ICFF, the system described herein needs to use more complete records (e.g., these more complete records include a timestamp and an object ID on each audit record) so that the system described herein can efficiently search for each record (e.g., each record based on the content of the audit record).
[0096] In some examples, database 175 includes a relational database rather than an ICFF database because the implementation goal of database 175 is to be able to query event sequences. To this end, the system transforms the stream of audit records into one or more records (e.g., ICFF records). Each record includes one or more indexable / searchable key fields and a data set representing the remaining audit events.
[0097] In some examples, the granularity level of the record is an event for a subscriber (e.g., user) of a chart. Thus, the system described herein extracts a sequence of audit records corresponding to an event for a subscriber of a chart (e.g., from an audit file in database 174). In this example, the events occur sequentially. Additionally, the audit records for each event are sequentially stored in database 174 (e.g., in a file in database 174). Using the extracted audit records, the system generates records with multiple key fields (e.g., including an event type field, a subscriber ID field, a timestamp field, a chart instance ID field, etc.). In some examples, the first audit record in the sequence includes values for the multiple key fields. The system uses these included values when generating new records. Additionally, the system constructs a data block corresponding to the remaining audit records of the sequence. The system stores the data block in database 175 as a data set (e.g., as a binary large object (BLOB)) as part of the generated new record or associated with the generated new record (e.g., by making the generated new record reference the data set or blob).
[0098] Using the generated records and / or data sets, the system can query ICFF (or database 175) for various information, such as, for example, "What happened when subscriber 12345 encountered event 7 on June 8?", and then display the decision sequence inside the event engine and the rule set associated with the event. In another example, the system queries ICFF (or database 175) with the following query: "What happened when subscriber 12345 sent a message on June 8?" In this example, to answer this query, the new record generated includes another key field to identify that the message action has been triggered (or not triggered). In this way, the system selects which key fields to include in the record based on the scope of the query to be supported.
[0099] In this example, database 175 stores the raw metrics in the ICFF. In this example, environment 100 also includes a metrics database 184 and an interface for accessing the metrics. System 162 generates metrics database 184 by post-processing audit records from the ICFF, rather than reading from memory in execution module 164. Since ICFF 175 is not cleared, metrics database 184 can be recreated at any time, including using different aggregation periods.
[0100] Thus, the metric generation module 167 includes a batch graph module 186 that runs a batch graph (e.g., every 10 minutes) to aggregate raw metrics and write them to the metric database 184. The metric generation module 167 also includes a metric calculation module 188 that runs "on demand" to calculate metrics (e.g., one or more states that are prevalent in the system 162) for display, for example, in the user interface 190. In this example, the batch graph module 186 accesses metrics from the metric database 184 and / or database 175, and processes those accessed metrics (which in this example are output data sets 192) for display in the user interface 190. In this example, the graphs generated or executed by the batch graph module 186 do not have persistent state, while the metric state is persisted in the metric database 184. Thus, the batch graph module 186 queries the metric database 184 to obtain metrics for display in the user interface 190. In the example, various types of user interfaces are displayed, including, for example, dynamic displays of metrics, reports on metrics, and investigation interfaces, each of which is described below:
[0101] In a dynamic display of metrics, the console displays metrics (e.g., summary counts) and updates these metrics in the display in near real time. The system queries the metrics database 184 to obtain this data and metrics. In this example, the console is a web user interface (UI) that includes a polling interface that can only extract data from the server (e.g., on a timer or at a specified time). In this way, the UI (e.g., a web page) intermittently queries the server to obtain updated counts to be displayed. The updated counts will be stored in the metrics database 184. In this example, there is server-side logic to interface between the UI (web page) and the metrics database 184, including, for example, a Java application server located between the browser and the database.
[0102] In another example, user interface 190 displays a report on metrics queried from metrics database 184. In this example, the user will be able to view the metrics in reports and dashboards. The metrics specify how many people entered each activity, how many people traversed each branch (e.g., of a graph), how many people were offered an offer, how many people accepted the offer, etc. The user interface includes controls for "drilling down" into more detailed information in the report. To drill down (e.g., to answer a query for "what happened to these 10 objects that saw the trigger event but did not get the offer"), the system accesses records in ICFF in database 175. To implement these reports, the system implements a server-side module (which in this example is batch graph module 186) that can access metrics database 184 and database 175 when drilling down.
[0103] In yet another example, user interface 190 includes a survey interface in which a user can ask questions such as “What happened when subscriber 12345 encountered event 7 on June 8?” To answer these questions, the system uses a server-side module (e.g., Java, graph, etc.) between user interface 124 and the ICFF in database 175 to query the ICFF (e.g., in database 175).
[0104] The execution module 164 creates a variety of different types of audit records to be stored in the ICFF in the database 175. The following table 1 specifies different types of audit records and their corresponding fields. Depending on the level of the log record, some of the following audit record types are optional. In this example, the ICFF stores a list of fields corresponding to the fields shown in the following table 1, such as key fields (as described above) for generating ICFF records. The values of the fields in italics in the following table 1 are not generated by the execution module 164, but are inferred by the system described in this article by looking back at the audit flow when writing data to the ICFF. This means that the execution module 164 needs to generate significantly less data. In the following table 1, the "?" column indicates whether the audit record is optional. The "+" symbol indicates that the audit record is generated by the execution module 164. The letter indicates that the audit record is optionally generated and controlled by a set of related control points, and the asterisk indicates that the record type is enabled by default.
[0105] Table 1
[0106]
[0107]
[0108]
[0109] In the example, database 174 groups the audit records for each object (e.g., by key) (e.g., continuously or intermittently). That is, the audits are automatically grouped by object or key. As a result, execution module 164 does not need to copy (e.g., every pulse) timestamps, object_ids, or instance_ids to each audit record in the audit file. Instead, database 174 can obtain values from these fields based on the values that previously appeared for these fields in the group. In this example, no state needs to be retained between pulses. This means, for example, that a block exit audit record needs to have a block_name because a block entry audit record may have appeared many pulses ago. In database 174, the audit records will be completed using the relevant fields shown in Table 1 above.
[0110] In the example, the system detects the occurrence of an event for an object of the graph (e.g., a user or subscriber). In this way, the execution module 164 generates an "event seen" audit record, then generates an "event dispatched" audit record, and then generates a "block exited-event" audit record. From these audit records, the system (e.g., or the audit file processor module 176) obtains the values of the following fields: event_type, timestamp, and instance_id. In this example, the system collects these audit records together into a blob, and the system sets key fields (e.g., key fields of event_type, timestamp, and instance_id) in another record (e.g., an ICFF record) based on the data seen (e.g., based on these audit records).
[0111] In another example, a timeout occurs for an object of a chart. In this example, the execution module 164 generates the following audit records: an "event seen" audit record of type alarm, followed by an "alarm dispatched" audit record, and then a "block exited-alarm" audit record. The system determines the values of the following key fields from these audit records: event_type, timestamp, and instance_id. In this example, the system determines the value of the instance_id field, for example, from the block exit-alarm audit record, rather than inferring the value of the field. In this example, the system collects these audit records into a blob, and the system sets the key fields (e.g., event_type, timestamp, instance_id, etc.) based on the data seen (e.g., audit record).
[0112] In yet another example, an event occurs for an object that starts a graph. In this example, execution module 164 generates an "event seen" audit record (which may be of the type that started the graph or may be another type), then an "event dispatched" audit record, and then a "graph started" audit record. The system obtains the values of the following key fields from these audit records: event_type, timestamp, and instance_id. The system collects the audit records into a blob and sets the key fields (event_type, timestamp, instance_id, etc.) based on the data seen (e.g., audit record).
[0113] Because audit records are grouped together by key, a subset of audit records associated with a particular key does not include the key field. For example, database 174 or audit file processor module 176 may receive the following audit record sequence: event-seen audit record, event-scheduled audit record, block-exit-event audit record. In this example, database 174 or audit file processor module 176 obtains an object ID (e.g., key) from the event-seen audit record. In this way, execution module 164 does not need to copy the key to the event-scheduled audit record and the block-exit audit record, because these audit records will be in memory (e.g., in database 174 or other system memory) in order after the event-seen audit record. Similarly, the event-scheduled audit record includes an instance ID field. In this way, the event engine does not need to include a copy of this field in the block-exit audit record. Since the system groups audit records in memory (and on disk), the system does not need to copy the key to all audit records in the group, thereby saving system resources and reducing the amount of memory consumption.
[0114] To avoid generating an excessive amount of data, the user controls when and which audit records the event engine generates. Although auditing (and therefore metrics) can be turned off completely, the core set of audit records (marked with a plus sign in the preceding table) is not disabled if auditing is turned on. The core set of audit records tracks the input (events and alarms), output (actions), errors, and chart and wait block activity of the event engine. Typically, an alarm consists of a data record of a predefined type (for example, a data record that a chart sends itself) that will arrive at a precalculated time in the future. Other audit record types can be selectively enabled and disabled. Enabling optional audit records involves two aspects - what should be enabled and when it should be enabled. For what to enable, several different audit record groups can be selectively enabled / disabled. These audit record groups include:
[0115] A: Alarm settings (open by default)
[0116] D: Decision making (open by default)
[0117] P: Monitoring point (open by default)
[0118] R: ruleset invocation and ruleset events
[0119] S: variable setting
[0120] V: variable read
[0121] L: Find access and find variables
[0122] When enabled, logging of these audit events can be enabled in their entirety or limited based on one or more of the following:
[0123] Only for a set of specified object ids (e.g. keys)
[0124] Only for the specified chart name and / or monitoring name set
[0125] Only for specified event types (such as record types)
[0126] In the example, the execution module 164 configures the logging of these audit records through control points. Typically, a control point includes runtime parameters defined by a user (who generates the chart) and is used to change the runtime behavior of one or more charts (or chart instances). In this example, audit records are controlled (e.g., by the event engine) through a set of four interrelated control points. The five control points are:
[0127] Audit Types - This is a list of the types of audit records to be turned on (using the letters listed in Table 1 above).
[0128] Audit Entities - This is a list of object ids. If not empty, the event engine will only generate audit record types in the audit type list for object ids included in this list. (Otherwise, all object ids will be audited).
[0129] Audit Events - This is a list of event types (e.g., numbers corresponding to or representing event types). If not empty, the event engine will only generate audit record types in the audit type list for event types included in this list. (Otherwise, all events will be audited). There are various types of events, including, for example, an event specifying that an SMS message was sent, an event specifying that a user renewed their mobile plan, etc.
[0130] Audit Charts - This is a list of chart names and monitoring names. If not empty, the event engine will generate audit record types in Audit Type for charts whose chart names or monitoring names are included in this list.
[0131] (Otherwise, all diagrams will be reviewed).
[0132] Audit variables - This consists of a comma separated list of variable names. If not empty and the event engine is configured to generate audit record types for variables (e.g., variable set or variable read), the event engine will only generate audit records if the variable name is in this list.
[0133] In some examples, execution module 164 applies different rules to different audit record groups. In this example, there is a first audit record group (e.g., first audit group) with a first rule set (e.g., audit type 1, audit entity 1, audit event 1, audit chart 1, and audit variable 1), and a second audit record group (e.g., second audit group) with a second rule set (e.g., audit type 2, audit entity 2, audit event 2, audit chart 2, and audit variable 2), and so on.
[0134] In this example, each of the audit types, audit entities, audit events, audit graphs, and audit variables includes a type of filter. If any or all of these filters are set, only a subset of these events will include additional audits (e.g., audits that include optional audit forms).
[0135] In one example, the system is configured for minimal auditing by implementing a first audit group as follows:
[0136] Audit Type 1 = "" (do not enable any optional content)
[0137] Audit Entity 1 = "" (all object IDs)
[0138] Audit Event 1 = "" (all event types)
[0139] Review Chart 1 = "" (all charts)
[0140] Audit variables 1 = "" (all variables, although this is ignored because the system is configured to turn off logging for optional variables).
[0141] Now, in another example, the system is configured to evaluate why an offer for object_ID 12345 has not been sent despite the fact that all criteria for sending the offer are met for that object_ID. The chart implementing this offer is "My Offers Chart" and the triggering event is number 6. In order to get all the more details for this specific object ID as to why the offer was not sent, the system is configured to have a second audit group as shown below:
[0142] Audit Type 2 = "ADPRSVL" (enable all audit types)
[0143] Audit Entity 2 = "12345" (only this object ID)
[0144] Audit Event 2 = "6" (only this event type)
[0145] Review Chart 2 = "My Offers Chart" (this chart only)
[0146] Audit variable 2 = "" (all variables).
[0147] In this example, when an event occurs, the system first checks to see if the object ID is in the list of audit events 2, and if the event type is in the list of audit events 2, and if the chart is in the list of audit charts 2. If all of these attributes (e.g., filters) evaluate to true, the system will implement all audit record types as specified in audit type 2, which will begin optional logging for the duration of the event. If not, the system implements audit type 1, which only logs the default items. In this example, the system is configured to check the highest numbered group first, and then run backwards until a matching group is found. If a matching group is not found, the system implements and uses the default item. In a variation, the system can be configured to take the union of all matching groups, so that more optional logging is added.
[0148] In this example, the execution module 164 implements conditional audit logging (e.g., by implementing monitoring point audit records) and optimizes its execution performance. To optimize this performance, the execution module 164 pre-computes (e.g., pre-generates) a data structure for making conditional audit decisions. Typically, a conditional audit decision specifies that an audit record is generated when a condition evaluates to a specified value (e.g., true or false). In this example, the event engine performs conditional audit calculations at the beginning of event processing for an event or alarm. For example, each time the event engine detects an event or alarm, the event engine performs calculations (e.g., with the above-mentioned audit group) - but only at the beginning of processing for each event or alarm (e.g., rather than for each audit record). The event engine saves the conditional audit calculations in a package of global variables and queries them inside each audit function that can be used as a condition.
[0149] For each audit record type, execution module 164 implements a single flag that specifies whether to generate an audit record of that audit record type. Execution module 164 analyzes the contents of the audit entity list and the audit event list when implementing the single flag, because both lists are specified or generated when event processing begins. Because an event can be processed by many charts, the event engine determines whether to implement an audit function (e.g., audit record type) for each chart. If the audit chart is not empty, the event engine converts the contents of the audit chart into a list of instance ids (each id identifies a chart) to simplify the comparison between the ids of the implemented charts and the instance ids that specify which charts generate audit records (and what types of audit records are generated).
[0150] In an example, execution module 164 collects audit records in an array of memory (e.g., in execution module 164 or in a system that includes execution module 164) and then writes the audit records out in one block to an audit file at the end of processing an event or event stream. In this example, audit records from different events (i.e., events associated with different keys or user IDs) are not mixed together in memory. Instead, the memory is partitioned and each partition has its own audit file. In this way, execution module 164 is configured to automatically group audit records together by key.
[0151] In this example, the audit files are versioned. There will be one audit file instance per pulse. The audit files will accumulate on disk, and the audit file processing graph (executed by the audit file processor module 176) will extract each audit file (from each partition) at the end of the pulse and post-process the audit records to write them into the ICFF.
[0152] In this example, the system generates the metrics database 184, for example, by post-processing audit records from the ICFF (in the database 175) rather than reading the audit records from the memory in the execution module 164. In this example, the ICFF is not cleared. In this way, the metrics database 184 can be recreated at any time, including using different aggregation periods. In this example, the tables in the metrics database 184 have a cluster_name column so that one database can be used for multiple clusters. Typically, a cluster includes an event engine instance. Typically, an event engine instance includes one or more specifications for a specific implementation of a unique value for a key (for which the execution module 164 executes one or more graphs). That is, each object ID is processed by exactly one event engine instance. Therefore, for each unique key (e.g., an object ID), there will be only exactly one event engine instance with all its graphs. Audit records for the same cluster will have the same cluster_name value.
[0153] In this example, metrics database 184 can be queried to obtain metrics indicating the state of execution module 164, such as the total number of events processed by execution module 164 (e.g., based on event seen audit records), the number of events that resulted in errors (e.g., based on error dispatched audit records), the number of events that triggered activities (e.g., based on action sent audit records), which subscribers transitioned from one phase of a particular activity to another (e.g., based on block entry audit records), the number of subscribers deleted or added in the event engine on a particular day (e.g., based on chart start and chart complete audit records), and the number of total charts running in the event engine that track the state of execution module 164 as the event engine processes events (e.g., based on chart start audit records).
[0154] Additionally, metrics database 184 may be queried (e.g., via a client device or the system described herein) to retrieve data specifying reasons why a message was not sent to a particular key (e.g., each subscriber). To this end, the event engine is configured to log the links taken (and / or components executed within a graph) and the rule set use cases accessed and implemented via execution module 164 (e.g., by generating audit records specifying which components and / or links were executed). The system may then be configured to compare the path taken through the graph (e.g., specified by the audit record) to all possible paths through the graph to determine why a particular message was not sent (e.g., because the path corresponding to sending the message was never traversed due to traversing an alternative path (as specified by the audit record)).
[0155] refer to Figure 8 , process 200 is implemented by one or more data processing systems (or processing systems) to execute a data flow graph for processing one or more data records representing a transaction, wherein the data flow graph includes multiple components and links, wherein the components represent calculations applied to one or more input records to process the one or more input records, wherein the input records have a specified relationship with the one or more data records representing the transaction, and wherein the links connect the components and represent the flow of data between the components. Operation, system execution (202) executes the data flow graph including the multiple components and the links, wherein a given component of the multiple components includes an input port, an audit port, and an output port. The input port is used to receive one or more input records, and the one or more input records have a specified relationship with the one or more data records representing the transaction. The audit port is used to provide audit data, which describes at least a portion of the transaction. The output port is used to provide one or more output records. Components (for example, Figure 2A At least one of the components 23e) in the embodiment is configured to execute a given component (e.g., Figure 2A 23j) in the data flow graph. The system processes (204) one or more data records representing the transaction by a data flow graph having the components and the links, wherein the at least one of the components saves the state of one or more input records specified to be processed by the at least one of the components. When an error occurs during the processing of one or more input records by the given component, the system restores (206) the state of the at least one of the components to the saved state. Based on the restored state, the system recovers (208) at least some of the audit data for the given component of the data flow graph. The system provides (210) at least some of the recovered audit data that describes at least the portion of the transaction on the audit port. The system provides (212) one or more output records on the output port.
[0156] The above techniques can be implemented using software to be executed on a computer. For example, the software forms a process in one or more computer programs that is executed on one or more programming or programmable computer systems (which can be various architectures such as distributed, client / server or grid), each of which includes at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. The software can form one or more modules of a larger program (e.g., providing other services related to the design and configuration of diagrams and flow charts). The nodes, links, and elements of the diagram can be implemented as data structures stored in a computer-readable medium or other organized data that conforms to a data model stored in a data repository.
[0157] The techniques described herein may be implemented in digital electronic circuit systems, or in computer hardware, firmware, software, or a combination thereof. The apparatus may be implemented in a computer program product tangibly embodied or stored in a machine-readable storage device (e.g., a non-transitory machine-readable storage device, a machine-readable hardware storage device, etc.) for execution by a programmable processor; and the method actions may be performed by a programmable processor executing an instruction program to perform functions by operating on input data and generating outputs. The embodiments described herein, as well as the claims and other embodiments of the techniques described herein, may be advantageously implemented in one or more computer programs that may be executed on a programmable system comprising: at least one programmable processor coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to transmit data and instructions to a data storage system, at least one input device, and at least one output device. Each computer program may be implemented in a high-level procedural or object-oriented programming language, or in an assembly language or machine language if desired; and in any case, the language may be a compiled language or an interpreted language.
[0158] Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. Essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to receive data from or transmit data to the one or more mass storage devices, or both receive and transmit data. Computer-readable media for embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit system. Any of the above may be supplemented by or incorporated into an ASIC (Application Specific Integrated Circuit).
[0159] To provide interaction with a user, embodiments may be implemented on a computer having a display device (e.g., an LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound input, voice input, or tactile input.
[0160] Embodiments may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the embodiments), or any combination of these back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).
[0161] The system and method or part thereof can use a collection of servers on the Internet that utilizes the Hypertext Transfer Protocol (HTTP) - the "World Wide Web" (Web or WWW). HTTP is a known application protocol that provides users with access to resources, which can be information in different formats, such as text, graphics, images, sounds, videos, Hypertext Markup Language (HTML), and programs. After the user specifies a link, the client computer sends a TCP / IP request to the Web server and receives information, which can be another Web page formatted according to HTML. The user can also access other pages on the same or other servers by following instructions on the screen, entering specific data, or clicking on a selected icon. It should also be noted that any type of selection device known to those skilled in the art, such as check boxes, drop-down boxes, etc., can be used for embodiments that use web pages to allow users to select options for given components. The server runs on a variety of platforms, including UNIX machines, but other platforms such as Windows 2000 / 2003, Windows NT, Sun, Linux, and Macintosh can also be used. Computer users can view information available on a server or network on the Web by using browsing software (such as Firefox, Netscape Navigator, Microsoft Internet Explorer, or Mosaic browser). The computing system can include a client and a server. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
[0162] Other embodiments are within the scope and spirit of the specification and claims. For example, due to the nature of software, the functions described above may be implemented using software, hardware, firmware, hardwiring, or any combination of these. Features that implement the functions may also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. The use of the term "a" herein and throughout the application is not used in a restrictive manner, and therefore is not meant to exclude multiple meanings or "one or more" meanings of the term "a". In addition, with respect to claiming priority to a provisional patent application, it should be understood that the provisional patent application is not restrictive, but includes examples of how the technology described herein may be implemented.
[0163] Several embodiments of the present invention have been described. However, it will be appreciated by those skilled in the art that various modifications may be made without departing from the spirit and scope of the claims and the technology described herein.
Claims
1. A method implemented by a data processing system for executing a data flow graph, the data flow graph processing one or more data records representing a transaction, in, The dataflow graph includes a plurality of components and links, wherein the components represent computations applied to one or more input records to process the one or more input records, wherein the input records have specified relationships with one or more data records representing the transaction, and wherein the links connect the components and represent data flows between the components, the method comprising: Executing a data flow graph including the plurality of components and the links, wherein the execution of the data flow graph includes executing a given component of the plurality of components, the given component including an input port, an audit port, and an output port; wherein the input port is used to receive one or more of the input records, the input records having a specified relationship with one or more data records representing the transaction; The audit port is used to provide audit data, and the audit data describes at least a part of the transaction; Wherein, the output port is used to provide one or more output records; wherein at least one component of the plurality of components is configured to be executed prior to executing the given component; processing one or more data records representing the transaction through a data flow graph having the plurality of components and the links, wherein the at least one component of the plurality of components saves a state of one or more of the input records specified for processing by the at least one component of the plurality of components; When an error occurs during the processing of one or more of the input records by the given component during execution of the given component, restoring the state of the at least one component of the plurality of components to a saved state; and restoring at least some of the audit data for the given component of the dataflow graph based on the restored state; providing at least some of the recovered audit data describing at least the portion of the transaction on the audit port; and The one or more output records are provided on the output port.
2. The method according to claim 1, in, The restored audit data is first audit data, the audit port is a first audit port, the given component is a first component, and wherein the method further comprises: The first audit data is combined with second audit data output on a second audit port of the second component at specified time intervals.
3. The method according to claim 1, in, The given component is a first component, the restored audit data is first audit data, and wherein the method further comprises: executing the first component on a first data processing system; executing a second component of the data flow graph on a second data processing system; and The restored audit data and the second audit data output from the second audit port of the second component are received on a third data processing system.
4. The method of claim 3, further comprising: include: storing the received restored audit data and the second audit data in a memory; as well as Dynamically generate one or more metrics based on stored audit data.
5. The method of claim 1, further comprising: include: Aggregating data from the first audit data and the second audit data of the audit data; as well as Stores aggregated data.
6. The method of claim 1, further comprising: include: categorizing the recovered audit data by assigning the recovered audit data to a particular category; as well as Audit metrics indicating user interactions with the particular activity are generated based on the audit data assigned to the particular category.
7. The method according to claim 1, in, In response to the occurrence of the error, the given component outputs an error log on the error log port, the error log including the status of the at least one component among the multiple components and specifying which one or which of the input records were last successfully processed by the at least one component among the multiple components, wherein the status specifies which of the components last successfully completed the processing.
8. The method according to claim 7, in, The given component outputs the error log to an error handler component, wherein the error handler component transmits a message to a rollback component based on the error log, and wherein the rollback component uses the message to return the state of the data flow graph to a state in which at least one of the multiple components is executing and one or more of the input records that were last successfully processed by the at least one of the multiple components are being processed by the at least one of the multiple components.
9. One or more machine-readable hardware storage devices for executing a data flow graph that processes one or more data records representing a transaction, in, The dataflow graph includes a plurality of components and links, wherein the components represent computations applied to one or more input records to process the one or more input records, wherein the input records have specified relationships to one or more data records representing the transaction, and wherein the links connect the components and represent data flow between the components, the machine-readable hardware storage device storing instructions executable by one or more processing devices to perform operations including: executing a data flow graph including the plurality of components and the links, wherein a given component of the plurality of components includes an input port, an audit port, and an output port; wherein the input port is used to receive one or more input records, the one or more input records having a specified relationship with one or more data records representing the transaction; The audit port is used to provide audit data, and the audit data describes at least a part of the transaction; Wherein, the output port is used to provide one or more output records; wherein at least one of the components is configured to be executed prior to executing the given component; processing one or more data records representing the transaction through a data flow graph having the components and the links, wherein the at least one of the components saves a state of one or more input records specified for processing by the at least one of the components; When an error occurs during the processing of one or more input records by this given component, restoring the state of the at least one of the components to a saved state; and restoring at least some of the audit data for the given component of the dataflow graph based on the restored state; providing at least some of the recovered audit data describing at least the portion of the transaction on the audit port; and The one or more output records are provided on the output port.
10. The machine-readable hardware storage device of claim 9, in, The recovered audit data is first audit data, the audit port is a first audit port, the given component is a first component, and wherein the operations further include: The first audit data is combined with second audit data output on a second audit port of the second component at specified time intervals.
11. The machine-readable hardware storage device of claim 9, in, The given component is a first component, the restored audit data is first audit data, and wherein the operations further include: executing the first component on a first data processing system; executing a second component of the data flow graph on a second data processing system; and The restored audit data and the second audit data output from the second audit port of the second component are received on a third data processing system.
12. The machine-readable hardware storage device of claim 11, in, These operations further include: storing the received restored audit data and second audit data in a memory; and Dynamically generate one or more metrics based on stored audit data.
13. The machine-readable hardware storage device of claim 9, in, These operations further include: Aggregating data from the first audit data and the second audit data of the audit data; and Stores aggregated data.
14. The machine-readable hardware storage device of claim 9, in, These operations further include: categorizing the recovered audit data by assigning the recovered audit data to a particular category; and Audit metrics indicating user interactions with the particular activity are generated based on the audit data assigned to the particular category.
15. A data processing system for executing a data flow graph, the data flow graph processing one or more data records representing a transaction, in, The dataflow graph includes a plurality of components and links, wherein the components represent computations applied to one or more input records to process the one or more input records, wherein the input records have specified relationships to one or more data records representing the transaction, and wherein the links connect the components and represent data flow between the components, the system comprising: one or more processing devices; and one or more machine-readable hardware storage devices storing instructions executable by the one or more processing devices to perform operations including: executing a data flow graph including the plurality of components and the links, wherein a given component of the plurality of components includes an input port, an audit port, and an output port; wherein the input port is used to receive one or more input records, the one or more input records having a specified relationship with one or more data records representing the transaction; The audit port is used to provide audit data, and the audit data describes at least a part of the transaction; Wherein, the output port is used to provide one or more output records; wherein at least one of the components is configured to be executed prior to executing the given component; processing one or more data records representing the transaction through a data flow graph having the components and the links, wherein the at least one of the components saves a state of one or more input records specified for processing by the at least one of the components; When an error occurs during the processing of one or more input records by this given component, restoring the state of the at least one of the components to a saved state; and restoring at least some of the audit data for the given component of the dataflow graph based on the restored state; providing at least some of the recovered audit data describing at least the portion of the transaction on the audit port; and The one or more output records are provided on the output port.
16. The data processing system according to claim 15, in, The recovered audit data is first audit data, the audit port is a first audit port, the given component is a first component, and wherein the operations further include: The first audit data is combined with second audit data output on a second audit port of the second component at specified time intervals.
17. The data processing system according to claim 15, in, The given component is a first component, the restored audit data is first audit data, and wherein the operations further include: executing the first component on a first data processing system; executing a second component of the data flow graph on a second data processing system; and The restored audit data and the second audit data output from the second audit port of the second component are received on a third data processing system.
18. The data processing system according to claim 17, in, These operations further include: storing the received restored audit data and second audit data in a memory; and Dynamically generate one or more metrics based on stored audit data.
19. The data processing system according to claim 15, in, These operations further include: Aggregating data from the first audit data and the second audit data of the audit data; and Stores aggregated data.
20. The data processing system according to claim 15, in, These operations further include: categorizing the recovered audit data by assigning the recovered audit data to a particular category; and Audit metrics indicating user interactions with the particular activity are generated based on the audit data assigned to the particular category.
21. A computer program product, comprising a computer program, which implements the method according to any one of claims 1 to 8 when executed by a processor.
