Method, hardware storage device and computing system for identifying portions of executable logic

Abstract

The present application relates to methods, hardware storage devices, and computing systems for identifying portions of executable logic. A data processing system for identifying one or more portions of executable logic that are executed when processing one or more data items each associated with a given value of a key by: accessing a specification representing the executable logic; and generating a log record for the given value of the key, the generation comprising: for each of the one or more data items associated with the given value of the key: identifying a given state maintained by the specification for the given value of the key associated with the data item; processing the data item, including executing executable logic specified in the identified given state of the specification on the data item; and generating the log record specifying the execution of the one or more portions of the executable logic for the given value of the key.

Description

[0001] This application is a divisional application of an invention patent application with an application date of November 13, 2018, application number 201880081094.6, and invention name “Key-based logging for processing structured data items using executable logic”. Technical Field

[0002] The present specification relates to a computer-implemented method, a data processing system, and a machine-readable hardware storage device for generating a key-specific log from the execution of executable logic, which is currently applied to one or more keyed data items in a stream of data items, such as data items input from a networked application over a network connection. Background Art

[0003] Logs record events that occur when an operating system or other software is running, or messages between different users of communication software. Summary of the invention

[0004] In general aspect 1, a method implemented by a data processing system for identifying one or more portions of executable logic that are executed when processing one or more data items that are each associated with a given value of a key is described, wherein a specification represents the executable logic, wherein the specification is associated with the key, and wherein a state of the specification is maintained for corresponding values ​​of the key, the method comprising: accessing a specification representing the executable logic, wherein a state of the specification for a particular value of the key specifies one or more portions of the executable logic that are executable at the state; intermittently receiving data items of a data stream through an input device or port, wherein one or more of the data items are each associated with a given value of the key; and generating a state for the executable logic. A log record of a given value of the key, wherein the log record specifies the one or more portions of the executable logic executed when processing the one or more data items of the data items each associated with the given value of the key, the generating comprising: for each of the one or more data items associated with the given value of the key: identifying a given state of the specification maintained for the given value of the key associated with the data item; processing the data item by the data processing system, including executing on the data item one or more portions of the executable logic specified in the identified given state of the specification; and generating the log record of the execution of the one or more portions of the executable logic specified for the given value of the key.

[0005] In aspect 2 according to aspect 1, the method further comprises: grouping log records together by key, wherein first log records associated with a first key are grouped into a first group, and wherein second log records associated with a second key are grouped into a second group.

[0006] In aspect 3 according to any one of aspects 1 to 2, the log records are grouped in the memory.

[0007] In aspect 4 according to any of aspects 1 to 3, each log record includes one or more fields indicating one or more attributes of the processed data item, wherein, for at least one log record, the field is a key field specifying a value of a key associated with the data item.

[0008] In aspect 5 according to any one of aspects 1 to 4, a subset of log records associated with a particular key does not include the key field.

[0009] In aspect 6 according to any one of aspects 1 to 5, only one log record associated with a particular key includes the key field.

[0010] In aspect 7 according to any one of aspects 1 to 6, the method further comprises: writing the log records associated with the specific key in the specific group to the file at the end of the predefined time interval.

[0011] In aspect 8 according to any one of aspects 1 to 7, the method further comprises: reformatting the file; writing the reformatted file into an index compressed flat file (ICFF); and indexing the ICFF according to the key.

[0012] In aspect 9 according to any one of aspects 1 to 8, the specification includes a graph having multiple nodes, wherein each node represents one or more parts of the executable logic, wherein the graph is executable, wherein execution of the graph results in execution of the parts of the executable logic, and the method further includes: based on these log records, tracing back the traversal of one or more nodes of the graph when processing the one or more data items.

[0013] In aspect 10 according to any one of aspects 1 to 9, the method comprises parsing the log records to identify which nodes of the graph are traversed and which nodes of the graph are not traversed.

[0014] In aspect 11 according to any one of aspects 1 to 10, the log records specify which nodes of the graph are traversed and which nodes of the graph are not traversed.

[0015] In aspect 12 according to any one of aspects 1 to 11, the method includes: receiving a user-initiated selection of one of the nodes; in response to receiving the selection, causing a graphical user interface to display a configuration area, thereby providing the user with a configuration tool for configuring the one or more parts of the executable logic represented by the selected node, wherein the configuration tool includes a tool for turning on and off auditing; and receiving the turning on of the audit initiated by the user through the configuration tool, thereby generating a log record associated with the selected node.

[0016] In aspect 13 according to any one of aspects 1 to 12, generating the log record that specifies the execution of the one or more parts of the executable logic for a given value of the key includes: specifying the results of executing the one or more parts of the executable logic in the generated log record, specifying variables that are read or set when executing the one or more parts of the executable logic, or identifying the one or more parts of the executable logic that are executed.

[0017] In aspect 14 according to any one of aspects 1 to 13, the method further comprises: initiating an action to be performed by the computing system in response to executing the one or more portions of the executable logic.

[0018] In aspect 15 according to any of aspects 1 to 14, the action comprises requesting additional computing resources for executing the one or more portions of executable logic based on execution characteristics of the execution of the one or more portions of executable logic.

[0019] In aspect 16 according to any of aspects 1 to 15, the method further comprises maintaining the states by storing state data indicating which portion of the executable logic is currently being executed for each state in a data repository or in an in-memory data grid.

[0020] In aspect 17 according to any one of aspects 1 to 16, the method further comprises: updating state data specifying a current state of the value for the key in response to completing processing of the one or more of the data items each associated with a given value for the key.

[0021] In aspect 18 according to any one of aspects 1 to 17, the method further comprises: dynamically determining a metric indicating which nodes of the graph are traversed and how many times the traversed nodes are traversed by the data items.

[0022] In aspect 19 according to any one of aspects 1 to 18, the method further includes: generating a stream of log records recording the paths taken through the graph; transforming the generated log records; and storing the transformed log records in a relational database or an index compressed flat file, including batching the log records together at a specified granularity.

[0023] In aspect 20 according to any one of aspects 1 to 19, the method further comprises: dynamically outputting the determined metric via a user interface, wherein the outputted metric is updated in response to a change to the determined metric.

[0024] In aspect 21 according to any one of aspects 1 to 20, the method further comprises: receiving a user selection of a first data item of the stream; and causing display of an indication of one or more portions of the executable logic currently executing on the selected first data item of the data items.

[0025] In aspect 22 according to any one of aspects 1 to 21, the method further includes: comparing a selected first data item currently being processed among the data items of the stream with a predetermined reference data item; determining whether there is a deviation or whether there is no such deviation between the first data item currently being processed among the data items of the stream and the predetermined reference data item; and based on determining whether there is a deviation or whether there is no such deviation between the first data item currently being processed among the data items of the stream and the predetermined reference data item, outputting an indication of whether the first data item currently being processed among the data items of the stream is consistent with the predetermined reference data item.

[0026] In aspect 23 according to any one of aspects 1 to 22, the method further includes: comparing the one or more parts of the executable logic currently being executed on the first data item among the data items with a predetermined reference executable logic; determining whether there is a deviation between the one or more parts of the executable logic currently being executed on the first data item among the data items and the predetermined reference executable logic or whether such a deviation does not exist; and based on determining whether there is a deviation between the one or more parts of the executable logic currently being executed on the first data item among the data items and the predetermined reference executable logic or whether such a deviation does not exist, outputting an indication of whether the one or more parts of the executable logic currently being executed on the first data item among the data items are consistent with the predetermined reference executable logic.

[0027] In aspect 24 according to any one of aspects 1 to 23, the log record is a structured data record having a first field and a second field, the first field being used to store data representing the state of the specification for a specific value of the key, and the second field being used to store data specifying which parts of the executable logic represented by the specification have been executed for the specific value of the key from the time when the executable logic was started to the current time.

[0028] In aspect 25 according to any one of aspects 1 to 24, the log record is a structured data record having a field for storing data specifying which portion of the executable logic represented by the specification has been executed for a particular value of the key at the current time.

[0029] In aspect 26 according to any one of aspects 1 to 25, the specification is a graph, and wherein one or more log records specify a path through the graph, the path indicating which portions of the executable logic are executed when processing one or more data records associated with a particular value of the key.

[0030] In aspect 27 according to any one of aspects 1 to 26, the method further comprises: generating one or more output records based on processing the one or more data items; wherein each of the one or more output records comprises data not included in the log record.

[0031] In general aspect 28, a data processing system is described for identifying one or more portions of executable logic that are executed when processing one or more data items that are each associated with a given value of a key, wherein a specification represents the executable logic, wherein the specification is associated with the key, and wherein a state of the specification is maintained for corresponding values ​​of the key, the data processing system comprising: one or more processing devices; and one or more machine-readable hardware storage devices storing instructions executable by the one or more processing devices to perform operations including: accessing a specification representing the executable logic, wherein a state of the specification for a particular value of the key specifies one or more portions of the executable logic that are executable at the state; intermittently receiving a stream of data through an input device or port Data items, wherein one or more of the data items are each associated with a given value of the key; and generating a log record for the given value of the key, wherein the log record specifies the one or more portions of the executable logic executed when processing the one or more of the data items each associated with the given value of the key, the generating comprising: for each of the one or more data items associated with the given value of the key: identifying a given state of the specification maintained for the given value of the key associated with the data item; processing the data item by the data processing system, including executing on the data item one or more portions of the executable logic specified in the identified given state of the specification; and generating the log record specifying the execution of the one or more portions of the executable logic for the given value of the key.

[0032] In aspect 29 according to aspect 28, the operations further include grouping log records together by key, wherein first log records associated with a first key are grouped into a first group, and wherein second log records associated with a second key are grouped into a second group.

[0033] In aspect 30 according to any one of aspects 28 to 29, the log records are grouped in the memory.

[0034] In aspect 31 according to any of aspects 28 to 30, each log record comprises one or more fields indicating one or more attributes of a processed data item, wherein, for at least one log record, the field is a key field specifying a value of a key associated with the data item.

[0035] In aspect 32 according to any one of aspects 28 to 31, a subset of log records associated with a particular key does not include the key field.

[0036] In aspect 33 according to any one of aspects 28 to 32, only one log record associated with a particular key includes the key field.

[0037] In aspect 34 according to any one of aspects 28 to 33, the operations further include: writing the log records associated with the specific key in the specific group to the file at the end of the predefined time interval.

[0038] In aspect 35 according to any one of aspects 28 to 34, the operations further include: reformatting the file; writing the reformatted file to an index compressed flat file (ICFF); and indexing the ICFF according to the key.

[0039] In aspect 36 according to any one of aspects 28 to 35, the specification includes a graph having multiple nodes, wherein each node represents one or more portions of the executable logic, wherein the graph is executable, wherein execution of the graph results in execution of portions of the executable logic, and the operations further include: based on the log records, tracing back the traversal of one or more nodes of the graph when processing the one or more data items.

[0040] In aspect 37 according to any one of aspects 28 to 36, the operations further include parsing the log records to identify which nodes of the graph are traversed and which nodes of the graph are not traversed.

[0041] In aspect 38 according to any of aspects 28 to 37, the log records specify which nodes of the graph are traversed and which nodes of the graph are not traversed.

[0042] In aspect 39 according to any one of aspects 28 to 38, the operations further include: receiving a user-initiated selection of one of the nodes; in response to receiving the selection, causing the graphical user interface to display a configuration area, thereby providing the user with a configuration tool for configuring the one or more parts of the executable logic represented by the selected node, wherein the configuration tool includes a tool for turning on and off auditing; and receiving the turning on of the audit initiated by the user through the configuration tool, thereby resulting in the generation of a log record associated with the selected node.

[0043] In aspect 40 according to any one of aspects 28 to 39, generating the log record that specifies the execution of the one or more parts of the executable logic for a given value of the key includes: specifying the results of executing the one or more parts of the executable logic in the generated log record, specifying variables that are read or set when executing the one or more parts of the executable logic, or identifying the one or more parts of the executable logic that are executed.

[0044] In aspect 41 according to any of aspects 28 to 40, the operations further include initiating actions to be performed by the computing system in response to executing the one or more portions of the executable logic.

[0045] In aspect 42 according to any of aspects 28 to 41, the action comprises requesting additional computing resources for executing the one or more portions of executable logic based on execution characteristics of the execution of the one or more portions of executable logic.

[0046] In aspect 43 according to any of aspects 28 to 42, the operations further include maintaining the states by storing state data in a data repository or in an in-memory data grid indicating which portion of the executable logic is currently being executed for each state.

[0047] In aspect 44 according to any one of aspects 28 to 43, the operations further include: updating state data specifying a current state of the value for the key in response to completing processing of the one or more of the data items each associated with a given value for the key.

[0048] In aspect 45 according to any one of aspects 28 to 44, the operations further include dynamically determining a metric indicating which nodes of the graph are traversed and how many times the traversed nodes are traversed by the data items.

[0049] In aspect 46 according to any one of aspects 28 to 45, the operations further include: generating a stream of log records recording the paths taken through the graph; transforming the generated log records; and storing the transformed log records in a relational database or an index compressed flat file, including batching the log records together at a specified granularity.

[0050] In aspect 47 according to any of aspects 28 to 46, the operations further comprise: dynamically outputting the determined metric via a user interface, wherein the outputted metric is updated in response to a change to the determined metric.

[0051] In aspect 48 according to any one of aspects 28 to 47, the operations further include: receiving a user selection of a first data item in the stream of data items; and causing display of an indication of one or more portions of the executable logic currently executing on the selected first data item in the stream.

[0052] In aspect 49 according to any one of aspects 28 to 48, the operations further include: comparing a selected first data item currently being processed among the data items of the stream with a predetermined reference data item; determining whether there is a deviation or whether there is no such deviation between the first data item currently being processed among the data items of the stream and the predetermined reference data item; and based on determining whether there is a deviation or whether there is no such deviation between the first data item currently being processed among the data items of the stream and the predetermined reference data item, outputting an indication of whether the first data item currently being processed among the data items of the stream is consistent with the predetermined reference data item.

[0053] In aspect 50 according to any one of aspects 28 to 49, the operations further include: comparing the one or more parts of the executable logic currently being executed on the first data item among the data items with a predetermined reference executable logic; determining whether there is a deviation or no such deviation between the one or more parts of the executable logic currently being executed on the first data item among the data items and the predetermined reference executable logic; and based on determining whether there is a deviation or no such deviation between the one or more parts of the executable logic currently being executed on the first data item among the data items and the predetermined reference executable logic, outputting an indication of whether the one or more parts of the executable logic currently being executed on the first data item among the data items are consistent with the predetermined reference executable logic.

[0054] In aspect 51 according to any one of aspects 28 to 50, the log record is a structured data record having a first field and a second field, the first field being used to store data representing the state of the specification for a specific value of the key, and the second field being used to store data specifying which parts of the executable logic represented by the specification have been executed for the specific value of the key from the time when the executable logic was started to the current time.

[0055] In aspect 52 according to any one of aspects 28 to 51, the log record is a structured data record having a field for storing data specifying which portion of the executable logic represented by the specification has been executed for a particular value of the key at the current time.

[0056] In aspect 53 according to any one of aspects 28 to 52, the specification is a graph, and wherein one or more log records specify a path through the graph, the path indicating which portions of the executable logic are executed when processing one or more data records associated with a particular value of the key.

[0057] In aspect 54 according to any one of aspects 28 to 53, the operations further include: generating one or more output records based on the processing of the one or more data items; wherein each of the one or more output records includes data not included in the log record.

[0058] In general aspect 55, one or more machine-readable hardware storage devices are described for identifying one or more portions of executable logic that are executed when processing one or more data items that are each associated with a given value of a key, wherein a specification represents the executable logic, wherein the specification is associated with the key, and wherein a state of the specification is maintained for corresponding values ​​of the key, the one or more machine-readable hardware storage devices storing instructions executable by one or more processors to perform operations comprising: accessing a specification representing the executable logic, wherein a state of the specification for a particular value of the key specifies one or more portions of the executable logic that are executable at the state; intermittently receiving data items of a data stream through an input device or port, wherein one of the data items or multiple data items each associated with a given value for the key; and generating a log record for the given value for the key, wherein the log record specifies the one or more portions of the executable logic executed when processing the one or more data items of the data items each associated with the given value for the key, the generating comprising: for each of the one or more data items associated with the given value for the key: identifying a given state of the specification maintained for the given value of the key associated with the data item; processing the data item by the data processing system, including executing on the data item one or more portions of the executable logic specified in the identified given state of the specification; and generating the log record specifying the execution of the one or more portions of the executable logic for the given value for the key.

[0059] In aspect 56 according to aspect 55, the operations further include grouping log records together by key, wherein first log records associated with a first key are grouped into a first group, and wherein second log records associated with a second key are grouped into a second group.

[0060] In aspect 57 according to any one of aspects 55 to 56, the log records are grouped in the memory.

[0061] In aspect 58 according to any of aspects 55 to 57, each log record comprises one or more fields indicating one or more attributes of a processed data item, wherein, for at least one log record, the field is a key field specifying a value of a key associated with the data item.

[0062] In aspect 59 according to any one of aspects 55 to 58, a subset of log records associated with a particular key does not include the key field.

[0063] In aspect 60 according to any one of aspects 55 to 59, only one log record associated with a particular key comprises the key field.

[0064] In aspect 61 according to any one of aspects 55 to 60, the operations further include: writing the log records associated with the specific key in the specific group to the file at the end of the predefined time interval.

[0065] In aspect 62 according to any one of aspects 55 to 61, the operations further include: reformatting the file; writing the reformatted file to an index compressed flat file (ICFF); and indexing the ICFF according to the key.

[0066] In aspect 63 according to any one of aspects 55 to 62, the specification includes a graph having multiple nodes, wherein each node represents one or more portions of the executable logic, wherein the graph is executable, wherein execution of the graph results in execution of portions of the executable logic, and the operations further include: based on the log records, tracing back the traversal of one or more nodes of the graph when processing the one or more data items.

[0067] In aspect 64 according to any one of aspects 55 to 63, the operations further include: parsing the log records to identify which nodes of the graph are traversed and which nodes of the graph are not traversed.

[0068] In aspect 65 according to any of aspects 55 to 64, the log records specify which nodes of the graph are traversed and which nodes of the graph are not traversed.

[0069] In aspect 66 according to any one of aspects 55 to 65, the operations further include: receiving a user-initiated selection of one of the nodes; in response to receiving the selection, causing the graphical user interface to display a configuration area, thereby providing the user with a configuration tool for configuring the one or more portions of the executable logic represented by the selected node, wherein the configuration tool includes a tool for turning on and off auditing; and receiving an opening of the audit initiated by the user through the configuration tool, thereby resulting in the generation of a log record associated with the selected node.

[0070] In aspect 67 according to any one of aspects 55 to 66, generating the log record that specifies the execution of the one or more parts of the executable logic for a given value of the key includes: specifying the results of executing the one or more parts of the executable logic in the generated log record, specifying variables that are read or set when executing the one or more parts of the executable logic, or identifying the one or more parts of the executable logic that are executed.

[0071] In aspect 68 according to any of aspects 55 to 67, the operations further include initiating actions to be performed by the computing system in response to executing the one or more portions of the executable logic.

[0072] In aspect 69 according to any of aspects 55 to 68, the action comprises requesting additional computing resources for executing the one or more portions of executable logic based on execution characteristics of the execution of the one or more portions of executable logic.

[0073] In aspect 70 according to any of aspects 55 to 69, the operations further include maintaining the states by storing state data in a data repository or in an in-memory data grid indicating which portion of the executable logic is currently being executed for each state.

[0074] In aspect 71 according to any one of aspects 55 to 70, the operations further include: updating state data specifying a current state of the value for the key in response to completing processing of the one or more of the data items each associated with the given value for the key.

[0075] In aspect 72 according to any of aspects 55 to 71, the operations further include dynamically determining a metric indicating which nodes of the graph are traversed and how many times the traversed nodes are traversed by the data items.

[0076] In aspect 73 according to any one of aspects 55 to 72, the operations further include: generating a stream of log records recording the paths taken through the graph; transforming the generated log records; and storing the transformed log records in a relational database or an index compressed flat file, including batching the log records together at a specified granularity.

[0077] In aspect 74 according to any one of aspects 55 to 73, the operations further include: dynamically outputting the determined metric via a user interface, wherein the outputted metric is updated in response to a change to the determined metric.

[0078] In aspect 75 according to any of aspects 55 to 74, the operations further include: receiving a user selection of a first data item in the stream of data items; and causing display of an indication of one or more portions of the executable logic currently executing on the selected first data item in the stream.

[0079] In aspect 76 according to any one of aspects 55 to 75, the operations further include: comparing a selected first data item currently being processed among the data items of the stream with a predetermined reference data item; determining whether there is a deviation or whether there is no such deviation between the first data item currently being processed among the data items of the stream and the predetermined reference data item; and based on determining whether there is a deviation or whether there is no such deviation between the first data item currently being processed among the data items of the stream and the predetermined reference data item, outputting an indication of whether the first data item currently being processed among the data items of the stream is consistent with the predetermined reference data item.

[0080] In aspect 77 according to any one of aspects 55 to 76, the operations further include: comparing the one or more parts of the executable logic currently being executed on the first data item among the data items with a predetermined reference executable logic; determining whether there is a deviation between the one or more parts of the executable logic currently being executed on the first data item among the data items and the predetermined reference executable logic or whether such a deviation does not exist; and based on determining whether there is a deviation between the one or more parts of the executable logic currently being executed on the first data item among the data items and the predetermined reference executable logic or whether such a deviation does not exist, outputting an indication of whether the one or more parts of the executable logic currently being executed on the first data item among the data items are consistent with the predetermined reference executable logic.

[0081] In aspect 78 according to any one of aspects 55 to 77, the log record is a structured data record having a first field and a second field, the first field being used to store data representing the state of the specification for the specific value of the key, and the second field being used to store data specifying which parts of the executable logic represented by the specification have been executed for the specific value of the key from the time when the executable logic was started to the current time.

[0082] In aspect 79 according to any one of aspects 55 to 78, the log record is a structured data record having a field for storing data specifying which portion of the executable logic represented by the specification has been executed for a particular value of the key at the current time.

[0083] In aspect 80 according to any one of aspects 55 to 79, the specification is a graph, and wherein one or more log records specify a path through the graph, the path indicating which portions of the executable logic are executed when processing one or more data records associated with a particular value of the key.

[0084] In aspect 81 according to any one of aspects 55 to 80, the operations further include: generating one or more output records based on the processing of the one or more data items; wherein each of the one or more output records includes data not included in the log record.

[0085] The above-described aspects may include one or more of the following advantages.

[0086] The logging system described herein can process log records in a way that has the least impact on system resources (e.g., memory and processing power). In addition, the system described herein processes these log records with reduced latency. The metrics of the execution of executable logic are dynamically determined and output so that users can track these metrics in real time. In addition, users are enabled to control which log records of executable logic are to be generated to focus log records on the parts of interest while reducing the generation of redundant data volumes. This helps users understand, track, and correctly operate the underlying data processing system.

[0087] Other features and advantages of the invention will be apparent from the following description, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is a schematic diagram of a database management system.

[0089] Figures 2A to 2G and Figure 5 Each is an illustration of a flow chart.

[0090] Figure 3 is a diagram of a flowchart instance.

[0091] Figure 4 is an illustration of a system for key-based logging of structured data items using executable logic.

[0092] Figure 6 is an illustration of an example process for processing key-based logging of structured data items using executable logic.

[0093] 7A to 7H is a diagram showing the generation and indexing of ICFF records and associated data blocks. DETAILED DESCRIPTION

[0094] refer to Figure 1, the system 10 for logging includes a data source 12, which may include one or more data sources (such as a storage device or a connection to an online data stream), each of which may store data in any of a variety of storage formats (e.g., a database table, a spreadsheet file, a flat text file, or a native format used by a host). An execution environment 14 (e.g., a data processing system) includes an execution module 16 and a user interface module 22. The execution environment 14 may be hosted on one or more computers under the control of a suitable operating system such as a UNIX operating system. For example, the execution environment 14 may include a multi-node parallel computing environment that includes a configuration of a computer system using multiple central processing units (CPUs), which may be local (e.g., a multi-processor system such as an SMP computer), or locally distributed (e.g., multiple processors coupled as a cluster or MPP), or remote, or remotely distributed (e.g., multiple processors coupled via a LAN or WAN network), or any combination thereof.

[0095] The execution module 16 implements logging (e.g., graph logging) that provides, for each unique key (e.g., identifier (ID)), detailed information about the state of the data and the path the data takes through a computer program (e.g., a graph such as a data flow graph described in U.S. Pat. No. 5,966,072, entitled “Executing Computations Expressed as Graphs,” the entire contents of which are incorporated herein by reference), which processes data records (also referred to herein as “data items”) for each unique key. In some examples, the execution module 16 implements data logging in a graph-based computation as described in U.S. Patent Publication 2009 / 0327196, the entire contents of which are incorporated herein by reference. In this application, the terms “data flow graph” or “graph” are used as examples or alternatives to “computer program.” In some examples, for convenience, the terms “data flow graph” and “graph” are used interchangeably and without distinction herein. In some examples, the unique key represents a customer ID for each customer, whose records are processed through the graph. The user interface module 22 displays logging information (e.g., metrics indicating which parts of the graph were traversed, and summary metrics indicating how many times a set of records traverse a particular part of the graph). The metrics described throughout this specification may be determined dynamically (i.e., during execution of the executable logic). In this example, the graph (e.g., a data flow graph) may be stored in a data store 26. The storage device providing the data source 12 may be local to the execution environment 14, e.g., stored on a storage medium connected to a computer running the execution environment 14 (e.g., hard drive 18), or may be remote from the execution environment 14, e.g., hosted on a remote system (e.g., host 20) that communicates with the computer running the execution environment 14 via a local area network or a wide area network.

[0096] The execution environment 14 is in communication with a data store 26 that stores data used by the user interface module 22 to display one or more user interfaces. The data store 26 may also be accessible by a development environment 28 (e.g., which may be optional) for developing user interfaces and diagrams (with logging capabilities) stored in the data store 26 that are accessed by the execution module 16 (when executing the diagrams) and used by the user interface module 22 to display the user interfaces.

[0097] In some examples, a chart or data flow graph is a computer program executed in a data flow graph execution environment that processes data from one or more data sources. The data from the data source is operated and processed according to the computer program and exported to one or more data receivers. For example, the data source and the receiver may include a file, a database, a data stream, or a queue. The data flow graph is represented as a directed graph, which includes: nodes representing data processing components, each node including program code (e.g., executable logic) for processing data from at least one data input and providing data to at least one data output; and nodes representing data set objects for accessing data sources and / or receivers. These nodes are connected by directed links, which represent the data flow between components, starting from the data source and ending at the data receiver. The data output port of the upstream component is connected to the data input port of the downstream component. The data flow graph can be reused for different data sources and different data receivers represented by the data set object. For example, the data structure and program code used to implement the data flow graph can support a variety of different configurations by parameterization, such as to make different sources and receivers easy to replace. In addition, in some arrangements, the flow of data in the data flow graph can be changed by using parameters so that a component or a series of components can be bypassed. Typically, a parameter represents a property of a component of a data flow graph that can be configured with a value and whose value can be changed. For example, such a property can be changed between uses of a data flow graph, and due to the change, the data flow graph can perform operations in different ways.

[0098] In some cases, the construction of a data flow graph can be highly technical in nature. When written to achieve a specific business purpose, the underlying structure and construction of the graph is determined based on technical considerations. For example, graph components may be selected to maximize reusability, or to support parallel processing. On the other hand, how and where the graph is used may be largely a business decision. Some of the parameters associated with such parameterized data flow graphs can be used to enable business users to customize the data flow graph without requiring the user to understand the technical complexities behind its implementation. Parameterized data flow graphs simplify customization and facilitate reuse.

[0099] In other words, Figure 1The computer system 10 shown in the figure may include: an optional development environment 28 coupled to a data store 26, wherein the development environment 28 is configured to build a computer program associated with a graph or data flow graph that implements a calculation that is performed on data flowing from one or more input data sets through a graph or graph of data processing components to one or more output data sets, wherein the graph or data flow graph is specified by a data structure in the data store 26, the graph or data flow graph having a plurality of nodes specified by the data structure and representing data processing components connected by one or more links, the links specified by the data structure and representing the flow of data between the data processing components; an execution environment 14 coupled to the data store 26 and hosted on one or more computers, the execution environment 14 including an execution module 16, the execution module configured to read a stored data structure specifying the graph or data flow graph and to allocate and configure computing resources for performing calculations on the data processing components assigned to the graph or data flow graph by the execution module 16, wherein the execution module 16 is configured to schedule and control the execution of the assigned calculations on the data processing components to enable the execution of any of the methods or processes described herein.

[0100] As described in further detail below, FIG. 2A to FIG. 2D A step-by-step progression of the flowchart is shown and the execution of various nodes at various states of the flowchart is demonstrated. Figure 2E to Figure 2G Shows various audit records generated in various states.

[0101] refer to Figure 2A , the diagram 29a shows a specification 30, which includes a flowchart 32 with nodes 32a to 32g representing data processing components. Typically, the specification represents executable logic and specifies various states, wherein, based on the state reached by executing the executable logic on the previous data item, the state of the specification specifies one or more parts of the executable logic that can be executed in the state. Typically, the executable logic includes source code and other computer instructions. In this example, one or more of the data processing components refers to a specific grouping or part of the executable logic. Typically, the chart includes a template for processing data records. This template includes a logic graph unit for reacting to input data records and generating output data records, for example, data records generated based on the logic contained in the specification. The chart is an example of a specification. Typically, the logic graph unit includes logic generated at least in part in a graphical manner, for example, by dragging and dropping various nodes from a template (not shown) to a window for building a chart. In the example, the node includes logic (not shown) that specifies the following items: how to process input data records, how to set variable values ​​used by the executable logic, for example, which output data records are generated when the conditions specified by the logic are met, etc. In an example, a node may be programmable by a user who inputs parameter values ​​and / or variable values ​​used in the logic of the node.

[0102] The chart itself is executable because the logic in the node is compiled into executable logic, and each node corresponds to one or more parts of the executable logic. For example, the system transforms the specification (and / or the chart in the specification) by compiling the logic in the node into executable logic. Because the chart itself is executable, the chart itself can process data records and can be stopped, started and paused. The system also maintains the state of the flowchart 32, for example, by tracking which of the nodes 32a to 32g is currently being executed. The state of the flowchart 32 corresponds to the state of the executable logic represented by the flowchart 32. For example, each node in the flowchart 32 represents a specific state of the executable logic (wherein one or more parts of the executable logic are executable in this state). When the flowchart 32 is being executed for multiple values ​​of a key, the system maintains the state of the flowchart 32 for each value of the key, for example, by maintaining the state of each instance as described in further detail below. Typically, the instance of the specification includes a specific implementation of the specification for the unique value of the key, for example, by executing the executable logic represented in the specification and maintaining the state of the executable logic for each unique value of the key. In this example, the flowchart 32 includes a state transition diagram in which each incoming data record drives a transition between nodes, and the data record is evaluated based on the state reached by processing the previous data record. The links between the nodes in the flowchart 32 represent the temporal flow of logic.

[0103] The specification 30 also includes a key 32h for a data record that identifies the flow chart 32 process including or associated with the key 32h. In this example, a custom identifier (ID) is used as the key. The key 32h may correspond to one of the fields of the data record, such as, for example, a subscriber_ID field, a customer_ID field, a session_ID field, etc. In this example, the customer_ID field is the key field. For a particular data record, the system determines the value of the key of the data record by identifying the value of the key field of the data record.

[0104] In this example, flowchart 32 subscribes to data records of a specified type (e.g., specified when configuring flowchart 32). In this example, flowchart 32 subscribes to data records including key 32h. In this example, flowchart 32 and data records share keys. Typically, a flowchart subscribes to a type of data record by including logic for processing those data records including the key of the flowchart. When data record processing begins, the system starts a new flowchart instance for each new value of the key of the flowchart, for example, by maintaining the state of executable logic for each new value of the key (represented in the flowchart). Typically, a flowchart instance includes a specific implementation of the flowchart for a unique value of a key, for example, by executing executable logic represented in the specification and maintaining the state of executable logic for each unique value of the key. The system performs data record processing by configuring the flowchart instance (and therefore the underlying executable logic) to respond to data records of a specific key value. In the example, the flowchart subscribes to customer short message service (SMS) data records. The flowchart instance for a specific customer ID manages the data records for that customer. The number of flowchart instances can be as many as the number of customer IDs encountered in the incoming data records.

[0105] refer to Figure 2B , diagram 29b shows the start of execution of flowchart 32. In this example, node 32a represents the start of executable logic and is being executed (e.g., as indicated by the black line around node 32a). In this example, flowchart 32 is in a state where node 32a is being executed. Figure 2C , Figure 29c shows the state of the flowchart 32 after completing node 32a. In this example, the state of the flowchart 32 is transferred to node 32b representing one or more other parts of the executable logic. Node 32b includes a wait node (hereinafter referred to as wait node 32b). Wait node 32b represents a wait state in which a part of the executable logic (corresponding to wait node 32b) waits for an input data record that satisfies one or more conditions. In the example, the wait state can be part of another state of the flowchart 32, for example, the system executes the wait node (to implement the wait state) and then executes the state of one or more other nodes. After completing the part of the executable logic represented by the wait node 32b, the system exits the wait state and executes node 32c representing the executable logic for implementing the decision. In this example, node 32c includes a decision node. Node 32c is referred to as decision node 32c hereinafter. Typically, a decision node includes a node containing logic for executing a decision (e.g., logic for finding a Boolean value). '

[0106] refer to Figure 2D, diagram 29d shows the state of the result of the decision represented by node 32c of flowchart 32. In particular, based on the result of the decision represented by node 32c, the state of flowchart 32 is transferred to node 32d, which is another waiting node. (In a variant, the state of flowchart 32 can be transferred to node 32g (which causes the state to transfer back to node 32a)). After completing the part of the executable logic represented by the waiting node 32d, the state of flowchart 32 is transferred to node 32e including the sending node. Typically, the sending node includes a node representing the executable logic for transmitting data to another system. After completing the part of the executable logic represented by node 32e, the state of flowchart 32 is transferred to node 32f including the completion node. Typically, the completion node represents the completion of the execution of the executable logic.

[0107] In this example, the flowchart 32 includes two states, a first state represented by nodes 32b, 32c, 32g, and a second state represented by nodes 32d, 32e, 32f. In this first state, the system waits for a particular data record (as represented by node 32b), and then executes node 32c, which in turn causes a transition (of the specification 30 and / or diagram 32) to a second state (which is initially represented by node 32d), or causes execution of node 32g. Once in the second state, the system again waits for a particular data record (as represented by node 32d), and then executes nodes 32e and 32f. By including nodes other than the wait node, the flowchart 32 includes a logical diagram of the temporal processing of data records.

[0108] refer to Figure 2E, diagram 29e shows the start state of flowchart 32, wherein an audit record is generated when start node 32a is executed. In particular, start node 32a represents a data processing component that is configured to generate audit record 32i, for example, when flowchart 32 is in a state where start node 32a is being executed (as shown in this example). Typically, an audit record (or also referred to as a "log record") includes data items (e.g., structured data items) that specify nodes traversed when processing a data record or event (and / or data processing components executed) and one or more attributes of the processed data record or event. In some examples, the log record is a structured data record having a first field and a second field, the first field being used to store data representing the state of the specification for a specific value of the key, and the second field being used to store data specifying which parts of the executable logic represented by the specification have been executed for a specific value of the key from the time the executable logic was started to the current time. In this example, when processing a data record for a particular key, the system generates an audit record (or one or more audit records) and stores data representing the state of the specification for a particular value of the key in a first field, and stores data specifying which parts of the executable logic represented by the specification have been executed for the particular value of the key from the time when the executable logic was started to the current time in a second field. In other examples, the log record is a structured data record with such a field, which is used to store data specifying which part of the executable logic represented by the specification has been executed for the particular value of the key at the current time. In this example, when generating the audit record, the system populates the fields of the audit record with data specifying which part of the executable logic represented by the specification has been executed for the particular value of the key at the current time. In yet other examples, the specification is a graph, and one or more log records specify a path through the graph, which path indicates which parts of the executable logic are executed when processing one or more data records associated with the particular value of the key. In this example, the system generates a log record specifying a path through the graph, which path indicates which parts of the executable logic are executed when processing one or more data records associated with the particular value of the key. In yet other examples, the system generates one or more output records based on processing of the one or more data items. In this example, each of the one or more output records includes data not included in the log record.

[0109] As described in further detail below, various types of audit records are predefined using various fields, and each field type specifies one or more attributes of processing data records or events. Typically, an event includes a data record (e.g., in a predefined format) representing a specific occurrence or non-occurrence of the event. In this example, node 32a is editable, such as by clicking on node 32a, which makes it possible to display a configuration screen that can be used to configure the node and therefore can also be used to configure the data processing component represented by the node. A part of the configuration can include turning on auditing (or also referred to as "logging") and specifying one or more types of audit records to be generated when traversing the node. In another example, the system described herein generates data for a user interface that displays visual representations of various types of audit records and provides controls for turning on / off various types of audit records. In this example, the system then generates a type of audit record (e.g., when logging of audit records of this type is enabled), which corresponds to or is associated with a specific type of node or data processing component. For example, audit record 32i is a chart start audit record for logging data for the start of a specified chart. Thus, if the user specifies that a graph start audit record is enabled, the system described herein identifies the correspondence between the graph start audit record and the start node (e.g., node 32a) based on the fact that the start node starts the graph and the graph start audit record logs the start of the graph, and generates a graph start audit record. In this example, the audit record 32i includes a key field (for specifying the key for which the graph was instantiated or executed), a graph_ID for specifying an identifier that uniquely identifies the graph 32, and a "start" value for specifying that the graph has started.

[0110] refer to Figure 2F, diagram 29f shows a second state of flowchart 32, in which audit record 32j is generated. In particular, the data processing component represented by decision node 32c is configured to generate audit record 32j, for example, when flowchart 32 is in a state in which decision node 32c is being executed (as shown in this example). (In this example, wait node 32b is also being executed in this state. However, wait node 32b is not configured to generate audit records. In this example, one of the audit records 32j specifies the result of decision node 32c, for example, by specifying whether link 32 or link 32m is traversed. Other audit records 32j can specify, for example, variables that are read or set when the data processing component represented by decision node 32c is executed. In this example, audit records 32j do not need to include a key field because the audit records are transmitted to a processing system (e.g., an internal or external computing system that processes the audit records) at specified pulses (e.g., times or predefined time intervals), resulting in the audit records being transmitted in groups, blocks, or as part of a file.

[0111] For example, the system groups audit records together by key, wherein a first audit record associated with a first key is grouped into a first group, and wherein a second audit record associated with a second key is grouped into a second group. In this example, the audit records are grouped in a memory of a system (e.g., system 10). To improve space efficiency, the system groups audit records in memory so that the system does not have to duplicate data on sequentially related audit data, for example, by including a key in each audit record. Each audit record includes one or more fields that indicate one or more attributes of a processed data item. For at least one audit record, a field is a key field that specifies a value of a key associated with the data item. Because the audit records are grouped together by key, a subset of audit records associated with a particular key does not include a key field. In some examples, only one audit record in the audit records included in the group of a particular key includes a key field. In this example, the system writes the audit records associated with a particular key in a particular group to a file at the end of a predefined time interval (e.g., a pulse).

[0112] For example, the system identifies the key for the audit records in the group (which may be represented as a block or file) by parsing the contents of the first audit record or the chart start audit record. Because not every audit record needs to include the key field (as well as other fields such as the chart_ID field, which other audit records included in the same pulse as the audit record 32i can look up in the chart start audit record), the system described herein can process the audit records in a manner that minimizes the impact on system resources (e.g., memory and processing power) relative to the impact on system resources (e.g., memory and processing power) if every audit record needs to include the key field to enable keyed logging of events. In addition, the system described herein reduces the latency of processing these audit records relative to the latency of processing audit records where each audit record has the key field, because the size of the audit records is reduced because each audit record does not need to include the key field. In addition, the system is configured to store the audit records in a partition (in memory) assigned to a particular key, thereby further reducing the need for each audit record to include the key field. At the beginning of a new pulse, the system described herein is configured to include the key field in the first audit record of the pulse.

[0113] refer to Figure 2G , diagram 29g shows a third state of flow chart 32, wherein audit record 32k is generated. In particular, sending node 32e is also configured to generate audit record 32k. In this example, at least one of audit records 32k specifies an action performed by the data processing component represented by sending node 32e. As with audit record 32j, audit record 32k also need not include a key field, because audit record 32k is transmitted as part of the same pulse that includes audit record 32i. Other audit records 32k may specify the time of sending an action, the value of a variable read, the value of a variable set, etc.

[0114] In this example, the system analyzes the contents of the audit record to trace back the traversal of one or more nodes of graph 32 while processing one or more data items. In addition, the system parses the audit record to identify which nodes of graph 32 were traversed, which nodes of graph 32 were not traversed, and why these nodes were not traversed (e.g., which links or paths of these nodes were not traversed).

[0115] refer to Figure 3 , FIG. 37 shows, for example, a system according to flowchart 32 ( Figure 2A ) The flowchart instances 33, 34 and the data items of the data records 36a, 36b generated are created. That is, a new copy or instance of the flowchart 32 is created for each new key detected in the data records 36a, 36b.

[0116] Each of flowchart instances 33, 34 is associated with a "customer_id" key. Flowchart instance 33 processes a data record that includes a value "VBN3419" in its "customer_id" field, which is a key field in this example. Flowchart instance 34 processes a data record that includes a value "CND8954" in its "customer_id" field. In this example, the system does not re-execute the executable logic for each flowchart instance. Instead, the system executes the executable logic and then implements the flowchart instance by maintaining the state of the corresponding value of the key. Therefore, the example of a "flowchart instance for processing data records" is a system that executes executable logic (represented by a flowchart), maintains the state of each value of the key, and processes data records associated with a specific value of the key (based on the state of the state machine for the specific value of the key).

[0117] In this example, flowchart instance 33 includes Figure 2A Nodes 33a to 33g and links 33l, 33m corresponding to nodes 32a to 32g in FIG. Figure 2A The nodes 32a to 32g in FIG. 3 correspond to the subnodes 34a to 34g and the links 341 and 34m.

[0118] The flowchart instance itself is executable. After the system receives the input data record associated with the specific value of the key, the flowchart instance for the specific value of the key processes the input data record, for example, by the system executing one or more parts of the executable logic corresponding to the flowchart instance (or one or more nodes of the flowchart instance) on the data record. The flowchart instance continues to process the input data record until the input data record reaches the completion node or the waiting node. In this example, the flowchart instance continues to process the input data record, for example, by the system continuing to process the input data record until the part of the executable logic corresponding to the completion node or the waiting node is reached. If the input data record arrives at the waiting node, the flowchart instance will pause until a certain amount of time has passed or a suitable new input data record arrives. Typically, suitable data records include data records that meet one or more specified conditions or criteria (for example, included in the logic of the node). If the input data record arrives at the completion node, the execution of the flowchart instance is completed.

[0119] A flowchart instance has its own lifecycle. When a data record arrives, one or more current states of the flowchart instance change: the data record triggers a decision, either returning to the beginning of the flowchart instance or sending a message to the client. When the data record reaches the completion node, the client's flowchart instance ends.

[0120] In this example, the system starts a flowchart instance 33 for a "VBN3419" value for a key field (e.g., customer_id=VBN3419). The flowchart instance 33 processes a subset of data records 36a, 36b that include customer_id VBN3419. In this example, the flowchart instance 33 processes data record 36a having a value "VBN3419" in the customer_id key field. Nodes 33a, 33b, 33c, 33d of the flowchart instance 33 process data record 36a. In this example, upon execution of the processing component represented by node 33a, the system generates an audit record 33i that specifies that the graph 32 (which is assigned graph_ID: 3454) and that it has been started for key VBN3419. Upon execution of the data processing component corresponding to node 33c, the system generates an audit record 33j, at least one of which specifies a link 331 that traverses the instance 33. The current state of the flowchart instance 33 is the state of sending or executing an action, as represented by the dotted line of node 33e. In this example, the data processing component represented by node 33e generates an audit record 33k, at least one of which specifies the action. The action may include sending an SMS message to be sent (e.g., the message includes a discount or promotion). Other such "actions" described herein may be one or more of the following: initiating execution of another computer program to further process the processed data record, initiating storage of the processed data record in a memory, initiating transmission of the processed data record to another computer system via a network, or requesting additional computing resources for one or more data processing components represented by one or more of the nodes in the flowcharts 32, 33, and 34. For example, the result of the flowchart instance 33 may be to perform an action of requesting additional computing resources for one of the data processing components represented by the nodes in the flowchart instance 33 (such as one of nodes 32b, 32c, 32d, 32e, 32f, 33b, 33c, 33d, 33e) based on one or more execution characteristics of the flowchart instance 33. An example of such an execution characteristic may be a waiting time of one or more waiting nodes 32b, 32d, 33a, or 33d. The request for additional computing resources may be granted, resulting in receiving an allocation of the requested additional computing resources. The term "computing resources" as used herein includes one or more of the following: access to a portion of memory accessible by the execution environment 14 (such as main memory), access to a data store accessible by the execution environment 14 (e.g., data source 12 or data store 26), and access to processing capabilities required by the execution environment 14 (such as access to a CPU or virtual core accessible by the execution environment 14).

[0121] In this example, the system maintains the state of instance 33, which is executed for a particular key value (i.e., VBN 3419). In this way, the system is able to log which nodes of instance 33 are traversed for that particular key value - when processing events or data records associated with that key value. In addition, by grouping audit records together into pulses, audit records 33j, 33k do need to include the key field, and when audit records 33i, 33j, 33k are included in the same pulse, these audit records can still be identified as audit records for the key specified in audit record 33i.

[0122] The system begins a flowchart instance 34 for a key value of "CND8954" (e.g., customer_id=CND8954). Flowchart instance 34 processes a subset of data records 36a, 36b that include customer_id CND8954. In this example, flowchart instance 34 includes wait nodes 34b and 34d. Each data record can only satisfy the conditions of one wait node per flowchart instance. Therefore, flowchart instance 34 processes data record 36b with customer_id=CND8954 through nodes 34a, 34b to node 34d, and then waits for a second data record with the same key before proceeding to node 34e (e.g., when the output of decision node 34c is "yes"). When processing data record 36b using node 34a, the system described herein generates an audit record 34i that specifies that chart 32 (identified by chart_ID: 3454) was started for key: CND8954. However, in this example, the output of decision node 34c is "no", as represented by the dotted line around node 34g. Thus, the data processing component represented by node 34c generates audit records 34j, one of which specifies traversing link 34m based on the result of the decision represented by decision node 34c. In this example, sending node 34e is not traversed. Thus, the data processing component represented by sending node 34e does not generate an audit record. In the aforementioned examples, references to nodes that process data records refer to data processing components that represent nodes that process data records. In addition, in these examples, the graph itself is executable-executing the graph results in executing executable logic corresponding to the nodes of the graph, or executing a data processing component that encapsulates part of the execution logic.

[0123] exist Figure 3 In a variation of , the system generates more than one flowchart instance for a single key value. For example, for the same customer, there may be multiple flowchart instances with different start and end dates, or for the same user, there may be multiple flowchart instances for different purposes (such as different industrial processes or different marketing campaigns).

[0124] In this example, the system maintains the state of the instances by storing state data (e.g., data indicating which node is currently executing for each instance) in a data repository or in-memory data grid. Typically, the state data includes data indicating the state. In this example, the instances are associated with values ​​for a key. The data repository or in-memory data grid stores the values ​​for the key. The system maintains the state of the instances by storing state data for each value of the key. Upon completing processing of a data record for a particular value of the key, the system updates the state data in the data repository to specify (in Figure 2A The next node in the flowchart 32 in FIG. 3 represents the current state of the value for the key. Then, when another data record arrives, the system looks up the current state of the value for the key in the data repository and executes the portion of the executable logic corresponding to the node representing the current state of the executable logic for the value for the key.

[0125] refer to Figure 4 , environment 100 illustrates an architecture for audit logging. Environment 100 includes a system 101 (e.g., a data processing system) having an event engine 102, which is configured to process data records (and / or other structured data items). In this example, event engine 102 includes rule set log 104, chart execution module 106, and chart 108. Environment 100 also includes databases 110, 112. In this example, rule set log 104 includes logs (e.g., logs indicating which rules are executed, when they are executed, parameter values ​​and / or values ​​set during execution, etc., or rule use case traceability information (also referred to herein as traceability information)) generated during the execution of rules in a business rule environment (BRE) as described in U.S. Patent No. 7,877,350, the entire contents of which are incorporated herein by reference. In this example, event engine 102 collects rule use case traceability information from the business rule environment to optionally include the information in the audit record (e.g., to explain the business logic processed by the rule set rather than in the chart decision block). In an example, the BRE implements a logging data stream by writing its log to a port on a graph component, and then writing the log from the port to a file. In this example, while the event engine 102 is collecting audit records (and other audit information) for the current event, the event engine 102 intercepts the logging data stream by replacing function calls that write out traceback information in the BRE with function calls that store traceback information in the event engine's internal data structures. In some examples, the event engine 102 intercepts the logging data stream by providing an alternative implementation of the logging functionality that the BRE is using (e.g., write_to_log). In other examples, the BRE supports the option of using different functionality.

[0126] In this example, the graph execution module 106 is configured to execute one or more data flow graphs (e.g., one of the graphs 108). In this example, one of the graphs 108 is an event engine graph that generates cluster metrics and writes them directly to a file in the database 110. In general, an event engine graph is a graph used to monitor the performance of the event engine 102. In general, cluster metrics refer to detailed information at an aggregate level of the system and / or event engine 102, such as execution details, for example, metrics indicating which parts of the data flow graph were traversed, and aggregate metrics indicating how many times a set of records traversed a particular part of the data flow graph. In some examples, this file is appended (e.g., appended to other previously stored files, or updated) on each pulse (e.g., data push), and a new file is opened each day.

[0127] In this example, another data path comes from the event engine business logic (e.g., based on the execution of the chart execution module 106 that executes the chart). As the chart executes, the chart execution module 106 generates a stream of audit records 111 that record the path taken by the chart (e.g., exiting wait blocks, traversing flows, and sending actions). In this example, the audit records include the activities that are logged and stored in the audit file (i.e., database 112) or in an indexed compressed flat file (ICFF) stored in database 116. Typically, the audit file includes files for storing various audit records. In this example, the audit records 111 are written to the file at the end of each pulse. In this example, the database 112 receives a new file per pulse.

[0128] Environment 100 includes an audit file processor module 114 that includes a diagram that processes raw information (i.e., audit files stored in database 112) and reformats the raw information into a format for ICFF. In this example, database 112 transmits the audit file to audit file processor module 114 for reformatting. In turn, audit file processor module 114 transmits the reformatted file to database 116, which writes the reformatted file to ICFF record 116a, which is indexed (in index 116) by one or more key fields (e.g., object ID) and merged by time (e.g., hourly or daily). In this example, ICFF record 116a references (e.g., via a pointer or another reference data structure) data block 116c. In this example, while portions of the original information are reformatted to obtain ICFF record 116a (which is then compiled in index 116b), other portions of the original information are stored in data block 116c, which is referenced by ICFF record 116a and is thereby identifiable and accessible via ICFF record 116a.

[0129] In some examples, audit records and / or audit files are reformatted by batching audit records together at a specified granularity (e.g., one event for one subscriber to one chart), computing key fields to be stored on the ICFF record (e.g., fields for event type, subscriber ID, timestamp, chart instance ID, etc.), and then grouping the remaining audit records into collections, as shown below:

[0130] In an example, event engine 102 generates audit records that are optimized for space efficiency (e.g., rather than audit records that are optimized for query efficiency). In this example, the audit records are optimized for space efficiency, for example, by not having each audit record include a timestamp, subscriber_ID, etc. In order to put these audit records into ICFF, the system described herein needs to use more complete records (e.g., these more complete records include a timestamp and object ID on each audit record) so that the system described herein can efficiently search for each record (e.g., each record based on the content of the audit record).

[0131] In some examples, database 116 includes a relational database rather than ICFF, because the implementation goal of database 116 is to be able to query event sequences. To this end, the system transforms the stream of audit records into one or more records (e.g., ICFF records). Each record includes one or more indexable / searchable key fields and a data set representing the remaining audit events.

[0132] In some examples, the granularity level of the record is one event for one subscriber (e.g., user) of one chart. Thus, the system described herein extracts a sequence of audit records corresponding to one event for one subscriber of one chart (e.g., from an audit file in database 112). In this example, the events occur sequentially. In addition, the audit records for each event are sequentially stored in database 112 (e.g., in a file in database 112). Using the extracted audit records, the system generates records with multiple key fields (e.g., including an event type field, a subscriber ID field, a timestamp field, a chart instance ID field, etc.). In some examples, the first audit record in the sequence includes values ​​for the multiple key fields. The system uses these included values ​​when generating new records. In addition, the system constructs a data block corresponding to the remaining audit records of the sequence. The system stores the data block in database 116 as a data set (e.g., as a binary large object (BLOB)) as part of the generated new record or associated with the generated new record (e.g., by making the generated new record reference the data set or blob).

[0133] Using the generated records and / or data sets, the system can query ICFF (or database 116) for various information, such as, for example, "What happened when subscriber 12345 encountered event 7 on June 8?", and then display the decision sequence inside the event engine and the rule set associated with the event. In another example, the system queries ICFFF (or database 116) with the following query: "What happened when subscriber 12345 sent a message on June 8?" In this example, to answer this query, the new record generated includes another key field to identify that the message action has been triggered (or not triggered). In this way, the system selects which key fields to include in the record based on the scope of the query to be supported.

[0134] In this example, database 116 stores raw metrics in ICFF. In this example, environment 100 also includes a metrics database 120 and an interface for accessing metrics. Therefore, environment 100 includes a batch graph module 118 that runs a batch graph (e.g., every 10 minutes) to aggregate raw metrics and write them to the metrics database 120. Environment 100 also includes a batch graph module 122 that runs "on demand" to calculate metrics (such as one or more states that are prevalent in system 100) for display, for example, in user interface 124. In this example, batch graph module 122 accesses metrics from metrics database 120 and / or database 116 and processes these accessed metrics for display in user interface 124. In this example, the graphs generated or executed by batch graph module 118 do not have persistent state, while the metric state is persisted in metric database 120. Thus, matching graph module 120 queries metric database 120 to obtain metrics for display in user interface 122. In the example, various types of user interfaces are displayed, including, for example, dynamic displays of metrics, reports on metrics, and investigation interfaces, each of which is described below:

[0135] In a dynamic display of metrics, the console displays metrics (e.g., summary counts) and updates these metrics in the display in near real time. The system queries the metrics database 120 to obtain this data and metrics. In this example, the console is a web user interface (UI) that includes a polling interface that can only extract data from the server (e.g., on a timer or at a specified time). In this way, the graphs generated by the batch graph module 118 cannot push data to the UI. Instead, the UI (e.g., a web page) intermittently queries the server to obtain updated counts to be displayed. The updated counts will be stored in the metrics database 120. In this example, there is server-side logic to interface between the UI (web page) and the metrics database 120, including, for example, a Java application server located between the browser and the database.

[0136] In another example, user interface 124 displays a report on metrics queried from metrics database 120. In this example, the user will be able to view the metrics in reports and dashboards. The metrics specify how many people entered each activity, how many people traversed each branch (e.g., of a graph), how many people were offered an offer, how many people accepted the offer, etc. The user interface includes controls for "drilling down" into more detailed information in this report. To drill down (e.g., to answer a query for "what happened to these 10 objects that saw the triggering event but did not get the offer"), the system accesses records in ICFF in database 116. To implement these reports, the system implements a server-side module (in this example, batch graph module 122) that can access metrics database 122 and database 116 when drilling down.

[0137] In yet another example, user interface 124 includes a survey interface in which a user can ask questions such as “What happened when subscriber 12345 encountered event 7 on June 8?” To answer these questions, the system uses a server-side module (e.g., Java, graph, etc.) between user interface 124 and the ICFF in database 116 to query the ICFF (e.g., in database 116).

[0138] The event engine 102 creates a variety of different types of audit records to be stored in the ICFF in the database 116. Table 1 below specifies different types of audit records and their corresponding fields. Depending on the level of the log record, some of the following audit record types are optional. In this example, the ICFF stores a list of fields corresponding to the fields shown in Table 1 below, such as key fields for generating ICFF records (as described above). The values ​​of the fields in italics in Table 1 below are not generated by the event engine 102, but are inferred by the system described in this article by looking back at the audit stream when writing data to the ICFF. This means that the event engine 102 needs to generate significantly less data. In Table 1 below, the "?" column indicates whether the audit record is optional. The "+" symbol indicates that the audit record is generated by the event engine 107. The letter indicates that the audit record is optionally generated and controlled by a set of related control points, and the asterisk indicates that the record type is enabled by default.

[0139] Table 1

[0140]

[0141]

[0142]

[0143] In the example, database 112 groups audit records for each object (e.g., by key) (e.g., continuously or intermittently). That is, audits are automatically grouped by object or key. As a result, event engine 102 does not need to copy (e.g., every pulse) timestamps, object_ids, or instance_ids to each audit record in the audit file. Instead, database 112 can obtain values ​​from these fields based on the values ​​of these fields that previously appeared in the group. In this example, no state needs to be retained between pulses. This means, for example, that a block exit audit record needs to have a block_name, because a block entry audit record may have appeared many pulses ago. In database 112, the audit records will be completed using the relevant fields shown in Table 1 above.

[0144] In an example, the system detects the occurrence of an event for an object of the graph (e.g., a user or subscriber). In this way, the event engine 102 generates an "event seen" audit record, then generates an "event dispatched" audit record, and then generates a "block exited-event" audit record. From these audit records, the system (e.g., the audit file processor module 114) obtains the values ​​of the following fields: event_type, timestamp, and instance_i. In this example, the system collects these audit records together into a blob, and the system sets key fields (e.g., key fields of event_type, timestamp, and instance_id) in another record (e.g., an ICFF record) based on the data seen (e.g., based on these audit records).

[0145] In another example, a timeout occurs for an object of a chart. In this example, the event engine 102 generates the following audit records: an "event seen" audit record of type alarm, followed by an "alarm dispatched" audit record, and then a "block exited-alarm" audit record. The system determines the values ​​of the following key fields from these audit records: event_type, timestamp, and instance_id. In this example, the system determines the value of the instance_id field, for example, from the block exit-alarm audit record, rather than inferring the value of the field. In this example, the system collects these audit records into a blob, and the system sets the key fields (e.g., event_type, timestamp, instance_id, etc.) based on the data seen (e.g., audit record).

[0146] In yet another example, an event occurs for an object that starts a graph. In this example, the event engine 102 generates an "event seen" audit record (which may be of the type that started the graph or may be another type), then an "event dispatched" audit record, and then a "graph started" audit record. The system obtains the values ​​of the following key fields from these audit records: event_type, timestamp, and instance_id. The system collects the audit records into a blob and sets the key fields (event_type, timestamp, instance_id, etc.) based on the data seen (e.g., audit record).

[0147] Because audit records are grouped together by key, a subset of audit records associated with a particular key does not include the key field. For example, database 112 or audit file processor module 114 may receive the following audit record sequence: event-seen audit record, event-scheduled audit record, block-exit-event audit record. In this example, database 112 or audit file processor module 114 obtains an object ID (e.g., key) from the event-seen audit record. In this way, event engine 102 does not need to copy the key to event-scheduled audit records and block-exit audit records because these audit records will be in memory (e.g., in database 112 or other system memory) in order after the event-seen audit record. Similarly, event-scheduled audit records include an instance ID field. In this way, the event engine does not need to include a copy of this field in the block-exit audit record. Since the system groups audit records in memory (and on disk), the system does not need to copy the key to all audit records in the group, and thereby saves system resources and reduces the amount of memory consumption.

[0148] To avoid generating an excessive amount of data, the user controls when and which audit records the event engine generates. Although auditing (and therefore metrics) can be turned off completely, the core set of audit records (marked with a plus sign in the preceding table) is not disabled if auditing is turned on. The core set of audit records tracks the input (events and alarms), output (actions), errors, and chart and wait block activity of the event engine. Typically, an alarm consists of a data record of a predefined type (for example, a data record that a chart sends itself) that will arrive at a precalculated time in the future. Other audit record types can be selectively enabled and disabled. Enabling optional audit records involves two aspects - what should be enabled and when it should be enabled. For what to enable, several different audit record groups can be selectively enabled / disabled. These audit record groups include:

[0149] A: Alarm settings (open by default)

[0150] D: Decision making (open by default)

[0151] P: Monitoring point (open by default)

[0152] R: ruleset invocation and ruleset events

[0153] S: variable setting

[0154] V: variable read

[0155] L: Find access and find variables

[0156] When enabled, logging of these audit events can be enabled in their entirety or limited based on one or more of the following:

[0157] Only for a set of specified object ids (e.g. keys)

[0158] Only for the specified chart name and / or monitoring name set

[0159] Only for specified event types (such as record types)

[0160] In the example, the event engine 102 configures the logging of these audit records through control points. Typically, a control point includes runtime parameters defined by a user (who generates the chart) and is used to change the runtime behavior of one or more charts (or chart instances). In this example, audit records are controlled (e.g., by the event engine) through a set of five interrelated control points. The five control points are:

[0161] Audit Types - This is a list of the types of audit records to be turned on (using the letters listed in Table 1 above).

[0162] Audit Entities - This is a list of object ids. If not empty, the event engine will only generate audit record types in the audit type list for object ids included in this list. (Otherwise, all object ids will be audited).

[0163] Audit Events - This is a list of event types (e.g., numbers corresponding to or representing event types). If not empty, the event engine will only generate audit record types in the audit type list for event types included in this list. (Otherwise, all events will be audited). There are various types of events, including, for example, an event specifying that an SMS message was sent, an event specifying that a user renewed their mobile plan, etc.

[0164] Audit Charts - This is a list of chart names and monitoring names. If not empty, the event engine will generate audit record types in Audit Type for charts whose chart names or monitoring names are included in this list.

[0165] (Otherwise, all diagrams will be reviewed).

[0166] Audit variables - This consists of a comma separated list of variable names. If not empty and the event engine is configured to generate audit record types for variables (e.g., variable set or variable read), the event engine will only generate audit records if the variable name is in this list.

[0167] In some examples, event engine 102 applies different rules to different audit record groups. In this example, there is a first audit record group (e.g., first audit group) with a first rule set (e.g., audit type 1, audit entity 1, audit event 1, audit chart 1, and audit variable 1), and a second audit record group (e.g., second audit group) with a second rule set (e.g., audit type 2, audit entity 2, audit event 2, audit chart 2, and audit variable 2), and so on.

[0168] In this example, each of the audit types, audit entities, audit events, audit graphs, and audit variables includes a type of filter. If any or all of these filters are set, only a subset of these events will include additional audits (e.g., audits that include optional audit forms).

[0169] In one example, the system is configured for minimal auditing by implementing a first audit group as follows:

[0170] Audit Type 1 = "" (do not enable any optional content)

[0171] Audit Entity 1 = "" (all object IDs)

[0172] Audit Event 1 = "" (all event types)

[0173] Review Chart 1 = "" (all charts)

[0174] Audit variables 1 = "" (all variables, although this is ignored because the system is configured to turn off logging for optional variables).

[0175] Now, in another example, the system is configured to evaluate why an offer for object_ID 12345 has not been sent despite the fact that all criteria for sending the offer are met for that object_ID. The chart implementing this offer is "My Offers Chart" and the triggering event is number 6. In order to get all the more details for this specific object ID as to why the offer was not sent, the system is configured to have a second audit group as shown below:

[0176] Audit Type 2 = "ADPRSVL" (enable all audit types)

[0177] Audit Entity 2 = "12345" (only this object ID)

[0178] Audit Event 2 = "6" (only this event type)

[0179] Review Chart 2 = "My Offers Chart" (this chart only)

[0180] Audit variable 2 = "" (all variables).

[0181] In this example, when an event occurs, the system first checks to see if the object ID is in the list of audit events 2, and if the event type is in the list of audit events 2, and if the chart is in the list of audit charts 2. If all of these attributes (e.g., filters) evaluate to true, the system will implement all audit record types as specified in "Audit Type 2", which will begin optional logging for the duration of the event. If not, the system implements Audit Type 1, which only logs the default items. In this example, the system is configured to check the highest numbered group first, and then run backwards until a matching group is found. If no matching group is found, the system implements and uses the default item. In a variation, the system can be configured to take the union of all matching groups, so that more optional logging is added.

[0182] In this example, the event engine 102 implements conditional audit logging (e.g., by implementing monitoring point audit logging) and optimizes its execution performance. To optimize this performance, the event engine 102 pre-computes (e.g., pre-generates) data structures for making conditional audit decisions. Typically, a conditional audit decision specifies that an audit record is generated when a condition evaluates to a specified value (e.g., true or false). In this example, the event engine performs conditional audit calculations at the beginning of event processing for an event or alarm. For example, each time the event engine detects an event or alarm, the event engine performs calculations (e.g., with the above-mentioned audit group) - but only at the beginning of processing for each event or alarm (e.g., rather than for each audit record). The event engine saves the conditional audit calculations in a package of global variables and queries them inside each audit function that can be used as a condition.

[0183] For each audit record type, the event engine 102 implements a single flag that specifies whether to generate an audit record of that audit record type. The event engine 102 analyzes the contents of the audit entity list and the audit event list when implementing the single flag, because both lists are specified or generated when event processing begins. Because an event can be processed by many charts, the event engine determines whether to implement an audit function (e.g., audit record type) for each chart. If the audit chart is not empty, the event engine converts the contents of the audit chart into a list of instance ids (each id identifies a chart) to simplify the comparison between the ids of the implemented charts and the instance ids that specify which charts generate audit records (and what types of audit records are generated).

[0184] In an example, the event engine 102 collects audit records in an array of memory (e.g., in the event engine 102 or in a system that includes the event engine 102), and then writes the audit records out in one block to an audit file at the end of processing an event or event stream. In this example, audit records from different events (i.e., events associated with different keys or user IDs) are not mixed together in memory. Instead, the memory is partitioned, and each partition has its own audit file. In this way, the event engine 102 is configured to automatically group audit records together by key.

[0185] In this example, the audit file is versioned. There will be one audit file instance per pulse. The audit files will accumulate on disk, and the audit file processing graph (executed by the audit file processor module 114) will extract each audit file (from each partition) at the end of the pulse and post-process the audit records to write them to the ICFF.

[0186] In this example, the system generates the metrics database 120, for example, by post-processing audit records from the ICFF (in the database 116) rather than reading the audit records from the memory in the event engine 102. In this example, the ICFF is not cleared. In this way, the metrics database 120 can be recreated at any time, including using different aggregation periods. In this example, the table in the metrics database 120 has a cluster_name column so that one database can be used for multiple clusters. Typically, a cluster includes an event engine instance. Typically, an event engine instance includes one or more specifications for a specific implementation of a unique value for a key (for which the event engine 102 executes one or more graphs). That is, each object ID is processed by exactly one event engine instance. Therefore, for each unique key (e.g., an object ID), there will be only exactly one event engine instance with all its graphs. Audit records for the same cluster will have the same cluster_name value.

[0187] In this example, the metrics database 120 can be queried to obtain metrics indicating the state of the event engine 102, such as the total number of events processed by the event engine 102 (e.g., based on event seen audit records), the number of events that resulted in errors (e.g., based on error dispatched audit records), the number of events that triggered activities (e.g., based on action sent audit records), which subscribers transitioned from one phase of a particular activity to another (e.g., based on block entry audit records), the number of subscribers deleted or added in the event engine on a particular day (e.g., based on graph start and graph complete audit records), and the number of total graphs running in the event engine that track the state of the event engine 102 as the event engine processes events (e.g., based on graph start audit records).

[0188] In addition, the metrics database 120 can be queried (e.g., via a client device or the system described herein) to retrieve data specifying reasons why a message was not sent to a particular key (e.g., each subscriber). To this end, the event engine is configured to log the links taken (and / or components executed within a graph) and the rule set use cases accessed and implemented via the event engine 102 (e.g., by generating audit records that specify which components and / or links were executed). The system can then be configured to compare the path taken through the graph (e.g., specified by the audit record) with all possible paths through the graph to determine why a particular message was not sent (e.g., because the path corresponding to sending the message was never traversed due to traversing an alternative path (as specified by the audit record)).

[0189] refer to Figure 5, the diagram 250 includes nodes 252 to 266. In this example, each of the nodes 252 to 264 represents a data processing component that includes code for automatically generating one or more audit records. For example, node 252 represents a data processing component configured to generate a diagram start audit record 252a. Node 254 represents a data processing component configured to generate a variable setting audit record 254a and a variable reading audit record 254b. In this example, the variable setting audit record 254a specifies the name of the variable that is set. In this example, the name of the variable is "next message date". The variable reading audit record 254b specifies the name of the variable that is read. In this example, the name of the variable that is read is "start date". The data processing component represented by node 256 generates audit records 256a to 256i. The data processing component represented by node 258 generates audit records 258a to 258c. The data processing component represented by node 260 generates audit records 260a to 260g. The data processing component represented by node 262 generates audit records 262a to 262c. The data processing component represented by node 264 generates audit records 264a to 264g. In this example, "next message date" is the next date that the event engine will process the message. In this example, "next message date" is set to the first date of the activity. In this example, as soon as the chart 250 begins (this may be before the start of the activity), the event engine sends the start, discount and IVR actions represented by nodes 256, 260, 264 respectively. In this example, start, discount and IVR actions are three actions sent by the chart. When the object triggers the activity, the system sends the "start" action. The system sends the "discount" action to configure the external subsystem for the discount (for example, get 10 minutes for free when recharging an account), and sends the "IVR" action to configure the external subsystem for processing incoming calls (for example, "tell me what discounts are available for me."). In this example, if the corresponding metadata is null or the subscriber (represented by the key) is in the control group (as represented by node 258), the graph 250 does not proceed to node 260 (and therefore does not send an offer action), and instead proceeds to node 262. In this example, if the corresponding metadata for the IVR is null or the subscriber (represented by the key) is in the control group (as represented by node 262), the graph 250 does not proceed to node 264 (and therefore does not send an IVR action), and instead proceeds to node 266 to check to see if the event engine needs to perform message processing daily (as represented by node 266).

[0190] refer to Figure 6, process 280 is implemented by a data processing system for identifying one or more portions of executable logic that are executed when processing one or more data items that are each associated with a given value of a key, wherein a specification represents the executable logic, wherein the specification is associated with the key, and wherein a state of the specification is maintained for the corresponding value of the key.

[0191] In operation, a data processing system accesses (282) a specification representing executable logic, wherein a state of the specification for a particular value of the key specifies one or more portions of the executable logic executable at the state. The data processing system also intermittently receives (284) data items of a data stream, such as through an input device or port, wherein one or more of the data items are each associated with a given value of the key. The data processing system generates (286) a log record for the given value of the key, wherein the log record specifies one or more portions of the executable logic executed when processing one or more of the data items each associated with the given value of the key. In this example, the generating includes the following operations for each of the one or more data items associated with the given value of the key: The data processing system identifies (288) a given state of the specification for the given value of the key associated with the data item. The data processing system processes (290) the data item according to the execution of the one or more portions of the executable logic represented in the specification as being associated with the given state. The data processing system generates (292) a log record of the execution of the one or more portions of the executable logic specified for the given value of the key.

[0192] 7A to 7H Demonstrates the generation of ICFF records and data blocks based on the audit records generated during chart execution. Fig. 7A , diagram 300 includes a chart 301 having components 304 to 316. In a real-world example, chart 301 is configured to be executed, for example, by a telecommunications company that wants to compensate a customer who has experienced multiple dropped calls. Specifically, if a customer has experienced two dropped calls in the last two hours (as specified by component 308), or 4 dropped calls in a full day (as specified by component 318), the telecommunications company will add 5 minutes to the customer's account (as specified by components 310, 314). The company will only pay out once every 24 hours (as specified by components 306 and 312). Chart 301 operates without an expiration date. In this example, the system described herein executes chart 301.

[0193] refer to Figure 7B, diagram 303 shows a state of diagram 301, in which diagram 301 has begun execution. When diagram 301 begins, diagram 301 is in a first state where component 304 is being executed (as shown by the black line surrounding component 304), which causes the system to generate audit record 324. Figure 7C , diagram 305 shows that diagram 301 advances to a second state, in which the system waits for a specified event and then makes various decisions and performs various actions when an event is detected. In this example state, the system executes component 306 (as specified by the black line around component 306), and the system generates an audit record 326. At component 306, the system is configured to wait for a specified event, for example, an event that specifies the last call dropped (i.e., the event type is "last call dropped"). In this example, the system receives event 302 (e.g., a structured data item or data record), which includes data that specifies the event type of this event is "last call dropped". When event 302 is detected, the system generates an audit record 322, and component 306 (or a system that executes executable logic represented by component 306) determines whether an expense has occurred within the last 24 hours (i.e., current time-payment time of the last dropped call>=24 hours). To this end, the system generates audit records 328, 330, each of which specifies the value of the variable read. When event 302 is detected and no payment has occurred within the last 24 hours, the system generates an audit record 332 and proceeds to component 308 while remaining in the second state.

[0194] refer to Fig.7D , diagram 307 illustrates a state of diagram 301 in which the system executes component 308, as indicated by the black line surrounding component 308. In this example, component 308 (or a system executing executable logic represented by component 308) generates audit records 334, 336 when reading the value of a variable. In this example, the logic represented by component 308 evaluates to a value of "no" based on the value of the variable read.

[0195] refer to Fig. 7E , diagram 309 illustrates another state of diagram 301, in which component 318 is executed (as illustrated by the black line around component 318), during which it remains in the second state. At component 318, the system generates an audit record 362, which specifies the read value of the "calls dropped per hour" variable. In this example, the system determines that the logic represented by component 318 evaluates to a value of "yes", and generates an audit record 364, the contents of which specify that component 318 evaluates to a value of "yes" (referred to as "Decision 2").

[0196] refer to Figure 7F, diagram 311 shows that diagram 301 has advanced to component 310, which generates audit record 340. Figure 7G , diagram 313 shows that diagram 301 then proceeds to component 312 , which generates an audit record 342 .

[0197] refer to Figure 7H , diagram 315 shows that diagram 301 then proceeds to component 314, which generates audit record 344. Diagram 301 then proceeds to component 316. In this example, the system batches together audit records 322, 324, 326, 328, 330, 332, 334, 336, 338, 340, 342, 344, 362, 364, identifies key fields (e.g., fields for event type, object ID, timestamp, instance ID) to be stored in the ICFF record, and then groups the remaining audit records into sets. In this example, the system generates ICFF record 348 that includes, for example, the key fields from audit records 322, 324. Included in data block 350 are the contents of audit records 326, 328, 330, 332, 334, 336, 338, 340, 342, 344, 362, 364 and the contents of audit records 322, 324 that are not included in ICFF record 348. The system stores ICFF record 348 and data block 350 in database 346. In this example, database 346 stores index 352, and ICFF record 348 is organized in index 352 according to the following fields: timestamp, object_ID, and event type, as shown in row 354 of index 352. In this example, index 352 also includes rows 356, 358, 360 that index other ICFF records for other charts and / or other object IDs, for example. In this example, ICFF record 348 references data block 350 via reference data structure 366 (e.g., pointer). In a variation, ICFF record 348 and data block 350 are combined into an indexed ICFF data block (or data blocks).

[0198] The above techniques can be implemented using software to be executed on a computer. For example, the software forms a program in one or more computer programs that is executed on one or more programming or programmable computer systems (which can be various architectures such as distributed, client / server or grid), each of which includes at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. The software can form one or more modules of a larger program, for example, to provide other services related to the design and configuration of diagrams and flow charts. The nodes, links, and elements of the diagram can be implemented as data structures stored in a computer-readable medium or other organized data that conforms to a data model stored in a data repository.

[0199] The techniques described herein may be implemented in digital electronic circuit systems, or in computer hardware, firmware, software, or a combination thereof. The apparatus may be implemented in a computer program product tangibly presented or stored in a machine-readable storage device (e.g., a non-transitory machine-readable storage device, a machine-readable hardware storage device, etc.) for execution by a programmable processor; and the method actions may be performed by a programmable processor executing an instruction program to perform functions by operating on input data and generating outputs. The embodiments described herein and other embodiments of the claims and the techniques described herein may be advantageously implemented in one or more computer programs executable on a programmable system, the programmable system comprising at least one programmable processor coupled to receive data and instructions from and transmit data and instructions to a data storage system, at least one input device, and at least one output device. Each computer program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language if desired; and in any case, the language may be an assembly or interpreted language.

[0200] Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. Essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to receive data from or transfer data to a mass storage device, or both receive and transfer data. Computer-readable media for embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit system. Any of the above may be supplemented by or incorporated into an ASIC (Application Specific Integrated Circuit).

[0201] To provide interaction with a user, embodiments may be implemented on a computer having a display device (e.g., an LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input.

[0202] Embodiments may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the embodiments), or any combination of these back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0203] The system and method or part thereof can use a collection of servers on the Internet that utilizes the Hypertext Transfer Protocol (HTTP) - the "World Wide Web" (Web or WWW). HTTP is a known application protocol that provides users with access to resources, which can be information in different formats, such as text, graphics, images, sounds, videos, Hypertext Markup Language (HTML), and programs. After the user specifies a link, the client computer sends a TCP / IP request to the Web server and receives information, which can be another Web page formatted according to HTML. The user can also access other pages on the same or other servers by following instructions on the screen, entering specific data, or clicking on a selected icon. It should also be noted that any type of selection device known to those skilled in the art, such as check boxes, drop-down boxes, etc., can be used for embodiments that use web pages to allow users to select options for given components. The server runs on a variety of platforms, including UNIX machines, but other platforms such as Windows 2000 / 2003, Windows NT, Sun, Linux, and Macintosh can also be used. Computer users can view information available on a server or network on the Web by using browsing software (such as Firefox, Netscape Navigator, Microsoft Internet Explorer, or Mosaic browser). The computing system can include a client and a server. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0204] Other embodiments are within the scope and spirit of the specification and claims. For example, due to the nature of software, the functions described above may be implemented using software, hardware, firmware, hardwiring, or any combination of these. Features that implement the functions may also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. The term "one" is not used in a limiting manner herein and throughout the application, and thus is not meant to exclude multiple meanings or "one or more" meanings of the term "one". In addition, to the extent that priority to a provisional patent application is claimed, it should be understood that the provisional patent application is not restrictive, but includes examples of how the technology described herein may be implemented.

[0205] Several embodiments of the present invention have been described. However, it will be appreciated by those skilled in the art that various modifications may be made without departing from the spirit and scope of the claims and the technology described herein.

Claims

1. A method implemented by a data processing system for identifying one or more portions of executable logic that are executed when processing one or more data items each associated with a given value of a key, wherein: A specification represents the executable logic, the specification is associated with the key, and a state of the specification is maintained for a corresponding value of the key, the method comprising: accessing the specification representing the executable logic, wherein a state of the specification for a given value of the key specifies one or more portions of the executable logic that are executable at that state; intermittently receiving data items of a data stream through an input device or port, wherein one or more of the data items of the data stream are each associated with a given value of the key; and generating a log record, wherein the log record maintains a state of the specification for a given value of the key, the log record specifying one or more portions of the executable logic executed when processing the one or more of the data items of the data stream that are each associated with the given value of the key, the generating comprising: For each of the one or more data items in the data stream associated with a given value for the key: identifying a given state of the specification maintained for a given value of the key associated with the data item of the one or more data items of the data stream; processing, by a data processing system, the data item of the one or more data items of the data stream, including executing, on the data item of the one or more data items of the data stream, one or more portions of executable logic specified at the identified given state of the specification; and generating a log record that maintains a state of the specification for a given value of the key by specifying execution of the one or more portions of the executable logic for the given value of the key; wherein the log record maintains the state of the specification for each value of the key, and Based on a given value for the key and further based on the one or more executions of the executable logic, one or more log records identify one or more portions of the executable logic that have been executed.

2. The method according to claim 1, wherein: Each log record includes one or more fields indicating one or more attributes of a processed data item, and for at least one log record, a field is a key field specifying a value of a key associated with the data item of the data flow.

3. The method according to claim 1, wherein: The specification includes a graph having a plurality of nodes, each node representing one or more portions of the executable logic, the graph being executable, execution of the graph resulting in execution of the portions of the executable logic, and the method further comprising: Based on the log records, traversal of one or more nodes of the graph is traced back while processing one or more data items.

4. The method according to claim 3, further comprising: The log records are parsed to identify which nodes in the graph are traversed.

5. The method according to claim 3, further comprising: receiving a user-initiated selection of one of the nodes; in response to receiving the selection, causing a graphical user interface to display a configuration area to provide the user with configuration tools for configuring the one or more portions of the executable logic represented by the selected node, wherein the configuration tools include tools for turning auditing on and off; as well as A user-initiated opening of the audit is received through the configuration tool, so that the log record associated with the selected node is generated.

6. The method according to claim 1, further comprising: Actions to be performed by the computing system are initiated in response to executing the one or more portions of the executable logic.

7. The method according to claim 1, further comprising: The states are maintained by storing state data in a data repository or in an in-memory data grid indicating which portion of the executable logic is currently being executed for each state.

8. The method according to claim 7, further comprising: In response to completing the processing of one or more of the data items of the data stream that are each associated with a given value for the key, state data specifying a current state of the value for the key is updated.

9. One or more machine-readable hardware storage devices for identifying one or more portions of executable logic that are executed when processing one or more data items each associated with a given value for a key, wherein The specification represents the executable logic, the specification is associated with the key, and the state of the specification is maintained for the corresponding value of the key, and the one or more machine-readable hardware storage devices store instructions that can be executed by one or more processing devices to perform operations including the following: accessing the specification representing the executable logic, wherein a state of the specification for a given value of the key specifies one or more portions of the executable logic that are executable at that state; intermittently receiving data items of a data stream through an input device or port, wherein one or more of the data items of the data stream are each associated with a given value of the key; and generating a log record, wherein the log record maintains a state of the specification for a given value of the key, the log record specifying one or more portions of the executable logic executed when processing the one or more of the data items of the data stream that are each associated with the given value of the key, the generating comprising: For each of the one or more data items in the data stream associated with a given value for the key: identifying a given state of the specification maintained for a given value of the key associated with the data item of the one or more data items of the data stream; processing, by a data processing system, the data item of the one or more data items of the data stream, including executing, on the data item of the one or more data items of the data stream, one or more portions of executable logic specified at the identified given state of the specification; and generating a log record that maintains a state of the specification for a given value of the key by specifying execution of the one or more portions of the executable logic for the given value of the key; wherein the log record maintains the state of the specification for each value of the key, and Based on a given value for the key and further based on the one or more executions of the executable logic, one or more log records identify one or more portions of the executable logic that have been executed.

10. The machine-readable hardware storage device of claim 9, wherein: Each log record includes one or more fields indicating one or more attributes of a processed data item, and for at least one log record, a field is a key field specifying a value of a key associated with the data item of the data flow.

11. The machine-readable hardware storage device of claim 9, wherein: The specification includes a graph having a plurality of nodes, each node representing one or more portions of the executable logic, the graph being executable, execution of the graph resulting in execution of the portions of the executable logic, and the operations further comprising: Based on the log records, traversal of one or more nodes of the graph is traced back while processing one or more data items.

12. The machine-readable hardware storage device of claim 11, wherein: The operations also include: The log records are parsed to identify which nodes in the graph are traversed.

13. The machine-readable hardware storage device of claim 11, wherein: The operations also include: receiving a user-initiated selection of one of the nodes; In response to receiving the selection, causing a graphical user interface to display a configuration area to provide the user with configuration tools for configuring the one or more portions of the executable logic represented by the selected node, wherein the configuration tools include tools for turning auditing on and off; and A user-initiated opening of the audit is received through the configuration tool, so that the log record associated with the selected node is generated.

14. The machine-readable hardware storage device of claim 9, wherein: The operations also include: Actions to be performed by the computing system are initiated in response to executing the one or more portions of the executable logic.

15. The machine-readable hardware storage device of claim 9, wherein: The operations also include: The states are maintained by storing state data in a data repository or in an in-memory data grid indicating which portion of the executable logic is currently being executed for each state.

16. The machine-readable hardware storage device of claim 15, wherein: The operations also include: In response to completing the processing of one or more of the data items of the data stream that are each associated with a given value for the key, state data specifying a current state of the value for the key is updated.

17. A computing system for identifying one or more portions of executable logic that are executed when processing one or more data items each associated with a given value of a key, wherein: A specification represents the executable logic, the specification is associated with the key, and a state of the specification is maintained for a corresponding value of the key, the computing system comprising: one or more processing devices; and One or more machine-readable hardware storage devices storing instructions executable by the one or more processing devices to perform operations including: accessing the specification representing the executable logic, wherein a state of the specification for a given value of the key specifies one or more portions of the executable logic that are executable at that state; intermittently receiving data items of a data stream through an input device or port, wherein one or more of the data items of the data stream are each associated with a given value of the key; and generating a log record, wherein the log record maintains a state of the specification for a given value of the key, the log record specifying one or more portions of the executable logic executed when processing the one or more of the data items of the data stream that are each associated with the given value of the key, the generating comprising: For each of the one or more data items in the data stream associated with a given value for the key: identifying a given state of the specification maintained for a given value of the key associated with the data item of the one or more data items of the data stream; processing, by a data processing system, the data item of the one or more data items of the data stream, including executing, on the data item of the one or more data items of the data stream, one or more portions of executable logic specified at the identified given state of the specification; and generating a log record that maintains a state of the specification for a given value of the key by specifying execution of the one or more portions of the executable logic for the given value of the key; wherein the log record maintains the state of the specification for each value of the key, and Based on a given value for the key and further based on the one or more executions of the executable logic, one or more log records identify one or more portions of the executable logic that have been executed.

18. The computing system of claim 17, wherein: Each log record includes one or more fields indicating one or more attributes of a processed data item, and for at least one log record, a field is a key field specifying a value of a key associated with the data item of the data flow.

19. The computing system of claim 17, wherein: The specification includes a graph having a plurality of nodes, each node representing one or more portions of the executable logic, the graph being executable, execution of the graph resulting in execution of the portions of the executable logic, and the operations further comprising: Based on the log records, traversal of one or more nodes of the graph is traced back while processing one or more data items.

20. The computing system of claim 19, wherein: The operations also include: The log records are parsed to identify which nodes in the graph are traversed.

21. The computing system of claim 19, wherein: The operations also include: receiving a user-initiated selection of one of the nodes; In response to receiving the selection, causing a graphical user interface to display a configuration area to provide the user with configuration tools for configuring the one or more portions of the executable logic represented by the selected node, wherein the configuration tools include tools for turning auditing on and off; and A user-initiated opening of the audit is received through the configuration tool, so that the log record associated with the selected node is generated.

22. The computing system of claim 17, wherein: The operations also include: The states are maintained by storing state data in a data repository or in an in-memory data grid indicating which portion of the executable logic is currently being executed for each state.

23. The computing system of claim 17, wherein: The operations also include: In response to completing the processing of one or more data items of the data stream that are each associated with a given value for the key, state data specifying a current state of the value for the key is updated.

24. A computer program product comprising a computer program which, when run by a processor, performs the method according to any one of claims 1 to 8.

Images