Dynamic execution of parameterized applications for processing keyed network data streams

Abstract

A method is described for processing keyed data items, each associated with a value of a key, the keyed data items coming from a plurality of different data streams, the processing comprising collecting the keyed data items, determining the satisfaction of one or more specified conditions for performing one or more actions based on the content of at least one of the keyed data items, and causing at least one of the one or more actions to be performed in response to the determination.

Description

Technical Field

[0001] The present application relates to a network-ready computer-implemented method, a computer system and a computer-readable medium for acquiring, validating, formatting and further processing (near) real-time network data streams in parameterized applications to perform certain actions, such as actions performed across a network such as a telecommunications network.

[0002] Priority claim

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 462,498, filed on February 23, 2017, and U.S. Patent Application No. 15 / 688,587, filed on August 28, 2017, the entire contents of each of which are incorporated herein by reference. Background Art

[0004] In a general method for processing data records received from different sources, multiple different systems collect data records, each system being configured to process a specific type of data. Once a specific type of data is processed by a system configured to process that type of data, the system then passes the data downstream to another application for further processing. In a conventional system, this downstream application then performs additional formatting (and / or reformatting) on ​​the received data record to convert it into an acceptable format for the application. Therefore, this conventional method results in delays and delayed interactions between the system that receives and collects data and the downstream application that processes the data. Since a single data record must be reformatted multiple times, this conventional method also results in increased delays. For example, a data record is formatted (and / or reformatted) by each system that receives it and processes it, resulting in an accumulation of increased delays. This non-integrated system framework (of different systems that each perform their own data formatting and processing) also results in delays when processing data because the data cannot be processed or analyzed in real time or at least in near real time. Therefore, if the current situation has changed again, operations or actions that rely on processed data will also be increasingly delayed, or ultimately not performed at all, thereby making the execution obsolete. Summary of the invention

[0005] In contrast to these common methods, the methods, systems, and computer-readable media described herein perform preparations for data integration and data record management in a simplified and accelerated manner. The described capabilities allow for the collection, verification, formatting, and further processing of batch data streams and real-time data streams (e.g., in real time or near real time when receiving data, e.g., without storing the collected data to disk). In addition, by having a single execution system that performs the operations of collection, detection, and action, the execution system eliminates the complexity involved in integrating data into one system for data collection and then reintegrating the collected data into another system to perform detection and action. The capabilities described herein can provide an immediate response to data records or data items (e.g., when they are received), which also provides instant visibility of application results. Since the desired final action usually depends on the rapid processing of data, the final action (such as actions in logistics or telecommunications) greatly benefits from the faster processing of large amounts of data from various different sources, thereby generating data in various different formats. The system described herein can process more than two billion data records or data items for 50 million users every day. In contrast to common methods, the system described herein provides increased bandwidth and reduced memory consumption.

[0006] A method for processing keyed data items each associated with a value of a key, the keyed data items coming from a plurality of different data streams, the processing comprising: collecting the keyed data items, determining the satisfaction of one or more specified conditions for performing one or more actions based on the content of at least one of the keyed data items, and causing at least one of the one or more actions to be performed in response to the determination, the method comprising: accessing a first parameterized application, a second parameterized application, and a third parameterized application comprising corresponding first, second, and third specifications; wherein the first specification specifies one or more parameters defining one or more properties of the first parameterized application and one or more corresponding values ​​of the one or more parameters; wherein the second specification specifies one or more parameters defining one or more properties of the second parameterized application and one or more corresponding values ​​of the one or more parameters, wherein the second specification comprises rules and corresponding conditions for the rules; wherein the third specification specifies one or more parameters defining one or more properties of the third parameterized application and one or more corresponding values ​​of the one or more parameters; maintaining a state of the second specification for the corresponding value of the key, wherein the state for a particular value of the key specifies that one or more portions of the second specification are to be performed by the second parameterized application in the state; using the one or more parameters specified by the first specification The first parameterized application is configured to execute the first parameterized application with one or more values ​​to perform processing including the following operations: collecting data items from one or more data sources and one or more data streams, multiple data sources, or multiple data streams, wherein the format of the collected first data items is different from the format of the collected second data items, and wherein the data items are associated with values ​​of a key; transforming the first data items and the second data items according to a first specification of the first parameterized application to obtain transformed data items; and populating a queue with the transformed data items; executing the second parameterized application using one or more values ​​of one or more parameters specified by the second specification to process the transformed data items in the queue, wherein processing the transformed data The items include: for one or more of the transformed data items associated with the specific value of the key, identifying a current state of the second specification with respect to the specific value of the key; identifying one or more rules in a portion of the second specification to be executed in the current state; executing the identified one or more rules; determining that at least one of the one or more transformed data items satisfies one or more conditions of at least one of the one or more rules executed in the current state; in response to the determination, generating a data structure specifying the execution of one or more actions; causing the second specification with respect to the specific value of the key to transition from its current state to a subsequent state; and transmitting the generated data structure to the third parameterized application;and executing the third parameterized application using one or more values ​​of one or more parameters specified by the third specification to perform operations, the operations comprising: based on at least one of the one or more actions specified in the data structure, sending one or more instructions to cause at least one of the one or more actions to be performed. A system of one or more computers may be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system, which in operation causes or causes the system to perform actions. One or more computer programs may be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform an action. In certain aspects, the aforementioned method, computer program, and / or system include one or more of the following features and / or actions. ;

[0007] The actions further include, during operation of one or more user interfaces, displaying one or more user interface elements for specifying one or more values ​​of the one or more parameters for each of the first parameterized application, the second parameterized application, and the third parameterized application. Executing the first parameterized application includes executing the first parameterized application using one or more values ​​specified by the one or more user interface elements for the one or more parameters of the first parameterized application. Executing the second parameterized application includes executing the second parameterized application using one or more values ​​specified by the one or more user interface elements for the one or more parameters of the second parameterized application, wherein the one or more specified values ​​are used as inputs by the rules when determining whether the one or more conditions are satisfied. Executing the third parameterized application includes executing the third parameterized application using one or more values ​​specified by the one or more user interface elements for the one or more parameters of the third parameterized application. The data item includes a data record, and wherein the transformation includes reformatting the data record according to a format specified by a first specification of the first parameterized application.

[0008] The actions further include enriching the data record with data from a profile of a user associated with the data record based on execution of the second parameterized application, wherein the enrichment is performed according to instructions specified by a second specification of the second parameterized application to retrieve profile data of the user from a memory and populate one or more fields of the data record with the retrieved profile data. The parameterized application includes an application for data processing, wherein the application includes one or more parameters that can be configured with one or more values. The actions further include executing a feedback loop (e.g., a synchronous or asynchronous feedback loop) to one or more third-party systems to request confirmation of the execution of the one or more actions. The actions further include generating one or more key performance indicators (KPIs) for a specific value of the key based on execution of the second parameterized application, wherein the KPI specifies one or more values ​​of a data item associated with the specific value of the key. For example, the KPI may include performance data, such as the performance data specifying the performance of one or more parts or logical branches of an activity or a set of predefined logic.

[0009] The actions further include receiving data for a specific value of the key, wherein the received data indicates feedback about at least one of the one or more actions; and updating the KPI for the specific value of the key using the feedback data. The one or more actions include one or more of the following: sending a text message to an external device, sending an email to an external system, opening a ticket for a work order in a case management system, disconnecting a mobile phone connection, providing a web service to a targeted device, transmitting a data packet of the one or more transformed data items using a notification, and executing a data processing application hosted on one or more external computers on the one or more transformed data items. The one or more instructions are sent via a network connection so that at least one of the one or more actions is performed on an external device, and the method further includes: receiving a feedback message indicating whether at least one of the one or more actions is (i) successfully completed or (ii) failed. If a portion of at least one of the one or more actions is not completed, then at least one of the one or more actions is considered to have failed, wherein the feedback message indicates which portion of at least one of the one or more failed actions is not completed. The feedback message indicates result data of successful completion and / or failure of at least one of the one or more actions.

[0010] The actions further include changing one or more specified values ​​of one or more parameters of the first parameterized application, the second parameterized application, and / or the third parameterized application based on the result data; and re-executing the first parameterized application, the second parameterized application, and / or the third parameterized application using the changed one or more specified values. The one or more instructions are sent via a network connection so that the one or more actions are performed on the external device, and the method further includes: receiving a feedback message from the external device, the feedback message including result data of at least one of the one or more actions performed; comparing the result data with predetermined data associated with successfully completing the execution of at least one of the one or more actions; and determining, based on the comparison, that the execution of at least one of the one or more actions was successfully completed, or that the execution of at least one of the one or more actions failed.

[0011] The actions further include changing one or more specified values ​​of one or more parameters of the first parameterized application, the second parameterized application, and / or the third parameterized application based on the result data; and re-executing the first parameterized application, the second parameterized application, and / or the third parameterized application using the changed one or more specified values. If the result data deviates from the predetermined data by less than a predetermined amount, the execution of at least one of the one or more actions is determined to be successfully completed, and wherein if the result data deviates from the predetermined data by at least a predetermined amount, the execution of at least one of the one or more actions is determined to be a failure.

[0012] The actions further include displaying one or more user interface elements for specifying the predetermined data and the predetermined amount during operation of the one or more user interfaces. The actions further include outputting, via one or more displayed user interface elements, whether at least one of the one or more actions is (i) successfully completed or (ii) failed during operation of the one or more user interfaces. The method of the present application further includes: outputting the result data via one or more displayed user interface elements during operation of the one or more user interfaces. The actions further include receiving, via one or more displayed user interface elements, a user-specified change of one or more specified values ​​of one or more parameters of the first parameterized application, the second parameterized application, and / or the third parameterized application based on the result data; and re-executing the first parameterized application, the second parameterized application, and / or the third parameterized application using the changed one or more specified values. The third parameterized application automatically executes the sending of the one or more instructions to cause the execution of at least one of the one or more actions by using the output specifying the execution of the one or more actions as input.

[0013] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a diagram of a system implementing parameterized logic for processing keyed data.

[0015] Figure 2A is an illustration of an exemplary data flow graph.

[0016] Figure 2B and Figure 2C is an illustration of a portion of an interface for customizing a dataflow graph.

[0017] Figure 3A is a diagram of a system for computing near real-time data record aggregates.

[0018] Figure 3B , Figure 3C , Figure 4 , Figure 5A , Figure 5B Each is a diagram of a system for processing keyed data.

[0019] Figure 6 , Figure 7 , Figure 8 , Fig. 9 Each is an example of a graphical user interface for configuring parameterized logic.

[0020] Fig.10 is a diagram of a flow chart.

[0021] Fig.11 is a diagram of a flowchart instance.

[0022] Fig.12 is an illustration of near real-time logic execution with wide records.

[0023] Fig.13 is a diagram of an example process that uses parameterized logic to process keyed data. DETAILED DESCRIPTION

[0024] refer to Figure 1, shows a system 100 for collecting data and data records from various sources, such as from different servers located in different locations and interconnected via a network, and for integrating the data with modules for performing data detection and performing actions. Typically, data items include data records, data indicating data records, or events (e.g., records including data indicating that an action has occurred (e.g., a voice call was made or the length of a voice call) or data indicating that an action has occurred). Although the techniques described herein are primarily described with respect to data records, these techniques can also be used to process events. System 100 includes a code management storage device 102, a development environment 104, a data source 106, and an execution system (or runtime environment) 108. Execution system 108 includes a system for implementing a collection-detection-action (CDA) environment, which is used to configure and execute various applications and programs for performing the collection, detection, and actions described above. These various applications and programs include, for example, reusable collections of maps, plans, applications, etc. (e.g., for accelerating development and simplifying maintenance of applications and programs). Typically, a graph (e.g., a data flow graph) includes vertices or nodes (components or data sets) connected by directed links between vertices (representing flows of work elements), for example, as described in U.S. Publication No. 2007 / 0011668, entitled "Managing Parameters for Graph-Based Applications," which is incorporated herein by reference. Typically, a plan includes an application that represents a process flow in which processing steps, called tasks, are connected by flows (e.g., dependencies) that indicate an order of execution.

[0025] Each method described herein can be performed by a system 100, which includes: a development environment 104 coupled to a code management storage device 102. The development environment 104 is configured to build any of the applications described herein associated with a data flow graph that implements a graph-based calculation performed on data that flows from one or more input data sets through a graph of processing graph components to one or more output data sets. The data flow graph is specified by a data structure in the code management storage device 102. The data flow graph has a plurality of vertices or nodes specified by the data structure and representing graph components connected by one or more links. These links are specified by the data structure and represent the data flow between the graph components. In addition, a runtime environment 108 is coupled to the storage device 102 and hosted on one or more computers. The runtime environment 108 also reads a stored data structure that specifies the data flow graph and is used to allocate and configure computing resources such as processes, which are used to perform calculations on the graph components allocated to the data flow graph by the runtime environment 108. The runtime environment 108 also includes an execution module for scheduling and controlling the execution of the allocated processes so that the operations according to the method are performed.

[0026] The execution system 108 includes a collection / integration module 114 (hereinafter referred to as the collection module 114), which is used to collect data records (and other data), transform them and distribute the transformed data to downstream applications, including, for example, a detection module 116 and an action module 118. In this example, the action module 118 includes an interface to a third-party system and / or an external system. Specifically, the collection module 114 collects data from various sources (such as, for example, data sources 106) or from different servers located in different locations and interconnected via a network in batch or real-time, and collects data from real-time data streams 107, for example, real-time data from different servers located in different locations and interconnected via a network. The storage device providing the data source 106 can be local to the system 100, for example, stored on a storage medium (e.g., a hard drive) connected to the computer running the execution system 108, or can be remote from the execution system 108, for example, hosted on a remote system that communicates with the execution system 108 via a local area data network or a wide area data network.

[0027] The acquisition module 114 is configured to parse, validate, and enrich data in batch data 109 and / or real-time data streams 107 received from the data source 106. For the purpose of convenience, without limitation, the terms "real-time" and "near real-time" may be collectively referred to as "real-time" or "near real-time" herein. The acquisition module 114 stores the enriched data records in memory and also writes the enriched data records to disk for archiving and to ensure recovery of the records. Because the acquisition module 114 is able to process any source, the acquisition module 114 enables fast and independent integration of data (e.g., additional applications are required for the acquisition module 114 to successfully perform the integration) into the CDA environment, while handling the complexity of arbitrarily large data volumes, low latency, and multiple data formats.

[0028] In the example, records are pushed from a remote system to the system 108. In this example, thousands of records arrive every day. In this example, the acquisition module 114 detects the records in the local directory pushed into the system 108. The acquisition module 114 also performs record delivery by queuing the records and copying the data to an archive (not shown). The acquisition module 114 also deletes the original record after parsing. In this example, the archive stores an unaltered and optionally compressed copy of the input data. The archive stores the data for a specified number of days. In this example, the acquisition module 114 publishes or transmits the acquired data to a data queue, with one data queue for each data feed. The acquisition module 114 also parses the records by reading the records, removing duplicates, enriching the records (e.g., by adding profile data to the records), partitioning the records by keys (e.g., for further enrichment), and writing the records to the queue. In this example, the acquisition module 114 removes duplicates by storing an ICFF (indexed compressed flat file) archive, which keeps a copy of the hash for each record for a specified number of days for removing duplicates. In this example, the acquisition module 114 also performs maintenance by executing a scheduled (eg, daily) chart of purging (eg, deleting) old files from the archive and the ICFF archive.

[0029] In this example, the queues (to which records are posted) include a single centralized queue partitioned by key, having a standardized format including a data record type (indicating a source data feed name), standard fields (such as a key), and feed-specific fields. The queues include parallel queues that can be processed in parallel within a server or across servers and / or can be processed in parallel across one or more systems.

[0030] In this example, the acquisition module 114 spots the data feed after a specified number of received records or files (e.g., a set of records) or after a specified number of seconds. The acquisition module 114 performs this spot check by executing a graph with configurable parameters that push data through the system 108 with near real-time latency as input files arrive.

[0031] The execution system 108 also includes a detection module 116 for generating logic for detecting predefined occurrences. After the acquisition module 114 acquires and integrates the acquired data, the execution system 108 executes the detection module 116 on the acquired and integrated data. In an example, the acquisition module 114 sends one or more (e.g., different) acquired data streams (e.g., acquired data streams that have been verified and formatted / reformatted) to the detection module 116 for further processing and / or execution of one or more predefined rules for the acquired data streams in real time as the data is received. As described in further detail below, the detection module 116 has unique flexibility, allowing users to generate simple and complex detection patterns (e.g., activities and / or a series of rules with various conditions that are met before the rules are executed) based on multiple and dynamic data record types, aggregation types, state definitions and transitions, complex functions and timers. The detection module 116 also implements detection of "synthetic" data (e.g., data records). Typically, synthetic data includes data indicating the absence of a certain condition or occurrence, for example, detecting that a user has not visited a portal within a specified amount of time. When detecting data records, the detection module 116 publishes instructions or messages to the queue, and the contents of these instructions or messages are received and processed by the action module 118.

[0032] The action module 118 performs the action that has been triggered, such as sending a text message or email, opening a ticket for a work order in a case management system, immediately disconnecting service, providing a Web service to a target system or device, transmitting packet data using one or more notifications, etc. In another example, the action module 118 generates instructions and / or content for a message and sends these instructions (and / or content) to a third-party system, which in turn performs an action based on the instructions, such as transmitting a text message, providing a Web service, etc. In an example, the action module 118 is configured to generate customized content for each recipient. In this example, the action module 118 is configured with rules or instructions that specify which customized content is sent or transmitted to which recipients.

[0033] In conventional models for data integration and data record detection, one system performs data acquisition and batches data and stores the data in a data warehouse. Then, to perform data record detection, a different system retrieves and queries the batch data from the data warehouse and performs data record detection on the stored data. This conventional model has many limitations, including, for example, that it cannot support real-time data acquisition. In addition, different and inconsistent applications and / or engines are built to perform different types of data record detection (e.g., for different segments). In contrast to these conventional models, the CDA environment described herein performs data integration and preparation for data record management without the need for additional technology (e.g., no separate system for data integration and then another separate system for data record management). This capability simplifies and accelerates end-to-end integration. In addition, the CDA environment described herein is configured to process batch and real-time data streams (e.g., real-time or near real-time when the data is received). For example, the CDA environment is capable of processing real-time data by processing, validating, and / or formatting data in near real-time (e.g., when the execution system 108 receives the data) by the acquisition module 114, for example, without the need to store the acquired data to disk. The acquisition module 114 is configured to, for example, validate and process the data stream as it is received, and then publish the processed data to a queue for further processing by the detection module 116 - all without storing the data in a data warehouse for subsequent retrieval, which introduces latency. In addition, by having a single system that performs the operations of acquisition, detection, and action, the execution system 108 eliminates the complexity involved in integrating data into one system for data acquisition and then reintegrating the acquired data into another system for performing detection and action. Since the desired final operation or action often depends on the processed data and will also be increasingly delayed (or even ultimately will not be performed if the current situation has changed again, making the execution obsolete), final operations or actions such as those in logistics or telecommunications greatly benefit from such rapid processing of large amounts of data in a variety of different formats.

[0034] In this example, the code management storage device 102 is configured to communicate with the execution system 108 and store a parameterized acquisition application 120, a parameterized detection application 122, and a parameterized action application 124. Typically, a parameterized application includes an electronic template or record that is preconfigured to perform specified functions and operations with respect to one or more parameters, the values ​​of which are received by the parameterized application from one or more other applications and / or from user input. Once parameter values ​​are specified or specified, the parameterized application (e.g., a parameterized template) represents a parameterized template specification (hereinafter referred to as a "specification"), for example, because the parameterized application specifies state values, action values, values ​​for transitions between states, etc. Typically, the specification represents executable logic and specifies values ​​for parameters of the executable logic and various states of the executable logic based on the state reached from executing the executable logic on a previous data item. In a variant, the system includes an unparameterized action application.

[0035] In an example, for example, when an electronic application or record is pre-configured with logic expressions (or other logic) for performing specified functions and operations with respect to one or more parameters, the parameterized application is referred to as parameterized logic.

[0036] In an example, a parameterized application includes a generic data flow graph (or other generic data processing program or application) having various parameters whose values ​​are specified as inputs to the parameterized application. In this example, a parameterized acquisition application includes a parameterized application for performing data acquisition and integration. A parameterized detection application includes a parameterized application for performing data detection. A parameterized action application includes a parameterized application for performing an action.

[0037] In the example, each of the modules 114, 116, 118 is implemented by executing an instance of a parameterized application 120, 122, 124, respectively. In this example, the instance of the parameterized application includes executing a parameterized application in which the values ​​of the parameters are specified. The execution of the acquisition module 114 is based on one or more parameterized acquisition applications 120, which specify how to collect, format, verify and integrate data. By parameterizing the incoming data streams, there is no need to write a new program to process each incoming data stream. Instead, the execution system 108 maintains a program or application for processing the incoming data. The program or application includes various parameters, and the values ​​of these parameters can be set to configure the program or application to process a specific incoming data stream. In some examples, the program or application is a general data flow graph, such as a parameterized application. Using the user interface module 126, the user customizes the general data flow graph of a specific incoming data stream by specifying the values ​​of the parameters of the general data flow graph. These specified values ​​specify how the acquisition module 114 processes the specific data stream. Compared to conventional techniques of having a dedicated program handle each incoming data flow, parameterization of the generic data flow graph (or application) enables reuse, which reduces the memory requirements of the execution system 108 and also reduces data flow graph errors in data collection and integration because a single generic (and error-checked) data flow graph is used and reused.

[0038] In this example, the parameterized acquisition application facilitates reuse of a given acquisition application because the parameters of the parameterized acquisition application provide for easy modification of values ​​in the parameterized acquisition application, e.g., rather than having a (non-parameterized) acquisition application where the code itself must be modified each time an associated change in the value of the acquisition application occurs. In this example, the execution system 108 compiles the parameterized acquisition application 120 into executable code. The acquisition module 114 is implemented by executing this executable code.

[0039] The execution detection module 116 is based on one or more parameterized detection applications 122, which are specified to detect one or more predefined rules for specifying the occurrence or lacking these specified occurrences. In this example, the parameterized detection application 122 includes one or more parameters for specifying one or more values ​​(e.g., the value used by the rule when performing the detection). In this example, the parameterized detection application promotes the reuse of the specified detection application, because the parameters of the parameterized detection application provide a simple modification of the value in the parameterized acquisition application, for example, instead of having a (non-parameterized) detection application, whenever the value of the acquisition application is associated with a change, the code itself must be changed. In addition, in general, parameterized applications and especially parameterized detection applications 122 provide flexible and instant detection implementation methods across (e.g., user's) various segments and across various types and data records of data records. For example, one of the parameterized detection applications 122 can be configured to detect the occurrence of data records of a specified type (e.g., by setting the value of the parameter in the parameterized detection application). The parameterized detection application can then be reused to detect the occurrence of another type of data record (e.g., by specifying other values ​​of the parameters of the parameterized detection application). This reuse of parameterized applications reduces errors when executing detection module 116, for example, because error-free applications can be reused. These parameterized applications also promote the flexibility of data record detection. Specifically, the deployment of conventional detection modules (e.g., detection engines) involves extensive modeling and configuration of various instructions and relationships (e.g., between the detection of various data record types and the rules of various actions to be taken when specifying successful detection) before performing detection. In contrast, detection module 116 can be started without such tasks and on the contrary in the case of instant configuration, such as by specifying (e.g., via user interface module 126) the value of the parameter of parameterized detection application 122 by the user. In this example, parameterized detection application 122 is compiled into executable code by execution system 108. Detection module 116 is implemented by executing this executable code.

[0040] The execution action module 118 is based on specifying one or more predefined actions to be performed or one or more parameterized action applications 124 of instructions to be transmitted or executed, for example, based on instructions or triggers received from the detection module 116. In this example, the parameterized action application 124 includes one or more parameters for specifying one or more values ​​(for example, the values ​​used by the rule when performing the action). In this example, the parameterized action application promotes the reuse of the specified action application, because the parameters of the parameterized action application provide a simple modification of the values ​​in the parameterized action application, for example, instead of having a (non-parameterized) action application, whenever the value of the action application is associated with a change, the code itself must be changed. In addition, the parameterized action application 124 provides a flexible and instant execution implementation method for the action of various types and data records across various segments (for example, users) and across data records. For example, one of the parameterized action applications 124 can be configured to perform a specified action (for example, sending a text message to warn the user) by setting the value of the parameter in the parameterized action application. The parameterized action application can then be reused to transmit instructions to a third party to perform an action, such as sending a specific message. The parameterized action application is reused by specifying or modifying values ​​for the parameters of the parameterized action application. In this example, the execution system 108 compiles the parameterized action application 124 into executable code. The action module 118 is implemented by executing this executable code.

[0041] The acquisition module 114, the detection module 116, the action module 118 represent an improvement over the prior art (e.g., having separate systems for performing acquisition, detection, and action) by achieving benefits over conventional techniques, such as increased flexibility, faster processing time of real-time data streams, and smaller memory requirements. As previously described, the execution system 108 includes an integrated system that includes the acquisition module 114, the detection module 116, and the action module 118. By integrating these modules into a single system, the execution system 108 is able to process data in real time (or near real time) because the acquisition module 114 processes the received data stream in memory and then publishes the verified data to a queue for further processing, for example, rather than submitting the received data to a data storage device for subsequent retrieval. In addition, integrating the acquisition module 114, the detection module 116, and the action module 118 into the system results in smaller memory requirements because the system does not need to store multiple different applications to perform acquisition, detection, and action for each data stream from which records are received. Instead, for example, when various values ​​of various parameters are specified, the corresponding parameterized applications can be reused.

[0042] The execution system 108 also changes the configuration of the parameterized application and also sets the values ​​of various parameters in the parameterized application, for example, based on user input of the value of the specified parameter or based on the executed action that causes such change or such setting. The user interface module 126 displays configuration information to the user and receives configuration actions from the user, for example, data specifying the parameter value and / or data specifying the action of the configuration of the parameterized application. In this example, each of the parameterized acquisition application 120, the parameterized detection application 122, and the parameterized action application 124 is stored in the code management storage device 102. The code management storage device 102 is also accessible to the development environment 104, in which the developer can develop a user interface stored in the code management storage device 102 and used by the user interface module 126 to display the user interface. The development environment 104 also enables the developer to develop parameterized applications, including, for example, the parameterized acquisition application 120, the parameterized detection application 122, and the parameterized action application 124. The executed actions generated by executing one or more applications allow the developer to determine whether the one or more applications operate correctly for one or more given parameter values. That is, one or more applications can be tested before the user uses them to prevent the user from facing one or more incorrectly operated applications.

[0043] refer to Figure 2A , the data flow graph 202 may include data sources 206a, 206b, components 208a to 208c, 210, and a data sink 212. In this example, the data flow graph 202 is an example of a parameterized application. As described in further detail, the data flow graph 202 includes various parameters whose values ​​are specified by user input. For example, when the data flow graph 202 is a parameterized acquisition application, each of the data sources 206a, 206b, components 208a to 208c, 210, and the data sink 212 specifies how to acquire, process, and integrate (e.g., integrate into the execution system 108) data in near real time. In another example where the data flow graph 202 is a parameterized detection application, each of the data sources 206a, 206b, components 208a to 208c, 210, and the data sink 212 specifies various detection operations and functions to be performed when detecting various data records. In yet another example where the dataflow graph 202 is a parameterized action application, each of the data sources 206a, 206b, components 208a-208c, 210, and data sink 212 specifies various action operations and functions that are performed in response to triggers and / or instructions received from a detection module.

[0044] In this example, each of the sources, components, and sinks can be associated with a set of parameters 204a to 204g. The user can use the user interface module 126 ( Figure 1) to input or otherwise specify the values ​​of these parameters. The parameters for one source, component or sink can be used to evaluate the parameters of different sources, components or sinks. Sources 206a, 206b are connected to the input ports of components 208a, 208c. The output port of component 208a is connected to the input port of component 208b. The output port of component 210 is connected to data sink 212. The connections between sources, components and sinks define the data flow.

[0045] Some of the data sources, components or sinks may have input parameters 204a to 204g that may define some behaviors of the graph. For example, a parameter may define the location of a data source or sink on a physical disk. Parameters may also define the behavior of a component, for example, a parameter may define how a sorting component sorts inputs. In another example, a parameter defines how a data record is formatted or validated. In some arrangements, the value of one parameter may depend on the value of another parameter. For example, a source 206a may be stored in a file in a particular directory. A parameter set 204a may include a parameter called "directory" and another parameter called "file name". In this case, the file name parameter will depend on the directory parameter. (For example, a directory may be " / usr / local / " and a file name may be " / usr / local / input.dat"). Parameters may also depend on parameters of other components. For example, the physical location of sink 212 may depend on the physical location of source 206a. In this example, sink 212 includes a set of parameters 204g, which includes a file name parameter, which depends on the directory parameter of source 206a. (For example, the file name parameter in set 204g may be " / usr / local / output.dat", where the value " / usr / local / " is obtained from the directory parameter in set 204a.)

[0046] Within the user interface on the client, the parameters of parameter sets 204a to 204g can be combined and reorganized into different groups for user interaction, which reflects business considerations rather than technical considerations. The user interface for receiving values ​​of parameters based on user input can display different parameters according to the relationship between the parameters in a flexible manner that is not necessarily limited by aspects of the development environment on the server.

[0047] refer to Figure 2B, a user interface can be presented in which icons are displayed with relationships representing dependencies between parameters. In this example, the parameters are divided into a first group of parameters, which are represented by a first source icon 224 representing the parameters of the first source data set, a second source icon 226 representing the parameters of the second source data set, a host icon 230 representing the parameters of the host data set, and a transformation icon 228 representing the parameters of one or more components of the data flow graph being configured, showing their relationship with the source data set and the host data set. This parameter grouping can be performed based on a stored user interface specification 222, which defines how a user will interact with the parameters of the data flow graph from the user interface on the client and how user interface elements such as icons 224, 226, 228, 230 will be related to each other and arranged for presentation in the user interface. In some embodiments, the user interface specification is an XML document. The user interface specification can also identify data flow graph components, and can identify specific components that can perform certain functions when the user configures the graph, such as viewing sample data, as described in more detail below.

[0048] In some cases, the user interface specification may include instructions for how to display the parameters. Figure 2B and Figure 2C , the user interface specification 222 can define the user interface 250 displayed to the user. Further, the user interface specification 222 can indicate that in response to an interaction with a source data set icon 224, one parameter should be displayed in the user interface 250 as a text box 252 that the user can fill in, while another parameter should be displayed in the user interface 250 as a drop-down list 254 with pre-populated values, and still another parameter can also be displayed in the user interface 250 as a radio button 256, etc. Thus, the user interface specification provides flexibility in how parameters are presented to the user so that the data flow diagram can be customized in a manner that can be customized for business users and / or non-technical users.

[0049] In some cases, the user interface specification may constrain the order in which the business user fills in the parameter values. The parameters associated with the sink 230, represented by the dashed line, may not be visible to the user until the user meets certain predefined conditions. For example, the user may have to provide a specific parameter value or fill in a set of parameters before the data sink parameter set appears.

[0050] In some implementations, the user interface specification may also include variables that define characteristics of user interface elements (in contrast to parameters that define characteristics of components of a data flow diagram). For example, a variable may be used to control the order in which a business user uses user interface elements. A variable references at least one data value. In some examples, a variable references multiple data values, and each data value is defined as a property of the variable. Thus, a single variable may have multiple properties, each property being associated with a data value.

[0051] The user interface 250 defined by the user interface specification may be presented in a manner in which user interface elements (e.g., text boxes 252, drop-down lists 254, radio buttons 256) do not directly correspond to parameters for customizing a data flow diagram. Instead, some user interface elements may correspond to configuration options relevant to a user, such as a business user and / or a non-technical user who may not be aware of the parameters.

[0052] In these examples, the user interface 250 need not be associated with the specific component 224 of the dataflow graph. Further, the user interface 250 may be associated with multiple dataflow graphs and other data processing and data storage constructs.

[0053] For example, a user interface element may allow a user to change a configuration option that has business implications rather than technical implications. A configuration option may be an option for converting between currency types used in business transactions, or an option for updating information associated with inventory of a particular category of products, or another option that is not related to the configuration of a single parameter. The user interface specification 222 may be defined in such a way that a business user / non-technical user can make changes to the configuration options in a way that he / she understands, and changes to the parameters are made through the associations and dependencies defined in the user interface specification 222.

[0054] The user interface specification 222 may define how configuration options correspond to configuration of parameters of a dataflow graph and other data elements that may be configured through the user interface 250. For example, interactions between a user and a user interface element may trigger changes to parameters in multiple dataflow graph components and changes to data stored in a database, data file, metadata repository, or another data storage device. The user interface specification 222 may define relationships between user interface elements and data that changes in association with changes to the user interface elements during operation of the user interface 250.

[0055] The user interface specification 222 may also define user interface elements based on data received from a database, a data file, a metadata repository, or another data storage device, or another data source such as a web service. When the user interface 250 is displayed, the received data is used to determine the manner in which the user interface element is displayed. In some embodiments, during operation of the user interface 250, data is received from an external source, such as a database, a data file, a metadata repository, or another data storage device, or another data source such as a web service, and the data received from the external source is defined in the user interface specification 222 as being associated with a parameter (e.g., the parameter is updated to include the data received from the external source).

[0056] The user interface may also display component output data associated with at least one data flow represented by a link of the data flow graph. Figure 2C , data flows from one component 224 to another component 228. The data flow between components can be viewed in the user interface 250. In some examples, sample data (e.g., data retrieved for testing purposes, not for processing or transformation purposes) is provided to one component 224 to determine how the component 224 processes the data.

[0057] exist Figure 2C In the variant, the user interface 250 (or multiple user interfaces) displays a parameterized acquisition application 120 ( Figure 1 ), parameterized detection application 122( Figure 1 ) and parameterized action applications 124 (see Figure 1 ) in each of the parameter values ​​in the . In this example, user interface 250 displays user interface elements that are independent of specifying which user interface elements correspond to which parameterized application. Instead, user interface elements are presented to enable a user to specify how to perform, for example, data acquisition and integration on a particular data stream, how to perform detection, and how to perform actions.

[0058] refer to Figure 3A , environment 300 includes a collection detection action (CDA) system 320 for collecting data records, detecting the satisfaction of one or more predefined conditions (as specified in rules) in the data records, and performing appropriate actions on the detected data records. In this example, Figure 1The execution system 108 of the CDA system 320 is shown as a CDA system 320 and the flow of data through the CDA system is shown. The CDA system 320 intermittently (e.g., periodically or continuously) receives data from various data sources, such as different servers interconnected in a network. When receiving data intermittently, the system collects the data into a single data stream (e.g., by posting the received data to a queue multiple times) and joins the data together in near real time (e.g., within one millisecond, two milliseconds, etc.) in a single, wide record, such as by generating a wide record that includes the data posted to the queue multiple times. The data is collected from the data source in near real time, rather than being retrieved (in batches) from a data warehouse. This collected data includes data records that include, for example, data indicating the occurrence of an event or action (e.g., a voice call made or the length of a voice call) or records indicating the occurrence of an event or action. By joining the data from these various data sources together, the wide record includes different types of data records (e.g., short message service (SMS) data records, voice data records, etc.). The CDA system 320 enriches this wide record with data record aggregations, data indicating that an event did not occur, state data, and various aspects such as customer data (e.g., customer profile), account data, etc. As data records are received, the environment 300 generates wide records with different types of data records (e.g., records that include and / or point to various sub-records) in near real-time.

[0059] Typically, the data collected from the data stream does not include all the information required for CDA system processing, such as user names and profile information. In such cases, the data (i.e., the data collected from the data stream) is enhanced by combining the profile data with the data received in the real-time data stream and by computing near real-time aggregates. By combining the profile data with the data from the real-time data stream and by computing near real-time aggregates, the search and retrieval system generates meaningful data records (e.g., data records including received near real-time data associated with a key, profile data for the key, and near real-time aggregates for the key) that are suitable for the processing needs of the search and retrieval system. Typically, the processing needs include various operations to be performed by the system (and / or rules to be executed) and various data required to perform those operations. In addition, such pre-computation or generation of data records that include "all data records" or fields (and / or a predefined set of fields) pre-populated with data corresponding to each data record in the data record helps avoid and reduce congestion in network bottlenecks when processing real-time data streams, for example. This is because all data required for processing is included in a single record (e.g., a record of records), for example, thereby eliminating or reducing data retrieval, calculations, and database queries at each stage or step of processing a data record or collecting a record. In addition, by keeping large amounts of enhanced data (e.g., profile data) in a memory or cache index in the CDA system, the system is able to access that data more quickly because it generates pre-computed records (of records).

[0060] For example, the systems described herein are configured to load enriched and enhanced data into memory (or into an indexed cache) when the system is under reduced load (e.g., relative to load at other times). Because the system has flexibility to preload enhanced data when the system is otherwise under reduced load, the system achieves load distribution by loading enhanced data into memory at reduced load, e.g., rather than having to do so in real time as processing data records occurs (which would be a period of increased load).

[0061] In the example, the CDA system 320 processes more than two billion data records per day for fifty million users and calculates aggregates for each data record type. In this example, the CDA system 320 receives a real-time data stream 340 (e.g., multiple different data streams, each having a unique format) from a data source 360. As used herein, real-time includes, but is not limited to, near real-time and substantially real-time, for each of which there may be a time lag between the time when the data is received or accessed and the time when the data is actually processed, but the data is still processed in real time when it is received. The CDA system 320 intermittently receives data including data records, also referred to as data items, from the real-time data stream 340. The received data also includes data records of different types (e.g., different formats). In the example, a first real-time data stream in the real-time data stream includes data representing data records of a first type / format, and a second real-time data stream in the real-time data stream includes data representing data records of a second type / format. The CDA system 320 includes a collection module 420 for collecting different types of data records received in the real-time data stream 340. Because the collection module 420 acts as real-time data records, rather than data extracted from the EDW, the CDA system 320 is able to provide immediate response to data records (when they are received) and near real-time aggregation of data records, and the system also provides immediate visibility of application results. The collection module 420 collects data records into a single data stream and publishes the data records to the queue multiple times. In the example, the collection module 420 collects data records by using a continuous stream to continuously process the received data records.

[0062] As the collection module 420 continues to intermittently receive data records from the real-time data stream 340, the collection module 420 detects (e.g., in a queue) two or more specific data records that share a general quality, such as included in a data record palette or associated with a specific user attribute (e.g., a user identifier (ID), a user key, etc.). In an example, the general quality is a corresponding value of a specific field (e.g., a user ID field) of two or more specific data records, the two or more specific data records having a specified data record type and / or the two or more specific data records defined by the data record palette.

[0063] The acquisition module 420 creates a data record set including two or more specific data records detected. In this example, the acquisition module 420 generates a data record 460 including a set of detected data records. The acquisition module 420 also inserts enriched data 440 into the data record 460, for example, a wide record. Typically, enriched data is data stored in a data warehouse (previously received or pre-calculated) associated with the data record. For example, a data record may specify multiple SMS messages that a user has sent, and may also include a user ID for the user. In this example, the data warehouse 380 receives data record 461 and stores data including (or associated with) the same user ID. This stored data includes user profile data, which includes, for example, the user's latest mobile phone type. The acquisition module 420 appends or inserts customer profile data of a customer associated with a specific data record included in the data record 460 into the data record 460.

[0064] The acquisition module 420 filters the received data, for example, so that only a subset of the received data is enriched and added to the data record 460. The acquisition module 420 is configured to filter based on a key (associated with the record) and / or based on a specified value of a specified field of the record. The acquisition module 420 also associates the received data and / or data records so that records associated with the same or similar keys are grouped together, for example, to enable complex data record processing (e.g., processing of records associated with a particular key and separated by time). In another example, the acquisition module 420 associates data records based on records having certain fields, certain values ​​of certain fields, etc. In this example, the values ​​of the fields of the associated records are inserted or added to the wide record.

[0065] The acquisition module 420 also calculates one or more aggregations (i.e., data record aggregations) of one or more data records included in the data record 460. For a particular data record for a particular user (as specified by a user ID included in the data record), the acquisition module 420 retrieves batch data 400 of the particular data record for the particular user from the data warehouse 380. The batch data 400 includes historical aggregations related to the particular data record, wherein the historical aggregation is a pre-computed aggregation of data record data from a previous time period, for example, a time period from a start time to a particular time before the detection data record is performed. Typically, the data record data includes data indicating a particular quality, attribute, or characteristic of the data record (e.g., data usage of a data usage data record). For example, the quality of the data record includes a particular field (the particular field is included in the data record), a particular value of a field included in the data record, a particular user ID key included in or associated with the data record, a lack of a particular field or a lack of a value of a particular field of the data record, etc. Based on the data included in the real-time data stream 340 of the particular data record for the particular user and based on the historical aggregations, the acquisition module 420 calculates combined data record data, for example, near real-time aggregations of the data record. Acquisition module 420 enriches data record 460 with the combined data record data of the at least one specific data record.

[0066] In the example, one of the data records in the data records 460 is data usage for JohnDoe associated with user ID 5454hdrm. In this example, the collection module 420 retrieves batch data 400 of the data record for “data usage” associated with user ID 5454hdrm from the data warehouse 380. In order to calculate the near real-time aggregate for this data record for this particular user, the collection module 420 aggregates the batch data 400 with the incremental data 410 to calculate the near real-time aggregate 430 for this data record.

[0067] In this example, the incremental data 410 includes a portion of the data received from the real-time data streams 340 that relate to the type of data record aggregated for this particular user. The incremental data 410 occurs during the last time that the historical aggregation was calculated close to the current time, for example, when a near real-time data stream is received. For example, the batch data 400 specifies that the user John Doe has used sixty-five megabytes of data in the last month, and the incremental data 410 specifies that the user John Doe has used one megabyte of data in the last five minutes. By aggregating the batch data 400 with the incremental data 410, the acquisition module 420 calculates the near real-time aggregation 430 for this particular data usage data record of the customer John Doe. The acquisition module 420 inserts the near real-time aggregation 430 into the data record 460, for example, as part of the record of this particular data record for this particular user. The acquisition module 420 also attaches an appendable lookup file (ALF) having the historical aggregation of the particular data record, for example, as specified by the batch data 400, to the data record 460. The acquisition module 420 appends the ALF to the historical aggregates to facilitate using the historical aggregates in computing new near-real-time aggregates, eg, when new data records are received.

[0068] In this example, acquisition module 420 transmits data records 460 to detection module 480. Detection module 480 includes rules 500, including, for example, rules for implementing various different applications for different types of entities. Detection module 480 includes a single module for implementing various applications and for performing aggregation.

[0069] The detection module 480 calculates one or more aggregations (i.e., data record aggregations) of one or more data records in the data records included in the data records 460. For a particular data record of a particular user (as specified by a user ID included in the data record), the detection module 480 retrieves batch data 400 of the particular data record of the particular user from the data warehouse 380. The batch data 400 includes historical aggregations related to the particular data record, wherein the historical aggregation is a pre-computed aggregation of data record data from a previous time period, for example, a time period from a start time to a specific time before the detection data record is performed. Typically, the data record data includes data indicating a specific quality, attribute, or characteristic of the data record (e.g., data usage of a data usage data record). For example, the quality of the data record includes a specific field (the specific field is included in the data record), a specific value of a field included in the data record, a specific user ID key included in or associated with the data record, a lack of a specific field or a lack of a value of a specific field of the data record, etc. Based on the data included in the real-time data stream 340 of the specific data record of the particular user and based on the historical aggregation, the detection module 480 calculates the combined data record data, for example, a near real-time aggregation of the data record. Detection module 480 enriches data record 460 with the combined data record data of the at least one specific data record.

[0070] In the example, one of the data records in data records 460 is data usage for JohnDoe associated with user ID 5454hdrm. In this example, detection module 480 retrieves batch data 400 of the data record for “data usage” associated with user ID 5454hdrm from data warehouse 380. In order to calculate a near real-time aggregate for this data record for this particular user, detection module 480 aggregates batch data 400 with incremental data 410 to calculate a near real-time aggregate 430 for this data record.

[0071] In this example, the incremental data 410 includes a portion of the data received from the real-time data streams 340 that relate to the type of data record aggregated for this particular user. The incremental data 410 occurs during the last time that the historical aggregation was calculated close to the current time, for example, when a near real-time data stream is received. For example, the batch data 400 specifies that the user John Doe has used sixty-five megabytes of data in the last month, and the incremental data 410 specifies that the user John Doe has used one megabyte of data in the last five minutes. By aggregating the batch data 400 with the incremental data 410, the detection module 480 calculates the near real-time aggregation 430 for this particular data usage data record of the customer John Doe. The detection module 480 inserts the near real-time aggregation 430 into the data record 460, for example, as part of the record of this particular data record for this particular user. The detection module 480 also attaches an appendable lookup file (ALF) having the historical aggregation of the particular data record, for example, as specified by the batch data 400, to the data record 460. The detection module 480 appends the ALF to the historical aggregate to facilitate using the historical aggregate in computing new near-real-time aggregates, eg, when new data records are received.

[0072] In this example, the CDA system 320 receives data representing one or more rules defining an application from a user's client device. For example, the user can use a data record palette to define the rules. The CDA system 320 generates one or more rules defining the application based on the received data. The CDA system 320 passes these one or more rules to a process configured to implement the one or more rules, such as a detection module 480. The detection module 480 implements the application based on the execution of the rule 500 for the data record 460. The detection module 480 also includes a state transition 530, including, for example, data specifying the state that the user has transitioned or entered in the application. Based on the state transition 530, the detection module 480 identifies which actions in the application are executed and / or which decision branches in the application are to be executed. For example, based on a particular user state in the application, as specified by the state transition 530, the user detection module 480 identifies which component of the application has been executed and which component of the application is to be executed next according to the user's application state.

[0073] Data records 460 include different types of data records, such as SMS data records, voice data records, etc. Therefore, rules 500 include rules with conditions for various different types of data records. Generally, a rule includes a condition, the satisfaction of which causes the execution of an action. In this example, one rule ("Rule 1") may have a condition that the user has sent thirty SMS messages in the past six months. After satisfying this condition, Rule 1 specifies an action to issue a $5 credit to the user. Another rule ("Rule 2") may have a condition that the user has used less than fifty megabytes of data in the past month. After satisfying this condition, Rule 2 specifies an action to provide a usage discount to the user, for example, to incentivize increased data usage. In this example, both Rule 1 and Rule 2 use different types of data records (i.e., SMS data records and voice data records, respectively). Detection module 480 is able to execute a program that includes rules that rely on different types of data records because data record 460 is a single wide record that includes different data record types. Additionally, detection module 480 is a single module that executes the application of multiple different applications because detection module 480 receives data record 460, which includes all data record types for all different operating levels. That is, detection module 480 is configured to execute multiple different applications for a single wide record (i.e., data record 460), rather than having different modules execute different applications for different data records (each including data types suitable for the corresponding application).

[0074] Upon detecting a data record (or aggregation of data records) in data records 460 that satisfies at least one of the conditions in rule 500, detection module 480 publishes action trigger 510 to queue 520 to initiate one or more actions (e.g., actions specified by the rule having the satisfied conditions). In the example, the action trigger includes data specifying which actions are to be performed, which application is being executed, and the user (e.g., the user for whom the action is to be performed). Detection module 480 transmits queue 520 to action module 540 to perform the actions specified in action trigger 510. In this example, action module 540 is configured to perform various actions, such as issuing a credit to a user account, transmitting a message, transmitting a discount message, etc.

[0075] refer to Figure 3B, Figure 541 illustrates the lifecycle of executable logic (e.g., for an activity). In this example, the executable logic is part of a detection engine. In this example, the system does not start the activity until a triggering event occurs. Typically, a triggering event includes an event that satisfies one or more specified conditions or attributes. The triggering event is based on the incoming event stream and the subscriber profile. As described below, the system delays the calculation of the control group until the triggering event occurs. In addition, the system can configure an activity to end early, for example, if an acceptance event occurs. The system can chain activities together - so that another activity can be triggered after an offer is accepted or expires.

[0076] In this example, start node 542 represents the start of an activity. In this example, start node 542 represents executable logic that specifies the start date and end date of an activity. The executable logic also specifies which rules (as part of a rule set and referred to herein as "decision rules") are to be executed for the activity. In this example, node 543 represents a decision rule (e.g., executable logic) for detecting a triggering event, such as an event that represents the activity should start. In this example, the decision rule includes logic for triggering an activity, such as by specifying which event to start the activity. The decision rule also includes logic for specifying an offer to be made, an activity duration, and a message priority. The decision rule also includes logic for detecting when to accept an invitation.

[0077] At the start of the campaign, the system executes a decision rule for each event received. Based on the event, the system determines whether to trigger the campaign. Possible results (of executing the decision rule for the event) are to ignore the event, exit the campaign, ignore the event until tomorrow, or trigger the campaign. As described in further detail below, the decision rule also specifies what to do if the message cannot be sent. For example, the rule specifies to cancel the campaign, trigger the campaign, or specify to continue searching for triggers at a later time (e.g., tomorrow).

[0078] Once the campaign has been triggered, the system executes control group logic, as represented by node 544, to determine whether the event (e.g., data record) will receive an invitation or will be assigned to a control group that does not receive an invitation. Control group assignments are immediate, for example, because they are completed only after the campaign is triggered. In addition, as described in further detail below, the system calculates control groups using matching panel cubes.

[0079] As represented by node 545, after the target group (e.g., the group receiving the invitation) or the control group is assigned, the system performs message arbitration. In this example, message arbitration refers to the process by which the system performs a relative message priority lookup. In this example, the system is configured to comply with the new message restrictions in the subscriber profile. For example, the subscriber profile can specify a predefined (e.g., maximum) amount of "normal" messages that a subscriber can receive every day. The subscriber profile can also specify a predefined (e.g., maximum) amount of "urgent" messages that a subscriber can receive on a given date. In this example, each message has an urgency and a priority. Various types of urgency include normal, urgent, or unlimited. In this example, if there are too many messages to be sent at one time, the system will prioritize the messages so that urgent messages are transmitted before normal messages. In this example, the decision rule specifies the message sending time. If the time is in the future, the system will wait before sending the message to allow the opportunity to send a higher priority message instead.

[0080] In the example, when the activity is first triggered, the system performs message arbitration. If the initial message cannot be sent (for example, because the maximum message limit has been exceeded), the triggering event can be canceled. In this example, for subsequent messages, if message arbitration fails, canceling the activity is not an option. In this example, the subscriber simply does not see the message.

[0081] Part of the arbitration process is a message (e.g., invitation) priority lookup. In this example, the message priority and urgency are specified in a lookup (reference) file. The keys to the lookup fields include a subject field, a type field, and a priority key field. That is, each invitation or message is pre-configured with a specified priority and urgency. In this example, based on the results of the message arbitration, the message can be sent, as represented by node 546.

[0082] After sending the message, the system is configured to perform further waiting actions (e.g., by waiting for tomorrow or accepting (uptake) (i.e., inviting fulfillment)), as represented by node 547. The system performs this further waiting action by executing additional decision rules. In this example, the decision rules are executed for each event. The rules are configured to determine whether to end the activity based on the event identifier (e.g., based on the detection of the occurrence of the target event). In this example, the target event is usually a notification that the fulfillment has occurred. However, any event and condition can be used to trigger the end of the activity phase. In this example, possible decision options are to ignore the event, exit the activity, and start a new phase. Typically, the phase of the activity refers to different invitations provided by the activity. In this example, when the activity phase ends, the decision rule can start a new phase. The decision rule is also executed after the activity phase has been executed for a specified number of days (e.g., N days), for example, if the fulfillment event is not detected. When the activity phase ends, the decision rule starts a new phase. In this example, possible options are to exit the activity or start a new phase.

[0083] After executing the subsequent wait event, the system also performs one or more actions to execute the calendar rules, as represented by node 548. For example, the calendar rules specify that after the initial message, the activity is configured to send a reminder message every day. These rules can also be used to suppress messages on some days. Based on the calendar rules, the system determines whether to resend the reminder. If the system determines that the message is to be resent, the system performs message arbitration again, as represented by node 549. Based on the result of the message arbitration, the system can send the message again, as represented by node 550. Generally, the executable logic represented by the nodes in this figure can be configured by a business rule editor (e.g., as described in U.S. Patent No. 8,069,129, the entire contents of which are incorporated herein by reference) or by a flowchart user interface editor (e.g., as described in U.S. Application No. 15 / 376,129, the entire contents of which are incorporated herein by reference).

[0084] refer to Figure 3C, the networked environment 560 includes data sources 562, 570 and a CDA system 577. In this example, the CDA system 577 includes parameterized acquisition applications 573, parameterized detection applications 580, and parameterized action applications 590. In this example, one of the parameterized acquisition applications 573 includes a data flow graph 573a, which includes, for example, nodes that each represent one or more data processing operations. One of the parameterized detection applications 580 includes a data flow graph 580a, which includes nodes 582, 583, 584, 585. In some examples, the data flow graph 580a includes a state diagram, for example, when the data flow graph 580a specifies various executable logic to be executed in various states. In this example, the state diagram can be parameterized, for example, so as to enable the values ​​of various parameters to be entered in the state diagram. In addition, a single key can be associated with multiple state diagrams (not shown here), for example, to maintain the state of the key across different data flow diagrams. Each node represents one or more parts of the executable logic to be executed in a specific state (of the executable logic). As such, nodes 582, 583, 584, 585 are hereinafter referred to as states 582, 583, 584, 585, respectively. In this example, one of parameterized action applications 590 includes a data flow graph 590a.

[0085] In this example, parameterized acquisition application 573 includes specification 574 that specifies one or more parameters (e.g., parameters A and C) that define one or more attributes of parameterized acquisition application 573 (or one of parameterized acquisition applications 573) and one or more corresponding values ​​(e.g., values ​​B and D) of those one or more parameters. For example, parameter A can be a parameter for specifying a data format, and parameter C can be a parameter for specifying a data source from which data items are collected. In this example, value B (for parameter A) specifies a data format into which the collected data items are transformed. Value D (for parameter C) specifies a data source from which the data items are collected (e.g., for later transformation). In the example, transformation includes association. In this example, data records are transformed by associating those data records associated with the same key together, and then publishing data records (e.g., master records) indicating related records to a queue. In this example, parameterized acquisition application 573 is also configured to perform enrichment, filtering, and formatting. In this example, the parameterized acquisition application 573 enriches the event (e.g., data record) itself, for example, rather than enriching the data record with profile data - which is performed by the parameterized detection application 580. For example, an event for an SMS message is enriched by including data specifying where a cell phone tower that relayed the SMS message is located. In this example, the parameterized acquisition application 573 enriches the event with this data, for example, by retrieving this data from one or more data sources - internal or external.

[0086] The parameterized detection application 580 includes a specification 586 that specifies one or more parameters (e.g., parameters E and G) that define one or more attributes of the parameterized detection application 580 (or one of the parameterized detection applications 580) and one or more corresponding values ​​(e.g., values ​​F and H) of those one or more parameters. For example, the parameters E, G can be parameters included in a rule (e.g., a rule for detecting a specified event) executed by the parameterized detection application 580, and the values ​​F, H are the values ​​of those parameters (e.g., values ​​that specify different types of events to be detected). The specification 586 also includes rules 587 and corresponding conditions for the rules. The specification 586 also includes various states of the parameterized detection application, including, for example, state data 592 of the data flow graph 580a. In this example, the state data 592 specifies that the parameterized detection application 580a has four states, states 1 to 4, which correspond to states 582 to 585, respectively. In this example, certain rules in the rules 587 are executed in certain states according to data structures 593, 594, 595 (e.g., pointers). For example, data structure 593 specifies that rule 1 is executed in state 1. No rule is executed in state 2. Rule 2 is executed in state 3 according to data structure 594. Rule 3 is executed in state 4 according to data structure 595.

[0087] The parameterized detection application 580 stores profile data 575, which includes KPI 576 and keyed state data 589. Typically, KPIs include data obtained by detection (e.g., various events included in or represented in collected data records). In this way, the value of the KPI is updated and changed in real time in operation. For example, the CDA system 577 defines a KPI for tracking data usage. In this example, when a new data record is received, the parameterized detection application 580 detects whether the new data record includes or specifies a data usage event. If the data record specifies a data usage event, the CDA system 577 updates the KPI (for the key associated with the data usage event) with data specifying the updated data usage. In this example, each KPI is a keyed KPI associated with a specific key. In the example, the KPI is used to execute a rule, for example, to determine whether various conditions of the rule are met by determining whether the content of the KPI of a specific key meets the conditions of the rule to be executed in the current state of the key in the specification 586. In addition, the KPI can be calculated based on multiple events. For example, a KPI can be defined to specify when a user uses a specified amount of data and a specified amount of voice usage. In this example, the KPI is based on two events - a data event and a voice usage event. The KPI itself can also be aggregated, and the aggregated KPI is used to detect events and / or detect the satisfaction of rule conditions. The KPI is stored as part of a customer profile. In addition, in some examples, the KPI is an attribute defined by a user and / or a system administrator.

[0088] In this example, keyed state data 589 specifies the state of the parameterized instrumentation application for each value of a key. In this example, each data item received by the CDA system 577 is associated with a value of a key. For example, a key can be a unique identifier such as a subscriber identifier. The CDA system 577 maintains the state of the parameterized instrumentation application 580 for each of the key values. For example, the key value "349jds4" is associated with state 1 of the parameterized instrumentation application 580. The key value "834edsf" is associated with state 3 of the parameterized instrumentation application 580 at a first time (T1) and is associated with state 4 at a second time (T2).

[0089] In this example, parameterized action application 590 includes specification 597 that specifies parameters I, K and corresponding values ​​J, L of those parameters that define one or more properties of parameterized action application 590. For example, parameter I can be a parameter that specifies profile data to use for customizing an action for a user. In this example, the value J of parameter I specifies which profile data to use in the customization.

[0090] In operation, the CDA system 577 executes one or more of the parameterized acquisition applications 573 to perform processing of data records with one or more values ​​of one or more parameters specified by the specification 574. In this example, the processing includes acquiring data items 566a, 566b, ..., 566n in a data stream 564 from a data source 562 by the parameterized acquisition application 573. In this example, each of the data items 566a, 566b, ..., 566n is a keyed data item, e.g., a data item associated with a key. One or more of the parameterized acquisition applications 573 also acquires data items 572a, 572b, ..., 572n from a data source 570 (these data items are part of the batch data 568). In this example, each of the data items 572a, 572b, ..., 572n is a keyed data item. In this example, the format of the data items 566a, 566b, ..., 566n is different from the format of the data items 572a, 572b, ..., 572n.

[0091] One or more of the parameterized acquisition applications 573 transform the data items 566a, 566b, ..., 566n and the data items 572a, 572b ... 572n according to the specification 574 to obtain transformed data items 579a, ..., 579f. In this example, each of the transformed data items 579a, ..., 579f is transformed into a data format suitable for the parameterized detection application 580. One or more of the parameterized acquisition applications 573 fills the queue 579 with the transformed data items 579a, ..., 579f and transmits the filled queue 579 to the parameterized detection application 580.

[0092] The CDA system 577 executes one or more of the parameterized detection applications 580 with one or more values ​​of one or more parameters specified by the specification 586 to process the transformed data items 579a, ..., 579f in the queue 579 as follows. The CDA system 577 can enrich the transformed data items with one or more portions of the profile data 575 and / or with the KPI 576. For example, the CDA system 577 generates an enriched transformed data item 598 by adding profile data (e.g., such as data usage, SMS usage, geolocation, and data plan data) to the transformed data item 579f. In this example, the enriched transformed data item 598 is associated with a specific value of the key ("834edsf"). The parameterized detection application 580 identifies the current state of one or more of the parameterized detection applications 580 with respect to the specific value of the key. In this example, at time T1, the current state of one or more of the parameterized detection applications 580 is state 3, as specified in the keyed state data 589. Parameterized detection application 580 identifies one or more of rules 587 in a portion of specification 586 to be executed in the current state. In this example, rule 2 is executed in state 3, such as Figure 3C 2 in the example. The parameterized detection application 580 executes the identified one or more rules (e.g., rule 2). The parameterized detection application 580 determines that at least one of the one or more transformed data items (e.g., enriched transformed data items 598) satisfies one or more conditions of at least one of the one or more rules (e.g., rule 2) executed in the current state. In response to this determination, the parameterized detection application 580 generates a data structure 521 specifying the execution of one or more actions (represented by action data 511). The parameterized detection application 580 also causes the specification 586 of a specific value (e.g., 834edsf) of the key to transition from its current state to a subsequent state. In this example, the transition is shown as a transition from time T1 to time T2, wherein the state of the specification 586 is transitioned from state 3 to state 4 for the key value 834edsf. The parameterized detection application 580 also transmits the generated data structure 521 to the parameterized action application 590.

[0093] The CDA system 577 executes the parameterized action application 590 with one or more values ​​of one or more parameters specified by the specification 597 to perform operations including: based on at least one of the one or more actions specified in the data structure 521, sending one or more instructions 591 to cause at least one of the one or more actions to be performed.

[0094] In a variation, parameterized acquisition application 573 accesses profile data (e.g., profile data 575) for each received data record and enriches the data record with the accessed profile data. In this example, parameterized detection application 580 compares the content of the profile data with one or more rules and / or applications included in parameterized detection application 580 to detect the occurrence of one or more predefined events. Upon detection, parameterized detection application 580 updates the KPI accordingly, for example, with the detected information and / or information specifying the detected event.

[0095] In yet another variation, there are multiple queues between the acquisition application and the detection application. For example, there may be a priority queue for certain types of events, such as events that should not be delayed when processing. Other events may be delayed when processing. These other events are assigned to another queue, for example, a non-priority queue, and an alarm is inserted into the queue to specify the delay of processing of these events. Typically, an alarm occurs when a chart sends itself a special type of event (e.g., an alarm event) that will arrive at a pre-calculated time in the future. Whenever the chart logic wants to wait, an alarm is used. By having multiple queues, the system is able to load balance events and also reduce the latency of processing events - by first processing those events in the priority queue and waiting to process events in the non-priority queue. In this example, the parameterized acquisition application is parameterized and configured with rules for inserting alarms and rules for specifying various event types associated with alarms.

[0096] refer to Figure 4, the system 600 includes an acquisition unit 610 for receiving 615 data records 601 to 608, for example, in batches and / or from a real-time data stream. In this example, the acquisition unit 610 stores the records 601 to 608 in a memory, for example, in a data structure 620 (e.g., an index) stored in the memory. In this example, the data structure 620 is not a static data structure. Instead, the data structure 620 is a dynamic data structure that is intermittently updated and modified when new records are received in the data stream 619. In addition, the entries in the data structure 620 are removed after the entries (e.g., logical rows) are processed, for example, by being assigned to the data structure 630 or filtered from the data structure 620. The data structure 620 includes logical rows 621 to 628. In this example, each of the logical rows 621 to 628 corresponds to one of the data records 601 to 608. The data structure 620 also includes logical columns 620a to 620d, which each correspond to a field and / or a value of a field in the data records 601 to 608. In this example, each logical row of the data structure 620, such as logical rows 621 to 628, corresponds to a subset of related information extracted from a particular data record 601 to 608 by the acquisition unit 610. Each logical column of the data structure 620 conceptually defines a particular data attribute of a particular data record associated with the particular logical row. In the example where the data structure 620 is an index, each of the logical rows 621 to 628 is an index entry.

[0097] The acquisition unit 610 also stores a data structure 617 in a memory. In an example, the data structure 617 represents a static data structure stored in a data storage device or repository. In this example, the data structure 617 stores enriched data, such as profile data. In this example, the enriched data stored in the data structure 617 can be added to the received data record in real time, for example, by adding or appending specific enriched data to the logical row of the data structure 620. The data structure 617 also defines which data types are to be included in the data record 630, for example, which data types are dynamically added to the received record in real time and when those records are received. In this example, the data structure 617 includes logical rows 617a to 617h, each of which includes enriched data for a specific ID. In this example, the data structure 617 includes logical columns (not shown) for ID data. By matching the ID data in the data structure 620 with the ID data in the data structure 617, the acquisition unit 610 generates a rich record for a specific ID, for example, a record including the received data from the real-time data stream and then enriched with the enriched data.

[0098] The collection unit 610 also filters and associates the data records 601 to 608 received in the data stream 619. In a variant, the collection unit 610 collects the records received from a batch retrieval, such as from a data storage device. In this example, the collection unit 610 filters the records that include a non-Boston value in the "location" field. That is, the collection unit 610 filters the data records 601 to 608 to include only those data records that have a value "Boston" in the location field. The collection unit 610 also associates the remaining filtered records 602, 603, 606 together, such as by associating record 602 with record 603 - because both records are associated with the same ID. The collection unit 610 associates the records 602, 603 together by including logical rows 631, 632 representing these records adjacent to each other in the data structure 630. In a variant, the associated records 602, 603 are merged into a single record. In yet another example, the collection unit 610 performs the association by generating an association aggregation.

[0099] In a correlation aggregation, the value of the aggregation is different from the value returned. Instead, the collection unit 610 uses individual fields in the records (separated in time) to perform the correlation. In this example, the correlation is particularly complex because the collection unit 610 connects records separated in time together, for example, based on specific values ​​of fields in those records. The following is an example correlation performed by the collection unit 610:

[0100] For each client ,for trade Each data record of type Transaction.Action = "Buy" (trade.action="buy" ), calculate the most recent 20 minutes The largest seen in trade.amount of symbol .

[0101] In the preceding examples, the underlined portions represent the portions of the aggregate definition that are parameters that can be specified. The following is a list of possible parameters for associating an aggregate. In this example, the parameters can be specified as part of the collection application:

[0102]

[0103] In this example, the collection unit 610 uses the field or expression along with the filter expression and the selection function to select the appropriate data record from the time window. We then calculate a value from the data record and use that value as the aggregate value. In the example, the data record has two fields - an amount field and a sign field. For the collection unit 610, the sign corresponding to the data record with the largest amount (within a given time period) is determined, and then the selection function is set to "latest maximum value", the field or expression is set to "amount", and the calculated expression is set to "sign" in the collection application.

[0104] In still another variation, the collection unit 610 associates data records that are separated in time, for example, by collecting a particular data record associated with or including a particular key, and then waiting a specified amount of time for another data record that includes / associated with the same key. When the collection unit 610 collects related data records (e.g., data records associated with the same key), the collection unit 610 groups the related data records, for example, by merging the data records into a single record and generating a data group that includes a single, merged record. In an example, because the data records are separated in time, the collection unit 610 can store data in memory indicating the collected data records and wait for a specified period of time to determine whether to receive another data record using a key that matches the key of the data record in which the data is stored (e.g., transient in memory). Due to the amount of data records that the collection unit 610 receives, the collection unit 610 implements an in-memory grid (or another in-memory data structure) to track and store keyed data representing the data records that have been received and (optionally) timestamps representing the times when those records were received. When a new record is received, the collection unit 610 identifies the key of the newly received record and searches the in-memory data grid for a matching key in the in-memory data grid. If a matching key exists, the collection unit 610 associates the data record with the matching key, for example, by merging the records into a single record. The collection unit 610 may also update the in-memory grid with data specifying that a second data record has been received for the key. In addition, the collection unit 610 is configured to delete or otherwise remove entries in the in-memory grid after a specified amount of time has passed since the time indicated in the timestamp.

[0105] In this example, data structure 630 represents the filtered associated records and includes logical rows 631 to 633 and logical columns 630a to 630e. Data structure 630 includes a dynamic data structure, rather than a static data structure, which is updated intermittently, for example, when new records are received. Once dynamic logic (as described in further detail below) is applied to a particular entry, the entry (e.g., logical row) is removed from data structure 630. After the dynamic logic is applied to a particular entry, data structure 630 does not retain and continues to store the entry. In this example, logical column 630e represents, for example, enriched data for a particular ID. Logical row 631 represents an enriched version of record 602 by including enriched data associated with ID "384343".

[0106] The acquisition unit 610 transmits the data structure 630 to the detection module 634. In this example, the detection module 634 executes executable logic 644 as described herein. The detection module 634 also executes dynamic logic (e.g., the detection module performs dynamic segmentation), for example, by executing logic included in the dynamic logic data structure 640. In an example, the dynamic logic includes executable logic that is configured to dynamically process incoming data records on the fly—as they arrive. In this example, the data structure 640 includes logic rows 646, 648 and logic columns 640a to 640d that specify rules. In this example, the detection module 634 is configured to execute the data structure 640 against the data structure 630 to determine which records represented in the data structure 630 satisfy the logic defined by the data structure 640. Upon detecting a record that satisfies the logic, the detection module 634 executes one or more portions of the executable logic 644 associated with a portion of the logic included in the data structure 640 (e.g., a segment defined by the data structure 640). That is, executable logic 644 has various parts (e.g., rules) associated with various parts of dynamic logic. Not all parts of executable logic 644 are executable for all parts of dynamic logic. Detection module 634 executes a part of executable logic 644 for a detected record that satisfies a part of dynamic logic. For example, record 603 represented by logic row 632 satisfies the dynamic logic in logic row 646 included in data structure 640. In this example, a part of executable logic 644 is defined as executable for the logic defined by logic row 646 in data structure 640. Therefore, detection module 634 executes a part of executable logic 644 for record 603 (or for the data included in logic row 632). Based on the execution of a part of executable logic, detection module 634 generates an instruction 642 for transmitting a target message (e.g., an invitation to reload mobile minutes) to a client device associated with a user represented by an ID included in logic row 632 at logic column 630a.

[0107] In the example, dynamic logic included in the data structure 640 specifies various segments, such as fill segments. For example, the logic included in the logic row 646 specifies a specific segment. In this example, the segmentation performed by the detection module 634 is dynamic because the segmentation is performed "on the fly" as the system 600 receives and processes data records in real time. Traditionally, customer records are stored to disk (e.g., in a data repository) and then the customers are "segmented", for example, by applying various segmentation rules to the customer records. This is an example of static segmentation because a static group of customer records is being segmented. In contrast, here, there is no static group of records. Instead, the system 600 receives records continuously and / or intermittently. When the records are received, the system 600 segments them dynamically, for example, by using a continuous stream to process the records on the fly, where part of the processing includes segmentation. Because the records are dynamically segmented, the system 600 can detect when a user or client device enters a specific geographic location (e.g., a mall) and send a targeted message to the client device at that time while the user is still in the mall. In an example, dynamic segmentation rules may be specified by configuring a parameterized application (eg, one of the parameterized detection applications).

[0108] In the example, the acquisition unit 610 or another component of the CDA system generates a target group (TG) and a control group (CG), for example, to which dynamic logic is performed. Typically, the control group includes a group of users (e.g., subscribers) that match the specified criteria of the dynamic logic and are therefore candidates for inclusion in a particular grouping (e.g., inclusion in an activity), but these users have been explicitly excluded from the grouping so that their behavior can be compared with other users - the target group - who actually participated in the grouping (e.g., and therefore received invitations to the activity). In this example, the acquisition unit determines the target group and the control group dynamically and in real time upon receiving the data records. That is, the acquisition unit 610 does not calculate the control group and the target group from a static data set, but determines the target group and the control group from a dynamically changing data set (e.g., these data sets are intermittently updated and changed when new records are received).

[0109] In the example, the control group is defined as follows: if the data records are associated with the specified logic (for example, these data records are in a given campaign), then these data records are assigned to the TG or CG by the acquisition unit 610. In the example, if the key of the specified data record is associated with the logic, then the data record is associated with the specified logic. In order to ensure that the TG and CG have similar characteristics and attributes, the acquisition unit 610 generates a matching panel and stores it in a memory to segment the data records. The matching panel (also called a matching panel cube) includes a multidimensional grid representing the keys for which records are being received (for example, including the entire subscriber base), where each key (for example, subscriber) is assigned to a "cube" (cell). Dimensions describe different aspects associated with the keys, such as data fields indicating the following: average revenue per user (ARPU), geography (rural or urban), network age, etc. Keys with the same dimension values ​​appear in the same cube of the matching panel.

[0110] Determining which keys (or data records) are in the TG or CG can then be performed, for example, by the acquisition unit 610 for each cube matching the panel using the specified logic described later. The membership decision of the target group and the control group will also be made in the specified chart (e.g., generated from the acquisition application). If late records affect the (active) logic, these late records are handled in the chart logic. For example, if the logic depends on data record A arriving before data record B, but due to operational delays, data record B may arrive first, then the chart can be written to take this possibility into account. If there is a possibility of time deviation between the systems sending these data records, the chart logic will also take time deviation into account. In the example, the logic for assigning keys (or records) to TG or CG (or neither - in data records where both TG and CG are saturated) is performed by the acquisition unit 610 as follows. If the logic is configured so that the CG is not required, the acquisition unit 610 assigns the key to the TG.

[0111] The logic is as follows: For each key, determine whether the key is pre-designated (e.g., in a table and / or memory) as belonging to the CG. If so, the acquisition unit 610 assigns the key to the CG. For each cube of the matching panel, the acquisition unit 610 modifies the entry in the CG as follows: If less than a specified amount (e.g., a specified percentage) of a predefined group of keys belonging to the cube (e.g., representing an active group) is in the CG, there is a shortage. If there is a shortage, the TG keys in the cube are moved to the CG of the cube. If more than a specified amount of a predefined group of keys belonging to the cube is in the CG, there is an excess. If there is an excess, the acquisition unit 610 marks the keys as neither in the TG nor in the CG. In the example, the acquisition unit 610 calculates a measurement of how well the CG is rated green, yellow, or red. This measurement is a dynamic measurement that changes intermittently because the CG changes intermittently.

[0112] refer to Figure 5A , an example networked environment 700 for performing real-time CDA functions is shown. In this example, the networked environment 700 includes a CDA system 702, a client system 704, an external data source 706, a network data source 708, and an external system 710. The CDA system 702 includes a collection / integration module 712, which, for example, performs the functions of the collection module described above. In this example, the collection module 712 stores data (as shown by arrow 756) in a data warehouse 732. The collection module 712 stores data in near real time (e.g., as the data is received) and in batches. The collection module 712 also retrieves (as shown by arrow 758) data from the data warehouse 732, for example, for enriching data records and for building wide records. In this example, the collection module 712 includes an in-memory archive 715 for archiving data retrieved from the data warehouse 732. In this example, as shown by arrow 717, the collection module 712 can send some of the collected data back to the external data source 706, for example, to facilitate data feedback and form a data feedback loop. The CDA system 702 also includes applications 714, which in turn include detection modules 716 and action modules 730, each of which performs the operations and functions described above.

[0113] In this example, the detection module 716 includes an in-memory data store 718 for storing profile data (e.g., of a user of the CDA system 702) and also for storing state data (e.g., the state data specifies the execution state of an application, a data flow graph, an activity (e.g., the activity includes a series of applications and / or data flow graphs), etc. As shown in this example, the profile is stored in memory, rather than to disk, to reduce and / or eliminate latency in data retrieval and enable real-time maintenance of state. In this example, the detection module 716 performs enrichment, for example, by enriching the data record 723 with the profile data stored in the in-memory data store 718. In this example, the detection module 716 executes a specification for a parameterized detection application. The enrichment performed by the detection module 716 is to retrieve profile data associated with a key (e.g., profile data for a particular user) from the memory 718 according to instructions specified by the specification of the parameterized detection application and to populate one or more fields of the data record 723 (for the key) with the retrieved profile data.

[0114] In addition, the CDA system 702 continuously or intermittently updates the user's profile, for example, by updating profile data in the in-memory data store 718. For example, if the action module 730 sends an invitation to the user, the CDA system 702 updates the user's profile data with data specifying that the invitation was sent and which invitation was sent. For example, each invitation includes or is associated with a key or other unique identifier. The profile data also includes or is associated with the key or unique identifier. Once the action module 730 sends the invitation, the CDA system 702 identifies the key associated with the invitation and updates the profile data associated with the same key with data specifying that the invitation was sent.

[0115] In addition, based on the profile data, the CDA system 702 generates KPIs. In the example, the KPI is a measurable value that indicates how effectively a predefined object is achieved. In the example, the KPI is generated by the detection module 716 and stored in the detection module 716. In the example, the KPI includes data specifying when an invitation was sent to a customer, which customer the invitation was sent to, whether the customer responded to the invitation, etc. In another example, the KPI also represents other metrics, such as, for example, the number of times a user deposited more than $10.00 into an ATM, or other predefined metrics or events. In this example, the CDA system 702 receives input data records. The CDA system 702 is configured to specify which metrics are KPIs. In this way, the KPI (and / or its definition) is integrated as part of the initialization of the CDA system 702. In still another example, the CDA system 702 updates the profile data and / or KPI (for a specific key) with other data specifying each event represented by a specific received record for that specific key. As such, the profile data tracks and includes data representative of all received events (for a particular user) and / or predefined events or event types.

[0116] In another example, the detection module 716 (or parameterized detection application) generates one or more KPIs for a specific value of a key. In this example, the KPI specifies one or more data item values ​​associated with the specific value of the key. The CDA system 702 receives data for the specific value of the key, wherein the received data indicates feedback about one or more actions initiated by the action module 730 and / or wherein the received data includes an incoming data record. In this example, the detection module 716 updates the KPI for the specific value of the key with the feedback data and stores the KPI in the in-memory data storage device 718.

[0117] The CDA system 702 transmits its KPIs to other systems, for example, to enable those systems to track and manage invitation validity and / or customer journeys. As previously described, the profile data is maintained in memory along with the KPIs, for example, by storing the profile data in memory and by storing the KPIs as part of the profile data.

[0118] In addition, the detection module 716 is configured to add external data to the in-memory profiles (which are stored in the in-memory data storage device 718). The CDA system 702 retrieves this external data from one or more of the external data sources 706. In this example, the detection module 716 adds the external data to the in-memory profiles, for example, to further reduce the latency of constructing records including enrichment and profile data (e.g., for detecting data records). In this example, the detection module 716 performs in-memory key-based processing (e.g., of the acquired data records). In this example, the processing is key-based processing because the detection module 716 processes data records associated with a particular key (or identifier) ​​and processes those keyed data records (e.g., data records associated with a particular key) according to the in-memory state (of an application or a series of applications) for the key, as described in U.S. Application No. 62 / 270,257. In this example, the acquisition module 712 transmits the record 723 to the queue 721 for retrieval by the detection module 716. In this example, the acquisition module 712 transmits data and / or data records to the detection module 716 in batches and in real time (e.g., as the acquisition module 712 receives the data records). The acquisition module 712 transmits data to the detection module 716 in batches, for example, by transmitting (as part of the record 723) batches of data retrieved from one or more data repositories.

[0119] In this example, the detection module 716 executes various rules (e.g., rules specified by various applications and / or flow charts). Based on the execution of these rules, the detection module 716 determines whether it is necessary to update the state (e.g., for a specific key). For example, based on the execution of the rules, the state of the application (for a specific key) can be changed from one state to another. When this example, the detection module 716 updates the state of the key accordingly. In the example, the detection module 716 stores the state as a shared variable separated from any specific running data flow graph or application, for example, via a keyed data storage device in a persistent memory. This reduces the latency required to determine the state, for example, because each application and / or module can retrieve the value in memory of the shared variable.

[0120] The detection module 716 also includes applications 720, 722, 724, 726, each of which is used to perform various types of data processing. In this example, the acquisition module 712 generates a data record 723, for example, from one or more acquired data records and by including the acquired data record as a sub-record in the data record 723 and also by enriching the data record 723 with profile data and / or other stored data. The detection module 716 transmits the data record 723 to each of the applications 720, 722, 724, 726, each of which is configured to perform data processing and apply rules for performing data record detection. In conventional ways of performing data record detection, a separate data record is generated and transmitted to each application (e.g., in a format appropriate for each application), resulting in, for example, an increase in system resources, an increase in memory storage, and an increase in system latency relative to the amount of system resources consumed, memory consumed, and latency incurred when a single wide record of data records (e.g., record 723) is transmitted to each of applications 720, 722, 724, 726, as described in U.S. Application No. 62 / 270,257. However, when each application has been integrated into a specific system or module, each application shares a common format, and thus a single record can be transmitted to each of the applications.

[0121] Based on the execution of one or more of the applications 720, 722, 724, 726, the detection module 716 detects one or more predefined data records. For each detected data record, the detection module 716 publishes an action trigger (e.g., an instruction to perform or cause the performance of one or more actions) to the queue 728, which in turn sends the action trigger to the action module 730. Based on the content of the action trigger, the action module 730 causes one or more actions to be performed (e.g., such as sending an email, a text message, an SMS message, etc.). In some examples and based on the content of the action trigger, the action module 730 generates a message or content and customizes the message / content for the user to whom the message / content is directed. The action module 730 transmits the customized message (as shown by arrow 766) to one or more of the external systems 710, which then transmit the message / content or perform further actions based on the customized message.

[0122] In this example, the data warehouse 732 includes a query engine 734, a data warehouse 738, and an analysis engine 736. The query engine 734 queries data (e.g., data indicating detected data records, data indicating processed data records, etc.) from the detection module 716, the data warehouse 738, or other data sources, and transmits the queried data (as shown by arrow 760) to the analysis engine 736 for performing data analysis. In this example, the analysis engine 736 stores the data analysis (as shown by arrow 762) in the data warehouse 738. In addition to performing data record detection, the detection module 716 is also configured to process warehouse data (e.g., data stored in the data warehouse 738 or another data storage device) to perform various analyses. In this example, the analysis engine 736 uses various personalization rules 713, which specify which rules apply to data records associated with which keys. In this example, the personalization rules also include segmentation rules, which specify how to personalize instructions or how to target records associated with various predefined segments. Analysis engine 736 transmits (as indicated by arrow 764) personalization rules 713 to application 714, for example, for execution of the personalization rules.

[0123] In this example, the action module 730 transmits data and / or messages (as indicated by arrow 740) to the network data sources 708, which then feed the data back (as indicated by arrow 752) to the CDA system 702. In the example, the action module 730 transmits customized messages to the network data sources 708, which then retransmit the customized messages (as indicated by arrow 744) to the client system 704, which in turn sends a confirmation message (not shown) (as indicated by arrow 746) to the network data sources 708, which then transmit the confirmation message as feedback (as indicated by arrow 752) back to the CDA system 702. In this way, the networked environment 700 implements a feedback loop (via one or more of the paths indicated by arrows 740, 744, 746, 752) that enables the CDA system 702 to audit or track received messages.

[0124] In this example, the action module 730 also transmits (as indicated by arrow 742) data (e.g., a customized message or other customized data) to the external data source 706 to perform a data feedback loop. In turn, one or more of the external data sources 706 transmits (as indicated by arrow 748) data (received from the action module 730) to the client system 704, which in turn can send a confirmation (as indicated by arrow 750) back to one or more of the external data sources 706, which sends the confirmation as feedback (as indicated by arrow 754) back to the CDA system 702. In this way, the networked environment 700 implements another feedback loop (via one or more of the paths indicated by arrows 742, 748, 750, 754). In this example, the analysis engine 736 or another module of the CDA system 702 tracks the end-to-end customer journey, for example, by tracking whether the invitation is delivered as is (e.g., the invitation is executed), the user's response to the invitation, the invitation is fulfilled, etc. In this example, analytics engine 736 tracks customer journeys (e.g., invitation delivery and invitation fulfillment) through one of the feedback loops described above and / or by obtaining data from multiple systems (e.g., from external data source 706, external system 710, or other external systems that track invitation fulfillment and delivery).

[0125] In some examples, the CDA system 702 can evaluate data received as part of one of the feedback loops described above. For example, the received data can specify the validity of an invitation or action output by an action module. Based on this received data, the CDA system 702 evaluates whether a parameter value (e.g., a parameter value input by a user) results in a desired reaction of the system, e.g., whether a desired action is performed. Using this feedback, the CDA system 702 or the user can modify or adjust one or more parameter values ​​of an acquisition, detection, or action application. For example, the detection application can be configured with parameter values ​​that specify the detection of an event (or data record) indicating that the user's data plan is less than a threshold amount of remaining data. In response to detecting this event, the action application can be configured with one or more parameterized values ​​that specify a specific action or output to be generated. In this example, the output can be a message notifying the user of a special promotional opportunity to reload or purchase more data. Using one of the feedback loops described above, the CDA system 702 tracks the validity of the output, for example, by tracking the fulfillment of the invitation. In an example, the output from the action module includes a key (or other identifier) ​​that uniquely identifies the output or the user to whom the output is directed or transmitted. Actions related to the output (e.g., clicking a link or other selectable portion of the output) are tracked (e.g., by an external system), for example, via a cookie that includes a key or identifier (or another identifier associated with a key for the output), by associating the action with a digital signature that includes the key or identifier of the output, etc.

[0126] In this example, based on data received from the feedback loop, the CDA system 702 may identify that a particular output is not particularly effective. For example, the output may not result in at least a threshold amount of users purchasing additional data. Based on the feedback data, the CDA system 702 or the user may adjust one or more parameter values ​​of a detection application (e.g., to specify detection of a different or changed event) and / or an action application (e.g., to specify a changed or different output in response to a detected event). In this example, the CDA system 702 executes a correction loop that adjusts the parameter values ​​by changing the values ​​to obtain the correct action to be performed after determining that the action performed is incorrect (e.g., does not achieve the desired result) or is not performed correctly. In some examples, the correction loop includes a set of rules that are heuristic-based and specify modifications to one or more of the applications to modify and / or obtain a specified result.

[0127] In another example, the detection module 716 sends feedback directly to the data warehouse 732, as shown by arrow 701. For example, the detection module 716 can send data indicating the success of the activity to the data warehouse 732. The detection module 716 can also send data (e.g., feedback data) indicating the path taken (in the detection application), the decision made by the detection application, and the branch logic followed by the detection application to the data warehouse 732. In this example, the CDA system 702 can execute one or more of a machine learning algorithm, a heuristic algorithm, or a neural network on the feedback data received by the data warehouse 732 from the detection module 714. Based on this execution, the detection module 716 can update one or more parameter values ​​in the detection application or the type of parameters included in the detection application. For example, if a particular area of ​​the branch logic is not fully used (as indicated by the feedback data), the neural network can use the feedback data specifying which logic is being accessed, and based on the feedback data, adjust the parameters of the underutilized branch logic to facilitate the traversal of the logic.

[0128] The feedback loop and correction loop described above can also be performed in the following manner to ensure the correct operation of the underlying system, for example, to ensure that the one or more actions are performed in the correct manner, i.e., as needed. The one or more actions may involve data processing tasks or network communications, such as one or more of the following: sending a text message to an external device, sending an email to an external system, opening a ticket for a work order in a case management system, immediately disconnecting a mobile phone connection, providing a web service to a target device, and using a notification to transmit a data packet of the one or more transformed data items, and executing a data processing application hosted on one or more external computers for the one or more transformed data items. In some examples, the one or more actions include providing an invitation to the user, for example, an additional ten minutes of voice or additional data. In this example, the system described herein is configured to communicate with a supply system (e.g., a system that maintains or controls minute and / or data usage rates, such as a telephone network) to indicate to the supply system that the user has additional voice minutes and / or data. In another example, the one or more actions include a system that provides benefits to the user, such as, for example, providing money. In this example, the system is configured to provide money to a user account maintained at a financial entity through a financial transaction. In another example, the one or more actions include downgrading the user's service while the user is roaming to prevent the user from incurring roaming bills.

[0129] The underlying system and / or user may receive a feedback message indicating whether the one or more actions are (i) successfully completed or (ii) failed in a feedback loop. If a portion of one of the one or more actions is not completed, the one or more actions may be considered to have failed. The feedback message may optionally indicate which portion of the one or more failed actions is not completed. For example, the feedback message may indicate that a data processing application hosted on one or more external computers is correctly or incorrectly executed for the one or more transformed data items, for example, some data processing tasks of the application are not executed for some of the data items. The feedback message may indicate which data processing tasks (e.g., which portion of the programming code) are not executed. For example, the action of immediately cutting off a mobile phone connection may not be correctly executed because the mobile phone connection is only cut off after a certain delay, which exceeds the allowed predetermined delay. The feedback message may indicate this specific delay. The feedback message may also indicate the result data (e.g., a specific delay or result generated by the data processing application) of one or more actions that are successfully completed and / or failed.

[0130] The result data may be compared to predetermined data associated with successful completion of execution of the one or more actions (e.g., a predetermined delay considered to be an allowable or expected result generated by the data processing application). Successful completion of execution of the one or more actions may then be determined based on the comparison, or failure of execution of the one or more actions. Successful completion of execution of the one or more actions may be determined if the result data deviates from the predetermined data by less than a predetermined amount (e.g., in absolute value or as a percentage), and failure of execution of the one or more actions may be determined if the result data deviates from the predetermined data by at least a predetermined amount.

[0131] That is, the correct execution of the one or more actions may be represented by predetermined data and / or predetermined quantities. In view of the possible actions to be performed as described above, such deviations of the result data from the predetermined data may occur in the following aspects: receipt confirmation data, transmission time or number of characters sent for a text / email message, ticket number for a ticket, time required to disconnect a mobile phone connection, available network bandwidth for a web service, amount of data transmitted for a data packet, data processing tasks performed in a data processing application on an external device, or similar parameters or features characterizing the action.

[0132] Subsequently, one or more specified values ​​for one or more parameters of the first parameterized application, the second parameterized application, and / or the third parameterized application may be changed (by the system 702 or by the user) based on the result data, and the first parameterized application, the second parameterized application, and / or the third parameterized application may be re-executed by the system with the changed one or more specified values. In order to provide the user with a further possibility to evaluate and / or initiate the correct operation of the underlying system, one or more of the following interactions may be provided during the operation of the one or more user interfaces: displaying one or more user interface elements for specifying predetermined data and predetermined amounts; outputting via one or more displayed user interface elements whether the one or more actions are (i) completed successfully or (ii) failed; and outputting the result data via one or more displayed user interface elements. The user interface may receive the predetermined data and predetermined amounts from the user (e.g., via icons as graphical elements).

[0133] Alternatively, one or both of the predetermined data and the predetermined amount are stored in a memory accessible and retrievable by the system. The user interface may graphically output (e.g., via an icon as a user interface element) whether the one or more actions are (i) successfully completed or (ii) failed. The user interface may graphically output (e.g., via an icon as a user interface element) result data for user inspection or automatic use by the system. Via one or more displayed (graphical) user interface elements, one or more specified values ​​(these specified values ​​may be based on the result data) of a user-specified change of one or more parameters of the first parameterized application, the second parameterized application, and / or the third parameterized application are received and the first parameterized application, the second parameterized application, and / or the third parameterized application may be automatically re-executed by the system (e.g., system 702) with the one or more specified values ​​of the change. One or more instructions for causing (re)execution of the one or more actions are automatically executed by the third parameterized application using the output of the execution of the one or more actions specified as input. The one or more instructions may be sent via a network connection to cause (re)execution of the one or more actions on an external device.

[0134] The user interface may provide a graphical element that automatically initiates re-execution of the changed value upon activation of the graphical element initiated by the user. For example, the result data may be provided to the user on the user interface, and the user may identify or be provided with information that the result data is not as expected based on the correct execution of the one or more actions. The user or system (e.g., system 702) may then initiate a re-execution action of the changed value in a correction loop, wherein the changed value is calculated by the system (or input by the user) based on the result data to ensure the correct / desired execution of the action. This may help ensure the correct operation of the underlying system, for example, by ensuring that the one or more actions are correctly executed (i.e., as required).

[0135] As described herein, CDA system 702 provides end-to-end operational robustness (e.g., because CDA system 702 can be utilized to process data for warehousing / analysis, as well as for operational systems) and virtually unlimited scalability on commodity hardware, e.g., due to reuse of the application, due to in-memory state and profiles, and due to the wide record of data records generated and then published multiple times to various applications including, e.g., applications 720, 722, 724, 726.

[0136] refer to Figure 5B , the networking environment 751 is Figure 5A. In this example, networked environment 751 includes client system 792, networking system 768, external input system 769 (for streaming data or inputting data), external output system 770 (for receiving output data), and CDA system 753 with acquisition application 780, application 755 (including detection application 775 and action application 776), profile data structure 771, and data analysis application 782. In this application, data analysis application 782 includes data repository 765 (e.g., data lake) for storing data (e.g., in native format), data warehouse 784, analysis application 786 (e.g., for implementing machine learning and data-related analysis engines), and visualization data 763 (e.g., data for generating visualizations of data analysis, profile data, etc.). The data analysis application 782 also includes a query engine 767 for querying the data repository 765, the data warehouse 784, the profile data structure 771, etc., for example, to query data for processing or analysis by the analysis application 786 and / or inclusion in the visualization data 763. In the example, the profile data structure 771 is stored in volatile memory (e.g., to reduce memory storage requirements and, for example, to reduce the latency of retrieving profile data relative to the latency of retrieving profile data when the profile data is stored to disk) and / or in non-volatile memory (e.g., stored in the data repository 765 and / or the data warehouse 784). The CDA system 753 also includes a data layer 788 (e.g., a service layer) for transmitting portions of near real-time profile data from the profile data structure 771 (e.g., in batch or real-time form) to the external output system 770 and / or the external input system 769.

[0137] In operation, one or more of the client systems 792 transmit data (e.g., data packets) including events (or indicating events) to the networking system 768 and the external input system 769. In an example, one of the client systems 792 is a smart phone. One of the external input systems 769 is a telecommunications system of a telephone company. In this example, the smart phone sends a text message. When sending a text message, the smart phone also sends data (specifying the sending of a text message) to the telecommunications system. In this example, the data specifying the sending of a text message is an event. In turn, the telecommunications system transmits the event to the collection application 780. Typically, the collection application 780 is configured to collect data records (e.g., in batch and / or real-time form) from data sources and data streams. In this example, the collection application 780 transmits the received data records to the application 755. In this example, the collection application 780 transmits the received data records in batches and in real time by filling the queue 759 with the collected data records.

[0138] In this example, the detection application 775 is configured to execute various rule sets 790a, 790b, 790c, ..., 790n. In this example, each rule set executes a specific activity, a loyalty program, a fraud detection program, etc. The detection application 775 executes various rule sets to the data records received from the acquisition application 780. In this example, the detection application 775 includes a profile repository 773, which is used to dynamically update the user profile (e.g., in a memory) with the data included in the received data record. In this example, for example, when profile data is needed to execute a rule set and / or determine whether one or more conditions of a rule set are met, the profile data structure 771 intermittently pushes the profile data (e.g., associated with a specific key and / or all keys) to the profile repository 773, which stores the profile data in a memory to reduce the detection application 775 to the profile data retrieval time. When the detection application 775 receives a new data record, the detection application 775 updates the profile repository 773 with the profile data of the appropriate profile (e.g., by matching the key associated with the received data record with the key associated with the profile data stored in the profile repository 773). In the example, specific fields in the data record include the profile data. In this example, the profile repository 773 is updated with the contents of those fields (e.g., for the appropriate keys). The profile repository 773 intermittently pushes its updated profile data to the profile data structure 771 to update the profile data structure 771.

[0139] Based on the execution of one or more rule sets, detection application 775 identifies one or more actions to be performed. In this example, detection application 775 issues instructions to queue 761 so that those one or more actions are performed. The instructions in queue 761 are transferred to action application 776, which includes profile repository 772, and profile data is stored in memory by this profile repository. In this example, profile repository 772 retrieves profile data from profile data structure 771 intermittently, and stores the retrieved profile data in memory. Profile repository 772 does this to realize the time delay retrieval profile data by action application 776 with the reduction of the time delay of the time delay retrieved relative to the profile data disk. For each of the actions specified in the instruction, action application 776 performs the action or causes the action to be performed (for example, by filling queue 774 with execution instructions and transmitting those execution instructions in queue 774 to external output system 770). In either example, the action application 776 uses the profile data in the profile repository 772 to customize the executed action (e.g., using profile data specific to the recipient of the action) or customize the execution instruction (e.g., by adding profile data specific to the recipient of the action). By storing the profile data in the profile repository 772 in memory, rather than storing it to disk, the action application 776 retrieves the profile data with a reduced latency relative to the latency when retrieving the profile data from the disk. Based on this reduced latency, the action application 776 can add profile data to instructions and / or actions in near real time. In some examples, the action application 776 updates the profile repository 772 with new profile data (e.g., data specifying that an invitation is sent to a specific user). The profile repository 772 intermittently pushes updates to the profile data and / or copies of the profile data stored in the profile repository 772 to the profile data structure 771. Based on updates and / or profile data received from the profile repositories 773, 772, the profile data structure 771 maintains near real-time customer profiles. In this example, the profile data structure 771 also receives profile data from the data repository 784, further enabling the profile data structure 771 to maintain versions of the profile data for specific and / or all keys.

[0140] In this example, the CDA system 753 is configured to push (e.g., in batch and / or real-time) the profile data in the profile data structure 771 to the collection application 780, which in turn transmits the profile data to the external input system 769 and / or the networked system 768. The CDA system 753 is also configured to push (e.g., in batch and / or real-time) the profile data from the profile data structure 771 to the data layer 788 for transmission to the external output system 770 as shown by the data flow 757, the networked system 768 (e.g., to implement operational scheduling, monitoring, audit trails, and execution tracking based on the data included in the profile data (e.g., data specifying what actions were performed and when those actions were performed)), and the external input system 769 as shown by the data flow 791 (e.g., to implement data governance and data management (e.g., data archiving and quality and data lineage) based on the metadata and reference data included in the profile data). In this example, the profile data stored in the profile data structure 771 includes data lineage data and data representing transformations and other operations performed on one or more portions of the profile data (eg, associated with a particular key).

[0141] In this example, the collection application 780 and the data analysis application 782 are configured to transfer data to each other in batches. In doing so, the collection application 780 pushes the collected data records and other data received from the systems 768, 769 to the data analysis application 782 for storage in the data repository 765 and / or the data warehouse 784. Similarly, the data analysis application 782 transmits analysis data and / or other stored data to the collection application 780, for example, for subsequent transmission to one or more of the systems 768, 769, for example, to facilitate data integration across various systems.

[0142] refer to Figure 6 , graphical user interface 800 shows a flexible wizard for configuring a detection module, such as by defining initial settings for a program for performing a detection, such as by configuring parameters such as duration of an activity or program, data logging, data sources, and definition of a target population. In this example, graphical user interface 800 includes a portion 802 for specifying one or more properties of a program (e.g., a program for activity execution), a portion 804 for specifying one or more properties and / or parameter values ​​of an event defined for detection, a portion 806 for specifying one or more properties and / or parameter values ​​for performing the detection, and a portion 808 for specifying a state transition of one or more properties and / or parameter values ​​for viewing results (e.g., the transition is part of the detection).

[0143] In this example, portion 804 includes sub-portions 804a, 804b, 804c for specifying values ​​of various parameters for performing detection (e.g., included in parameterized logic or parameterized applications). For example, sub-portion 804a enables a user to specify which types of data records are to be detected (e.g., data records or data records with specific properties). Sub-portion 804b enables a user to specify keys for which detection is not to be performed, for example, to filter records associated with a specific key. Sub-portion 804c enables a user to specify which records will be used as test records, for example, to test the processing of data records by an application in real time. Portion 806 implements configuring which rules and / or logic will be executed by the detection module.

[0144] refer to Figure 7 , showing Figure 6 In this example, portion 804 includes a selectable portion 812 (e.g., a link), selection of which causes overlay 810 to be displayed. In this example, overlay 810 includes one or more selectable portions (e.g., check boxes) for specifying which data records are to be processed (e.g., by applying rules) with respect to those data records. In this example, the data entered into overlay 810 is displayed in the form of Figure 2C The described approach functionally specifies parameter values, for example, in the same manner that the interface 250 sets parameter values ​​for a base map and / or parameterized applications (or parameterized logic) and / or applications.

[0145] refer to Figure 8 , the graphical user interface 900 provides the configuration of the acquisition module and shows the correspondence between the parts 902, 904, 906, 908 of the configuration interface 901 and the parts 912, 914, 916, 918 of the data flow diagram 910 for setting parameter values ​​through the configuration interface 901, for example, implementing the above FIG. 2A to FIG. 2C In this example, graphical user interface 900 illustrates a general purpose application that can be configured by a non-developer. In this example, data flow graph 910 includes an application for collecting, transforming, and enriching data records received from multiple sources. In this example, data flow graph 910 includes an application with various parameters, the values ​​of which are set through configuration interface 901. When parameter values ​​of data flow graph 910 are specified, the CDA system generates an instance of the data flow graph, e.g., an instance of the data flow graph 910 in which the parameter values ​​in the data flow graph are set to the values ​​specified by the configuration interface.

[0146] In this example, section 902 of configuration interface 901 specifies which source files the configuration module collects. Input to section 902 specifies parameter values ​​included in section 912 of data flow graph 910. Based on the selection of one or more selectable sections in section 902, the user specifies the source for collecting data and data records. Section 904 specifies conversion, for example, reformatting. As described herein, the CDA system handles the complexity of arbitrarily large data volumes, low latency, and multiple data formats. In this example, section 904 provides processing for multiple data formats. Input to section 904 specifies parameter values ​​included in section 914 of data flow graph 910. Input to section 906 specifies data transformation, for example, enriching and adding profile data to wide records. In this example, section 906 corresponds to section 916 of data flow graph 910 and specifies the values ​​of one or more parameters included in section 916 of data flow graph 910. In this example, section 908 includes one or more selectable sections for specifying the values ​​of one or more parameters included in section 918 of data flow graph 910. Section 908 is used to specify parameter values ​​related to the output of data (eg, what data is to be output) and instructions for transmission to various devices.

[0147] refer to Fig. 9 , graphical user interface 1000 displays controls for configuring the detection module. Graphical user interface 1000 includes a portion 1002 for displaying inputs available in graphical user interface 1000, for example, inputs to defined logic. In this example, portion 1002 defines an option board, as described in U.S. application Ser. No. 62 / 270,257. The option board includes all data records included in a wide record (e.g., a wide record generated from a collection record received by a CDA system). In this example, the option board includes inputs 1026, 1054 (e.g., data records) and enriched inputs 1052 (e.g., enriched data records). Typically, enriched inputs include inputs based on data retrieved or collected from a data storage device or storage device in memory, for example, rather than data received from outside the system. In this example, the inputs include pre-computed aggregations, for example, as described in U.S. application Ser. No. 62 / 270,257. For input 1026, there are various types of inputs, including inputs 1028, 1030, 1032, 1034, 1036, 1038, 1040, 1042, 1044, 1046, 1048, 1050. Each of these types of inputs is included in a wide record. Each input displayed in portion 1002 is selectable (e.g., by dragging and dropping) for input into a cell in portion 1004, for example, for defining logic.

[0148] The graphical user interface 1000 also includes a portion 1004 for generating logic (e.g., rules) to be executed by the detection module. In the example, portion 1004 includes a business rule editor. In this example, portion 1004 displays a visualization of one or more parameterized applications (e.g., detection applications and action applications). Portion 1004 also displays editable units for specifying parameter values ​​in (multiple) parameterized applications. In the example, for example, when inputting parameter values ​​of (multiple) parameterized applications, the business rule editor and / or portion 1004 are used to define specifications. That is, the input to portion 1004 and the rules specified by portion 1004 together form specifications.

[0149] In this example, portion 1004 includes a logic portion 1006 for specifying one or more triggers or conditions, the satisfaction of which causes one or more actions to be performed (e.g., by causing an output to be implemented). Logic portion 1008 specifies various outputs to be initiated and / or performed when the conditions or triggers are met. In this example, logic portion 1006 includes a state portion 1006a that specifies one or more state triggers, for example, a condition that specifies when the state of a particular key corresponds to a particular value or state type for which the detection module should perform a corresponding action. Logic portion 1008 also includes a state portion 1008a that specifies a new state to which the logic transitions, for example, when the corresponding trigger is met.

[0150] In this example, logic portion 1006 includes logic sub-portions 1010, 1012, 1014, 1016, each of which specifies one or more conditions. For example, logic sub-portion 1010 specifies a condition that the user associated with the keyed data being processed is a new user. Logic sub-portion 1012 specifies a condition that the state of the keyed data has a "CampaignState.Eligible" value of "is_null". Logic sub-portion 1014 specifies a condition that the keyed data being processed has any data record type. Logic sub-portion 1016 specifies a condition that the SMS usage rate is greater than 500. In this example, the combination of logic sub-portions 1010, 1012, 1014, 1016 together form a trigger, the satisfaction of which causes the detection module to execute and / or implement the corresponding output. In this example, logical subsections 1010, 1012, 1014, 1016 correspond to logical subsections 1018, 1020, 1022, 1024, 1025, which together specify a particular output. The detection module generates instructions indicating a particular output and sends those instructions to the action module, for example, to implement the output and / or customize the output and send data indicating the customized output (or the customized output itself) to the external system for execution. In this example, logical subsection 1018 specifies that the state of the executable logic is updated to a "New Campaign State.Send Offer" value. Logical subsection 1020 specifies the action of sending an SMS message. Logical subsection 1022 specifies the content of the message. Logical subsection 1024 specifies an expiration date or value for which the message is valid. Logical subsection 1025 specifies a fulfillment plan code. In this example, the combination of logical subsections 1018, 1020, 1022, 1024, 1025 together form an output that is implemented, for example, when the conditions specified in logical subsections 1010, 1012, 1014, 1016 are met.

[0151] In the example, the palette displayed in section 1002 is used to generate the logic included in logic subsections 1012, 1016, 1018, 1020, 1022, 1024. For example, the logic in logic subsection 1016 is generated by dragging and dropping data record 1040 into logic subsection 1016 (which comprises an editable cell in this example). The user would then further edit logic subsection 1016 by entering text of ">500" into the editable cell that is logic subsection 1016. In this example, data record 1040 corresponds to a parameter in the associated dataflow diagram or application. The user editing logic subsection 1016 then enters a value for the parameter, i.e., a value of ">500". Using the logic displayed in section 1004, the CDA system generates an application or dataflow diagram to implement the logic, for example, by setting the parameter values ​​in the parameterized logic to those values ​​displayed in section 1004. In some examples, the executable logic may be specified as a flow chart rather than in a table.

[0152] refer to Fig.10 , specification 1100 includes a flow chart 1102 having nodes 1102a to 1102g. Typically, the flow chart includes an application for processing data records, such as a parameterized application. In this example, the specification of the parameter values ​​of the parameterized application (e.g., input) generates the specification. As described above, the specification represents executable logic and specifies various states of the executable logic based on the state reached from executing the executable logic on the previous data item. Typically, the executable logic includes source code and other computer instructions. Each node in the figure represents one or more parts of the executable logic. For example, the node includes one or more logical expressions (hereinafter referred to as "logic") that generate the executable logic. In another example, when the executable logic is in a specific state, the node corresponds to one or more specific parts of the executable logic. In this example, the executable logic in the figure is generated using an option board (described above), for example, to select various inputs to be included in one or more nodes of the figure.

[0153] This application (represented by the diagram) includes a logic graphical unit for reacting to input data records and producing output data records, e.g., data records generated based on logic contained in a specification. Typically, the logic graphical unit includes logic generated at least in part graphically, e.g., by dragging and dropping various nodes from an application (not shown) into a window for building the diagram. In an example, the node includes logic (not shown) that specifies how to process input data records, how to set variable values ​​used by executable logic, e.g., which output data records are generated when conditions specified by the logic are met, etc. In an example, the node is programmable by a user who enters parameter values ​​and / or variable values ​​used in the node's logic.

[0154] The graph itself is executable, because the logic in the node is compiled into executable logic, and each node corresponds to one or more parts of the executable logic. For example, the system transforms the specification (and / or the graph in the specification) by compiling the logic in the node into executable logic. Because the graph itself is executable, the graph itself can process data records, and can be stopped, started and paused. The system also maintains the state of the flowchart 1102, for example, by tracking which one of the nodes 1102a-1102g is currently being executed. The state of the flowchart 1102 corresponds to the executable state represented by the flowchart 1102. For example, each node in the flowchart 1102 represents a specific state of executable logic (wherein one or more parts of the executable logic are executable in this state). When the flowchart 1102 is executed for multiple values ​​of a key, the system maintains the state of the flowchart 1102 for each value of the key, for example, by maintaining the state of each instance described in further detail below. In this example, the flowchart 1102 includes a state transition diagram, wherein each incoming data record drives the transition between nodes, and the data record is evaluated based on the state reached by processing the previous data record. The links between the nodes in the flowchart 1102 represent the temporal flow of logic.

[0155] Node 1102a represents the beginning of executable logic.After node 1102a is completed, the state of flow chart 1102 is transformed to the node 1102b representing one or more other parts of executable logic.Node 1102b includes a waiting node (hereinafter referred to as waiting node 1102b).Wait for a part of node 1102b representing executable logic (corresponding to wait for node 1102b) to wait for the waiting state of the input data record that satisfies one or more conditions.In an example, the waiting state can be a part of another state of flow chart 1102, for example, the system executes the waiting node (to implement the waiting state) and then executes the state of one or more other nodes.After completing the part represented by the waiting node 1102b of executable logic, the system exits the waiting state and executes the node 1102c representing the executable logic for implementing decision-making.In this example, node 1102c includes a decision node.Usually, a decision node includes a node comprising the logic (for example, the logic of seeking Boolean value) for executing decision-making.

[0156] Based on the result of the decision, the state of the flowchart 1102 transitions to node 1102g (which causes the state to transition back to node 1102a) or node 1102d, which is another wait node. After completing the portion of the executable logic represented by the wait node 1102d, the state of the flowchart 1102 transitions to node 1102e, which includes a send node. Typically, a send node includes a node representing executable logic for transmitting data to another system. After completing the portion of the executable logic represented by node 1102e, the state of the flowchart 1102 transitions to node 1102f, which includes a finish node. Typically, a finish node indicates that the execution of the executable logic is complete.

[0157] In the example, the wait node represents a transition between states, for example, a transition from one state to another state, the beginning of which is a wait node. In this example, the flowchart 1102 is different from a state transition diagram because not every node in the flowchart 1102 represents a wait node representing a state transition. Instead, for example, when the flowchart 1102 is already in a particular state, some nodes represent actions to be performed. In some examples, the system processes the flowchart 1102 to generate a state machine diagram or state machine instructions.

[0158] In this example, the flowchart 1102 includes two states, a first state represented by nodes 1102b, 1102c, 1102g, and a second state represented by nodes 1102d, 1102e, 1102f. In this first state, the system waits for a particular data record (as represented by node 1102b), and then executes node 1102c, which in turn causes a transition (of specification 1100 and / or diagram 1102) to a second state (which begins to be represented by node 1102d), or causes the execution of node 1102g. Once in the second state, the system again waits for a particular data record (as represented by node 1102d), and then executes nodes 1102e, 1102f. By including nodes other than the wait node, the flowchart 1102 includes a logical diagram of the temporal processing of data records. In this example, graph 1102 includes link 2i, which represents the transition of graph 1102 from a first state to a second state and also represents the flow of data from node 1102c to node 1102d.

[0159] Figure 1102 also includes a link 1102j between nodes 1102a, 1102b and a link 1102k between nodes 1102b, 1102c to represent a user-specified execution order for the portion of the executable logic corresponding to nodes 1102a, 1102b, 1102c in the first state. In this example, the portion of the executable logic in the first state (hereinafter referred to as "first state executable logic") includes statements (e.g., logical statements, instructions, etc. (collectively referred to as "statements" herein, but not limited to this)). Typically, the execution order includes the order in which the executable logic and / or statements are executed. Each of nodes 1102a, 1102b, 1102c corresponds to one or more of those statements (e.g., corresponding to one or more portions of the first state executable logic). In this way, link 1102j represents the execution order of the first state executable logic by indicating that the statements in the first state executable logic represented by node 1102a are executed by the system before other statements in the first state executable logic represented by node 1102b are executed. Link 1102k also represents the execution order of the first-state executable logic by representing that statements in the first-state executable logic represented by node 1102b are executed by the system before other statements in the first-state executable logic represented by node 1102c are executed.

[0160] The specification 1100 also includes a key 1102h for a data record that includes or is associated with a key 1102h, as identified in the flow chart 1102. In this example, a custom identifier (ID) is used as the key. The key 1102h may correspond to one of the fields of the data record (i.e., a data record field), such as, for example, a subscriber_ID field, a customer_ID field, a session_ID field, etc. In this example, the customer_ID field is the key field. For a particular data record, the system determines the value of the key of the data record by identifying the value of the key field of the data record.

[0161] In this example, flowchart 1102 subscribes to a data record of a specified type (e.g., specified when configuring flowchart 1102). In this example, flowchart 1102 subscribes to a data record including key 1102h. In this example, flowchart 1102 and data record share a key. Typically, a flowchart subscribes to a type of data record by including logic for processing those data records including the key of the flowchart. When data record processing begins, the system starts a new flowchart instance for each new value of the key of the flowchart, for example, by maintaining the state of the executable logic for each new value of the key (represented in the flowchart). The system performs data record processing by configuring the flowchart instance (and therefore the underlying executable logic) to respond to data records of specific key values. In the example, the flowchart subscribes to a customer short message service (SMS) data record. The flowchart instance for a specific customer ID manages the data record of the customer. The number of flowchart instances can be as many as the number of customer IDs encountered in the incoming data record. In some examples, the system described herein provides a user interface for the configuration of flowchart 1102, for example, to enable a user to easily enter values ​​into the various components of the flowchart.

[0162] refer to Fig.11 , FIG. 1107 shows, for example, a system according to flowchart 1102 ( Fig.10 ) The flowchart instances 1103, 1104, 1105 and data records 1106a, 1106b, 1106c generated by the flowchart 1102. That is, a new copy or instance of the flowchart 1102 is created for each new key detected in the data records 1106a, 1106b and 1106c.

[0163] Each of the flowchart instances 1103, 1104, 1105 is associated with a "customer_id" key. Flowchart instance 1103 processes a data record that includes a value "VBN3419" in its "customer_id" field, which is a key field in this example. Flowchart instance 1104 processes a data record that includes a value "CND8954" in its "customer_id" field. Flowchart instance 1105 processes a data record that includes a value "MGY6203" in its "customer_id" field. In this example, the system does not re-execute the executable logic for each flowchart instance. Instead, the system executes the executable logic and then implements the flowchart instance by maintaining the state of the corresponding value of the key. Therefore, the example of a "processing data record flowchart instance" is a system that executes executable logic (represented by a flowchart), maintains the state of each value of the key, and processes data records associated with a specific value of the key (based on the state of the state machine of the specific value of the key).

[0164] In this example, flowchart instance 1103 includes nodes 1103a to 1103g, which correspond to Fig.101102a to 1102g in FIG. 1104 includes nodes 1104a to 1104g, which correspond to Fig.10 1102a to 1102g in FIG. 1105 includes nodes 1105a to 1105g, which correspond to Fig.10 Nodes 1102a to 1102g in.

[0165] The flowchart instance itself is executable. After the system receives the input data record associated with the specific value of the key, the flowchart instance of the specific value of the key processes the input data record, for example, by executing one or more parts of the executable logic corresponding to the flowchart instance (or one or more nodes of the flowchart instance) through the system. The flowchart instance continues to process the input data record until the input data record reaches the completion node or the waiting node. In this example, the flowchart instance continues to process the input data record, for example, by the system continues to process the input data record until the part corresponding to the completion node or the waiting node of the executable logic is reached. If the input data record arrives at the waiting node, the flowchart instance will pause until a certain amount of time has passed or a suitable new input data record arrives. Typically, suitable data records include data records that meet one or more specified conditions or criteria (for example, included in the logic of the node). If the input data record arrives at the completion node, the execution of the flowchart instance is completed.

[0166] A flowchart instance has its own lifecycle. When a data record arrives, one or more current states of the flowchart instance change: the data record triggers a decision, either returning to the beginning of the flowchart instance or sending a message to the client. When the data record reaches the completion node, the client's flowchart instance ends.

[0167] In this example, the system starts flowchart instance 1103 to obtain a "VBN3419" value for a key field (e.g., customer_id=VBN3419). Flowchart instance 1103 processes a subset of data records 1106a, 1106b, 1106c that include customer_id VBN3419. In this example, flowchart instance 1103 processes data record 1106a having a value "VBN3419" in the customer_id key field. Nodes 1103a, 1103b, 1103c of flowchart instance 1103 process data record 1106a. The current state of flowchart instance 1103 is a state of waiting for a data record, as indicated by the dashed line of node 1103d. When node 1103d is reached, flowchart instance 1103 waits for another data record with customer ID=VBN3419 to be processed by nodes 1103d, 1103e, 1103f of flowchart instance 1103.

[0168] The system starts flowchart instance 1104 for the "CND8954" value of the key (e.g., customer_id=CND8954). Flowchart instance 1104 processes a subset of data records 1106a, 1106b, 1106c that include customer_id CND8954. In this example, flowchart instance 4 includes wait nodes 1104b and 1104d. Each data record can only meet the conditions of one wait node per flowchart instance. Therefore, flowchart instance 1104 processes data record 1106b with customer_id=CND8954 through node 1104b to node 1104d, and then waits for a second data record with the same key before proceeding to node 1104f. The system starts flowchart instance 1105 for the "MGY6203" value of the key (e.g., customerID=MGY6203). Flowchart instance 1105 processes a subset of data records 1106a, 1106b, 1106c including customer_id MGY6203. In this example, flowchart instance 1105 processes data record 1106c with customer_id=MGY6203 through nodes 1105b to 1105d, and then waits for a second data record with the same key before proceeding to node 1105e, which sends a message. In this example, the system does not stop at a non-wait node, and therefore does not stop at node 1105e, but instead sends a message and then proceeds to node 1105f.

[0169] exist Fig.11 In a variation of , the system generates more than one flowchart instance for a single key value. For example, for the same customer, there may be multiple flowchart instances with different start and end dates, or for the same user, there may be multiple flowchart instances for different marketing campaigns.

[0170] In this example, the system maintains the state of the instance by storing state data (e.g., data indicating which node is currently being executed for each instance) in a data repository or a data grid in memory. Typically, the state data includes data indicating the state. In this example, the instance is associated with the value of the key. The data repository or the data grid in memory stores the value of the key. The system maintains the state of the instance by storing state data for each value of the key. After completing the processing of the data record for a specific value of the key, the system updates the state data in the data repository to specify that the next node (in flowchart 1102) represents the current state of the value for the key. Then, when another data record arrives, the system looks up the current state of the value for the key in the data repository and executes the portion of the executable logic corresponding to the node representing the current state of the executable logic for the value of the key.

[0171] refer to Fig.12, the logic flow 1112 is based on the execution of logic (e.g., rule-based logic generated using the above-mentioned palette) that uses wide records generated using the above-mentioned techniques in its execution. In this example, the logic flow 1112 specifies various data record triggers and actions based on data records included in the data records of one or more specific subscribers. The logic flow 1112 includes different decision points (e.g., "Did the subscriber consume fifty SMS messages?"). For a specific key, the detection module decides which branch of the logic flow 1112 to traverse based on the data records (or lack of data records) included in the data records of the subscriber and based on the state of the executable logic for the key. Generally, the state refers to a specific component of the logic that has been transformed during the execution of the logic (e.g., a specific data record trigger or a specific action). For example, the state specifies which data record trigger or action in the logic is currently being executed for a specific key. In some examples, the detection module waits for a specified period of time before selecting a branch in the logic flow 1112. By waiting for these specified time periods, the detection module analyzes new data records inserted into the data records.

[0172] In this example, the logic flow 1112 includes a data record trigger 1119 that specifies that upon activation of a particular subscriber's service, the action module performs a start action 1120 that monitors the amount of SMS messages consumed by the particular subscriber over a two-day period. In this example, the data record trigger 1119 is a prerequisite for a rule executed by the logic flow 1112. When the data record trigger 1119 is satisfied, the action module performs the start action 1120. The detection module determines when a particular subscriber satisfies the data record trigger 1119 by detecting an activation data record in a wide record, and determines the subscriber associated with the activation data record (via a subscriber ID).

[0173] In this example, when a subscriber has consumed at least fifty SMS messages in the past two days (e.g., as specified by an SMS usage data record aggregation in the data record), a data record trigger 1113 is executed. The data record trigger 1113 executes a suggested reload action 1114, which causes the action module to prompt this particular subscriber to reload. When the subscriber does perform a reload, the entry in the data record for the particular subscriber is updated with a data record indicating the reload. This update of the data record causes the logic flow 1112 to execute a data record trigger 1115, which specifies that upon a successful reload an action 1116 is executed to send a package suggestion SMS to the subscriber. Typically, a package suggestion is a suggestion to purchase a package or bundle of services.

[0174] When the user sends a response to the package suggestion SMS, the data record is updated with a data record representing the response and a data record representing that the response was received in less than three hours. The detection module detects the update in the data record and causes a data record trigger 1117 to execute. When the subscriber has completed the package purchase, the data record trigger 1117 specifies when the response (for that particular subscriber) is received in less than ten hours of executing an action 1118 that ends the activity. When the entry in the data record for a particular subscriber specifies that the particular subscriber did not send a response to action 1116, the logic flow 1112 also specifies an action 1125 that ends the activity for that particular subscriber.

[0175] In an example, an entry in the data record for a particular subscriber specifies that the subscriber has not performed a reload, for example, by the absence of a reload data record or by specifying that there is no export data record for a reload. In this example, the logic flow 1112 specifies a data record trigger 1123 that waits for three hours, for example, to monitor whether the user performs a reload within the next three hours. After three hours, the data record trigger 1123 causes a reminder action 1124 to be executed that sends a reminder SMS for the reload to the subscriber. If the subscriber does not respond to the reminder SMS, the logic flow 1112 specifies an action 1126 that ends the activity for the particular subscriber.

[0176] In response to action 1120, the entry for a particular subscriber may specify that the subscriber has not consumed at least fifty SMS in the past two days. The entry may specify this by specifying an exported data record that does not consume fifty SMS or by specifying an SMS usage aggregate data record that consumes less than fifty SMS. In this example, logic flow 1112 includes a data record trigger 1121 that waits five days and then implements an action 1122 to send a reminder SMS. If, after sending the reminder, the subscriber still does not consume fifty SMS within another five days (e.g., as specified by the data record for the subscriber in the data record), then logic flow 1112 specifies that a data record trigger 1127 executes an action 1128 that sends an alert to a customer recovery team (e.g., to notify the team that the consumer is not using the service) and ends the activity for the particular subscriber.

[0177] refer to Fig.13 , a system (e.g., a CDA system described herein, e.g., system 100) implements process 1200 when processing data items in a plurality of different data streams. In operation, the system accesses (1202) a first parameterized application, a second parameterized application, and a third parameterized application (e.g., parameterized applications). In this example, each of the parameterized applications includes one or more parameters that define one or more properties of the parameterized application.

[0178] The system (or the collection module 114 of the system) executes (1204) the first parameterized application using one or more specified values ​​of one or more parameters of the first parameterized application to implement a collection module (e.g., the collection module 114 of the system) for processing data records. In the example, the first parameterized application includes the data flow graph 910 ( Fig. 9 ). In this example, values ​​for one or more parameters of the dataflow graph 910 are specified by entering values ​​into one or more of the portions 902, 904, 906, 908. In this example, the portions 902, 904, 906, 908 provide for entering one or more values ​​into one or more selectable and / or editable regions of the portions 902, 904, 906, 908. These selectable and / or editable regions are mapped or otherwise associated with parameters of the dataflow graph 910, for example, to enable setting of values ​​for these parameters.

[0179] The system (or the acquisition module 114 of the above system) performs processing by acquiring (1206) data items (e.g., data records) from multiple different (e.g., different) data streams. In this example, the format of the first part of the data items acquired from one stream is different from the format of the second part of the data items acquired from a different stream. In addition, the data items are associated with the value of the key and can therefore be referred to as keyed data. The system (or the acquisition module 114 of the above system) also transforms (1208) the first part of the data item and the second part of the data item according to one or more specifications applied by the first parameterization. In the example, the system transforms the data item (or part thereof) by formatting the data into a format suitable for the acquisition module, the detection module and / or the action module (e.g., the acquisition module 114, the detection module 116 and / or the action module 118). By doing so, the system solves the problem of data integration and preparation for data record management without the need for additional technologies as described above. As previously described, this capability simplifies and accelerates end-to-end integration. Specifically, the received data record only needs to be formatted and verified once (e.g., by the acquisition module 114), and can then be processed by the acquisition module, the detection module, and the action module (e.g., the above-mentioned acquisition module 114, the detection module 116 and / or the action module 118). In conventional methods, independent systems perform acquisition, detection, and action. As such, the data needs to be formatted for the acquisition aspect, then formatted again into a format suitable for the detection aspect, and then formatted again into a format suitable for the action aspect. This repeated formatting and reformatting introduces significant latency in the execution of the acquisition, detection, and action, and records real-time actions (or near real-time) when receiving the data record. This repeated formatting and reformatting also consumes a large amount of bandwidth and system memory resources. Therefore, in contrast to conventional methods, the system described herein provides reduced bandwidth and memory consumption, as well as reduced latency in processing - which provides real-time action execution.

[0180] The acquisition module also performs transformations by parsing, validating, and enriching (as appropriate) the received data records with slower changing data (e.g., profile data from a data store). As previously described, such enrichment includes generating all data records in a data record palette (or a subset thereof) to enable real-time execution of logic for the received data records without the latency of performing a database lookup to retrieve the data required for logic execution. The system stores these validated and enriched data records in memory, for example, to enable real-time detection. The system also stores these validated and enriched data records to disk for archiving and to ensure graceful recovery. The acquisition module is configured to handle a large number of data sources (and / or virtually any data source), thereby enabling rapid and independent integration into the CDA system while handling the complexity of arbitrarily large data volumes, low latency, and multiple data formats. After transforming the acquired data records, the acquisition module populates (1210) a queue with entries representing the transformed data items. The queue then transmits the data items to a second parameterized application, for example, a detection module.

[0181] In this example, the system executes (1212) a second parameterized application, e.g., to implement a detection module (e.g., detection module 116 described above). Specifically, the system executes the second parameterized application using one or more specified values ​​for one or more parameters of the second parameterized application to process the transformed data items in the queue. In this example, the second parameterized application represents a specification including rules and corresponding conditions for the rules. Additionally, the state of the specification is maintained for corresponding values ​​of the keys.

[0182] In an example, processing transformed data items in a queue includes the following operations. For one or more transformed data items associated with a particular value of a key, the system (and / or a detection module of the system) detects (1214) that at least one of the one or more transformed data items satisfies one or more conditions of at least one of the rules of the specification. In response to the detection, the detection module generates an output specifying the performance of one or more actions and publishes the output to the queue. For example, the output can be a message or invitation to a user. In an example, the output and one or more user actions (or lack of user actions) with respect to the output are tracked by the CDA system. The tracking of these user actions provides a feedback lookup to the CDA system, wherein the CDA system tracks the output and then receives data (e.g., from an external system or the CDA system itself) specifying the delivery of the output, one or more user interactions with the output, one or more actions with respect to the output, and the like. The detection module also causes (1216) the specification with respect to the particular value of the key to transition from its current state to a subsequent state. For example, Fig. 9Portion 1004 of the diagram shows a state portion 1006a indicating a current state of the executable logic. Portion 1004 also shows a state portion 1008a indicating a subsequent state or state to which the executable logic may transition.

[0183] In this example, the system also executes (1218) a third parameterized application with one or more specified values ​​for one or more parameters of the third parameterized application to perform operations, including sending one or more instructions to cause one or more actions to be performed. In this example, the third parameterized application implements an action module. As previously described, this action module customizes output and sends it to a user and / or transmits the customized output to an external system.

[0184] The above techniques can be implemented using software so as to be executed on a computer. For example, the software forms a program in one or more computer programs that is executed on one or more programming or programmable computer systems (which can be various architectures such as distributed, client / server or grid), each of which includes at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. The software can form one or more modules of a larger program, for example, to provide other services related to the design and configuration of diagrams and flow charts. The nodes, links, and elements of the diagram can be implemented as data structures stored in a computer-readable medium or other organized data that conforms to a data model stored in a data repository.

[0185] The techniques described herein may be implemented in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The apparatus may be implemented in a computer program product tangibly presented or stored in a machine-readable storage device (e.g., a non-transitory machine-readable storage device, a machine-readable hardware storage device, etc.) for execution by a programmable processor; and the method actions may be performed by a programmable processor executing an instruction program to perform functions by operating on input data and generating outputs. The embodiments described herein and other embodiments of the claims and the techniques described herein may be advantageously implemented in one or more computer programs executable on a programmable system, the programmable system comprising at least one programmable processor coupled to receive data and instructions from and transmit data and instructions to a data storage system, at least one input device, and at least one output device. Each computer program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language if desired; and in any case, the language may be an assembly or interpreted language.

[0186] Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. Essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to receive data from or transfer data to a mass storage device, or both receive and transfer data. Computer-readable media for embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into dedicated logic circuits. Any of the above may be supplemented by or incorporated into an ASIC (Application Specific Integrated Circuit).

[0187] To provide interaction with a user, embodiments may be implemented on a computer having a display device (e.g., an LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input.

[0188] Embodiments may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the embodiments), or any combination of these back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0189] The system and method or part thereof can use a collection of servers on the Internet that utilizes the Hypertext Transfer Protocol (HTTP) - the "World Wide Web" (Web or WWW). HTTP is a known application protocol that provides users with access to resources, which can be information in different formats, such as text, graphics, images, sounds, videos, Hypertext Markup Language (HTML), and programs. After the user specifies a link, the client computer sends a TCP / IP request to the Web server and receives information, which can be another Web page formatted according to HTML. The user can also access other pages on the same or other servers by following instructions on the screen, entering specific data, or clicking on a selected icon. It should also be noted that any type of selection device known to those skilled in the art, such as check boxes, drop-down boxes, etc., can be used for embodiments that use web pages to allow users to select options for given components. The server runs on a variety of platforms, including UNIX machines, but other platforms such as Windows 2000 / 2003, Windows NT, Sun, Linux, and Macintosh can also be used. Computer users can view information available on a server or network on the Web by using browsing software (such as Firefox, Netscape Navigator, Microsoft Internet Explorer, or Mosaic browser). The computing system can include a client and a server. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0190] Other embodiments are within the scope and spirit of the specification and claims. For example, due to the nature of software, the above functions may be implemented using software, hardware, firmware, hard wiring, or any combination of these. Features that implement the functions may also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. The term "a" is not used in a limiting manner herein and throughout the application, and thus is not intended to exclude multiple meanings or "one or more" meanings of the term "a". In addition, to the extent that priority to a provisional patent application is claimed, it should be understood that the provisional patent application is not restrictive, but includes examples of how the technology described herein may be implemented.

[0191] Several embodiments of the present invention have been described. However, it will be appreciated by those skilled in the art that various modifications may be made without departing from the spirit and scope of the claims and the technology described herein.

Claims

1. A method for processing keyed data items each associated with a value for a key, the processing comprising: collecting the keyed data items, determining satisfaction of one or more specified conditions for performing one or more actions based on the content of at least one of the keyed data items, and causing at least one of the one or more actions to be performed in response to the determination, the method comprising: accessing a first parameterized application, a second parameterized application, and a third parameterized application including corresponding first specifications, second specifications, and third specifications; wherein the first specification specifies one or more parameters defining one or more properties of the first parameterized application and one or more corresponding values ​​of the one or more parameters; wherein the second specification specifies one or more parameters defining one or more properties of the second parameterized application and one or more corresponding values ​​of the one or more parameters, wherein the second specification includes rules and corresponding conditions for the rules; wherein the third specification specifies one or more parameters defining one or more properties of the third parameterized application, and one or more corresponding values ​​of the one or more parameters; maintaining a state of the second specification for a corresponding value of the key, wherein the state for a particular value of the key specifies that one or more portions of the second specification are to be executed by the second parameterized application for one or more data items associated with the particular value of the key, wherein the particular value of the key uniquely identifies an entity associated with the one or more data items; Executing the first parameterized application using one or more values ​​of the one or more parameters specified by the first specification to perform a process comprising: Collecting data items from one or more data sources and one or more data streams, multiple data sources or multiple data streams, wherein a format of a first collected data item is different from a format of a second collected data item, and wherein the data item is associated with a value of the key; transforming the first and second data items according to a first specification of the first parameterized application to obtain transformed data items; and populating a queue with the transformed data item; Executing the second parameterized application using one or more values ​​of one or more parameters specified by the second specification to process the transformed data items in the queue, wherein processing the transformed data items comprises: for one or more transformed data items of the transformed data items output from the first parameterized application that are associated with a particular value of the key, identifying a current state of the second specification with respect to a particular value for the key; identifying one or more rules in a portion of the second specification to be executed in the current state; executing the identified one or more rules; determining that at least one of the one or more transformed data items satisfies one or more conditions of at least one of the one or more rules executed in the current state; In response to the determination, generating a data structure specifying performance of one or more actions; causing the second specification to transition from its current state to a subsequent state with respect to a particular value for the key; and transmitting the generated data structure to the third parameterization application; and According to the result of the second parameterized application executing the one or more rules, the third parameterized application is executed using one or more values ​​of the one or more parameters specified by the third specification to perform operations including: Based on at least one of the one or more actions specified in the data structure, one or more instructions are sent to cause at least one of the one or more actions to be performed.

2. The method of claim 1, further comprising: During operation of the one or more user interfaces, one or more user interface elements are displayed for specifying one or more values ​​of the one or more parameters for each of the first parameterized application, the second parameterized application, and the third parameterized application.

3. The method of claim 2, wherein: Executing the first parameterized application includes executing the first parameterized application using one or more values ​​specified by one or more user interface elements for one or more parameters of the first parameterized application.

4. The method of claim 2, wherein: Executing the second parameterized application includes executing the second parameterized application using one or more values ​​specified by one or more user interface elements for one or more parameters of the second parameterized application, wherein the specified one or more values ​​are used as input by the rules in determining whether the one or more conditions are satisfied.

5. The method of claim 2, wherein: Executing the third parameterized application includes executing the third parameterized application using one or more values ​​specified by the one or more user interface elements for one or more parameters of the third parameterized application.

6. The method of claim 1, wherein: The data items include data records, and wherein the transforming includes reformatting the data records according to a format specified by a first specification of the first parameterized application.

7. The method of claim 6, further comprising: Based on execution of the second parameterized application, the data record is enriched with data from a profile of a user associated with the data record, wherein the enrichment is performed according to instructions specified by a second specification of the second parameterized application to retrieve the user's profile data from memory and populate one or more fields of the data record with the retrieved profile data.

8. The method of claim 1, wherein: A parameterized application includes an application for data processing, wherein the application includes one or more parameters that are configurable with one or more values.

9. The method of claim 1, further comprising executing a feedback loop to one or more third party systems to request confirmation of performance of the one or more actions.

10. The method of claim 1, further comprising: Based on the execution of the second parameterized application, one or more key performance indicators (KPIs) are generated for the particular value of the key, wherein the KPIs specify one or more values ​​of the data item associated with the particular value of the key.

11. The method of claim 10, further comprising receiving data for a specific value of the key, wherein: The received data indicates feedback regarding at least one of the one or more actions; and The KPI for the particular value of the key is updated with the feedback by aggregating one or more portions of the data included in or associated with the KPI with the received data.

12. The method of claim 1, wherein: The one or more actions include one or more of: sending a text message to an external device, sending an email to an external system, opening a ticket for a work order in a case management system, severing a mobile phone connection, providing a web service to a targeted device, transmitting a data packet of the one or more transformed data items using a notification, and executing a data processing application hosted on one or more external computers on the one or more transformed data items.

13. The method of claim 1, wherein: The one or more instructions are sent via a network connection so that at least one of the one or more actions is performed on the external device, the method further comprising: A feedback message is received indicating whether at least one of the one or more actions (i) completed successfully or (ii) failed.

14. The method of claim 13, wherein: If a portion of at least one of the one or more actions is not completed, then at least one of the one or more actions is considered to have failed, wherein the feedback message indicates which portion of the at least one of the one or more actions that failed was not completed.

15. The method of claim 13, wherein: The feedback message indicates result data of successful completion and / or failure of at least one of the one or more actions.

16. The method of claim 15, further comprising: changing one or more specified values ​​of one or more parameters of the first parameterized application, the second parameterized application, and / or the third parameterized application based on the result data; as well as The first parameterized application, the second parameterized application, and / or the third parameterized application are re-executed using the changed one or more specified values.

17. The method of claim 1, wherein: The one or more instructions are sent via the network connection so that the one or more actions are performed on the external device, the method further comprising: receiving a feedback message from the external device, the feedback message including result data of at least one action performed among the one or more actions; comparing the result data to predetermined data associated with successful completion of performance of at least one of the one or more actions; and Based on the comparison, it is determined that the execution of at least one of the one or more actions was completed successfully, or that the execution of at least one of the one or more actions failed.

18. The method of claim 17, further comprising: changing one or more specified values ​​of one or more parameters of the first parameterized application, the second parameterized application, and / or the third parameterized application based on the result data; as well as The first parameterized application, the second parameterized application, and / or the third parameterized application are re-executed using the changed one or more specified values.

19. The method of claim 17, wherein: If the result data deviates from the predetermined data by less than a predetermined amount, the execution of at least one of the one or more actions is determined to be successfully completed, and wherein, if the result data deviates from the predetermined data by at least a predetermined amount, the execution of at least one of the one or more actions is determined to be a failure.

20. The method of claim 19, further comprising: During operation of the one or more user interfaces, one or more user interface elements are displayed for specifying the predetermined data and the predetermined amount.

21. The method of claim 13, further comprising: During operation of the one or more user interfaces, outputting, via one or more displayed user interface elements, whether at least one of the one or more actions (i) completed successfully or (ii) failed.

22. The method of claim 15, further comprising: During operation of the one or more user interfaces, the result data is output via one or more displayed user interface elements.

23. The method of claim 22, further comprising: receiving, via one or more displayed user interface elements, a user-specified change to one or more specified values ​​of one or more parameters of the first parameterized application, the second parameterized application, and / or the third parameterized application based on the result data; as well as The first parameterized application, the second parameterized application, and / or the third parameterized application are re-executed using the changed one or more specified values.

24. The method of claim 1, wherein: Sending the one or more instructions to cause performance of at least one of the one or more actions is automatically performed by the third parameterized application using the output specifying performance of the one or more actions as input.

25. The method of claim 1, wherein: The performance of at least one of the one or more actions is performed by an external system.

26. The method of claim 1, wherein: The first parameterized application is executed upon receiving a designated event or at a designated time.

27. The method of claim 1, further comprising: After determining that at least one of the one or more transformed data items satisfies the one or more conditions, data representing a key performance indicator (KPI) is updated with data specifying that at least one of the one or more transformed data items satisfies the one or more conditions.

28. The method of claim 1, wherein: The one or more transformed data items include an updated key performance indicator (updated KPI), the updated key performance indicator including aggregated performance data for the value of the key, and wherein the method further comprises: A determination is made that the updated KPI satisfies one or more conditions of at least one of the one or more rules executed in the current state.

29. The method of claim 28, further comprising: Data specifying that the transformed data item satisfies one or more conditions of at least one of the one or more rules is recorded in the memory.

30. The method of claim 1, based on execution of the first parameterized application, enriching the transformed data item with data from a profile of a user associated with a value of a key also associated with the transformed data item.

31. An electronic system for processing keyed data items each associated with a keyed value, the keyed data items coming from a plurality of different data streams, the processing comprising: collecting the keyed data items, determining satisfaction of one or more specified conditions for performing one or more actions based on the content of at least one of the keyed data items, and causing at least one of the one or more actions to be performed in response to the determination, the electronic system comprising: one or more processing devices; and One or more machine-readable hardware storage devices storing instructions, wherein the instructions are executable by the one or more processing devices to perform the method according to any one of claims 1-30.

32. One or more machine-readable hardware storage devices for processing keyed data items each associated with a value of a key, the keyed data items coming from a plurality of different data streams, the processing comprising: Collect these keyed data items, determine the satisfaction of one or more specified conditions for performing one or more actions based on the content of at least one of these keyed data items, and perform at least one of the one or more actions in response to the determination, and the one or more machine-readable hardware storage devices store instructions that can be executed by one or more processing devices to perform the method according to any one of claims 1-30.

33. A computer program product for processing keyed data items each associated with a value of a key, the keyed data items coming from a plurality of different data streams, the processing comprising: Collect these keyed data items, determine the satisfaction of one or more specified conditions for performing one or more actions based on the content of at least one of these keyed data items, and perform at least one of the one or more actions in response to the determination. The computer program product stores instructions that can be executed by one or more processing devices to perform the method according to any one of claims 1-30.

Images