Generating optimized logic from schemas

Abstract

To generate logic from a schema, such as a database schema.SOLUTION: A method includes accessing a schema that specifies relationships among datasets (DSs), computations on the DSs, or transformations of the DSs, selecting a DS from among the DSs, and identifying, from the schema, other DSs that are related to the selected DS. Attributes of the DSs are identified, and logical data representing the identified attributes and relationships among the attributes is generated. The logical data is provided to a development environment, which provides access to portions of the logical data representing the identified attributes. A specification that specifies at least one of the identified attributes in performing an operation is received from the development environment. Based on the specification and the relationships among the identified attributes represented by the logical data, a computer program is generated to perform the operation by accessing, from storage, at least one DS having the at least one of the attributes specified in the specification.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] (Priority) This application claims priority to and the benefit of U.S. Patent Application No. 17 / 025,751, filed September 18, 2020, and U.S. Provisional Patent Application No. 62 / 986,374, filed March 6, 2020, the entire contents of each of which are incorporated herein by reference.

[0002] FIELD OF THE INVENTION The present disclosure relates to generating logic from a schema, such as a database schema. [Background technology]

[0003] Complex computations can often be represented as data flows through directed graphs, with components of the computation associated with the vertices of the graph and data flows between components corresponding to the links (arcs, edges) of the graph. A system for implementing such graph-based computations is described in U.S. Patent No. 5,966,072, entitled "Executing Computations Expressed as Graphs," the entire contents of which are incorporated herein by reference. In some cases, the computations associated with the vertices are described in a human-readable form called "business rules."

[0004] One technique for generating a data flow graph uses a business rules editor, an example of which is disclosed in U.S. Patent No. 8,069,129, entitled "Editing and Compiling Business Rules," the entire contents of which are incorporated herein by reference. Summary of the Invention

[0005] In general, in a first aspect, a method is implemented by a data processing system for providing a development environment and a storage device that stores datasets having one or more attributes, the development environment providing access to one or more attributes of the datasets, the method comprising: accessing a schema, the schema specifying relationships between the datasets represented in the schema, one or more calculations on one or more of the datasets, or one or more transformations on one or more of the datasets; and identifying a plurality of datasets in the storage device from among the datasets, the identifying being performed by selecting a dataset from among the datasets and identifying one or more other datasets related to the selected dataset from the schema. identifying; generating logical data representing the identified attributes of the plurality of datasets and further representing one or more relationships between the attributes; providing the logical data to a development environment; providing, by the development environment, access to one or more portions of the logical data representing the identified attributes of the plurality of datasets; receiving from the development environment a specification that specifies at least one of the identified attributes when performing an operation; and generating a computer program configured to perform the operation by accessing from a storage device at least one dataset from the plurality based on the specification and the one or more relationships between the identified attributes represented by the logical data, wherein the at least one dataset accessed has at least one of the attributes specified in the specification.

[0006] Generally, in a second aspect, combinable with the first aspect, the development environment provides access to one or more portions of logical data without accessing multiple data sets from a storage device.

[0007] In general, in a third aspect, combinable with the first or second aspect, the method includes identifying a dataset from a plurality of datasets that includes at least one of the attributes specified in the specification; and accessing the identified dataset from a storage device.

[0008] Generally, in a fourth aspect, combinable with any of the first to third aspects, the method includes executing a computer program using at least one data set accessed from a storage device.

[0009] In general, in a fifth aspect, combinable with any of the first to fourth aspects, the method includes optimizing a computer program to produce an optimized computer program configured to perform an operation by accessing from a storage device only those datasets in a plurality of datasets that have at least one of the attributes specified in a specification.

[0010] Generally, in a sixth aspect, which can be combined with any of the first to fifth aspects, the one or more attributes include field names of multiple datasets.

[0011] Generally, in a seventh aspect, combinable with any of the first to sixth aspects, the one or more attributes include information for accessing a plurality of data sets in the storage device.

[0012] Generally, in an eighth aspect, combinable with any of the first to seventh aspects, the method includes identifying, from the schema, one or more parameters for combining the selected dataset with one or more other datasets.

[0013] Generally, in a ninth aspect, combinable with any of the first to eighth aspects, the one or more parameters include a key for combining the selected data set with at least one of the one or more other data sets.

[0014] Generally, in a tenth aspect, combinable with any of the first to ninth aspects, the method includes receiving selection data from a client device specifying a selected data set.

[0015] Generally, in an eleventh aspect, combinable with any of the first to tenth aspects, the selected dataset includes a root node of the logical data, and at least one of the one or more other datasets is joined to the selected dataset.

[0016] Generally, in a twelfth aspect, combinable with any of the first to eleventh aspects, one or more calculations on one or more of the data sets, or one or more transformations of one or more of the data sets, define a virtual field for at least one of the plurality of data sets.

[0017] Generally, in a thirteenth aspect, combinable with any of the first to twelfth aspects, the method includes generating an executable dataflow graph configured to perform operations based on the specification and one or more relationships between identified attributes represented by the logical data, the executable dataflow graph including at least one of the one or more attributes as an input.

[0018] Generally, in a fourteenth aspect, combinable with any of the first to thirteenth aspects, the method includes excluding from the computer program an operation of accessing from a storage device at least one data set in the plurality of data sets that does not include at least one of the attributes specified in the specification.

[0019] Generally, in a fifteenth aspect, combinable with any of the first to fourteenth aspects, the computer program is configured to access at least some data from a storage device by a plurality of selection statements, the selection statements being minimized to select only at least one of the attributes specified in the specification.

[0020] Generally, in a sixteenth aspect, which can be combined with any of the first to fifteenth aspects, the development environment reads logical data as a data source.

[0021] Generally, in a seventeenth aspect, combinable with any of the first to sixteenth aspects, the computer program is configured to access from the storage device only data sets that have at least one of the attributes specified in the specification.

[0022] Generally, in an eighteenth aspect, combinable with any of the first to seventeenth aspects, there is provided a system for providing a development environment and a storage device that stores a dataset having one or more attributes, wherein the development environment provides access to the one or more attributes of the dataset, the system comprising one or more processors and one or more storage devices that store instructions operable when executed by the one or more processors to cause the one or more processors to perform an operation, the operation being accessing a schema, the schema specifying relationships between the datasets represented in the schema, one or more calculations on one or more of the datasets, or one or more transformations on one or more of the datasets; and identifying a plurality of datasets from among the datasets, wherein the identifying includes selecting a dataset from among the datasets; and identifying one or more other datasets related to the selected dataset from the schema. and one or more storage devices, the identifying being performed by: identifying attributes of the plurality of datasets; generating logical data representing the identified attributes of the plurality of datasets and further representing one or more relationships between the attributes; providing the logical data to a development environment; providing, by the development environment, access to one or more portions of the logical data representing the identified attributes of the plurality of datasets; receiving from the development environment a specification that specifies at least one of the identified attributes when performing an operation; and generating a computer program configured to perform the operation by accessing from storage at least one dataset from the plurality of datasets based on the specification and the one or more relationships between the identified attributes represented by the logical data, wherein the at least one dataset accessed has at least one of the attributes specified in the specification.

[0023] Generally, in a nineteenth aspect, combinable with any of the first to eighteenth aspects, a non-transitory computer-readable storage medium stores instructions, the instructions causing a computing system to access a schema, the schema specifying relationships between datasets represented in the schema, one or more calculations on one or more of the datasets, or one or more transformations of one or more of the datasets; and identifying a plurality of datasets in the storage device from among the datasets, the identifying being performed by selecting a dataset from among the datasets and identifying one or more other datasets related to the selected dataset from the schema; identifying attributes of the plurality of datasets; and identifying attributes of the plurality of datasets. generating logical data representing identified attributes of the plurality of datasets and further representing one or more relationships between the attributes; providing the logical data to a development environment; providing, by the development environment, access to one or more portions of the logical data representing the identified attributes of the plurality of datasets; receiving from the development environment a specification that specifies at least one of the identified attributes when performing an operation; and generating a computer program configured to perform the operation by accessing from a storage device at least one dataset from the plurality based on the specification and the one or more relationships between the identified attributes represented by the logical data, wherein the at least one dataset accessed has at least one of the attributes specified in the specification.

[0024] One or more of the above implementations may provide one or more of the following advantages: The techniques described herein use information about datasets and relationships between datasets to generate logical data that includes information about the attributes of the datasets. By providing logical data as a data source in a development environment, the logical data can provide logical access to datasets without the burden of accessing the physical datasets themselves. In this way, the consumption of computational resources associated with accessing physical datasets from database storage can be reduced. Separately, computational logic can be specified through the development environment using attributes of the datasets without the need to access the actual datasets, thereby protecting the datasets from unwanted access. That is, the datasets, including the datasets, are kept secure while still allowing computational logic to be specified and applications to be compiled from them. Additionally, the techniques described herein can improve the productivity of users of the development environment because they are empowered to flexibly develop computational logic using attributes in the logical data without the overhead and processing time required when accessing and processing physical datasets.

[0025] Once computational logic is developed, the techniques described herein enable the logic to be processed in a highly optimized manner. For example, when developing computational logic, a user may consider or access numerous data sets that, in principle, prove unnecessary. Using the information provided by the logical data, an application, such as a data flow graph, can be generated that minimally loads and combines only the subset of data required in the processing to create the desired output. In doing so, the techniques described herein increase the speed of generation and execution of computational logic while reducing the computational resources required to process the logic.

[0026] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features, objects, and advantages set forth herein will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is a block diagram of a schema. [Figure 2A] 1 is a block diagram of a system for creating logical data and using the logical data to generate a computer program. [Figure 2B] FIG. 1 is a block diagram of a system for producing logical data. [Figure 2C] FIG. 1 is a block diagram of a development environment. [Figure 2D] FIG. 1 is a block diagram of a development environment. [Figure 2E] FIG. 1 is a block diagram of a system for executing a computer program. [Figure 3] FIG. 1 is a diagram of a process for creating logical data and using the logical data to generate a computer program. [Figure 4A] FIG. 1 is a block diagram of a system for producing logical data. [Figure 4B] FIG. 1 is a block diagram of a system for producing logical data. [Figure 4C] FIG. 1 is a block diagram of a system for producing logical data. [Figure 5A-1] 1 is a block diagram of a system for creating logical data and using the logical data to generate a computer program. [Figure 5A-2] 1 is a block diagram of a system for creating logical data and using the logical data to generate a computer program. [Figure 5A-3] 1 is a block diagram of a system for creating logical data and using the logical data to generate a computer program. [Figure 5B] FIG. 1 is a block diagram of a system for producing logical data. [Figure 5C] FIG. 1 is a block diagram of a system for producing logical data. [Figure 5D] FIG. 1 is a block diagram of a system for generating a computer program. [Figure 5E] FIG. 1 is a block diagram of a system for generating a computer program. [Figure 5F] FIG. 1 is a block diagram of a system for generating a computer program. [Figure 5G] FIG. 1 is a block diagram of a system for generating a computer program. [Figure 5H] FIG. 1 is a block diagram of a system for generating a computer program. [Figure 5I] FIG. 1 is a block diagram of a system for generating a computer program. [Figure 5J] FIG. 1 is a block diagram of a system for testing computer programs. [Figure 5K] FIG. 1 is a block diagram of a system for testing computer programs. [Figure 5L] FIG. 1 is a block diagram of a system for testing computer programs. [Figure 5M] FIG. 1 is a block diagram of a system for testing computer programs. [Figure 5N] FIG. 1 is a block diagram of a system for testing computer programs. [Figure 5O] FIG. 1 is a block diagram of a system for testing computer programs. [Figure 5P] FIG. 1 is a block diagram of a system for testing computer programs. [Figure 5Q] FIG. 1 is a block diagram of a system for testing computer programs. [Figure 6A] FIG. 1 is a block diagram of a development environment. [Figure 6B] FIG. 1 is a block diagram of a system for generating a computer program. [Figure 6C] FIG. 1 is a block diagram of a system for executing a computer program. [Figure 7A] FIG. 1 is a block diagram of a development environment. [Figure 7B] FIG. 1 is a block diagram of a system for generating a computer program. [Figure 7C] FIG. 1 is a block diagram of a system for generating a computer program. [Figure 7D] FIG. 1 is a block diagram of a system for generating a computer program. [Figure 7E] FIG. 1 is a block diagram of a system for generating a computer program. [Figure 8] 1 is a flowchart of a process for creating logical data and using the logical data to generate a computer program. DETAILED DESCRIPTION OF THE INVENTION

[0028] Described herein is a system for generating logical data that represents physical datasets stored in a storage system or memory. The logical data represents these physical datasets, among other things, by including attributes of the physical datasets, by including pointers that specify addresses of storage locations of these physical datasets, or by specifying other information that describes how to access the physical datasets, or a combination thereof. In this example, the logical data, or portions of the logical data, are accessible in a development environment to enable the development of specifications that specify which datasets (or attributes of datasets) are to be used and accessed. Generally, the specifications specify operations (e.g., computational logic) to be performed on the datasets or attributes of the datasets. The specifications are compiled into or otherwise used to create computer programs (e.g., executable dataflow graphs) that can be executed on a computing system. In some examples, the computer programs include executable machine code. Because the logical data is accessible in the development environment without having to physically access the datasets or their attributes, the logical data provides logical access without any physical overhead.

[0029] For convenience, but not by way of limitation, visual representations of some of the features described herein may be referred to as the features themselves. For example, a visual representation of a dataflow graph may be referred to as a dataflow graph. A visual representation of logical data may be referred to as logical data. A visual representation of a database schema may be referred to as a database schema. A visual representation of a component may be referred to as a component, and so on.

[0030] Referring to FIG. 1 , a schema 2 is shown, which specifies relationships 4a, 4b, and 4c, such as hierarchical relationships, between datasets 6a, 6b, 6c, and 6d stored in a storage system. In some examples, the schema 2 is a database schema that describes the structure of a database in a formal language supported by a database management system (DBMS). The schema 2 can be generated based on information about the datasets 6a, 6b, 6c, and 6d stored in the storage system and the relationships 4a, 4b, and 4c between those datasets. In some examples, the information about each stored dataset includes, among other information, the name of the dataset, the access parameters of the dataset (e.g., file name, location), the record format of the dataset, the data types contained in the dataset, or a combination thereof. In some examples, the information about the relationships between the datasets includes information about how the datasets can be joined, such as information about the type of relationship between the datasets (e.g., one-to-one, one-to-many, such as relationships 4b and 4c, or many-to-many, such as relationship 4a), or the keys for joining data within the datasets (e.g., primary key, foreign key), or both.

[0031] The information used to generate Schema 2 can be specified by a user (e.g., a technical user), automatically obtained from the storage system (e.g., by one or more computing systems coupled to the storage system), or both. For example, in some examples, one or more computing systems communicatively coupled to the storage system can import metadata or other information about datasets 6a, 6b, 6c, 6d to generate Schema 2 using data discovery techniques, semantic discovery techniques, or other machine learning techniques. In some examples, processing information, calculations on one or more of datasets 6a, 6b, 6c, 6d, or transformations of one or more of datasets 6a, 6b, 6c, 6d, etc., is specified (e.g., by a technical user) and included in Schema 2. For example, Schema 2 includes instructions for performing calculations or operations (or instructions for invoking a computer program, such as an executable data flow graph, to perform the operations). These calculations or transformations can modify existing fields in datasets 6a, 6b, 6c, 6d, create new fields (sometimes referred to as virtual fields or calculated fields) in the datasets, or create entirely new datasets. In some examples, values ​​for modified or newly created fields or datasets are not populated until runtime (e.g., when executed by a computer program that uses the fields or datasets), as described below.

[0032] 2A illustrates an environment 10 having a storage system 12 and a client device 14. In this example, the environment 10 also includes a logical data generator 16. The logical data generator 16 is configured to generate logical data including, for example, information regarding attributes of an actual physical dataset (or a logical dataset based on a physical dataset). In this example, the logical data provides logical access to a physical dataset that may be stored, for example, on the storage system 12, without requiring physical access to the dataset from the storage system 12. In this example, the logical access refers to a list or other specification of attributes of a physical dataset that is itself stored on the storage system 12. In another example, the logical data may include a pointer or other information that identifies an address or location where the physical dataset represented by the logical data may be accessed from the storage system 12, or instructions or parameters for accessing the physical dataset, or both.

[0033] In this example, storage system 12 is configured to communicate with logical data generator 16 and provides the logical data generator with information to use in generating the logical data, such as, among other information, information specifying the locations of physical datasets, information specifying attributes of physical datasets, information specifying relationships between physical datasets, or the physical datasets themselves, or a combination thereof. Client device 14 is also configured to communicate with logical data generator 16, such that client device 14 may send logical data generator 16 information for generating the logical data, such as, among other information, information specifying which physical datasets or which attributes of physical datasets to include (or omit) from the logical data, information specifying the root node of the logical data, or a combination thereof.

[0034] Environment 10 also includes a development environment 18, which provides a user (e.g., a user of a client device 14, which may be communicatively coupled to development environment 18) with a graphical or other user interface for specifying which datasets (or attributes of datasets) represented in the logical data the user wants to access or use in generating a computer program, such as a dataflow graph. Development environment 18 is coupled to a graph generator 22, which is configured to generate a dataflow graph from information received from development environment 18. The information received from development environment 18 is often referred to as a specification, because this information specifies the functionality of the computer program (e.g., an executable dataflow graph) and specifies which datasets (or attributes) should be accessed during execution or compilation of the specification into the application itself.

[0035] Environment 10 also includes a compiler 24 configured to compile the specification and / or data flow graph into a computer program executable (e.g., in machine code) by data processing system 26. In this example, development environment 18 transmits the specification to graph generator 22, which generates a data flow graph. Graph generator 22 then transmits the data flow graph to compiler 24, which compiles the data flow graph into a computer program (e.g., an executable data flow graph). Compiler 24 transmits the computer program to data processing system 26 for execution and / or storage. In this example, the computer program is configured to access, from storage system 12, at least one of a plurality of data sets whose attributes were included in the logical data, specified in the specification, or both.

[0036] Referring to Figure 2B, environment 20 shows further details of environment 10. In this example, storage system 12 transmits schema 21 to logical data generator 16. Database schema 21 specifies relationships, such as hierarchical relationships, between data sets 21a, 21b, 21c, 21d stored in storage system 12. In the example, schema 21 is a database schema. Schema 21 may be the same as or similar to schema 2 described with reference to Figure 1.

[0037] The client device 14 also sends selection data 23 to the logical data generator 16. The selection data 23 specifies a root node, e.g., a dataset that will be the parent node in defining the logical data. In this example, the root node is a perspective that defines the initial dataset, which is the root node in the logical data. In this example, the perspective is an information abstraction that specifies a selected starting point in a schema. The perspective includes the selected starting point in the schema and represents the root logical entity of interest. To generate the selection data 23, the client device 14 displays a graphical user interface 27. The graphical user interface 27 includes a datasets portion 28 and a selected perspectives portion 29 that is updated with an icon 29a indicating that dataset 21d has been selected as the root node of the logical data. The datasets portion 28 displays visual representations 28a, 28b, 28c, and 28d of datasets 21a, 21b, 21c, and 21d, respectively. The selected perspectives portion 29 includes a selectable portion 29b, which may be a button. When selectable portion 29b is selected, the user can select one of visual representations 28a, 28b, 28c, and 28d as the root node of the logical data. In this example, the user selects visual representation 28d to specify that dataset 21d is the root node of the logical data. When visual representation 28d is selected and selectable portion 292b is interacted with, selected perspective portion 29 is updated to display icon 28a, specifying that dataset 21d is the root node of the logical data. Selection data 23 specifies that the root node is dataset 21d.

[0038] Once a target root logical entity is specified for logical data, the logical data is expanded to include information of other datasets related to the target root logical entity. As described herein, that other information may include attributes, fields, sources, instructions, parameters, or pointers to the target root logical entity and related datasets. In this example, the logical data may be embodied in a wide record having entries for fields of the target root logical entity and fields of other datasets related to the target root logical entity. Generally, a wide record contains a group of related data held within the same structure. The logical data may also be embodied in a wide record of other attributes, such as pointers to the physical locations in memory of the target logical entity and other related datasets.

[0039] Logical data generator 16 uses schema 21 and selection data 23 to generate logical data 25. For example, logical data generator 16 receives selection data 23 that identifies dataset 21d as the root node and includes information about the attributes or available fields of dataset 21d in logical data 25. In some examples, the information includes a vector of the available attributes or fields of dataset 21d. Logical data generator 16 uses schema 21 to identify other datasets related to dataset 21d. For example, in this example, logical data generator 16 determines that datasets 21a, 21b, and 21c are related to dataset 21d, and therefore includes information about the attributes or available fields of datasets 21a, 21b, and 21c in logical data 25. In this example, logical data 25 includes a vector specifying the attributes or available fields of each of datasets 21a, 21b, and 21c. These vectors of attributes or available fields specify, instruct how to access, or otherwise represent the attribute or field names without actually accessing the fields or data within the fields themselves. Thus, logical data 25 provides logical access to data sets 21 a, 21 b, 21 c, and 21 d without the physical burden of actually accessing these data sets from storage system 12.

[0040] Referring to FIG. 2C , environment 30 illustrates development environment 18 receiving logical data 25. For example, development environment 18 receives logical data 25 from logical data generator 16 or from a storage device (e.g., storage system 12). In this example, a business rule editor interface 32 is generated by development environment 18 to define business rules and other logical rules. Generally, editor interface 32 may graphically identify cells containing formulas. This may help a user understand the difference between an expression that will evaluate to true or false by itself and an expression that returns a value that is compared against a column variable. As the user is typing, the user may indicate that a particular cell is a formula cell by, for example, typing an asterisk at the beginning of the formula. In this example, editor interface 32 includes an input portion 33 and a rule definition portion 34. Input portion 33 provides a visual representation of the datasets represented in logical data 25, as well as other data sources (which may or may not correspond to the logical data), with their attributes (e.g., fields). For example, input portion 33 includes visual representation 35 representing dataset 21a (shown in FIG. 2B ). Input portion 33 also includes visual representation 35a representing “Field A” of dataset 21a. In this example, visual representation 35a is visually depicted as being a field within dataset 21a by being indented from visual representation 35. Input portion 33 also includes visual representations 36 and 36a representing dataset 21b and “Field B” within dataset 21b, respectively. Input portion 33 also includes visual representations 37 and 37a representing dataset 21c and “Field C” within dataset 21c, respectively. Input portion 33 also includes visual representations 38 and 38a representing dataset 21d and “Field D” within dataset 21d, respectively. In this example, the visual representations in input portion 33 represent the datasets and fields available to a user when defining business rules.The available data sets and fields represented in the input portion 33 are identified from the logical data 25, thereby providing the user with access to the data sets and fields without having to access the data sets (or fields) from physical memory.

[0041] The rule definition portion 34 includes a set of rule cases. In this example, the rule definition portion 34 includes a spreadsheet format. Trigger columns in the spreadsheet correspond to available data values, and rows correspond to rule cases, e.g., sets of criteria that associate available data values. A rule case applies to a given record if the record's data values ​​satisfy the trigger criteria for each trigger column for which the rule case has criteria. When a rule case applies, an output is generated based on the output columns. A rule case for which all of its input relationships are satisfied may be said to be "triggered." The output columns correspond to potential output variables, and the value in the corresponding cell of the appropriate row determines the output, if any, for that variable. The cells may contain a value assigned to the variable or, as discussed below, may contain an expression that must be evaluated to generate an output value. Although only one is shown in FIG. 2C, there may be more than one output column.

[0042] Once the rule definition is complete by specifying the cell inputs in the rule definition portion 34, the development environment 18 generates a rule specification 39a that specifies the rule cases and which fields need to be accessed to implement the rule. However, at this stage of defining the rule, the logical data 25 provides the user with logical access to those fields without physical access. For example, the user may be able to browse available fields from various data sets stored in the storage system 12 in the input portion 33, thereby providing logical access. The development environment 18 then transmits the rule specification 39a to the graph generator 22. The development environment 18 also transmits the logical data 25 to the graph generator 22.

[0043] Referring to FIG. 2D , environment 40 illustrates another example of development environment 18. In this example, development environment 18 renders a graphical user interface 41 having a component portion 42, an input portion 43, and a canvas pane 44. Component portion 42 includes visual representations 42a-42f that represent various operations available for defining computational logic. Input portion 43 displays visual representations 45, 45a, 46, 46a, 47, 47a, 48b, and 48a of datasets and attributes (e.g., fields) represented in logical data 25. Input portion 43 also displays visual representations 49 and 49a of datasets and fields represented in other data sources (e.g., data sources other than logical data 25). That is, the visual representations in input portion 43 represent datasets and fields available for defining computational logic.

[0044] The canvas portion 44 is used to define computational logic in the form of a dataflow graph, visually depicted as a visualization 44a (hereinafter, for convenience and without limitation, referred to as the "dataflow graph 44a"). The dataflow graph represented by the visualization 44a includes a data structure having nodes. Each of the nodes includes at least one operation placeholder field and at least one data placeholder field, which are populated with operations and data (e.g., logical data, other data sources such as "Dataset V") specified by a user in the canvas portion 44. In this example, the dataflow graph 44a is generated by dragging and dropping one or more of the visual representations 42a-42f from the component portion 42 onto the canvas portion 44. Each of the visual representations 42a-42f represents an operation to be performed by or on a data structure. When the visual representations are placed on the canvas portion 44, they become icons on the canvas portion 44. The development environment 18 generates the specification 39b using the computational logic visually represented by the dataflow graph 44a. Specification 39b specifies the computational logic that is visually depicted in canvas portion 44. Development environment 18 sends specification 39b and logical data 25 to graph generator 22. Graph generator 22 can use specification 39b and logical data 25 to populate the operation and data placeholder fields of each node in dataflow graph 44a, as described below.

[0045] 2E, environment 50 illustrates further details of environment 10. In this example, graph generator 22 generates a data flow graph 52 from information received from development environment 18 (e.g., the specification and logical data). Compiler 24 receives data flow graph 52 and compiles it into an executable program 54 (e.g., a computer program such as an executable data flow graph). Compiler 24 transmits executable program 54 to data processing system 26 for execution and / or storage of the computer program. In this example, the computer program is configured to access, from storage system 12, at least one of a plurality of data sets whose attributes were included in the logical data, specified in the specification, or both.

[0046] Referring to FIG. 3 , a swimlane diagram 300 illustrates a process for generating logical data and using the logical data to generate an optimized dataflow graph. During operation, the storage system 12 sends a schema to the logical data generator 16 (302). The logical data generator 16 receives the schema (304). The logical data generator 16 generates graphical user interface (GUI) data for presenting data representing the schema (306). The logical data generator 16 sends the GUI data to the client device 14 (308). The client device 14 renders the GUI data for display to the user (310). The client device 14 receives root node selection data (312) (e.g., from a user interacting with the GUI rendered on the client device). The root node selection data includes data specifying a dataset selected for a root node of the logical data. The client device 14 sends the root node selection data to the logical data generator 16 (314). The logical data generator 16 receives the root node selection data (316). The logical data generator 16 then generates logical data using the received root node selection data and schema (318). The logical data generator 16 sends the logical data to the development environment 18 and the graph generator 22 (320). In some examples, the logical data generator 16 sends the logical data to the development environment 18, which then passes the logical data to the graph generator 22.

[0047] The development environment 18 and the graph generator 22 each receive logical data (322, 324). The development environment 18 generates GUI data for displaying fields or other attributes of the logical data (326). The development environment 18 sends the GUI data to the client device 14 (328). The GUI data represents attributes, such as field attributes, or other attributes included in the logical data, thereby providing logical access without physical overhead. The client device 14 renders the received GUI data (330) and receives selection data specifying selected fields, data sets, or other attributes (332). For clarity, a selected field or data set, as described herein, refers to information selected from the logical data itself. In some examples, the selection data also specifies operations or logic to be performed on the selected fields. The client device 14 sends selection data specifying the selected files to the development environment 18 (334). The development environment 18 receives the selection data specifying the selected fields (336) and generates a specification with the selected fields (and the operations to be performed on the selected fields) (338). The development environment 18 sends the specification to the graph generator 22 (340).

[0048] The graph generator 22 receives (342) the specification (102). The graph generator 22 generates (344) a dataflow graph using the specification and logical data. Generally, a dataflow graph (or persistent computer program) is generated from a specification such that the specification specifies modules implemented by the computer program to process one or more values ​​of one or more fields in a structured data item (e.g., a data record). These modules may include rules, instructions, components of the dataflow graph, etc. The system described herein converts the specification into a computer program implementing a plurality of modules, the converting including: identifying, for each of one or more first modules of the plurality of modules, one or more second modules of the plurality of modules, each receiving input that is based at least in part on an output of the first module; formatting an output data format of the first module such that the first module outputs only one or more values ​​of one or more fields of structured data items that are each (i) accessible to the first module and (ii) specified as input to at least one of the one or more second modules based at least in part on the output of the first module; and storing the computer program in persistent memory, with the stored computer program specifying the formatted output data format for each of the one or more first modules as described in U.S. Patent Application Publication No. 2019 / 0130048 A1, entitled “Transforming a Specification into a Persistent Computer Program,” the entire contents of which are incorporated herein by reference. The system also includes various rules that specify that the contents of each module are to be included in a computer program and / or converted into instructions in the appropriate format for a computer program. In this example, the graph generator 22 first generates a data flow graph with the data sources represented in logical data.The graph generator 22 also adds data sinks to the dataflow graph, since the dataflow graph requires data sinks. The graph generator 22 then adds various components to the dataflow graph, such as a sort component, which the graph generator 22 is configured to automatically add to increase the computational efficiency of the dataflow graph. The graph generator 22 is also configured to add join components to appropriately combine data from various data sources. The logical data may include instructions, parameters, or other information for accessing or combining data sources. Finally, the graph generator 22 adds a transformation component that includes the computational logic specified in the specification. The transformation component itself may include various components or subcomponents that represent another dataflow graph when the specification is converted into a dataflow graph as described above.

[0049] In examples, graph generator 22 optimizes the dataflow graph to generate an optimized dataflow graph (346). Generally, graph generator 22 runs an optimizer that optimizes the dataflow graph by analyzing the specification to identify which fields and associated data sources the specification identifies are being accessed. The optimizer then identifies data sources whose fields are not referenced in the specification, and the optimizer removes from the dataflow graph those data sources whose fields are not referenced in the specification. In some examples, the optimizer minimizes selection statements (e.g., database selection statements issued in the language of the database) so that only data sets and fields referenced in the specification are retrieved. In some examples, the optimizer does this by applying a set of optimization rules, as described in U.S. Patent Application Publication No. 2019 / 0370407 A1, entitled "Systems and Methods for Dataflow Graph Optimization," the entire contents of which are incorporated herein by reference. In doing so, the optimizer can create a dataflow graph that minimally loads and combines only the subset of data to which the logical data provides logical access to create the desired output. The optimizer may also perform other optimizations, such as rearranging the order of components in the dataflow graph to improve computational efficiency. For example, it may be computationally more efficient for a filter component to come before a join component, so that the join component does not combine data that is ultimately filtered out. Thus, the optimizer may move the filter component to come before the join component.

[0050] Referring to FIG. 4A , environment 60 illustrates logical data generator 16 using logical data 23 to identify dataset 21d as the root node of the logical data. This is indicated by dataset 21d being starred and outlined in FIG. 4A . Logical data generator 16 also uses schema 21 to identify other datasets related to dataset 21d. The other related datasets include datasets 21a, 21b, and 21c. Using dataset 21d as the root node, logical data generator 16 generates logical data 25. As previously described, logical data 25 includes entry 25a that specifies that dataset 21d is the perspective or root node of logical data 25. Entry 25a includes fields and / or attributes of dataset 21d. Attributes of a dataset (e.g., dataset 21d) may include names of fields within the dataset or other information describing the fields within the dataset. By including names of fields within dataset 21d, among other information, logical data 25 provides access to fields within dataset 21d without having to physically access dataset 21d in storage. Logical data 25 also includes entries 25b, 25c, and 25d corresponding to datasets 21c, 21b, and 21a, respectively. In this example, entries 25b, 25c, and 25d are ordered according to their relationship to dataset 21d. In this example, dataset 21d is the root node, and dataset 21c is a child node. Thus, entry 25b, which represents dataset 21c, is ordered immediately below entry 25a in logical data 25. In addition, datasets 21a and 21b are children of dataset 21c. Thus, entries 25c and 25d are ordered below entry 25b to represent the relationship between datasets 21a, 21b, and 21c. Each of entries 25b, 25c, and 25d includes attributes and / or fields of the respective dataset.As previously mentioned, these attributes and / or fields may be names or other identifying information for the fields, allowing logical data 25 to provide logical access to data sets 21 a, 21 b, 21 c, and 21 d without the burden of actually accessing those data sets from storage. Logical data 25 may provide logical access because it includes information that may be used to identify attributes or fields of data sets 21 a, 21 b, 21 c, and 21 d and / or that may be used to access those data sets as needed.

[0051] Referring to FIG. 4B, environment 70 illustrates a variation of environment 60 (FIG. 4A), in which dataset 21b has been selected as the root node, as indicated by the dashed line and star outline. Dataset 21b is selected as the root node, for example, when a user selects visual representation 28b of FIG. 2B. When dataset 21b is selected as the root node, logical data generator 16 generates logical data 72, in which dataset 21b is designated as the root node and the order of the other datasets in the logical data has been changed relative to the order of the datasets in logical data 25, as shown in FIG. 4A. In this example, logical data 72 includes entry 72a representing dataset 21b. In this example, dataset 21b is a child of dataset 21c, and entry 72b is included in logical data 72 representing dataset 21c. Dataset 21a is a child of dataset 21c, and entry 72c is included in logical data 72 representing dataset 21a. Dataset 21c is a child of dataset 21d, and entry 72d is included in logical data 72 that represents dataset 21d. As discussed above with reference to Figure 4A, entries 72a, 72b, 72c, and 72d each contain information about attributes or fields for each of the respective datasets and / or other information that specifies the characteristics of the dataset or how to access the dataset.

[0052] 4C , environment 80 illustrates the generation of logical data 82 for database schema 84. In this example, logical data generator 16 receives database schema 84 and also receives selection data 23 that specifies that dataset 84d is the root node in schema 84 (as indicated by the dashed line and star outline). In this example, schema 84 includes datasets 84a, 84b, 84c, 84d, and 84e. In the example, schema 84 includes instructions for performing calculations on dataset 84d (e.g., fields or values ​​of fields in dataset 84d) or otherwise transforming dataset 84d to produce dataset 84e. For example, schema 84 may include instructions for performing one or more operations or instructions for invoking an executable program (e.g., a data flow graph) that includes dataset 84d (or a portion thereof) as input and produces dataset 84e (or a portion thereof) as output. In some examples, these calculations, transformations, or other operations are defined directly in schema 84, such as by including instructions specifying the operations in schema 84. In some examples, schema 84 may include links, pointers, or other information for accessing instructions to perform the operations. In some examples, these operations have been previously performed, and dataset 84e produced by the operations is a physical dataset stored in a storage system. In some examples, dataset 84e includes virtual data, such as one or more calculated or virtual attributes, virtual fields, or other virtual elements that are populated during execution (e.g., when dataset 84e is used in a computer program such as a dataflow graph).

[0053] Because dataset 84d is the root node, the logical data generator 16 generates logical data 82 having entries 82a-82e. Entry 82a represents dataset 84d, which is the root node. Entry 82a may include attributes of dataset 84d. As described above, attributes include field names, pointers to fields, etc. Because datasets 84c and 84e are children of dataset 84d, the next entries in the logical data 82 are entry 82b representing dataset 84e and entry 82c representing dataset 84c. Entries 82b and 82e each include field attributes. Because datasets 84a and 84b are children of dataset 84c, the next entries in the logical data 82 are entry 82d representing dataset 84b and entry 82e representing dataset 84a. Entries 82d and 82e each include field attributes.

[0054] Referring to FIG. 5A , an environment 90 illustrates an overview of an actual implementation for generating logical data and using the logical data to generate an optimized dataflow graph. In this example, logic generator 16 receives schema 91 from a storage system. Logical data generator 16 also receives selection data 92 from client device 14 indicating a selected root node. Using this schema 91 and selection data 92, logical data generator 16 generates logical data 94 in accordance with the techniques described herein. Logical data generator 16 transmits logical data 94 to development environment 18. Using this logical data 94, development environment 18 generates a graphical user interface or other user interface that makes information contained in logical data 94, such as attributes or fields, viewable or accessible by a user who can interact with development environment 18 (e.g., using client device 14) without accessing the underlying physical data set of storage. The user uses development environment 18 to select at least one of the attributes in logical data 94 and one or more operations for execution or use of the selected attribute. Based on this information, development environment 18 creates a specification 96a that specifies which attributes and / or fields of logical data 94 should be included in generating the dataflow graph. Graph generator 22 receives logical data 94 and specification 96a and creates a dataflow graph 98a that is optimized to access only the physical data sets associated with the attributes specified in specification 96a (or otherwise required to perform the operations in specification 96a).

[0055] The same or a different user may use development environment 18 to select one or more different attributes of logical data 94, one or more different operations to perform on the selected attributes, or both. For example, a user may change the selected attributes or operations specified in specification 96a in response to an error identified in the processing of dataflow graph 98a, or may select different attributes and operations to create an entirely new dataflow graph. Using this information, the development environment creates specification 96b, which differs from specification 96a. Graph generator 22 receives logical data 94 and specification 96b and creates dataflow graph 98b, which is optimized differently from dataflow graph 98a to access only the physical data sets associated with the attributes specified in specification 96b. In this way, logical data 94 provides logical access to all of the data sets and attributes contained therein, without the physical burden of doing so. This provides great flexibility to end users (e.g., users of development environment 18) who can view and select all of the physical data contained in logical data 94 (without the burden of physically accessing such data) and obtain highly optimized data flow graphs that are tailored to access only the physical data necessary to fulfill their specifications.

[0056] 5B, environment 100 shows further details of environment 90. In this example, storage system 12 stores schema 91 with datasets 101, 102, 103, and 104. "Offer Status" dataset 101 includes a "Key" field 101a and an "Offer Accepted" field 101b. Field 101a may include, for example, a primary key, a foreign key, or both (which may be defined in separate fields). "Minutes" dataset 102 includes fields 102a, 102b, 102c, and 102d. "Customer" dataset 103 includes fields 103a and 103b. "Reload Date" dataset 104 includes fields 104a and 104b. In this example, "Minutes Remaining" field 102d is, for example, a virtual or calculated field defined in schema 91 as described above. For example, schema 91 may specify one or more operations or other instructions that generate field 102d from one or more other fields in dataset 102 or another dataset. In particular, schema 91 may define field 102d as the difference between fields 102b and 102c. In this example, square brackets are used to indicate that field 102d is a virtual or calculated field. In this example, datasets 101, 102, 103, and 104 are related to each other through their key values. That is, each of datasets 101, 102, 103, and 104 has key values ​​that match each other, and data from one dataset can be used to join with another dataset.

[0057] The logical data generator 16 receives the schema 91 from the storage system 12. The client device 14 displays a graphical user interface 105 (e.g., based on GUI data received from the logical data generator 16 (not shown) specifying which datasets are included in the schema 91). The GUI 105 includes a datasets portion 106 and a selected perspective portion 107. The datasets portion 106 includes visual representations 106a, 106b, 106c, and 106d of datasets 101, 102, 103, and 104, respectively. The selected perspective portion 107 includes a button 107a, the selection of which allows the viewer to select one of the visual representations 106a-106d. In this example, the user selects visual representation 106c, which represents dataset 103. Upon selection, the selected perspective portion 107 is updated with an icon 107b specifying that dataset 107 has been selected as the root node of the logical data to be generated by the logical data generator 16. The client device 14 generates selection data 92 that specifies that the dataset 103 is to be selected as the root node. The client device 14 sends the selection data 92 to the logical data generator 16. The logical data generator 16 uses the schema 91 and the selection data 92 to produce logical data 94.

[0058] Referring to FIG. 5C , an example is shown in which environment 110 generates logical data 94 from schema 91 and selection data 92. In this example, logical data 94 includes a set of instructions, parameters, or other information that specify how to access datasets 101, 102, 103, and 104 and their respective fields, as shown in FIG. 5B . In some examples, logical data 94 includes instructions, parameters, or other information that specify how to create or otherwise access virtual or calculated fields, such as field 102d. In some examples, the logical data is embodied in a wide record that includes attributes, fields, or other characteristics of the underlying datasets. Logical data generator 16 sends logical data 94 to development environment 18.

[0059] Referring to FIG. 5D , environment 120 illustrates an example of a business rule editor interface 121 generated by development environment 18 for defining business rules and other logical rules. Generally, editor interface 121 may graphically identify cells containing formulas. This may help a user understand the difference between an expression that will evaluate to true or false by itself and an expression that returns a value that is compared against a column variable. As the user types, the user can indicate that a particular cell is a formula cell by, for example, typing an asterisk at the beginning of the expression. In this example, editor interface 121 includes an input portion 122 and a rule definition portion 123. Input portion 122 provides visual representations of fields and datasets (shown in expanded view, as indicated by the downward-pointing arrows) represented in logical data 94, as well as other data sources (shown in collapsed view, as indicated by the right-pointing arrows). For example, input portion 122 includes a visual representation 124 representing dataset 101 (shown in FIG. 5B ). Input portion 122 also includes visual representation 124a, which represents "Offer Accepted" field 101b in dataset 101. In this example, visual representation 124a is visually depicted as being a field in dataset 101 by being indented from visual representation 124. Input portion 122 also includes visual representations 125 and 125a, 125b, and 125c, which represent dataset 102 and fields 102b, 102c, and 102d, respectively. Input portion 122 also includes visual representations 126 and 126a, which represent dataset 103 and field 106b, respectively. Input portion 122 also includes visual representations 127 and 127a, which represent dataset 104 and field 104b, respectively. In this example, the visual representations in input portion 122 represent the datasets and fields available to a user when defining business rules. The available datasets and fields represented in input portion 122 are identified from logical data 94, thereby providing the user with access to the datasets and fields, without actually having to access those datasets (or fields) from physical memory.

[0060] Rule definition portion 123 includes a set of rule cases. In this example, rule definition portion 106 includes a spreadsheet format. Trigger columns 128a, 128b, and 128c in the spreadsheet correspond to available data values, and rows 129c-129g correspond to rule cases, e.g., sets of criteria that associate available data values. A rule case applies to a given record if the record's data values ​​satisfy the trigger criteria for each trigger column for which the rule case has criteria. When a rule case applies, an output is generated based on output column 129a. A rule case for which all of its input relationships are satisfied may be said to be "triggered." Output column 129a corresponds to potential output variables, and the value in the corresponding cell of the appropriate row determines the output, if any, for that variable. The cell may contain a value assigned to the variable or, as discussed below, may contain an expression that must be evaluated to generate an output value. Although only one is shown in FIG. 5D, there may be more than one output column.

[0061] In particular, row 129a specifies the relative inputs and outputs of the rule. Row 129b specifies what the fields and outputs are used in defining the rule. In this example, row 129b includes cells 128a, 128b, and 128c. Cell 128a is added to rule definition portion 123 upon user selection of visual representation 126a, as visually depicted by the dotted line around visual representation 126a in input portion 122. As a result of this selection, cell 128a specifies that "Name" field 103be (shown in FIG. 5B) is used as an input in defining the rule specified in rule definition portion 123. Cell 128b specifies that "Minutes Remaining" field 102d (shown in FIG. 5B) is also used as an input in defining the rule shown in rule definition 123. In this example, when visual representation 125c is selected, cell 128b is updated to indicate that "Minutes Remaining" field 102d is used as an input to the rule. Similarly, cell 128c specifies that after visual representation 125b representing "Minutes Used" field 102c (shown in FIG. 5B) is selected, "Minutes Used" 102c is also used as an input in defining the rule shown in rule definition 123. Cells 128a, 128b, and 128c illustrate that a user can access field attributes (such as the name of the field) from datasets stored in storage system 12, without having to physically access those datasets (or fields) themselves. Rule definition portion 123 also includes rows 129c, 129d, 129e, 129f, and 129g that specify various rule cases and outputs when various criteria of the rule cases are met.

[0062] Upon completing the definition of the rule by specifying cell inputs in the rule definition portion 123, the development environment 18 generates a rule specification 96a that specifies the rule cases and which fields need to be accessed to implement the rule. In this example, the rule specification 96a specifies that the "Name" field 103b, the "Minutes Remaining" field 102d, and the "Minutes Used" field 102c (each shown in FIG. 5B) are used as inputs to the rule. That is, the values ​​of these fields are used as inputs to the rule. Thus, when the rule itself is executed, those fields would require physical access when executing the rule. However, at this stage of defining the rule, the logical data 94 provides the user with logical access to those fields without physical access. For example, logical access is provided by the user being able to view available fields from various data sets stored in the storage system 12 in the input portion 122. The development environment 18 then sends the rule specification 96a to the graph generator 22. The development environment 18 also sends the logical data 94 to the graph generator 22.

[0063] 5E, an environment 130 illustrates an example of generating and optimizing a dataflow graph from rule specifications 96a and logical data 94. Graph generator 22 receives rule specifications 96a and logical data 94. Graph generator 22 applies optimizer 132 to both rule specifications 96a and logical data 94 in generating optimized dataflow graph 98a. In this example, graph generator 22 uses rule specifications 96a and logical data 94 to generate dataflow graph 134. In this example, dataflow graph 134 includes components 134a-134m. Graph generator 22 then applies optimizer 132 to dataflow graph 134. Generally, optimizer 132 reduces redundancy in a dataflow graph (e.g., dataflow graph 134) and eliminates data sources unused by the dataflow graph. That is, if rule specification 96a does not specify that a rule accesses a field (e.g., a data set) from a particular data source, optimizer 132 will remove that data source from the dataflow graph. In some examples, optimizer 132 does this by minimizing select statements (e.g., when the source data is stored in a relational database) so that only the data sets and fields specified in rule specification 96a and included in logical data 94 are accessed.

[0064] Initially, graph generator 22 generates dataflow graph 134 with datasets and fields included in logical data 94 as data sources, e.g., based on instructions, parameters, or other information for accessing datasets specified in logical data 94. In this example, components 134a-134m in dataflow graph 134 are based on data sources (e.g., datasets) represented in logical data 94. In some examples, graph generator 22 may also rely on built-in functionality that specifies how information included in specification 96a, logical data 94, or both, is transformed into dataflow graph 134. For example, built-in functionality may include functionality to insert various operations, such as sort, partition, or join operations, among others, into the dataflow graph, e.g., based on information from specification 96a, logical data 94, or both.

[0065] The dataflow graph 134 may also include one or more transformation components. Generally, a transformation component receives input records from one or more data sources, e.g., an input data set, and produces output records based on computational logic. To produce a transformation component, the graph generator 22 may receive a specification of logic (e.g., a rule set from the specification 96a or instructions, parameters, or other information from the logic data 94) to be applied to the input. The graph generator 22 may then generate and implement the transformation as a graph-based computation having data processing components connected by link elements representing the data flow. In this example, the dataflow graph 134 includes a transformation component 134l that includes logic to execute the rules specified in the rule specification 96a. In this example, the dataflow graph 134 also includes a transformation component 134i that includes logic to generate the computational field 102d. In this example, the generated transformation is a component (e.g., component 134l) in the dataflow graph 134. The graph generator 22 may also update the transformation when, for example, the rule set is edited. For example, when a rule set is edited, the editor (e.g., development environment 18) may provide the entire rule set to graph generator 22, or may provide only the new or modified rules or rule cases. Depending on the capabilities and needs of the system using the transformation, graph generator 22 may generate an entirely new transformation to replace the original transformation, or graph generator 22 may provide a new component that includes the transformation.

[0066] The graph generator 22 applies the optimizer 132 to the dataflow graph 134 to generate a dataflow graph 136. The optimizer 132 removes components 134a, 134c, 134f, 134g, and 134j from the dataflow graph 134, as indicated by the cross-out portions of the dataflow graph 136. The optimizer 132 determines to remove these components because they relate to datasets not referenced or used by the rule specification 96a. That is, the rule specification 96a does not contain references to any fields contained in the datasets being removed. Note that in some examples, a dataset serving as a root node (e.g., dataset 103 or component 134b in this example) may not be optimized, regardless of whether it is used by the rule specification 96a. The end result of the optimization is a dataflow graph 98a that has been optimized to remove all datasets not required to execute the rules specified by the rule specification 96a, as well as other components (e.g., sorts, joins, etc.) instantiated to access those datasets.

[0067] Referring to FIG. 5F, environment 140 illustrates another example of business rule editor interface 121 generated by development environment 18 for defining business rules and other logical rules. In environment 140, rule definition 123 has been modified relative to the rule definition in environment 120 shown in FIG. 5D. Specifically, trigger cells 128b and 128c have been removed, and cell 128a has been modified to specify that “minutes used” field 102c (shown in FIG. 5B) is the only input in defining the rule shown in rule definition 123 after selecting visual representation 125b representing “minutes used” field 102c. The rule cases in columns 129d, 129e, 129f, and 129g have also been updated. Thus, development environment 18 generates rule specification 142, which is a modified version of rule specification 96a shown in FIG. 5D. In this example, rule specification 142 specifies that “minutes used” field 102c is used as the only input for the rule. The development environment 18 sends the rule specification 142 to the graph generator 22. The development environment 18 also sends the logical data 94 to the graph generator 22.

[0068] Referring to FIG. 5G, environment 150 illustrates an example of generating and optimizing a dataflow graph from modified specification 142 and logical data 94. Initially, graph generator 22 generates dataflow graph 152 similar to dataflow graph 134 shown in FIG. 5E, except for transformation component 152l, which includes logic to enforce rules specified in modified specification 142. In this example, transformation component 152l differs from transformation component 134l (shown in FIG. 5E) because the rules specified in modified specification 142 and implemented by component 152l differ from the rules specified in specification 96a and implemented by component 134l. Graph generator 22 applies optimizer 132 to dataflow graph 152 to generate dataflow graph 154. In doing so, optimizer 132 removes components 134a, 134c, 134f, 134g, 134j, and 134i from dataflow graph 154, as indicated by the cross-out portions of dataflow graph 154. Optimizer 132 determines to remove these components because they relate to data sets not referenced or used by rule specification 142. Note that the data set serving as the root node (e.g., data set 103 or component 134b in this example) is not referenced in specification 140 but is not optimized. The end result of the optimization is dataflow graph 156 that has been optimized to remove all of the data sets not required to execute the rules specified by rule specification 142, as well as other components (e.g., sorts, joins, etc.) instantiated to access those data sets. Dataflow graph 156 differs from dataflow graph 98a despite using the same logical data 94 source due to different attributes relied upon in the respective graph specifications 96a, 142.

[0069] Referring to FIG. 5H, environment 160 illustrates yet another example of a business rule editor interface 121 generated by development environment 18 for defining business rules and other logical rules. In environment 140, rule definition 123 has been modified relative to the rule definitions in environments 120 and 140 shown in FIGS. 5D and 5F, respectively. Here, cell 128a specifies, after selecting visual representation 127a representing “Last Reloaded” field 103b (shown in FIG. 5B), that “Last Reloaded” field 103b is the only input in defining the rule shown in rule definition 123. The rule cases in columns 129c, 129d, 129e, 129f, and 129g have also been modified. Therefore, development environment 18 generates rule specification 96b, which differs from each of rule specifications 96a (first shown in FIG. 5A), 142. In this example, rule specification 96b specifies that “Last Reloaded” field 103b is used as the only input for the rule. The development environment 18 sends the rule specification 96b to the graph generator 22. The development environment 18 also sends the logical data 94 to the graph generator 22.

[0070] Referring to FIG. 5I, environment 170 illustrates an example of generating and optimizing a dataflow graph from specification 96b and logical data 94. Initially, graph generator 22 generates dataflow graph 172 similar to dataflow graphs 134 and 152 shown in FIGS. 5E and 5G, respectively, except for transformation component 172l, which includes logic for executing rules specified in modified specification 96b (in this example, different from each of transformation components 134l and 152l). Graph generator 22 applies optimizer 132 to dataflow graph 172 to generate dataflow graph 174. In doing so, optimizer 132 removes components 134a, 134c, 134f, 134e, 134h, and 134i from dataflow graph 154, as indicated by the cross-out portions of dataflow graph 174. Optimizer 132 determines to remove these components because they are associated with a data set not referenced or used by rule specification 96b. Note that the dataset that serves as the root node (e.g., dataset 103 or component 134b in this example) is not referenced in specification 140, but is not optimized. The end result of the optimization is dataflow graph 98b (first shown in FIG. 5A ) that has been optimized to remove all datasets that are not needed to execute the rules specified by rule specification 96b, as well as other components (e.g., sorts, joins, etc.) instantiated to access those datasets. Dataflow graph 98b differs from dataflow graphs 98a and 156, despite using the same logical data 94 source, due to the different attributes relied upon in the specifications of each graph.

[0071] 5J, environment 180 illustrates the results of executing dataflow graph 96a. Graph generation system 18 sends dataflow graph 96a to compiler 24. Compiler 24 compiles dataflow graph 96a into executable program 182 as follows:

[0072] A dataflow graph represents a computation as a plurality of vertices representing a computational process, each vertex having an associated access method, and a plurality of links, each link connecting at least two vertices to each other and representing the flow of data between the connected vertices. A dataflow graph is executed by (1) receiving the dataflow graph as user input to a computing system, (2) preparing the dataflow graph for execution by performing graph transformation steps on the computing system until each vertex is executable and each link is associated with at least one communication method that is compatible with the access methods of the vertices connected by the link, (3) activating each link by the computing system by creating a combination of communication channels and / or data stores according to the link's communication method, and (4) launching each process on the computing system by invoking process execution.

[0073] Generally, a dataflow graph is prepared for execution as follows.

[0074] A driver program (or simply "driver" for short) provides a means for drawing a dataflow graph based on input from a user received through a user interface. One or more dataflow graph data structures representing a visual representation of the dataflow graph are generated by the driver. The driver accesses the dataflow graph initially drawn by the user and prepares the dataflow graph for execution by applying graph transformations. In performing these transformations, the dataflow graph data structures defining the initial dataflow graph are traversed in a known manner to fetch each vertex and any associated links. In some examples, five dataflow graph transformations are used on the fetched data structures to prepare the dataflow graph for execution, as described below.

[0075] Although the dataflow graph is not yet in an executable form, the five dataflow graph transformations described below can be selected and applied in any order, as often as needed (including never at all), until an executable dataflow graph is obtained. The five dataflow graph transformations include: (1) inserting a file adapter, (2) inserting a communication adapter, (3) setting the state of a file vertex to Complete, (4) setting the state of a process vertex to Runnable or Unrunnable, and (5) setting the communication method of a data link. Each of these transformations and the conditions under which each may be executed are described here.

[0076] Inserting a file adapter In this transformation, the driver replaces links with file adapters (i.e., links, file vertices, and other links). That is, as each dataflow graph data structure representing a link is fetched or accessed during traversal of the dataflow graph data structure, a new data structure may be created that modifies, extends, or replaces the original data structure.

[0077] For a source (destination) file adapter, the host of the file vertex is the same as the host of the source (destination) vertex, and the file of the file vertex is a new file located in the working directory of the source (destination) vertex. This transformation is (1) The source is either a file vertex or a process vertex that is not in the Done state, and (2) It can be implemented when the destination is either a file vertex in the Incomplete state or a process vertex that is not in the Done state.

[0078] Inserting a communications adapter In this transformation, the driver replaces a link with a communication adapter (i.e., a link, a process vertex, and another link). The process vertex runs a copy program that copies data from its input to its output, and can read from / write to any of the communication channels or data stores supported by the underlying substrate. For the source (destination) communication adapter, the host of the process vertex is the same as the host of the source (destination) vertex, and the working directory is the same as the working directory of the source (destination) vertex. The process vertex is created in the Enabled state. This transformation is (1) The source is either a process vertex or a file vertex in a state other than Done, and (2) It can be created if the destination is either a process vertex in a state other than Done or a file vertex in an Incomplete state.

[0079] Setting the file vertex state to Complete This transformation sets the state of the file vertex to Complete. This transformation can be performed if the state of the file vertex is incomplete and all inputs to the file vertex are process vertices in the Done state.

[0080] Setting the state of a process vertex to Runnable or Unrunnable In this transformation, the state of the process vertex is set to either Runnable or Unrunnable. This transformation can be performed if the state of the process vertex is Enabled.

[0081] Setting the data link communication method In this conversion, the communication method is set to Data Link. This conversion can be performed when the communication method of Data Link is Unbound.

[0082] A dataflow graph is executed that has the following three properties: (1) All process vertices are in one of the following states: Done, Runnable, Unrunnable, or Disabled. (2) All data links meet all of the following criteria: 1) If the source or destination of a data link is a Runnable process vertex, the communication method of the data link must be bound to a specific communication method. 2) If the communication method of a data link is something other than File, both its source and destination must be process vertices, and if one process vertex is Runnable, both process vertices must be Runnable. 3) If the communication method of a data link is File, its source or destination must be a file vertex. If the destination is a Runnable process vertex, the source must be a Complete file vertex. If the source is a Runnable file vertex, the destination must be an Incomplete file vertex. (3) All links bound to a communication method satisfy the constraints specific to the communication method. 1) The communication method must be compatible with the access method of its source and destination ports (this can be determined by referring to the program template). In the case of the extended substrates mentioned above, all communication methods are compatible with SOC access, all except Shared Memory are compatible with File Descriptor access, NamedPipe and File are compatible with NamedPipe access, and only File is compatible with File access. 2) Some communication methods require that the source and destination nodes are the same. In the case of the extended substrate mentioned above, this is true for all communication methods except TCP / IP.

[0083] The dataflow graph transformations may be applied in any order until an executable graph is obtained (e.g., the dataflow graph data structure may be traversed repeatedly until all transformations are complete). In some examples, the dataflow graph transformations are applied in the following order: (1) inserting file adapters, (2) replacing inter-file links, (3) identifying Complete file vertices, (4) identifying Unrunnable process vertices, (5) identifying Runnable process vertices, (6) setting the remaining Enabled vertices to Unrunnable, (7) if conditions are met, inserting more file adapters, (8) selecting a communication method, and (9) inserting communication adapters. The steps of this example are now described in more detail.

[0084] (1) Insert the file adapter To insert a file adapter, the following steps are performed for every link in the dataflow graph: If the source port of the link has a data access method that requires the use of a file and the destination is not a file on the same node, insert a source file adapter. If the destination port of the link has a data access method that requires the use of a file and the source is not a file on the same node, insert a destination file adapter. If the destination of the link is a process vertex in the Disabled state and the source is a process vertex in the Enabled state, insert a destination file adapter.

[0085] (2) Replacing inter-file links To replace inter-file links, the following steps are performed for every link in the dataflow graph: If the source and destination are both file vertices, insert a source communication adapter (and if the source and destination are on different nodes, also insert a destination communication adapter, not shown).

[0086] (3) Identifying the complete file vertex To identify a Complete file vertex, the following steps are performed for every file vertex in the dataflow graph: If it is a process vertex with all upstream vertices in the Done state, set its state to Complete.

[0087] (4) Identifying unrunnable process vertices To identify Unrunnable process vertices, the following steps are performed for every link in the dataflow graph: The "Unrunnability" test is performed as follows: if the source of a link is an Incomplete file vertex and its destination is a process vertex in the Enabled state, set the state of the process vertex to Unrunnable; if the source is a process vertex in any state other than Enabled and the destination is a process vertex in the Enabled state, mark the destination process vertex as Unrunnable. This test is repeated until no vertices can be marked as Unrunnable.

[0088] (5) Identifying Runnable Process Vertices To identify Runnable process vertices, the following steps are performed for all process vertices in the dataflow graph: A "Runnability" test is performed as follows: if the vertex is in the Enabled state and all upstream vertices are either Complete file vertices or Runnable process vertices, then set the vertex's state to Runnable. This test is repeated until no vertices can be marked as Runnable.

[0089] (6) Set the remaining enabled vertices to unrunnable. To set the remaining Enabled vertices to Unrunnable, the following steps are performed for all process vertices in the graph: If the vertex is in the Enabled state, set its state to Unrunnable.

[0090] (7) Insert a file adapter To insert a further file adapter, the following steps are performed for every link in the dataflow graph: If the source of the link is a Runnable process vertex and the destination is an Unrunnable process vertex, insert a source file adapter.

[0091] (8) Selecting communication methods To select a communication method, the following steps are performed for all links in the dataflow graph. This step applies only to links that are attached to a runnable process at any time and are not bound to a communication method. If the link's source (destination) is a file vertex and its destination (source) is a process vertex on the same node, set the link's communication method to File. Otherwise, select one of the available communication methods so that all of the method's constraints are satisfied. For speed, communication methods may be considered in the following order: Shared Memory, NamedPipe, and TCP / IP. In some examples, the first method that satisfies the above constraints is selected. The criteria substrate may use the following rules: First, if the link is attached to a port that accepts SOC connections, the link will use Shared Memory if the source and destination are on the same node, and TCP / IP if they are on different nodes. Otherwise, if the source and destination are on the same node, the NamedPipe method will be used. In all other cases, a single communication method is not sufficient, and the system will revert to a communication adapter (below).

[0092] (9) Inserting a communication adapter If no single communication method was selected in the previous step of selecting a communication method and all have been tried, continue by inserting a source communication adapter and attempting to select a communication method for the adapter's two links. If this fails, attempt to replace the newly inserted source communication adapter with a destination communication adapter. If this fails, insert both the source and destination communication adapters and select communication methods for the three links in the resulting dual adapter. In the reference substrate, a communication adapter is only needed if the source and destination are on different nodes and the link is connected to either a file vertex or a process vertex that does not accept the SOC connection method. In this case, an adapter may be selected as follows:

[0093] If the source is a file vertex, insert a source communication adapter. The two links in the source communication adapter will then use the File and TCP / IP communication methods.

[0094] If the source is a port that does not accept the SOC communication method, a source communication adapter is inserted. The two links in the source communication adapter then use the TCP / IP and File communication methods.

[0095] If the destination is a file vertex, insert a destination communications adapter.

[0096] The two links within the adapter then use TCP / IP and File communication methods.

[0097] If the destination is a port that does not accept the SOC communication method, a destination communication adapter is inserted. The two links in the adapter then use the TCP / IP and NamedPipe communication methods.

[0098] Phase C: Initiating Data Link Data links must be created and activated in the Unlaunched state. To activate a link, the links are scanned to find one that is Unlaunched, bound to a communication method, and has a Runnable source or destination. For every such link, an identifier is generated that can be used by various communication methods. For the extended substrates described above, the identifiers are created as follows: Every link has two identifiers: a stream object identifier and a communication channel / file identifier. The stream object identifier is used by the SOC mechanism and is identical to the name of the link. The channel / file identifier is used to identify the File, NamedPipe, Shared Memory area, or TCP / IP connection used by the link. Additionally, if the process vertex requires the NamedPipe or File communication method, the channel / file identifier will be made available so that the process vertex, when launched, can attach to the channel / file using the UNIX file system.

[0099] After the identifier is generated, the substrate is called to create a channel or stream object. If the communication method is a NamedPipe, the substrate is called again to create a NamedPipe.

[0100] Once the executable program 182 is generated, the compiler 24 transmits the executable program 182 to the data processing system 26. The data processing system 26 receives the records from the storage system 12 and executes the executable program 182 in batch mode to produce, for example, batch results 184. The batch results 184 indicate the number of times a particular rule case "fired" (e.g., how many times a rule was triggered by the processed data records). In this example, the "gold" offer triggered a disproportionate amount of time compared to the other rules. Therefore, a user may wish to test the rules they created to determine whether changes can be made, for example, to reduce the number of times the gold offer is triggered.

[0101] Referring to FIG. 5K, environment 190 illustrates an example of a business rule editor and testing interface 191 generated by development environment 18 for defining and testing business rules and other logical rules. In this example, interface 191 enables various test categories 192, including testing by record 192a, testing by expression 192b, testing by error 192c, testing by baseline delta 192d, and testing by rule case 192e. In this example, a user of development environment 18 has selected to test by rule case 192e and specified case 2 (corresponding to golf offers). From here, the user can step through records 193 that triggered the specified rule case (i.e., rule case 2) by interacting with button 193a, as shown in FIG. 5L. In this example, record 4 triggered rule case 2, as indicated by the thick outline with shaded fill in the rule definition portion 123 of interface 191. As the dataflow graph is executed (thereby accessing physical data), the fields shown in input portion 122 are populated with data values ​​194 for the current record (record 4 in FIG. 5L). As can be seen from these values, record 4 falls well within the defined rule for trigger case 2. As shown in FIG. 5M, interacting with button 193a advances to the next record that triggered case 2. From data values ​​194, it can be seen that record 24 is significantly close to the "minutes used" threshold for rule case 2.

[0102] Therefore, to reduce the number of gold offers, the user may increase the "minutes used" threshold, as shown in FIG. 5N. In this example, the user has changed the "minutes used" trigger value in rule case 2 to ">400," as shown in bold-shaded cell 195. Before running another batch test to see the overall results of the change, the user may want to modify the rule for an individual record or a small number of records to ensure the rule is functioning as expected. To do so, the user can enter the record to be tested in window 193 and interact with the "Test Record" button 196. In this example, the user has selected record 24 to test. In response to selecting the "Test Record" button 196, the development environment 18 generates a modified specification 197 and sends the specification 197 to the graph generator 22. The development environment 18 can transmit the entire specification 197 or only the modified portion. Using the modified specification 197 and the logical data (not shown), the graph generator 22 has produced a modified dataflow graph 198. The data flow graph 198 is sent to a compiler for compilation and subsequent execution. The development environment 18 also sends data 199 specifying the records to be tested to the data processing system 26 for execution.

[0103] Referring to FIG. 5O, environment 200 illustrates the execution of dataflow graph 198 for a specified test record 204. In this example, compiler 24 receives dataflow graph 198 and compiles it to produce executable program 202, which is transmitted to data processing system 26. Data processing system 26 receives data 199 specifying the record to be tested and retrieves the specified test record 204 (e.g., record 24 in this example) from storage system 12. Data processing system 26 then executes executable program 202 using test record 204 to produce updated results 206, i.e., the results of processing the specified record with the updated executable dataflow graph. These results are presented to the user who ran the test in interface 191, as shown in FIG. 5P. As can be seen in FIG. 5P, record 24 triggers case 3 (representing a silver offer) under the modified rules, rather than case 2 (representing a gold offer).

[0104] Once it has been verified that the modified rule cases and data flow graphs are functioning as intended, batch testing can be performed, as shown in Figure 5Q. To do so, data processing system 26 receives records from storage system 12 and executes executable program 202 in batch mode to produce batch results 208. Batch results 208 show that the number of gold offers has been significantly reduced compared to the unmodified batch results 184 shown in Figure 5J.

[0105] 6A , environment 210 illustrates another example of business rule editor interface 121 generated by development environment 18 for defining business rules and other logical rules. In this example, logical data 211 includes a “Customer Data” dataset as a root node, which is related to various other datasets, including a “Transactions” dataset, a “Payment Data” dataset, and a “Withdrawal Data” dataset. Each of these datasets and their respective fields is visualized in input portion 122 of interface 121. Specifically, input portion 122 includes visual representations 212, 212a, 212b, and 212c of “Customer Data” and its fields, 213, 213a, and 213b of “Transactions” and its fields, 214 and 214a of “Payment Data” and its fields, and 215 and 215a of “Withdrawal Data” and its fields.

[0106] Rule definition portion 123 includes a set of inputs and rule cases. In this example, the "Price" and "Location" fields are used as inputs in defining the rule, as shown in cells 128a and 128b, respectively. The "Transaction Location" and "Transaction Account Balance" fields are used as part of an expression in defining the rule case specified in rule definition portion 120. When a rule case is applied, an output is generated based on output column 129a. As shown in this column, the output of each of rule cases 129c, 129d, and 129e is associated with approving, rejecting, or flagging a particular transaction for review based on specified trigger criteria. Upon completing the rule definition by specifying cell inputs in rule definition portion 123, development environment 18 generates a rule specification 216 that specifies the rule case and which fields need to be accessed to implement the rule. Development environment 18 sends rule specification 216 to graph generator 22. Development environment 18 also sends logical data 211 to graph generator 22.

[0107] Referring to FIG. 6B, environment 220 illustrates an example of generating and optimizing a dataflow graph configured for continuous operation from rule specification 216 and logical data 211. Graph generator 22 receives rule specification 216 and logical data 211. As in a batch or non-continuous setting, graph generator 22 initially generates dataflow graph 222 configured to access datasets and fields included in logical data 211 as data sources, for example, based on instructions, parameters, and other information for accessing the datasets specified in logical data 211. However, the components of dataflow graph 222 and the manner in which data is accessed and processed are different in a continuous setting. In this example, subscribe component 222a is used to subscribe to the flow of data from root node "AccountData." Each incoming flow unit (or portion thereof) from the root node is then replicated through replication component 222b for use in subsequent lookups of related records defined in logical data 211, for example, using lookup component 222c.

[0108] After generating initial dataflow graph 222, graph generator 22 applies optimizer 132 to dataflow graph 222 to generate dataflow graph 224. Optimizer 132 removes components 222d, 222f, 222g, 222h, and 222i from dataflow graph 222, as indicated by the cross-out portions of dataflow graph 224. Optimizer 132 determines to remove these components because they relate to data sets not referenced or used by rule specification 216. That is, rule specification 216 does not contain references to any fields contained in the removed data sets. The end result of the optimization is dataflow graph 226 that has been optimized to remove all data sets not required to execute the rules specified by rule specification 96a, as well as other components (e.g., sorts, joins, etc.) instantiated to access those data sets. This makes the logical data described herein useful for providing logical access without physical burden and facilitating optimization, regardless of whether the input data is continuous, semi-continuous, or discontinuous.

[0109] 6C, environment 230 illustrates the results of executing continuous data flow graph 226. Graph generation system 18 transmits data flow graph 226 to compiler 24, which compiles data flow graph 96a into executable program 232 (e.g., executable data flow graph). Compiler 24 transmits executable program 232 to data processing system 26. Data processing system 26 receives data stream 12 (e.g., continuous data) and executes executable program 232 to process the data stream and produce real-time or near-real-time results 234.

[0110] Referring to FIG. 7A , environment 240 illustrates another real-world example of a development environment 18 generating a specification 252. In this example, development environment 18 renders a graphical user interface 241 having a component portion 242, an input portion 243, and a canvas portion 244. Component portion 242 includes visual representations 242a-242f that represent various operations available for defining computational logic. Input portion 243 displays visual representations 245, 245a, 246, 246a, 246b, 246c, 247, 247a, 248, and 248a of datasets and fields represented in logical data 94. Input portion 243 also displays visual representations 249 and 249a of datasets and fields represented in other data sources. That is, the visual representations in input portion 243 represent datasets and fields available for defining computational logic.

[0111] The canvas portion 244 is used to define computational logic in the form of a dataflow graph, visually depicted as visualization 250 (hereinafter, for convenience and without limitation, referred to as "dataflow graph 250"). The dataflow graph represented by visualization 250 includes a data structure having nodes. Each of the nodes includes at least one operation placeholder field and at least one data placeholder field, which are populated with operations and data specified by a user in the canvas portion 244. In this example, dataflow graph 250 is generated by dragging and dropping one or more of visual representations 242a-242f from the component portion 242 onto the canvas portion 244. Each of visual representations 242a-242f represents an operation to be performed by or on the data structure. When visual representations are placed on the canvas portion 244, they become icons on the canvas portion 244. Some of these icons, such as icon 251a, specify an operation to perform on a particular data set or field (e.g., a filter operation). In this example, icon 251a specifies that a filter action is to be performed on the "minutes added" field represented by visual representation 246a in input portion 243. Icon 251b specifies that a filter action is to be performed on the "minutes remaining" field represented by visual representation 246c in logical data portion 243. Development environment 18 generates specification 252 using the computational logic visually represented by dataflow graph 250. Specification 252 specifies the computational logic visually depicted in canvas portion 244. Development environment 18 sends specification 252 and logical data 94 to graph generator 22. Graph generator 22 can use specification 252 and logical data 94 to populate the operation fields and data placeholder fields of each node in dataflow graph 250.

[0112] Referring to FIG. 7B, environment 260 illustrates an example of graph generator 22 generating an optimized dataflow graph, a visualization of which is shown by visualization 268 (referred to herein, for convenience and without limitation, as “dataflow graph 268”). Graph generator 22 receives specification 252 and logical data 94. Using specification 252 and logical data 94, graph generator 22 generates dataflow graph 262, including components 262a-262r, as shown in FIG. 7C. In particular, graph generator 22 generates dataflow graph 262 from specification 252 and logical data 94 by populating operation fields and data placeholder fields for each node of dataflow graph 252 and using the techniques described above. For example, unlike dataflow graph 98a, whose specified computational logic is implemented by transformation component 134l, dataflow graph 262 includes distinct components 262o, 262p, and 262q based on the computational logic specified in specification 252. Dataflow graph 262 represents the dataset represented in logical data 94 and combined with the separate "Offers" dataset 249 and its "Monthly" field 249a, and dataflow graph 262 also represents additional built-in functions required to generate the dataflow graph (e.g., sort, partition, etc.).

[0113] In this example, the graph generator 22 applies the optimizer 132 to the dataflow graph 262 shown in FIG. 7C to produce an optimized dataflow graph 268. Various intermediate stages of the optimization are shown in FIGS. 7D and 7E. The optimizer 132 analyzes the specification 252 or the logical data 94, or both, to identify fields used in the specification 252 and then identifies the datasets that contain those fields. The optimizer 132 removes from the dataflow graph 262 datasets that are not used or referenced by the specification 252. The optimizer 132 may also be responsible for adding partition components to the graph as needed. In some examples, the optimizer 132 does this by minimizing select statements so that only datasets and fields specified in the rule specification 252 and included in the logical data 94 are accessed. 7D , optimizer 132 removes components 262a, 262s, 262c, 262f, 262i, 262v, and 262h from dataflow graph 262 (thereby producing dataflow graph 264 at time T2). This is because component 262a represents the data set “Offer Status,” and its field “Offer Accepted” is not referenced or used by specification 252. Similarly, component 262c represents the data set “Reload Date,” and its field “Last Reload” is not referenced or used by the specification. By removing these input sources (i.e., those represented by components 262a and 262i), the remaining components are rendered unnecessary (sometimes referred to as “dead components”), and therefore these components (i.e., 262s, 262c, 262v, 262h) can also be removed.

[0114] Optimizer 132 also performs a further optimization that moves filter components 262o and 262p before the join operation specified by component 262k, thereby producing dataflow graph 266 at time T3, as shown in FIG. 7E. By doing so, optimizer 132 produces a dataflow graph that is faster, more efficient, and uses fewer computational resources because the filter operation is performed before the join operation, reducing the amount of data that needs to be combined. If the filter operation were performed after the join operation, more compositional resources would be used because the system would have to combine data that is ultimately filtered out. The result of the optimization is dataflow graph 268.

[0115] Generally, optimizer 132 performs optimizations or other transformations that may be required to process data in accordance with one or more of the operations specified in the dataflow graph, or improves the processing of data in accordance with one or more of the operations specified in the dataflow graph compared to processing the data without the optimizations or transformations, or both. For example, the optimizer adds one or more sort operations, data type operations, join operations including join operations based on keys specified in the dataflow graph, partition operations, automatic parallelization operations, or operations that specify metadata, among others, to produce a transformed dataflow graph 268 having the desired functionality of dataflow graph 262. In some implementations, transformed dataflow graph 268s is an optimized dataflow graph (or is transformed into an optimized dataflow graph) by applying one or more dataflow graph optimization rules to the transformed dataflow graph to improve the computational efficiency of the transformed dataflow graph compared to the computational efficiency of the transformed dataflow graph before applying the optimizations. Data flow graph optimization rules can include, among other things, removing dead or redundant components, early filtering, or record narrowing, as described, for example, in U.S. patent application Ser. No. 62 / 966,768, entitled "Editor for Generating Computational Graphs," the entire contents of which are incorporated herein by reference.

[0116] The techniques described herein use information about relationships between datasets to improve user (e.g., business user) productivity using a development environment and enable optimized data processing. While a user (e.g., technical user) may initially need to define the logical data to expose to the development environment (e.g., by selecting a dataset to use as a root node or defining a virtual field), business users are empowered to flexibly develop their own computational logic from the exposed logical data, and based on that logic, can generate a wide variety of data flow graphs to execute the logic in an optimized manner.

[0117] The techniques described herein empower users to quickly and powerfully expose logical data from complex sets of datasets stored in a storage system to a development environment. In some examples, a technical user selects a set of datasets of interest to work with, and schema definitions are discovered or otherwise obtained from within all of these datasets. For example, schemas can be exported from these datasets in a database and discovered using data discovery, semantic discovery, or other machine learning, or by receiving additional input from a technical user, or a combination thereof. In some examples, a technical user can create additional calculated or virtual fields in the schema, such as aggregations from among other data elements. In some examples, a technical user can select a root node or perspective of the logical data.

[0118] Business users operating in the development environment can then use any of the attributes contained in the logical data (which may correspond to actual physical data elements or logical data elements defined by technical users) to develop computational logic applicable to their business needs. In some examples, business users can view the output and test the logic (e.g., rules) they have written in the development environment.

[0119] Once a business user is satisfied with the computational logic they have developed (and optionally tested), an optimized dataflow graph can be generated by a graph generator that processes only the datasets needed for the dataflow graph. For example, when developing the computational logic, the business user may have access to many datasets that turn out to be unnecessary. Because the graph generator and optimizer have detailed information about the datasets from the logical data, the dataflow graph that the datasets generate can be dramatically optimized.

[0120] Once the optimized data flow graph is generated, it can be executed, for example, by a data processing system. In some examples, the data flow graph can be executed in two different modes: batch or real-time. In some examples, if a business user is interested in different sets of rules that depend on different sets of data, the business user can generate the desired data flow graph, which can also be optimized without the need for technical user involvement.

[0121] 8 illustrates a flowchart of an example process 800 for creating logical data and generating a computer program using the logical data. Process 800 can be implemented by one or more of the systems and components described herein, including one or more of the computing systems configured to implement the techniques described with reference to FIGS. 1-7.

[0122] The operations of the process 800 include accessing 802 a schema that specifies relationships between the datasets represented in the schema, one or more calculations for one or more datasets, or one or more transformations for one or more datasets. In an example, the schema is a database schema. In an example, the one or more calculations for one or more of the datasets or the one or more transformations for one or more of the datasets define at least one logical field, virtual field, or calculated field of the plurality of datasets.

[0123] A plurality of datasets from among the datasets in the storage device are identified by selecting a dataset from among the datasets and identifying one or more other datasets related to the selected dataset from a schema (802). In an example, the selected dataset is a root node of a logical dataset, and at least one of the one or more other datasets is joined to the selected dataset. In an example, selection data specifying the selected dataset is received from a client device. In an example, one or more parameters, such as one or more keys, for joining the selected dataset with the one or more other datasets are identified from the schema.

[0124] Attributes of the multiple datasets are identified (806). In an example, the one or more attributes include field names of the multiple datasets. In an example, the one or more attributes include information for accessing the multiple datasets. Logical data is generated (808) that represents the identified attributes of the multiple datasets and further represents one or more relationships between the attributes.

[0125] The logical data is provided to a development environment (810). The development environment provides access to one or more portions of the logical data representing identified attributes of the plurality of datasets (812). In an example, the development environment provides access to the one or more portions of the logical data without accessing the plurality of datasets from a storage device. In an example, the development environment reads the logical data as a data source.

[0126] A specification specifying at least one of the identified attributes when performing the operations is received from the development environment (814). A computer program configured to perform the operations by accessing at least one dataset from the plurality of datasets from storage based on the specification and one or more relationships between the identified attributes represented by the logical data (816), wherein the at least one accessed dataset has at least one of the attributes specified in the specification. In an example, the computer program is executed using the at least one dataset accessed from storage. In an example, the operations include identifying a dataset from the plurality of datasets that includes at least one of the attributes specified in the specification and accessing the identified dataset from storage.

[0127] In an example, the computer program is optimized to produce an optimized computer program configured to perform an operation by accessing from storage only those datasets in the plurality of datasets that have at least one of the attributes specified in the specification. In an example, an operation of accessing from storage at least one dataset in the plurality of datasets that does not include at least one of the attributes specified in the specification is removed from the computer program. In an example, the computer program is configured to access from storage at least some data from the plurality with a selection statement, the selection statement being minimized to select only at least one of the attributes specified in the specification. In an example, the operation includes generating an executable dataflow graph configured to perform the operation based on the specification and one or more relationships between the identified attributes represented by the logical data, the executable dataflow graph including at least one of the one or more attributes as an input.

[0128] Implementations of the subject matter and operations described herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in one or more combinations of these. Implementations of the subject matter described herein can be implemented as one or more computer programs (also referred to as data processing programs) (i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by or to control the operation of a data processing apparatus). A computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random-access or serial-access memory array or device, or a combination of one or more of these. A computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). The subject matter may be implemented in computer program instructions stored on a non-transitory computer storage medium.

[0129] The operations described herein may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0130] The term "data processing apparatus" encompasses all kinds of apparatuses, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system-on-chip, or a plurality or combination of the foregoing. An apparatus may include special-purpose logic circuitry (e.g., a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC)). In addition to hardware, an apparatus may also include code that provides an execution environment for the computer program in question (e.g., code comprising processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of these). The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0131] A computer program (also known as a program, software, software application, script, or code) can be written in any type of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple cooperating files (e.g., files storing one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0132] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)).

[0133] Processors suitable for executing a computer program include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer include a processor for performing actions in accordance with the instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or be operatively coupled to receive data from, transfer data to, or both, although a computer need not have such devices. Furthermore, a computer can be incorporated into another device (e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash drive)). Suitable devices for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0134] Implementations of the subject matter described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), middleware components (e.g., an application server), or front-end components (e.g., a user computer having a graphical user interface or web browser through which a user can interact with implementations of the subject matter described herein), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0135] A computing system may include users and servers. Users and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server may arise by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server sends data (e.g., HTML pages) to a user device (e.g., for the purpose of displaying the data to and receiving user input from a user interacting with the user device). Data generated at the user device (e.g., results of user interaction) may be received from the user device at the server.

[0136] While this specification contains details of many specific implementations, these should not be construed as limitations on any implementation or the scope of what may be claimed, but rather as descriptions of features unique to particular implementations. Certain features described herein in the context of separate embodiments can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single embodiment can also be implemented separately in multiple implementations or in any suitable subcombination. Furthermore, while features may be described above as operative in a particular combination and initially claimed as such, one or more features from a claimed combination can, in some cases, be removed from the combination, and the claimed combination may be directed to a subcombination or variations of the subcombination.

[0137] Similarly, although operations are depicted in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in any sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above implementations should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0138] Other implementations are within the scope of the following claims. [Explanation of symbols]

[0139] 2 Schema 10 Environment 12 Memory Systems 14 Client Devices 16 Logical Data Generator 18 Development Environment 20 Environment 22 Graph Generator 24 Compiler 26 Data Processing System 27 Graphical User Interface 30 Environment 32 Editor Interface 33 Input section 34 Rule definition part 40 Environment 42 Component Parts 43 Input section 44 Canvas part 50 Environment 52 Data Flow Graph 54 Executable Programs 60 Environment 70 Environment 72 Logical Data 80 Environment 82 Logical Data 84 Schema 90 Environment 91 Schema 92 Selected Data 94 Logical Data 96a specifications 96b specifications 98a Data Flow Graph 98b Dataflow graph 100 Environment 101 datasets 102 datasets 103 datasets 104 datasets 105 Graphical User Interface 106 Dataset Part 107 Perspective section 110 Environment 120 Environment 121 Interface 122 Input part 123 Rule Definition 124 Visual Representation 125 Visual Representation 126 Visual Representation 127 Visual Representation 130 Environment 132 Optimizer 134 Data Flow Graph 136 Data Flow Graph 140 Environment 142 Specifications 150 Environment 152 Data Flow Graph 154 Data Flow Graph 156 Data Flow Graph 160 Environment 170 Environment 172 Data Flow Graph 174 Data Flow Graph 180 Environment 182 Executable Programs 184 Batch Results 190 Environment 191 Interface 192 Test Categories 193 Window 194 Data Values 195 thick-bordered shaded cells 196 buttons 197 Specifications 198 Data Flow Graph 199 Data 200 Environment 202 Executable Programs 204 Test Records 206 results 208 Batch Results 210 Environment 211 Logical Data 213 Visual Representation 214 and visual representation of its field 215 and visual representation of its field 216 Rules Specifications 220 Environment 222 Data Flow Graph 224 Data Flow Graph 226 Data Flow Graph 230 Environment 232 executable programs 234 Near Real-Time Results 240 Environment 241 Graphical User Interface 242 Component Parts 243 Logical Data Part 243 Input part 244 Canvas part 245 Visual Representation 246 Visual Representation 247 Visual Representation 248 Visual Representation 249 Visual Representation 252 Specifications 260 Environment 262 Data Flow Graph 264 Data Flow Graph 266 Data Flow Graph 268 Data Flow Graph 300 Swimlane Diagram 800 processes

Claims

1. 1. A method implemented by a data processing system for providing one or more interfaces and a storage device for storing a dataset having one or more attributes, the one or more interfaces providing access to the one or more attributes of the dataset, the method comprising: storing a representation of attributes associated with a given dataset of the plurality of datasets, as represented in a schema; the representation specifies one or more relationships between the attributes, or the one or more relationships specified by the schema; or the representation is generated independent of accessing the plurality of data sets; the schema specifies relationships between the datasets represented by the schema, or one or more calculations on one or more of the datasets, or one or more transformations of one or more of the datasets; The attributes are identified by identifying one or more attributes of the given dataset and identifying one or more other attributes from the schema that are associated with the given dataset. Steps and providing access to one or more portions of the representation of the attribute via one or more interfaces; receiving a specification from the one or more interfaces specifying at least one of the attributes to be included in the representation when performing an action; generating, based on the specification, a computer program configured to perform the operation by accessing at least one dataset from the plurality of datasets from the storage device, wherein the at least one accessed dataset is associated with the at least one of the attributes specified in the specification; method.

2. The one or more other attributes include attributes of one or more elements associated with the given data set. The method of claim 1.

3. The one or more other attributes include attributes of one or more other datasets related to the given dataset. The method of claim 1.

4. the representation specifies the one or more relationships between the attributes or one or more relationships specified by the schema, and generating includes generating the computer program based on the specification and the one or more relationships specified in the representation. The method of claim 1.

5. identifying a dataset from the plurality of datasets that includes the at least one of the attributes specified in the specification; accessing the identified data set from a storage device. The method of claim 1.

6. executing the computer program using the at least one data set accessed from a storage device. The method of claim 1.

7. optimizing the computer program to produce an optimized computer program configured to perform the operation by accessing from a storage device only those datasets in the plurality of datasets that have the at least one of the attributes specified in the specification. The method of claim 1.

8. the one or more attributes include field names of the plurality of the datasets; The method of claim 1.

9. the one or more attributes include information for accessing the plurality of data sets in a storage device. The method of claim 1.

10. identifying the given data set from among the data sets; Identifying from the schema the one or more other datasets of the dataset that are related to the given dataset. identifying the plurality of data sets from among the data sets represented in the schema by The method of claim 1.

11. 11. The method of claim 10, further comprising identifying from the schema one or more parameters for joining the selected dataset with the one or more other datasets.

12. The method of claim 11 , wherein the one or more parameters include a key for combining the selected data set with at least one of the one or more other data sets.

13. The method of claim 10, comprising receiving data from a client device specifying the given data set.

14. the given dataset includes a root node, and at least one of the one or more other datasets is joined to the given dataset; The method of claim 10.

15. the one or more calculations on one or more of the data sets or the one or more transformations of one or more of the data sets define a virtual field of at least one of the plurality of data sets. The method of claim 1.

16. generating an executable dataflow graph based on the specification, the executable dataflow graph being configured to perform the operations, the executable dataflow graph including at least one of the attributes as an input; The method of claim 1.

17. excluding from the computer program an operation of accessing from a storage device at least one data set in the plurality of data sets that does not contain the at least one of the attributes specified in the specification. The method of claim 1.

18. the computer program is configured to access the at least one data set from storage by a selection statement, the selection statement being minimized to select only the at least one of the attributes specified in the specification; The method of claim 1.

19. The representation is retrieved as a data source The method of claim 1.

20. providing a development environment including one or more graphical user interfaces for viewing the one or more portions of the representation and for defining a specification specifying at least one of the attributes for performing the operation. The method of claim 1.

21. 1. A system for providing one or more interfaces and a storage device for storing a dataset having one or more attributes, the one or more interfaces providing access to the one or more attributes of the dataset, the system comprising:

21. A system comprising: one or more processors; and one or more storage devices storing instructions operable, when executed by the one or more processors, to cause the one or more processors to perform the method of any one of claims 1 to 20.

22. A non-transitory computer-readable storage medium storing instructions that cause a computing system to perform the method of any one of claims 1 to 20.

Images