Converting specifications to persistent computer programs

Abstract

To provide a computer program converting method, a computer system, and a computer-readable medium which processes one or a plurality of values in one or a plurality of fields in a structured data item of a specification.SOLUTION: A method according to the present invention has a step of accessing to a specification designating a plurality of modules for processing one or a plurality of values of one or a plurality of fields in a structured data item, a step of converting the specification into a computer program, and a step of storing the computer program in a permanent memory. The step of converting the specification into the program has a step of, with respect to each of one or a plurality of first modules of the plurality of modules, specifying each of one or a plurality of second modules for receiving an input at least partially based on an output of the first module, and a step of setting an output data format of the first module so that the first module should output only one or the plurality of values of the one or the plurality of fields of the structured data item.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] This application relates to a method, a computer system, and a computer-readable medium for converting a specification into a computer program that processes one or more values of one or more fields within structured data items.

Background Art

[0002] Complex data processing systems typically process data in multiple stages, with the results generated at one stage being fed into the next stage. The overall flow of information through such a system can be described by a directed data flow graph, where the nodes or vertices within the graph represent components (data files or processes), and the links or "edges" within the graph indicate the data flow between components. A system for performing calculations using such a graph is described in the prior art (Patent Document 1) entitled "EXECUTING COMPUTATIONS EXPRESSED AS GRAPHS", which is hereby incorporated by reference into this application.

[0003] Graphs can also be used to directly call computations. The graphs created by this system provide a way to input information into and extract information from the individual processes represented by the components of the graph, to move information between processes, and to define the execution order for those processes. Systems for calling these graphs include algorithms for selecting an inter-process communication method and for scheduling the execution of processes, and also provide a monitor for the execution of the graph.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] In a general embodiment 1, a method is described in which a computer system performs the following steps in converting a specification into a computer program that processes one or more values ​​of one or more fields of a structured data item: accessing a specification that specifies a plurality of modules to be implemented by the computer program to process one or more values ​​of one or more fields of a structured data item; converting the specification into a computer program that implements the plurality of modules, the step of for each of one or more first modules of the plurality of modules, identifying one or more second modules of the plurality of modules, each of which receives input based at least partially on the output of the first module; setting the output data format of the first module so that the first module outputs only one or more values ​​of one or more fields of a structured data item, each of which is specified as (i) accessible to the first module and (ii) as input to at least one of the one or more second modules based at least partially on the output of the first module; and storing the computer program in persistent memory, the step of which the stored computer program specifies the set output data format for each of the one or more first modules. One or more computer systems can be configured to perform a specific operation or action by installing software, firmware, hardware, or a combination thereof on the system that causes the system to perform an action when it is running. One or more computer programs can be configured to perform a specific operation or action by including instructions that cause the data processing device to perform an action when it is executed by the device.

[0006] In Embodiment 2 according to Embodiment 1, the content of one or more parts of the computer program is automatically generated without requiring user input, unlike in the specifications.

[0007] In Embodiment 3, which is any one of Embodiments 1 or 2, the computer program is persistent and editable.

[0008] In Embodiment 4, which is any one of Embodiments 1 to 3, the computer program is a dataflow graph, each module is a component of the dataflow graph represented by nodes of the dataflow graph, and the dataflow graph components are connected by data flows represented by links between nodes.

[0009] In Embodiment 5, which is any one of Embodiments 1 to 4, the method further includes the steps of: accessing a data structure in a data storage system that maps data representing a part of a specification to data flow graph components; identifying data items in the data structure that represent a particular part of the specification; identifying which data flow graph components in the data structure are mapped to that particular part of the specification; and adding the identified data flow graph components to the data flow graph in the order of execution of operations specified by the specification.

[0010] In embodiment 6, which is any one of embodiments 1 to 5, the method further includes the steps of identifying dependencies between data flow graph components and specifying the output data format of the data flow graph components based at least partially on these identified dependencies.

[0011] In Embodiment 7, which is one of Embodiments 1 to 6, the method further includes the steps of: identifying that, with respect to a particular first module, only one of one or more second modules accesses a particular one or more fields accessible to the particular first module; identifying that only one of the one or more second modules accesses a particular one or more fields from a third module of a plurality of modules, which is different from each of the particular first module and the one or more second modules; and setting the output data format so as not to output one or more values ​​for a particular one or more fields accessed by only one of the one or more second modules.

[0012] In an embodiment 8 according to any one of embodiments 1 to 7, the method further includes the steps of: providing data for displaying a user interface; receiving data via the user interface representing the selection of one or more modules from a plurality of modules to be included in a computer program; and receiving, for each selected module, the values ​​of parameters for the operation implemented by the selected one or more modules via the user interface.

[0013] In embodiment 9, which is any one of embodiments 1 to 8, the method further includes the steps of: determining whether a value received via a user interface conforms to a predetermined criterion associated with an operation; and outputting a notification via the user interface indicating whether the received value conforms to the predetermined criterion, wherein if the received value is determined not to conform to the predetermined criterion, the notification includes guidance on how the received value must be modified to conform to the predetermined criterion.

[0014] In embodiment 10, which is any one of embodiments 1 to 9, the method further includes the steps of configuring a selected module with parameter values, and then storing the configured module which will be included in and implemented by another computer program.

[0015] In embodiment 11, which is one of embodiments 1 to 10, the method further includes, before executing a computer program, the steps of executing first, second, and third passes for each specific module of a plurality of modules to identify and save a configured output data format for that specific module that specifies which fields are required by the modules downstream of that specific module, and to identify and save an input data format for that specific module that specifies which fields are accessible to that specific module, wherein during the first pass, the specific module is required by the modules upstream of that specific module by itself and by any of the modules downstream of that specific module The process involves broadcasting one or more messages containing data representing one or more fields, wherein the specific module performing the broadcast is a broadcast module, and the upstream modules receiving the broadcast are receiving modules; during the second pass, the receiving modules send one or more messages to the broadcast module specifying which receiving modules can provide the values ​​for the required fields; and during the third pass, the broadcast module analyzes the messages received from the receiving modules and, in response, sends back one or more messages to the receiving modules specifying which receiving modules are responsible for sending which fields to the broadcast module.

[0016] In embodiment 12, which is one of embodiments 1 to 11, the method further includes: receiving a structured data item in a data stream that includes one or more data records; running a computer program on at least one of the one or more data records to process at least one of the one or more data records; and removing one or more fields from at least one of the records that are not designated as input to at least one of the one or more second modules, at least partially based on the output of the first module, before submitting at least one of the one or more records to one of the one or more second modules, based on the stored output data format of at least one of the first modules.

[0017] In embodiment 13, which is one of embodiments 1 to 12, the method further includes the steps of defining one or more new modules in a specification, and storing at least one of the one or more new modules in a global palette that specifies entities that can be included in other specifications.

[0018] In an embodiment 14 according to any one of embodiments 1 to 13, the method further includes the steps of defining one or more new modules in a specification, and saving at least one of the new modules to a local palette that specifies entities that can only be used for inclusion in that specification and cannot be included in other specifications.

[0019] In embodiment 15, which is one of embodiments 1 to 14, the method further includes the steps of: accessing a predefined module included in a global palette for the purpose of defining a specification; modifying one or more attributes of the predefined module; and performing an operation which includes storing the modified predefined module in a local palette that specifies entities that can only be used for inclusion in that specification and cannot be included in other specifications, or saving the modified predefined module to the global palette so that the modified predefined module can also be accessed by other specifications.

[0020] In an embodiment 16 according to any one of embodiments 1 to 15, the method further includes the steps of defining one or more new first modules in a specification, defining one or more new second modules in a specification, and configuring at least one of the one or more new first modules to serve as a data source for at least one of the one or more new second modules.

[0021] In embodiment 17, which is one of embodiments 1 to 16, the module includes one or more other modules.

[0022] In embodiment 18, which is one of embodiments 1 to 17, the method further includes the steps of configuring a selected module with parameter values ​​and storing the configured module for later inclusion in another program.

[0023] In embodiment 19, which is one of embodiments 1 to 18, the method further includes the steps of: identifying that only one of one or more second modules accesses a particular one or more fields accessible to the first module; identifying that only one of one or more second modules accesses a particular one or more fields from a third module different from the first module; and setting the output data format so as not to output one or more values ​​for a particular one or more fields accessed only by one of one or more second modules.

[0024] In embodiment 20, which is one of embodiments 1 to 19, the structured data item includes a data record.

[0025] In general embodiment 21, the method is a method that a computer system performs in converting a specification into a computer program that processes one or more values ​​of one or more fields of a structured data item, the method being performed by a computer system in converting a specification that specifies a plurality of modules into a computer program that implements modules to process one or more values ​​of the one or more fields of a structured data item, the method being to identify, for each of the first modules of the plurality of modules, one or more second modules of the plurality of modules, each of which receives input based at least partially on the output of the first module, and for each of the first modules, one or more values ​​of a field that are processed by the one or more second modules, when the second module is configured to process one or more values ​​of one or more fields, and for each of the first or second modules, one or more values ​​of a field that are processed by the second module, and for each of the first or second modules, one or more values ​​of a field that are processed by the second module, and for one or A conversion step includes: identifying one or more fields from among one or more fields whose multiple values ​​are processed by one or more second modules, one or more of which values ​​are accessible to the first module; for each of the one or more fields whose multiple values ​​are accessible to the first module, whether the values ​​of the fields are specified as input to at least one of one or more second modules based at least partially on the output of the first module; and specifying the output data format of the first module by setting the first module to output only one or more values ​​of one or more fields, each of which is specified as input to at least one of one or more second modules based at least partially on the output of the first module; and storing a computer program in persistent memory, the computer program specifying the output data format for each of one or more first modules.Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the method. One or more computer systems can be configured to perform particular operations or actions during operation by virtue of software, firmware, hardware, or combinations thereof that are incorporated into the system. One or more computer programs can be configured to perform particular operations or actions by including instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0026] In general aspect 22, a computer system for converting a specification into a computer program that processes one or more values of one or more fields of structured data items, the computer system comprising: one or more processing devices; and accessing a specification that specifies a plurality of modules that are executable by the one or more processing devices and that are to be implemented by a computer program to process one or more values of one or more fields of structured data items; and converting the specification into a computer program that implements the plurality of modules, for each of one or more first modules of the plurality of modules, identifying one or more second modules of the plurality of modules, each of which receives an input that is at least partially based on the output of the first module; and setting the output data format of the first module such that the first module outputs only one or more values of one or more fields of the structured data items, each of which is (i) accessible to the first module and (ii) specified as an input to at least one of the one or more second modules at least partially based on the output of the first module; and storing the computer program in a persistent memory, the stored computer program storing instructions for performing operations including specifying the output data format set for each of the one or more first modules; and one or more machine-readable hardware storage devices.

[0027] In aspect 23 according to aspect 22, the content of one or more parts of the computer program is automatically generated without requiring user input, unlike the specification.

[0028] In aspect 24 according to any one of aspects 22 to 23, the computer program is persistent and editable.

[0029] In embodiment 25, which is any one of embodiments 22 to 24, the computer program is a dataflow graph, each module is a component of the dataflow graph represented by nodes of the dataflow graph, and the dataflow graph components are connected by data flows represented by links between nodes.

[0030] In embodiment 26, which is any one of embodiments 22 to 25, the operation further includes accessing a data structure in a data storage system that maps data representing a part of a specification to data flow graph components; identifying data items in the data structure that represent a particular part of the specification; identifying which data flow graph components in the data structure are mapped to that particular part of the specification; and adding the identified data flow graph components to the data flow graph in the order of execution of operations specified by the specification.

[0031] In embodiment 27, which is one of embodiments 22 to 26, the operation further includes identifying dependencies between dataflow graph components and specifying the output data format of the dataflow graph components based at least partially on these identified dependencies.

[0032] In an embodiment 28, which is any one of embodiments 22 to 27, the operation further includes: specifying that, for a particular first module, only one of one or more second modules accesses a particular one or more fields accessible to the particular first module; specifying that only one of the one or more second modules accesses a particular one or more fields from a third module of a plurality of modules, which is different from each of the particular first module and the one or more second modules; and setting the output data format so as not to output one or more values ​​for a particular one or more fields accessed by only one of the one or more second modules.

[0033] In embodiment 29, which is any one of embodiments 22 to 28, the operation further includes providing data for display on a user interface, receiving data via the user interface representing the selection of one or more modules from among a plurality of modules to be included in a computer program, and receiving, for each selected module, the parameter values ​​of the operation implemented by the selected one or more modules, via the user interface.

[0034] In embodiment 30, which is any one of embodiments 22 to 29, the operation further includes determining whether a value received via the user interface conforms to a predetermined criterion associated with the operation, and outputting a notification via the user interface indicating whether the received value conforms to the predetermined criterion, wherein if the received value is determined not to conform to the predetermined criterion, the notification includes guidance on how the received value must be modified to conform to the predetermined criterion.

[0035] In embodiment 31, which is one of embodiments 22 to 30, the operation further includes configuring a selected module with parameter values ​​and then storing the configured module which will be included in and implemented by another computer program.

[0036] In embodiment 32, which is any one of embodiments 22 to 31, the operation further includes, before executing a computer program, performing first, second and third passes for each specific module of a plurality of modules to identify and save a configured output data format for that specific module that specifies which fields are required by the downstream modules of that specific module, and to identify and save an input data format for that specific module that specifies which fields are accessible to that specific module, during the first pass, that the specific module is required by the upstream modules of that specific module by itself and by any downstream modules of that specific module The process involves broadcasting one or more messages containing data representing one or more fields, wherein the specific module performing the broadcast is a broadcast module, and the upstream modules receiving the broadcast are receiving modules; during a second pass, the receiving modules send one or more messages to the broadcast module specifying which receiving modules can provide the values ​​for the required fields; and during a third pass, the broadcast module analyzes the messages received from the receiving modules and, in response, sends back one or more messages to the receiving modules specifying which receiving modules are responsible for sending which fields to the broadcast module.

[0037] In an embodiment 33 according to any one of embodiments 22 to 32, the operation further includes receiving a structured data item in a data stream that includes one or more data records; executing a computer program on at least one of the one or more data records to process at least one of the one or more data records; and removing one or more fields from at least one of the records that are not designated as input to at least one of the one or more second modules, at least partially based on the output of the first module, before submitting at least one of the one or more records to one of the one or more second modules, based on the stored output data format of at least one of the first modules.

[0038] In an embodiment 34 relating to any one of embodiments 22 to 33, the operation further includes defining one or more new modules in a specification and storing at least one of the one or more new modules in a global palette that specifies entities that can be used for inclusion in other specifications.

[0039] In an embodiment 35 relating to any one of embodiments 22 to 34, the operation further includes defining one or more new modules in a specification, and saving at least one of the new modules to a local palette that specifies entities that can only be used to include in that specification and cannot be included in other specifications.

[0040] In a mode 36, which is any one of modes 22 to 35, the operation further includes performing an operation that includes accessing a predefined module included in the global palette for the purpose of defining a specification, modifying one or more attributes of the predefined module, storing the modified predefined module in a local palette that specifies entities that can only be used for inclusion in that specification and cannot be included in other specifications, or saving the modified predefined module to the global palette so that the modified predefined module can also be accessed by other specifications.

[0041] In an embodiment 37, which is one of embodiments 22 to 36, the operation further includes defining one or more new first modules in the specification, defining one or more new second modules in the specification, and configuring at least one of the one or more new first modules to be a data source for at least one of the one or more new second modules.

[0042] In embodiment 38, which is one of embodiments 22 to 37, the module includes one or more other modules.

[0043] In embodiment 39, which is one of embodiments 22 to 38, the operation further includes configuring a selected module with parameter values ​​and storing the configured module for later inclusion in another program.

[0044] In embodiment 40, which is one of embodiments 22 to 39, the operation further includes specifying that only one of the one or more second modules accesses a particular one or more fields accessible to the first module; specifying that only one of the one or more second modules accesses a particular one or more fields from a third module different from the first module; and setting the output data format so as not to output one or more values ​​for a particular one or more fields accessed by only one of the one or more second modules.

[0045] In embodiment 41, which is one of embodiments 22 to 40, the structured data item includes a data record.

[0046] In a general embodiment 42, one or more machine-readable hardware storage devices for converting a specification into a computer program that processes one or more values ​​of one or more fields of a structured data item, the one or more machine-readable hardware storage devices access a specification that specifies a plurality of modules which will be implemented by a computer program to process one or more values ​​of one or more fields in a structured data item, and convert the specification into a computer program which implements the plurality of modules, wherein for each of the first modules of the plurality of modules, each of the plurality of modules is at least partially an output of the first module The computer program stores instructions for performing operations including identifying one or more second modules that receive force-based inputs, converting the output data format of the first module so that the first module outputs only one or more values ​​of one or more fields of a structured data item, each of which is (i) accessible to the first module and (ii) at least partially designated as input to at least one of the one or more second modules based on the output of the first module, and saving a computer program in persistent memory, the saved computer program storing instructions for performing operations including saving the output data format set for each of the one or more first modules.

[0047] In the embodiment 43 according to embodiment 42, the content of one or more parts of the computer program is automatically generated without requiring user input, unlike in the specifications.

[0048] In embodiment 44, which is any one of embodiments 42 to 43, the computer program is persistent and editable.

[0049] In embodiment 45, which is any one of embodiments 42 to 44, the computer program is a dataflow graph, each module is a component of the dataflow graph represented by nodes of the dataflow graph, and the dataflow graph components are connected by data flows represented by links between nodes.

[0050] In a mode 46, which is any one of modes 42 to 45, the operation further includes accessing a data structure in a data storage system that maps data representing a part of a specification to data flow graph components; identifying data items in the data structure that represent a particular part of the specification; identifying which data flow graph components in the data structure are mapped to that particular part of the specification; and adding the identified data flow graph components to the data flow graph in the order of execution of operations specified by the specification.

[0051] In embodiment 47, which is one of embodiments 42 to 46, the operation further includes identifying dependencies between dataflow graph components and specifying the output data format of the dataflow graph components based at least partially on these identified dependencies.

[0052] In an embodiment 48 according to any one of embodiments 42 to 47, the operation further includes: specifying that, for a particular first module, only one of one or more second modules accesses a particular one or more fields accessible to the particular first module; specifying that only one of the one or more second modules accesses a particular one or more fields from a third module of a plurality of modules, different from each of the particular first module and the one or more second modules; and setting the output data format so as not to output one or more values ​​for a particular one or more fields accessed by only one of the one or more second modules.

[0053] In a manifestation 49, which is any one of manifestations 42 to 48, the operation further includes providing data for display on a user interface, receiving data via the user interface representing the selection of one or more modules from among a plurality of modules to be included in a computer program, and receiving, for each selected module, the values ​​of parameters for the operation implemented by the selected one or more modules, via the user interface.

[0054] In embodiment 50, which is any one of embodiments 42 to 49, the operation further includes determining whether a value received via a user interface conforms to a predetermined criterion associated with the operation, and outputting a notification via the user interface indicating whether the received value conforms to the predetermined criterion, wherein if the received value is determined not to conform to the predetermined criterion, the notification includes guidance on how the received value must be modified to conform to the predetermined criterion.

[0055] In embodiment 51 according to any one of embodiments 42 to 50, the operation further includes configuring a selected module with parameter values ​​and then storing the configured module which will be included in and implemented by another computer program.

[0056] In embodiment 52, which is one of embodiments 42 to 51, the operation further includes, before executing a computer program, performing first, second and third passes for each specific module of a plurality of modules to identify and save a configured output data format for that specific module that specifies which fields are required by the downstream modules of that specific module, and to identify and save an input data format for that specific module that specifies which fields are accessible to that specific module, during the first pass, that the specific module is required by the upstream modules of that specific module by itself and by any downstream modules of that specific module The process involves broadcasting one or more messages containing data representing one or more fields, wherein the specific module performing the broadcast is a broadcast module, and the upstream modules receiving the broadcast are receiving modules; during a second pass, the receiving modules send one or more messages to the broadcast module specifying which receiving modules can provide the values ​​for the required fields; and during a third pass, the broadcast module analyzes the messages received from the receiving modules and, in response, sends back one or more messages to the receiving modules specifying which receiving modules are responsible for sending which fields to the broadcast module.

[0057] In an embodiment 53 according to any one of embodiments 42 to 52, the operation further includes receiving a structured data item in a data stream that includes one or more data records; executing a computer program on at least one of the one or more data records to process at least one of the one or more data records; and removing one or more fields from at least one of the records that are not designated as input to at least one of the one or more second modules, at least partially based on the output of the first module, before submitting at least one of the one or more records to one of the one or more second modules, based on the stored output data format of at least one of the first modules.

[0058] In embodiment 54, which is one of embodiments 42 to 53, the operation further includes defining one or more new modules in a specification and storing at least one of the one or more new modules in a global palette that specifies entities that can be included in other specifications.

[0059] In an embodiment 55 relating to any one of embodiments 42 to 54, the operation further includes defining one or more new modules in a specification, and storing at least one of the new modules in a local palette that specifies entities that can only be used to include in that specification and cannot be included in other specifications.

[0060] In a manifestation 56, which is any one of manifestations 42 to 55, the operation further includes performing an operation that includes accessing a predefined module included in the global palette for the purpose of defining a specification, modifying one or more attributes of the predefined module, storing the modified predefined module in a local palette that specifies entities that can only be used for inclusion in that specification and cannot be included in other specifications, or saving the modified predefined module to the global palette so that the modified predefined module can also be accessed by other specifications.

[0061] In an embodiment 57 according to any one of embodiments 42 to 56, the operation further includes defining one or more new first modules in the specification, defining one or more new second modules in the specification, and configuring at least one of the one or more new first modules to be a data source for at least one of the one or more new second modules.

[0062] In embodiment 58, which is one of embodiments 42 to 57, the module includes one or more other modules.

[0063] In embodiment 59, which is one of embodiments 42 to 58, the operation further includes configuring a selected module with parameter values ​​and storing the configured module for later inclusion in another program.

[0064] In embodiment 60, which is one of embodiments 42 to 59, the operation further includes specifying that only one of the one or more second modules accesses a particular one or more fields accessible to the first module; specifying that only one of the one or more second modules accesses a particular one or more fields from a third module different from the first module; and setting the output data format so as not to output one or more values ​​for a particular one or more fields accessed by only one of the one or more second modules.

[0065] In embodiment 61, which is one of embodiments 42 to 60, the structured data item includes a data record.

[0066] There are numerous advantages to configuring the output data format of the first module so that the first module outputs only one or more values ​​of one or more fields of a structured data item, each of which is designated as an input to at least one of one or more second modules, each of which is (i) accessible to the first module and (ii) at least partially based on the output of the first module. These advantages include reduced consumption of computational resources (such as memory and CPU) because only the data fields necessary to satisfy the data processing requirements of downstream data processing within the computer program are sent to the computer program and processed more deeply. Furthermore, reduced latency in data record processing is achieved because the data processing system can process smaller records (e.g., by dropping fields) more quickly than the data processing speed if the data processing system had to process the entire data record through the data flow graph. Other advantages include reduced memory and resource consumption because smaller records (e.g., by dropping fields) require less memory space to be consumed during record processing.

[0067] Other features and advantages will become apparent from the following description and claims. [Brief explanation of the drawing]

[0068] [Figure 1A] This is a schematic diagram of a database management system. [Figure 1B] This is a schematic diagram of a data processing system. [Figure 2A1] This is an exemplary graphical user interface for defining modules. [Figure 2A2] This is an exemplary graphical user interface for defining modules. [Figure 2B] This is an example of converting specifications into a persistent computer program. [Figure 2C] This is an example of converting specifications into a persistent computer program. [Figure 2D] This is an example of converting specifications into a persistent computer program. [Figure 2E] This diagram shows the process of receiving user input for a specification document and generating entities from that specification document. [Figure 3A] This is a diagram of a data flow graph that includes entities. [Figure 3B] This diagram shows the process of receiving user input for a specification document and generating a computer program from that specification document. [Figure 4] This is a diagram illustrating the processing of data records through a data flow graph. [Figure 5] This is a diagram illustrating the process flow for field identification. [Figure 6] This is an example of a user interface for specifying parameter values ​​for a specification document. [Figure 7] This is an example of a user interface for specifying parameter values ​​for a specification document. [Figure 8] This is an example of a user interface for specifying parameter values ​​for a specification document. [Figure 9]This is a diagram of a data flow graph that includes entities. [Figure 10] This diagram illustrates the process of converting specifications into entities for generating a data flow graph. [Figure 11] This is a diagram illustrating the process of converting specifications into a computer program. [Modes for carrying out the invention]

[0069] Each of the examples and optional features described below may be combined with any of the accompanying claims. Furthermore, each of the examples and optional features described below may be combined with any of the other examples and optional features described below, and this combination may be combined with any of the accompanying claims.

[0070] Referring to Figure 1A, System 1, which converts a specification into a computer program such as a dataflow graph, includes a conversion environment 5 which includes a specification configuration module 7, an optional entity generation module 6, and a user interface module 8, as well as a data storage system 3 and a data processing system 4. Generally, an "entity" includes a part of a computer program (for example, a predefined portion of one computer program to be included in another computer program) or one or more dataflow graph components (for example, those encapsulated in a predefined module). Throughout this document, "entities" may be referred to as "modules" in an unrestricted and convenient manner. Dataflow graph components may include data processing components and / or datasets, such as data sinks and / or data sources. A dataflow graph can be represented as a directed graph, which includes nodes or vertices representing dataflow graph components, connected by directed links or dataflow connections that represent the flow of work elements (i.e., data) between dataflow graph components. A data processing component may include code that processes data from at least one data input and provides the data to at least one data output of the data processing component. A dataflow graph, therefore, can implement graph-based computations performed on data flowing from one or more input datasets through graph components to one or more output datasets.

[0071] Generally, specification configuration module 7 contains modules for constructing specifications. There are various types of specifications, including, for example, specifications for entities and specifications for data flow graphs. Generally, a specification for an entity may identify data flow graph components and specific components that can perform specific functions for that purpose while the user is constructing the graph, such as viewing sample data. A specification for a data flow graph may define which entities to include in the data flow graph (i.e., data flow graph components), the parameter values ​​for such included entities, and the data flow between the included entities.

[0072] System 1 also includes a data processing system 4 for executing one or more computer programs (such as data flow graphs), which are generated by converting specifications into computer programs using a conversion environment 5 and the techniques described herein. The data processing system 4 may be hosted on one or more general-purpose computers under the control of a suitable operating system, for example, a UNIX® operating system. For example, the data processing system 4 may include a multi-node parallel computing environment, which includes a configuration of a computer system using multiple central processing units (CPUs) that may be local (e.g., a multiprocessor system such as an SMP computer), locally distributed (e.g., multiple processors connected as a cluster or MPP), remotely distributed (e.g., multiple processors connected via a LAN or WAN network), or a combination thereof.

[0073] The entity generation module 6 (which may be optional in some examples) generates entities, which will be described in more detail below. The user interface module 8 (which may be optional in some examples) displays configuration information to the user and receives data from the user representing configuration actions. In particular, the user interface module 8 displays a visual representation that allows the user to select which entities to include in a computer program or data flow graph and to input parameter values ​​for these entities. The term “parameter” as used herein may affect the outcome of the actions performed or implemented by the entities, and therefore may also be called “operational parameters.” The user interface module 8 can also generate entities by the user inputting parameter values ​​that specify, for example, which components to include in an entity and the configuration of the components in relation to each other. The user interface module 8 also communicates with the specification configuration module 7, which configures the specification based on the user’s actions in the user interface (e.g., the user’s own selections or inputs described above). In this example, the specification configuration module 7 configures the specification based on the selected entities (or entity definitions if the user defines new entities) or entities otherwise specified in the user interface. Based on the specifications, the entity generation module 6 generates entities for inclusion in or implementation within the data flow graph, and for execution. Generally, entities, data flow graphs, and / or specifications can be stored in the data storage system 3, which is accessible to the transformation environment 5.

[0074] The conversion environment 5 communicates with the data storage system 3, which contains data used by the user interface module 8 to display the user interface. The data storage system 3 is also accessible to an optional development environment 2, where a developer 9 can develop a user interface, which is stored in the data storage system 3 and can then be used by the user interface module 8 to display user interfaces such as user interfaces 10, 11, 15, or 38 described later. In some implementations, the conversion environment 5 is a system for developing applications as a data flow graph containing entities.

[0075] As shown in Figure 1B, the data processing system 4 may include one or more data sources 9a, which, in addition to one or more data sources, include an execution environment 9 connected to the data storage system 3 (Figure 1A) and hosted on one or more computers, the execution environment 9 may include a pre-execution module 9b configured to allocate and configure computing resources (e.g., CPU and / or primary memory) for executing calculations (data processing operations, etc.) of a computer program, such as data processing operations performed by a data flow graph component. The execution environment 9 may also include an execution module 9c for scheduling and controlling the execution of components of the computer program. The computer program may be specified by programming code and / or data structures stored in the data storage system 3, which may be read by the pre-execution module 9b or the execution module 9c during the execution of the computer program.

[0076] An entity is part of a computer program that runs within the execution environment and processes data from one or more data sources 9a. The data from the data sources is manipulated and processed according to entities included in the computer program or data flow graph and exported to one or more data sinks 9d. In other words, the execution module 9c is configured to process data from data sources 9a according to a computer program, for example, a data flow graph, using the computing resources of the execution environment 9 allocated to the computer program by the pre-execution module 9b, thereby generating output data which may be exported to one or more data sinks 9d. The data sources and sinks, 9a and 9d, respectively, may include, for example, files, databases, data streams, or queues.

[0077] As mentioned above, each data processing component may contain code to process data from at least one data input and provide the data to at least one data output. Entities are included in a data flow graph, which also includes nodes representing dataset objects for accessing data sources and / or sinks. Nodes are connected by directed links representing data flows between components, starting at data sources and ending at data sinks. Data output ports of upstream data flow graph components are connected to data input ports of downstream data flow graph components. Data flow graphs and / or entities may be reused for different data sources and different data sinks represented by dataset objects. The data structures and computer program code used to implement the data flow graphs and / or entities can accommodate multiple different configurations by parameterizing them through parameters, for example, to allow easy substitution of different sources and sinks. Furthermore, in some arrangements, the flow of entities may be modified by the use of parameters, thereby bypassing certain components or sets of components. Generally, parameters represent characteristics of an entity, which are configurable or modifiable and affect the behavior performed by that entity. In general, the parameters or characteristics of an entity can be changed each time the entity is used, and the behavior of the entity (e.g., data processing) may differ as a result of the changed characteristics or parameters.

[0078] In some cases, constructed entities can be of a highly technical nature. The basic structure and construction of an entity are determined based on technical considerations. For example, graph components may be chosen to maximize reusability or to support parallel processing. Some of the parameters associated with an entity can be used to allow non-computationally savvy users to customize the entity without needing to understand the technical complexity behind its implementation. Parameterized entities simplify customization and can be easily reused for a wider variety of applications, situations, and user backgrounds.

[0079] A user interface for identifying entities and associated parameter values ​​for constructing a dataflow graph and / or entities can be presented on a client machine or user device. In some embodiments, the client may access a development environment 2 (Figure 1A) running on a server using a web browser on the client providing the parameter interface, and may use a scripting language that provides some capabilities for client-side processing. The scripting language may communicate with the server to update parameters and perform other necessary actions. This communication may occur via a bridge machine, which translates communication between the client and the server running the development environment that stores the objects and associated parameter values ​​for the graph and entities being constructed.

[0080] The user interface allows users to select entities to include in a dataflow graph and configure their parameters, even without technical knowledge of dataflow graphs and their configuration. In particular, the dataflow graph is transformed into executable logic for processing data records. That is, the execution of this executable logic generates code (e.g., compiled code) for processing the records.

[0081] In the variant, System 1 also includes data sources, which may include one or more data sources such as storage devices or connections to online data streams, each of which may store data in any of various storage formats (e.g., database tables, spreadsheet files, flat text files, or native formats used by the mainframe). These data sources may be the same as one or more data sources 9a, or they may be additional data sources. In either case, the storage device providing the data sources may be local to the data processing system 4, for example, on a storage medium connected to a computer running the data processing system 4 (e.g., a hard drive), or remote to the data processing system 4, for example, hosted on a remote system that communicates with a computer running the data processing system 4 locally or on a wide area network (e.g., a mainframe).

[0082] Referring to Figure 2A, schematic Figure 10 shows a graphical user interface (hereinafter referred to as "graphical user interface 10") for configuring one or more modules or entities. In this example, the graphical user interface 10 includes a palette section 10a for selecting one or more pre-configured modules (i.e., visual representations of modules 10c to 10f). Generally, the palette section 10a represents multiple modules that can be included in a computer program or data flow graph (for example, also called a module palette). The palette section 10a may also include additional controls 10i for creating new modules that will be added to the multiple modules. The graphical user interface 10 may also include a configuration section 10b for configuring new modules.

[0083] In this example, when the user selects an additional control 10i to create a new module that will be added to multiple modules, component 10b prompts the user to configure the new module by presenting an input area 10g for entering information that specifies the module, such as information specifying the module name. Component 10b may also include a data source input box 10h for specifying the data source for the module specified by input area 10g. In this example, the user may enter the name of a data file in the data source input box 10h, or drag and drop a data source module (represented, for example, by one of the visual representations 10d or 10e) into the data source input box 10h. In this example, the user drags and drops the visual representation 10d into the data source input box 10h, specifying that “A_customers” is the data source for the “Filtered Customer Deals” module. Component 10b may also include controls 10j-10p for adding and configuring one or more parameters, transformations, or behaviors relating to the module created or defined within component 10b. In this example, each of controls 10j to 10p allows the user to select values ​​for parameters, actions, or transformations, thereby configuring those parameters, actions, or transformations. For example, control 10j specifies that the module being created includes a filtering action, and the filtering action may specify the output data format of the module, for example, as described herein. In this example, the user selects a filter option from a dropdown box included in control 10j. Control 10o is placed alongside control 10j. By selecting control 10o, the user can configure the filtering action specified by control 10j. In this example, selecting control 10o displays overlay 10q. Overlay 10q includes controls 10r and 10s, and an input section 10t for specifying the conditions or expressions that will be performed (e.g., evaluated) by the filtering action.

[0084] Control 10k specifies that the module being created includes a join operation. In this example, the user selects a join option from a dropdown box in control 10k. Control 10p is placed alongside control 10k. By selecting control 10p, the user can configure the join operation specified by control 10k. In this example, selecting control 10p displays overlay 10u. Overlay 10u contains controls 10v~10z that specify the actions or operations to be implemented as part of the join operation. In this example, records output from the A_customer module (e.g., formatted records showing customer transactions) are joined with records output from the B_name_address module (e.g., formatted records specifying customer names and addresses) to create a dataset containing customer names and addresses for various transactions. Control 10v allows the user to specify whether the key name for the data source specified in data source input box 10h is different from the key name specified by the data source specified in control 10w. For example, the user may join two data sources that have different formats for a particular data field or key, such as the customer identifier key. In this example, the data source specified in control 10w is the B_name_address module specified by the visual representation 10e. In this example, if the user specifies a different key name via control 10v, the overlay 10ae is displayed. In this example, the overlay 10ae allows the user to specify various key names for various data sources. In this example, the control 10ag allows the user to specify a key name for the data source specified in control 10af. In addition, the control 10ai allows the user to specify a key name for the data source specified by control 10ah. Component 10b includes an additional control 10n, which, depending on its selection, allows the user to add additional transformations.In this example, the customer transaction module (represented by visual representation 10c) is identical to the filtered customer transaction module, except that the filtering operation is omitted.

[0085] Referring to Figure 2B, schematic diagram 11 shows a “real-world” example of the conversion of a specification to a program (e.g., a computer program), which in this case is a data flow graph. In this example, schematic diagram 11 shows a graphical user interface 12, which may be separate from or different from the graphical user interface 10 described above with respect to Figures 2A and 2B, and includes a palette section 12a and a component section 12e. Palette section 12a displays several modules that can be used to define a computer program. In this example, palette section 12a displays visual representations 12b, 12c, 12d, 12da, 12db, 12dc, and 12dd. In this example, visual representation 12b represents the filtered customer transaction module defined in Figure 2A. Visual representation 12c represents a previously defined customer transaction module, for example, the same customer transaction module represented by visual representation 10c in Figure 2A. Visual representation 12d represents the offer submission module, which will be described later. Visual representations 12da and 12db represent the A_customers module (for example, specifying a particular data source and how to format the data records of that data source) and the B_name_address module (for example, specifying another specific data source and appropriate reformatting). In this example, visual representations 12da and 12db represent the same data sources as those represented by visual representations 10d and 10e. Visual representation 12dc represents an output module that specifies the output file where the data will be stored. In this example, visual representation 12dc represents the same output module as represented by visual representation 10f in Figure 2A. Each of the visual representations 12b, 12c, 12d, 12da, 12db, 12dc, and 12dd is selectable and can be "drag-and-dropped" onto component 12e to add the module to a computer program, for example. Component 12e provides a user interface for specifying parameters for modules selected from palettes (e.g., visual representations 12b, 12c, 12d, 12da, 12db, 12dc, 12dd) displayed within palette component 12a, which represent modules to be included in a computer program or data flow graph.Component 12e includes component 12ac (represented by visual representation 12b, for modifying a previously defined filtered customer transaction module) and components 12aa and 12ab for defining new modules, namely the offer submission module (represented by visual representation 12d) and the output file module (represented by visual representation 12dd).

[0086] Section 12ac displays controls 12f, 12i, 12j, 12k, 12m, and 12n for modifying one or more parameters or attributes of the filtered customer transaction module represented by the visual representation 12b. In particular, each of 12f, 12i, 12j, 12k, 12m, and 12n is pre-populated based on the data specified in the data source input box 10h (Figure 2A) and the selections or data entered into controls 10r, 10s, input section 10t, and controls 10v, 10w, and 10y (Figure 2A), respectively. The data for each of controls 12f, 12i, 12j, 12k, 12m, and 12n is modifiable to allow user customization of the module.

[0087] In this example, after changing the values ​​of one or more parameters or attributes adjustable by one or more controls 12f, 12i, 12j, 12k, 12m, and 12n, the changes can be saved as a global save via the global save control 12g or as a local save via the local save control 12h. In this example, the global save updates the module across all palettes in the global palette (not just palette 12a, but currently and in the future used in the graph definition) and across the graphs and computer programs that access that module. Generally, the global palette is a definition or list of modules that are globally available for the graph definition. That is, these modules in the global palette are not limited only to the specification or graph being defined. A local save updates the module locally, i.e., only for the specific palette being displayed and only for the specific graph being defined. That is, with a local save, the scope of the module's changes is limited to that palette and graph only. In this example, the user has selected the global save control (indicated by the dotted line around the global save control 12g). Based on the selection of global save control 12g, changes to the filtered customer transaction module are made globally, i.e., across all palettes, and across all graphs and computer programs containing that module. If the user instead selects local save control 12h, the changes will apply only to palette 12a and the graphs defined in component 12e.

[0088] Component 12e also includes component 12aa, which has controls 12oa, 12o-12u for configuring and defining a new module - the Offer Sending Module. Control 12oa specifies the data source for the Offer Sending Module. In this example, the data source for the Offer Sending Module is the Filtered Customer Deals Module. In this example, controls 12o-12q specify the triggers that should occur, i.e., send an SMS message to the user asking if they want to update unlimited text. In this example, control 12r specifies a particular event to wait for (i.e., to be received before the action specified by the subsequent trigger is performed). Controls 12s-12u specify the action or trigger to be performed (e.g., sending an SMS message that says "Upgrade Texting for $12.99 / month"). In this example, since the Offer Sending Module is newly defined in component 12aa, there is no option for local save (which would mean that a previously defined version of the module would be saved locally for palette 12a). In this example, part 12aa contains the global save control 12v, and when selected, the offer send module is added to the global palette, which is the palette used in the generation of all graphs. In this example, the user has not selected the global save control 12v. Therefore, the offer send module is only available on palette 12a, which is the local palette. In this example, the configuration of the offer send module itself causes the visual representation 12d to be included in palette 12a as a module that is being defined locally and is available locally for the graph being defined through component part 12e.

[0089] Component 12e also includes component 12ab for defining an output file module (represented by visual representation 12dd) in which the results of sending the offer are stored. Component 12ab includes a control 12w for specifying the data source for the output file module. In this example, control 12w specifies that the data source is the offer sending module. The user may specify this by dragging and dropping visual representation 12d onto control 12w. Component 12ab also includes a control 12x for specifying the data target. In this example, the user drags and drops visual representation 12dd onto control 12x to specify that the output module (represented by visual representation 12dc) is the data target for the output file module defined through component 12ab.

[0090] Generally, and therefore applicable to all examples and embodiments, the contents of component 12e (and / or data generated from or based on the contents of component 12e) define a specification that specifies which modules should be included in the computer program and the parameter values ​​for those modules. Component 12e may also include a submission control 12t, which, if selected, the computing system, for example, the conversion environment 5 shown in Figure 1A, converts the specification specified by the contents of component 12e into a computer program, which in this example is represented by the computer program 14 (in this example, a data flow graph).

[0091] In this example, computer program 14 includes data sources 14a and 14b, a filtered customer transaction module 14c, an offer sending module 14d, and an output file source 14e. In this example, the computerized system configures data source 14a to access data records and / or files from “A_Customers,” specifying a particular external module (e.g., a data warehouse) from which to extract data. The computerized system configures data source 14b to be the B_name_address module. The computerized system configures the filtered customer transaction module 14c according to the information entered into part 12ac for the filtered customer transaction module 14c. In particular, the filtered customer transaction module 14c includes a data flow graph 14y having components (e.g., data flow graph components) 14f, 14h-14l. In this example, component 14f is a filter component (hereinafter referred to as filter component 14f). Components 14h and 14i are partition components (hereinafter referred to as partition components 14h and 14i). Components 14j and 14k are sorting components (hereinafter referred to as sorting components 14j and 14k). Component 14l is a join component (hereinafter referred to as join component 14l).

[0092] In this example, the filter component 14f is configured according to the inputs specified in controls 12i to 12k. That is, the filter component 14f is configured to exclude records that meet a predetermined condition, for example, data records with a total transaction amount less than $260. In this example, the join component 14l is configured to join data records that match or have similar values, according to the information specified by controls 12m to 12n, i.e., for the customer ID field. In this example, the partition components 14h, 14i and the sort components 14j, 14k are not specified in the specification, i.e., not specified by the input to component 12e. However, the computerized system includes the partition components 14h, 14i and the sort components 14j, 14k based on executable logic (by the computerized system) that specifies that there is a partition component before the join component and a sort component before that. In this example, the offer sending module 14d is configured according to the information entered in controls 12oa, 12o to 12u. In particular, the offer sending module 14d is configured to wait for a data record (e.g., an event) that indicates that a response has been sent. When that event is detected, the offer sending module 14d is configured to send an SMS message that says, "Upgrade to unlimited texting for $12.99 / month." The output file source 14e is configured to save the output of the execution of the computer program 14 to a data repository according to the contents of part 12ab.

[0093] In this example, each of the data sources 14a and 14b, the filtered customer transaction module 14c, and the offer sending module 14d includes output ports 14m through 14p, which specify the output data format that determines which fields (e.g., from structured data items such as data records received from data sources 14a and 14b) are output from that source or module. In other words, each module and source is configured to output only the values ​​of fields that are accessible to that module / source and accessed by one or more downstream modules. In this example, the filtered customer transaction module 14c includes input ports 14q and 14r (for the data sources specified by controls 12f and 12m) and output port 14o (e.g., to output data and / or data records that are input to the offer sender module 14d), the offer sending module 14d includes input port 14s and output port 14p, and the output file source 14e includes input port 14t. The input port is configured to receive one or more structured data items from other modules. The input port may also be configured with one or more data formats required for the received structured data items.

[0094] Referring to Figure 2C, schematic diagram 15 shows a variation of Figure 2B, in which the filtered customer transaction module (for example, the filtered customer transaction module generated through the graphical user interface 10 in Figure 2A) is used as part of a different computer program than the computer program 14 shown in Figure 2B.

[0095] Schematic Figure 15 shows a graphical user interface 16, which includes a palette section 16a and a component section 16e. The palette section 16a displays multiple modules available for use in the definition of a computer program. In this example, the palette section 16a displays visual representations 16b-16d and 16da-16dd. Each of the visual representations 16b-16d and 16da-16dd is selectable and can be added to the computer program or in the definition of a new module, for example, by "dragging and dropping" it onto the component section 16e. In this example, visual representation 16b represents a filtered customer transaction module, which was initially created through the input specification to the graphical user interface 10, as shown in Figure 2A, and then updated (through global save) in section 12ac of Figure 2B. In this example, the palette section 16a does not include a visual representation for the offer submission module shown in Figure 2B, because the offer submission module is defined only locally and was not added to the global palette. Component 16e provides a user interface for selecting data (e.g., one of the visual representations 16b-16d and 16da-16dd) representing modules to be included in a computer program or data flow graph from the palette displayed in palette component 16a. In this example, component 16e includes components 16h, 16aa, and 16ab.

[0096] In this example, part 16h provides changes (via controls 16f, 16i-16k, and 16m-16n) to the filtered customer transaction module, which is initially defined in Figure 2A and modified in Figure 2B. In this example, the user saves changes to the filtered customer transaction module locally via the selection of local save control 16o. In this example, part 16h also includes a global save control 16p, providing the option to save changes to the filtered customer transaction module globally (i.e., defining a global scope rather than a local scope for the changes). Each of controls 16f, 16i-16k, and 16m-16n is pre-populated based on the selection of controls 10r, 10s, 10t, 10v, 10w, 10y (Figure 2A) or the data entered into them, and is then modified via controls 12f, 12i-12k, and 12m-12n in Figure 2B. The data for controls 16f, 16i-16k, and 16m-16n can be modified to allow users to customize the filtered customer transaction module.

[0097] Component 16e also includes component 16aa, which has controls 16q~16v for configuring the deal aggregation module, newly defined in component 16aa. In this example, control 16q specifies the data source for the deal aggregation module. In this example, the data source is the filtered customer deal module. A visual representation 16b representing the filtered customer deal module is dragged and dropped onto control 16q. Control 16r is for specifying the data source to be merged (combined) with the output from the filtered customer deal module. Control 16s is for receiving data that specifies the key to perform a data rollup from records output from the filtered customer deal module, for example, contained in Warehouse_A.dat. Control 16t specifies the actions to be performed on the fields specified by control 16u (included in the rollup record) and other fields specified by control 16v (included in the rollup record). In this example, component 16aa includes a global save control 16w that adds the deal aggregation module to the global palette so that it is accessible to all users, for example, in all palettes. Since the transaction aggregation module is newly defined in part 16aa, there is no local save control, for example, to save a modified version of a globally accessible module locally.

[0098] Component 16e also includes component 16ab for defining an output file module for saving the output. In this example, control 16x specifies that the data source for the output file module is a transaction aggregation module. The user may specify this by dragging and dropping the visual representation 16d onto control 16x. Additionally, control 16y specifies that the output module represented by the visual representation 16dc is the data target for the data file module defined in component 16ab. In this example, when the user specifies values ​​for controls 16x and 16y, palette 16a is updated to include the visual representation 16dd representing the output file module.

[0099] Component 16e also includes a submission control 16ac, which, upon selection, enables the computing system, for example, the conversion environment 5 in Figure 1A, to convert the specifications specified by the contents of component 16e into a computer program, which in this example is represented by computer program 18 (which is a data flow graph in this example).

[0100] In this example, the computer program 18 includes data sources 18a and 18b, a filtered customer transaction module 18c, a transaction aggregation module 18d, and an output file source 18e. In this example, the computerized system configures data source 18a to access data records and / or files from the A_Customers module. The computerized system configures data source 18b to become the B_name_address module. The computerized system configures the filtered customer transaction module 18c according to the information entered into section 16h for the filtered customer transaction module 18c. In particular, the filtered customer transaction module 18c includes a data flow graph 18y having components 18f, 18h to 18l. In this example, component 18f is a filter component (hereinafter referred to as filter component 18f). Components 18h and 18i are partition components (hereinafter referred to as partition components 18h and 18i). Components 18j and 18k are sort components (hereinafter referred to as sort components 18j and 18k). Component 18l is a coupling component (hereinafter referred to as coupling component 18l).

[0101] In this example, the filter component 18f is configured according to the inputs specified in controls 16g, 16i-16k. That is, the filter component 18f is configured to exclude data records that meet a predetermined condition, for example, records with a total transaction amount less than $250. In this example, the join component 18l is configured to join data records that match or have similar values ​​for the customer ID field, according to the information specified by controls 16m-16n. In this example, the partition components 18h, 18i and the sort components 18j, 18k are not specified in the specification, i.e., they are not specified by the input to component 16e. However, the computerized system includes the partition components 18h, 18i and the sort components 18j, 18k, which is based on the execution (by the computerized system) of executable logic that specifies that the partition components precede the join components, and the sort components precede those. In this example, the transaction aggregation module 18d is configured according to the information entered into component 16aa. In particular, the transaction aggregation module 18d is configured to aggregate the value in the "Total Transaction Amount" field with the value in the "Total Transaction Amount to Today" field. The output file source 18e is configured to save the output of the execution of the computer program 18 to a data repository, as specified by the output file module defined in section 16ab. In this example, each of the data sources 18a, 18b, the customer transaction module 18c, and the transaction aggregation module 18d includes output ports 18m to 18p, each specifying an output data format that specifies which fields (e.g., from data records received from data sources 18a, 18b) are output from that source or module. As described above and below, each module and source is configured to output only the values ​​of fields that are accessible to that module / source and accessed by one or more downstream modules.In this example, the customer transaction module 18c includes input ports 18q and 18r, the transaction aggregation module 18d includes input port 18s, and the output file source 18e includes input port 18t. The input ports are configured to receive one or more structured data items from other modules. The input ports may also be configured with one or more data formats required for the received structured data items.

[0102] Referring to Figure 2D, schematic diagram 20 shows the conversion of the specification to a graph. In this example, the graphical user interface 19 is shown in schematic diagram 20. In this example, the graphical user interface 19 includes a palette section 19a and a specification section 19b for defining the specification. The palette section depicts visual representations 19c, 19d, 19e, 19f, 19g, 19h, 19i, 19j, 19k, and 19l of various modules, which are either previously defined (e.g., filtered customer transaction module, customer transaction module, transaction aggregation module, A_customers module, B_name_address module, and output module, represented by visual representations 19c, 19d, 19e, 19f, 19g, and 19h, respectively) or newly defined in the specification 19b and added to palette 19a for the first time (e.g., modules 1, 2, 3, and 4, represented by visual representations 19i, 19j, 19k, and 19l, respectively) that can be used to define the specification.

[0103] In this example, the user defines a new module, namely Module 1, in the Module 1 area 19m of specification section 19b. The Module 1 area 19m includes a control 19ma for defining the data source for Module 1. In this example, the user drags and drops a visual representation 19d onto control 19ma to specify that the Customer Transactions module is the data source for Module 1. The Module 1 area 19m also includes controls 19mb, 19mc, and 19md for defining the filtering operations to be performed on the data records output from the Customer Transactions module. In this example, the Module 1 area 19m also includes a control (not shown) for selecting the type of operation to be performed, which in this case is a filtering operation. The Module 1 area 19m also includes a global save control 19mf for adding Module 1 to the global palette. The Module 1 area 19m does not include a local save control because Module 1 is newly defined and therefore the system cannot save a local copy of a previously defined module. In this example, the user does not select the global save control 19mf. Instead, simply defining module 1 itself adds module 1 to the local palette (for the graph being defined) by adding the visual representation 19i to the palette portion 19a.

[0104] In this example, specification section 19b also defines another new module, namely module 2, in module 2 area 19o. In this example, module 2 area 19o includes control 19oa for defining the data source for module 2. In this example, visual representation 19e is dragged and dropped onto control 19oa to specify that the transaction aggregation module 19e is the data source for module 2. Module 2 area 19o also includes controls 19ob~19od for specifying attributes of filtering behavior, such as that it itself was specified through the selection of another control (not shown). Module 2 area 19o includes global save control 19of, which, when selected, adds module 2 to the global palette. In this example, due to the specification of the definition of module 2 in module 2 area 19o, visual representation 19j is added to palette section 19a.

[0105] Specification section 19b also defines a new module 3 within module 3 area 19n. Module 3 area 19n includes a control 19na for specifying the data source for module 3. In this example, module 1 is the data source for module 3, which is specified by dragging and dropping a visual representation 19i onto control 19na. Module 3 area 19n also includes controls 19nb, 19nc, and 19nd for specifying attributes of a join operation (which itself is specified through the selection of other controls for selecting the type of operation). In this example, the output of module 1 is joined with the output of module 2, as specified by control 19nb. In this example, the user may drag and drop a visual representation 19j onto control 19nb. Additionally, controls 19nc and 19nd specify the names of the keys (i.e., fields) of the records output from modules 1 and 2, respectively, on which the join should be performed. Module 3 area 19n also includes a global save control 19nf, which, when selected, adds module 3 to the global palette. In this example, the global save control 19nf is not selected. Instead, simply defining module 3 adds the visual representation 19k to the palette section 19a.

[0106] Specification section 19b also defines a new module 4 within module 4 area 19p, which includes a control 19pa for specifying the data source for module 4 and a control 19pb for specifying the data target for module 4. In this example, module 3 is the data source for module 4, and the output module (represented by visual representation 19h) is the data target. In this example, when module 4 is defined within module 4 area 19p, a visual representation 19l is added to palette section 19a.

[0107] Specification section 19b also includes submission controls 19s, which, when selected, cause the system, for example, the conversion environment 5 in Figure 1A, to convert the specification defined in specification section 19b into graph 19q. In this example, graph 19q includes customer transaction data source 19qa, transaction aggregation data source 19qb, module 1 19qc, module 2 19qd, module 3 19qe, and module 4 19qf. In this example, many of the data sources and modules included in graph 19q encapsulate other modules, data sources, and / or components. In this example, graph 19r shows an expanded view of graph 19q, within which the contents of each encapsulated module and / or data source are shown. Graph 19r includes components (some of which are data sources) 19ra~19rp. In this example, components 19ra~19rg are included in the customer transaction module, which is defined as a data source for module 1. In this example, component 19ri represents a filtering function defined by controls 19mb~19md. In this example, components 19rh and 19rj represent module 2. Components 19rk to 19ro represent module 3. As previously mentioned, the system is configured to insert partition and sort components before join components. Finally, component 19rp represents module 4. As shown in Figures 2B to 2D, the use of modules in different graph definitions differs depending on the specification. In some examples, the data processing system described here converts the specification into a data flow graph as follows: The data processing system accesses a data structure within the data storage system, which maps data representing parts of the specification to data flow graph components, and identifies data items within the data structure that represent specific parts of the specification. The data processing system also identifies within the data structure which data flow graph components are mapped to specific parts of the specification, and adds the identified data flow graph components to the data flow graph according to the execution order of operations specified by the specification.

[0108] Referring to Figure 2E, schematic diagram 30 includes a user interface module 32, a specification configuration module 34, and an entity generation module 36, which have also been previously described with respect to Figure 1A. The user interface module 32 displays (or makes display) a user interface 38 for configuring a specific entity (for example, "Entity 1" in Figure 2E). The user interface 38 includes a section 40 for receiving one or more values ​​for parameters for an operation to be included in the entity being configured through the user interface 38, such as a mapping operation. In this example, section 40 includes fields 40a, 40b for entering values ​​for parameters used in the operation. The user interface 38 may also include a section 42 for entering values ​​for parameters for other operations, such as a join operation. In this example, section 42 includes fields 42a, 42b for entering these values. In this example, the user interface 38 is for configuring an already defined entity. In a variant, the user interface can be rendered to define a new entity by, for example, specifying which parameters, operations, and / or data flow graph components should be included in the entity.

[0109] Based on the data entered into fields 40a-40b and 42a-42b of the user interface 38, the specification configuration module 34 generates a specification 44 that specifies a particular entity. A data processing system (for example, a conversion environment 5 (Figure 1A) that performs conversion from a specification to a data flow graph and includes one or more of the user interface module 32, the specification configuration module 34, and the entity generation module 36) may receive a request (for example, an XML message or other data) specifying one or more values ​​for fields 40a-40b and 42a-42b (for example, from a client device that displays or renders the user interface 38). This request may also specify the type of operation (for example, a mapping operation, a join operation, etc.) to which each value is specified. Based on the contents of the request, the specification configuration module 34 generates a specification 44. The conversion environment 5 (Figure 1A) may check values ​​received via the user interface 38 or via requests to conform to predetermined criteria related to the corresponding behavior of a particular entity, and may output a notification to the user (for example, via the user interface 38 or via a response message to the client) indicating whether the received values ​​conform to the predetermined criteria. If the conversion environment 5 identifies that some of the received values ​​do not conform to the predetermined criteria, the notification (warning, etc.) may also include information to guide the user to submit one or more new values ​​that conform to the predetermined criteria for the corresponding behavior of a particular entity. In other words, the conversion environment 5 (e.g., the user interface 38) may provide an guided user-machine interaction process that ensures the underlying system or entity operates correctly, for example, by ensuring that the final received values ​​conform to the predetermined criteria.

[0110] The above request may include the following:

[0111] [Table 1]

[0112] As shown in the table above, the request content includes data specifying various types of actions. For each type of action, the request specifies one or more parameters and one or more specified (e.g., user-entered) values ​​for each parameter. The specification configuration module 34 uses the request content to generate a specification 44 for a particular entity. For example, the specification configuration module 34 consists of a set of rules that parse the request to generate a specification, e.g., specification 44. In this example, the specification specifies various actions for a particular entity (e.g., performed by the entity) and various parameters for each action (along with the values ​​of the associated parameters).

[0113] The specification configuration module 34 may access templates for the specification (for example, for entities) as shown in Table 2 below:

[0114] [Table 2]

[0115] As shown in Table 2 above, the template for the specification includes fields that will be populated based on the content of the request, for example. In this example, there is a field for the type of action (i.e., ___ action), and another field (i.e., ___1) that specifies the first instance of that action. There are also other fields that specify the parameters (and associated values) for that instance of the action. The populated version of the template is shown in Figure 2E as specification 44. In the variant, the content of the request itself (e.g., shown in Table 1) is the specification, rather than having another template that will be populated from the content of the request, for example.

[0116] The entity generation module 36 may convert the specification 44 into an entity 46, for example, by selecting data graph components or data from the data repository that represent the components specified by the type of operation in the specification 44. For example, the specification 44 includes the types of operation "mapping" and "joining". Based on these types of operation, the entity generation module 36 selects the data flow graph components "mapping component 48" and "joining component 54" to include in the entity 46.

[0117] The entity generation module 36 may access a data structure in the data storage system 3 that maps data representing parts of a specification to data representing components or data flow graph components. In the mapping, the entity generation module 36 identifies data items that represent a specific part of the specification. In the mapping, the entity generation module 36 identifies which data representing a component or data flow graph component is mapped to the identified part of the specification, and adds the identified data representing the component or data flow graph component to the entity 46 in the order of operation execution specified by the user interface 38. In some examples, there is no one-to-one mapping between data representing a part of the specification and data representing a specific component. Instead, data representing a join can be mapped to multiple different join components (or data representing join components). In this example, the entity generation module 36 is configured to select the appropriate component.

[0118] The entity generation module 36 also selects other components, such as sort components 50, 53 and partition components 51, 52, which are not specified by the specification or by input to the user interface 38. In this example, the entity generation module 36 consists of rules that specify various types of operations before various other types of operations. For example, partition and sort operations are performed before key-based operations (e.g., join or roll-up operations) to save memory during the execution of key-based operations. In general, key-based operations include operations that operate on data based on a key (or other unique identifier) ​​for that data. In this example, the entity generation module 36 is programmed with rules that specify that partition and sort operations precede join operations. Based on these rules, the entity generation module 36 includes sort components 50, 53 and partition components 51, 52 in entity 46 before the occurrence of join component 54 in entity 46.

[0119] In another example, the entity generation module 36 is also programmed with a rule specifying that partition operations occur after mapping operations, thereby saving memory after the mapping operations have been performed, for example. Based on the execution of this rule, the entity generation module 36 includes a sort component 50 within the entity 46. The entity generation module 36 inserts edges 56, 58, 59, 60, and 61 (e.g., representing data flows) between components 48, 50, 52, and 54 based on the order in which the various operations are listed (or otherwise occur) in the specification 44 and / or the order specified by the rule. For example, based on the execution of the rule described above, the sort component 50 is inserted between the mapping component 48 and the join component 54. After this insertion, the entity generation module 36 inserts an edge 56 between the mapping component 48 and the sort component 50. The entity generation module 36 also inserts an edge 58 between the sort component 50 and the partition component 51. The entity generation module 36 inserts an edge 59 between the partition component 51 and the join component 54. Additionally, after inserting the partition component 52, the entity generation module 36 inserts an edge 60 between the sort component 53 and the partition component 52. The entity generation module 36 also inserts an edge 61 between the partition component 52 and the join component 54.

[0120] In the variant, the specification configuration module 34 is configured according to the rules described above. Based on the execution of these rules, the specification configuration module 34 is configured to identify additional operations (e.g., partition operations and sort operations) that should be included in the specification 44 and entity 46.

[0121] The system described herein is configured such that the user is presented with minimal complexity required to configure entities (through the user interface 38). That is, the user is prompted to specify parameter values ​​for data operations that require user input for configuration (e.g., join and filter operations that require the user to input key values ​​on which joins and / or filters are performed). However, other operations that can be configured independently of user input (e.g., partition and sort operations) are automatically configured and added to entities; that is, the complexity of these data operations is not presented to the user. Therefore, the user is presented with minimal complexity.

[0122] With respect to Figure 3A, schematic diagram 70 shows a data flow graph 71 generated from multiple entities 46, 76, 78, 80, and 82, each generated using the techniques described herein. In this example, each entity may itself contain multiple other entities. As will be explained in more detail below, the data processing system provides various user interfaces for selecting which entities should be included in the data flow graph and the execution order of those entities. In this example, the data flow graph 71 also includes data sources 72 and 74. In this example, the user interface also provides specifications for the data sources and the relationships between the data sources and entities included in the data flow graph. In this example, the data sources are modifiable fields in that they are customizable at the user end and do not need to be specified in advance (for example, they can be specified during deployment or installation). Referring to Figure 3B, schematic diagram 83 shows a variation of Figure 2E, in which case the specifications are generated based on user input for a data flow graph containing multiple specific entities, rather than for individual specific entities. However, the descriptions of individual entities in Figure 2E also apply accordingly to each entity described with respect to Figure 3. In this example, the user interface module 32 displays a user interface 85 for collecting all parameter values ​​(and / or configuration options for the data flow graph). Based on the inputs collected through the user interface 85, the specification configuration module 34 generates a specification 86 for all entities included in the data flow graph, using the techniques described in this specification, for example with respect to Figure 2E. Based on the specification 86, the graph generation module 84 generates a data flow graph 71, which itself contains various entities. By generating the entire data flow graph, the system can identify and generate the necessary dependencies and / or data flows between entities. Based at least partially on these dependencies and / or data flows, the system can specify input and output data formats for entities and for the components of entities.

[0123] In other words, the user interface module 32 displays (or makes displayable) a user interface 85 for configuring several specific entities (for example, the "Customer Transaction Entity" and the "Purchase Entity" in Figure 3B). The user interface 85 includes a section for receiving one or more values ​​for parameters for operations, such as mapping operations, that are included in the entities being configured through the user interface 85 (for example, the "Customer Transaction Entity" in Figure 3B). In this example, the user interface 85 includes fields 85d and 85e for inputting values ​​for parameters used in the operations. The user interface 85 may also include a section for inputting values ​​for parameters for other operations for the same entity, such as join operations. In this example, this section may also include fields for inputting values ​​for parameters for these join operations, etc. Based on the data entered into the fields of the user interface 85, the specification configuration module 34 generates a specification 86 that specifies several specific entities.

[0124] In this example, the user interface 85 displays a palette of entities (e.g., a given entity) for selection. In this example, the user interface 85 includes a control 85a, which, when selected, displays a list of various entities that can be included in the computer program and / or data flow graph. In this example, the user has selected, for example, the "Customer Transactions" entity 85b and the "Purchase" entity 85c to be included in the computer program. In this example, the "Customer Transactions" entity 85b includes mapping and join operations. For the "Customer Transactions" entity 85b, the user interface 85 allows the user to configure the values ​​of the mapping operation via input boxes 85d, 85e for entering, for example, the parameter values ​​of the join operation. Input boxes are provided for configuring the parameter values ​​of the join operation, which is part of the "Customer Transactions" entity 85b, and for configuring the parameter values ​​of the sort and join operations included in the "Purchase" entity 85c.

[0125] In this example, the user interface 85 includes a control 85d, and when selected, the client device rendering the user interface 85 sends a request (e.g., an XML message or other data) to a data processing system executing the specification configuration module 34, for example, a transformation environment 5 (Figure 1A). In this example, the request specifies, for each specific entity, the name of the entity, the type of data operation performed by that entity (e.g., a mapping operation, a join operation, etc.), and for each type of data operation, the values ​​of the respective parameters for that data operation. The request is sent to the specification configuration module 34. Based on the contents of the request, the specification configuration module 34 generates a specification 86. The specification configuration module 34 may identify whether there are any additional data operations to include in the specification using the techniques described herein. These additional data operations include data operations required to perform other data operations or data operations on which a specified data operation depends, but which the user is not prompted to input values ​​for. The data processing system may check values ​​received via user interface 38 (Figure 2E) or user interface 85, or via requests that conform to predetermined criteria related to the corresponding behavior of a particular entity, and may output a notification to the user (e.g., via user interface 85 or via a response message to the client) indicating whether the received values ​​conform to the predetermined criteria. If the data processing system identifies that some of the received values ​​do not conform to the predetermined criteria, the notification (e.g., a warning) may also include information that guides the user to submit one or more new values ​​for the corresponding behavior of a particular entity that conform to the predetermined criteria. In other words, the data processing system (e.g., user interface 85) may provide an guided user-machine interaction process that ensures the proper operation of the underlying system or entity, for example, by ensuring that the finally received values ​​conform to the predetermined criteria.

[0126] Based on the specification, the graph generation module 84 generates the data flow graph 71 by selecting entities (e.g., entities 76, 78, 80, 82, 46) from memory, for example, that correspond to the names of entities included in the specification 86, and configuring the parameter values ​​of these entities according to the values ​​entered into the user interface 85. In some cases where entities have not yet been predefined and stored in memory, the graph generation module 84 generates entities by selecting components from memory, for example, that have names corresponding to the names of various operations specified in the specification.

[0127] Referring to Figure 4, schematic diagram 90 shows how a data processing system, for example, data processing system 4 in Figure 1A, is configured to "drop" or otherwise remove a field from a data record when the data processing system determines that the field is no longer needed by any downstream or subsequent entity (or component contained within an entity) in the data flow graph 91, which may be data flow graph 71 (Figure 3A). In this example, the data flow graph 91 includes data sources 92, 94, a data sink 97 (e.g., a data repository for storing data), and an entity 98. The data flow graph 91 may also include one or more other entities, which are indicated by elliptical dots ("...") seen in the data flow graph 91.

[0128] There are many advantages to dropping fields when they are no longer used or processed by the data flow graph. These advantages include reduced latency in data record processing, because the data processing system can process smaller records (e.g., by dropping fields) more quickly than if the data processing system had to process all data records through the data flow graph. Other advantages include reduced memory and resource consumption, because smaller records (e.g., by dropping fields) consume less memory space during record processing.

[0129] In this example, the data flow graph 91 contains an entity 98, which in turn contains data flow graph components 100, 102, 104, and 106. Each of components 100 and 106 is a filter component, which filters records based on a location field. Therefore, each of components 100 and 106 requires that the data records have a location field in order to perform the filtering correctly. Component 102 is a sort component, which is configured to sort data records based on, for example, the ID or key of the data record. Therefore, component 102 requires an ID field in order to perform the sorting correctly. Component 104 is a join component, which is configured to perform joins on data records related to the same or similar key or ID. Therefore, component 104 requires an ID field in order to perform the join operation correctly. In this example, the data sink 97 is configured to store a data record (e.g., data record 96) having four fields (e.g., fields 96a to 96d) for each of the processed data records. In this example, the four fields include the ID field 96a, the SMS field 96b, the timestamp field 96c, and the data usage field 96d. Therefore, based on the fields specified for storage by the data sink 97, the processed record should include these four fields so that the values ​​of these fields can be stored in the data sink 97.

[0130] In this example, data source 92 receives a structured data item, for example, data record 108, which includes an ID field 108a, a location information field 108b, an SMS field 108c, and a timestamp field 108d. In this example, each of fields 108a through 108d is required by a downstream entity or data sink. For example, field 108a is required by components 102, 104, and data sink 97. Field 108b is required by component 100. Field 108c is required by data sink 97. Field 108d is required by data sink 97. Therefore, the data processing system does not drop any fields from data record 108, and the entire data record 108 is sent to the next entity in the data flow graph 91.

[0131] In this example, data source 94 receives data record 110, which includes an ID field 110a, a location field 110b, a data usage field 110c, a timestamp field 110d, and a voice field 110e. The ID field 110a is used by downstream entities and components in the data flow graph 91 (i.e., component 104 and data sink 96). The location field 110b is used by component 106. The data usage field 110c is used by data sink 97. The timestamp field 110d is used by data sink 97. However, the voice field 110e is not used by any downstream component or entity. Therefore, the data processing system drops the voice field 110e from data record 110 (this is visually indicated by the strikethrough on "voice"). In this example, data source 94 sends data record 112 (including fields 112a-112d, but not the dropped field 110e) to the next entity in the data flow graph 91. Data record 112 is a version of data record 110 with field 110e removed. That is, in this example, data record 112 is the same as data record 110 except that data record 112 does not contain field 110e. In this example, fields 112a to 112d are the same as fields 110a to 110d, respectively. This may also be done by setting the output data format of data source 94 before running the data flow graph for data processing while converting the specification associated with data source 94 to data source 94, thereby removing or dropping the voice field 110e from data record 110 before data record 110 is output by data source 94 as data record 112 for subsequent downstream processing by downstream entity 98.

[0132] In this example, entity 98 receives data record 108. The data processing system runs component 100 on data record 108 to determine, for example, whether to continue processing data record 108 or whether the data record is filtered out and not processed further. In this example, based on the value of location field 108b, the data processing system determines that data record 108 does not meet the criteria for a data record to be filtered out and not processed further. The data processing system also determines that location field 108b is not used or required by any downstream entity or component. For example, none of components 102, 104, or data sink 97 use the location field. Therefore, the data processing system drops location field 108b from data record 108, which is indicated by the strikethrough of "Location" above component 100. This may be done by setting the output data format of component 100 during the conversion of the specification associated with entity 98 to entity 98, thereby excluding or dropping the location information field 108b from data record 108 before it is output by component 100 as data record 114 for subsequent downstream processing by downstream components 102 and 104. The exclusion of the location information field 108b results in data record 114, which is a version of data record 108. That is, data record 114 is a version of data record 108 from which the location information field 108b has been removed. In this example, data record 114 contains fields 114a to 114c, which correspond to fields 108a and 108c to 108d, respectively. Components 102 and 104 each process data record 114. In this example, after processing by components 102 and 104, no fields are dropped from data record 114 because each of fields 114a to 114c is required by data sink 97.

[0133] In this example, data record 112 is passed to entity 98, which then executes component 106 on data record 112 to determine, for example, whether to continue processing data record 112 or to filter it out and not process it further. In this example, based on the value of the location field 112b, the data processing system determines that data record 112 does not meet the criteria for a data record to be filtered out and not processed further. The data processing system also determines that the location field 112b is not used or required by any downstream entity or component. For example, neither component 104 nor data sink 97 uses the location field. Therefore, the data processing system drops the location field 112b from data record 112, which is indicated by the strikethrough of "Location" under component 106. This may be done by setting the output data format of component 106 during the conversion of the specification associated with entity 98 to entity 98, thereby excluding or dropping the location information field 112b from data record 112 before data record 112 is output by component 106 as data record 116 for subsequent downstream processing by downstream component 104. The exclusion of location information field 112b results in data record 116, which is a version of data record 112. That is, data record 116 is a version of data record 112 from which the location information field 112b has been removed. In this example, data record 116 contains fields 116a to 116c, which correspond to fields 112a and 112c to 112d, respectively. Component 104 processes data record 116. In this example, after processing by component 104, no fields are dropped from data record 116 because each of fields 116a to 116c is required by data sink 97.

[0134] In this example, component 104 combines data records 114 and 116 (for example, each data record having the same ID). The output of component 104 is data record 118, which contains data records 114 and 116. In this example, each of data records 114 and 116 is a subrecord of data record 118. Entity 98 transmits data record 118 to data sink 97 for storage.

[0135] In one example, a data processing system (e.g., data processing system 4 or transformation environment 5) performs three passes for each entity in the data flow graph (e.g., data flow graph 71 in Figure 3A or data flow graph 91 in Figure 4) to determine which data fields can be deleted from storage or otherwise excluded, and when. That is, to reduce the consumption of computing and memory resources, the data processing system is configured to "drop" or delete fields as early as possible, for example, when the field is no longer needed by the data flow graph and / or by the entities in the data flow graph. In the first pass, an entity in the data flow graph broadcasts fields to other entities (e.g., other upstream entities) that are needed by itself and by any entity downstream of that entity. The entity that performs this broadcast is called the broadcast entity. In one example, the broadcast entity broadcasts the required fields to one or more receiving entities (e.g., upstream entities) by sending one or more messages containing data representing the required fields to these upstream entities.

[0136] In the second pass, the receiving entity transmits one or more messages to the broadcasting entity specifying which receiving entity can provide (e.g., transmit to the broadcasting entity) the values ​​of the required fields. In the third pass, the broadcasting entity parses (or otherwise parses) the messages received from the receiving entity. In response, the broadcasting entity sends back data (e.g., a message) to the receiving entity specifying which receiving entity is responsible for transmitting which fields (and / or field values) to the broadcasting entity. In some examples, the data processing system executes these three passes. In doing so, each entity consists of data (e.g., a configured output data format) that specifies which fields are required by downstream entities, including, for example, the broadcasting entity. Generally, an entity (e.g., each entity) consists of data that specifies the required fields by storing data representing the required fields in memory and / or a data repository, and by storing data associated with a unique identifier for that entity. Then, as the data flow graph is executed, the data processing system queries this stored data as the entities are executed. For example, when a specific entity is executed, the data processing system retrieves data for that entity in the database (for example, based on the entity's unique identifier). As mentioned earlier, the retrieved data specifies which fields are needed by downstream entities. In this example, the data processing system is configured to drop fields that are not needed by downstream entities when data processing by the current entity (for example, the currently executing entity associated with stored data that specifies which fields are needed and not needed by downstream entities) is complete (for example, by deleting them from storage or by ensuring that no data about these fields is stored).

[0137] Referring to Figure 5, schematic diagram 120 illustrates the process a data processing system performs when determining which fields of a data record should be dropped and when. In this example, the data flow graph includes data sources 122 and 124, and entities 126, 128, 130, 132, and 134. In this example, entities are transient in that they do not store the values ​​of their fields. In this example, entity 132 requires three fields, namely x, y, and z. Entity 130 requires one field, namely A. Entity 126 requires two fields, namely x and y. Additionally, in this example, data source 122 stores (or can otherwise access) a data record that has fields x, y, and A.

[0138] To track which fields are required by which entities and when the data processing system can drop fields, the data processing system performs the following techniques as part of configuring the data flow graph, before data processing by the data flow graph. First, entity 132 sends message 170 to entity 130 during the first pass, as shown by arrow 136. Message 170 contains data specifying the fields required by entity 132. In this example, message 170 contains data specifying that fields x, y, and z are required by entity 132. Entity 130 then receives message 170 and adds any fields required by entity 130 to message 170. In this example, entity 130 generates message 180, which is a version of message 170 and contains the contents of message 170 (i.e., data representing fields x, y, and z) as well as other data representing field A. During the first pass, the data processing system transmits message 180 from entity 130 to entity 126, as shown by arrow 138. In this example, entity 126 requires fields x and y (for example, because entity 126 reads fields x and y). Therefore, entity 126 can respond to message 180 by indicating that entity 126 will have values ​​for fields x and y. In the first pass, as indicated by arrow 193, entity 126 sends message 190 to data source 122. In this example, message 190 specifies that fields x, y, z, and A are required by, for example, one or more downstream entities. Since data source 122 stores (or otherwise accesses) values ​​for fields x, y, and A, the data processing system ensures that message 192 is transmitted from data source 122 to entity 126 in the second pass, as indicated by arrow 194. In this example, message 192 specifies that data source 122 stores (or otherwise accesses) values ​​for fields x, y, and A.Next, during the second pass, the data processing system causes entity 126 to transmit message 182 to entity 130, as indicated by arrow 140. In this example, message 182 specifies that entity 126 can transmit values ​​for fields x, y, and A to entity 130. In one example, the data processing system identifies which fields to specify or include in a message (transmitted by an entity) by identifying the fields specified in the received message (to that entity) and then identifying which of the specified fields are required by or accessible by the entity that received the message. In this example, message 180 (received by entity 126) specifies fields x, y, z, and A. Entity 126 requires fields x and y. Therefore, entity 126 can access values ​​for fields x and y (by receiving these values ​​from data source 122). Thus, fields x and y are included in message 182. Entity 126 does not require field A, but entity 126 can access field A through data source 122. Therefore, entity 126 includes field A in message 182.

[0139] In this example, entity 130 receives message 182 and identifies which of the fields specified in message 182, if any, are required by the downstream entity. To do this, entity 130 compares the contents of message 182 with the contents of message 170. In this example, the data processing system is configured to generate another message specifying the fields that represent overlaps between the fields specified in message 182 and the fields specified in message 170 (e.g., overlapping fields). Therefore, the data processing system generates message 185 with data representing fields x and y, specifying, for example, that entity 130 can provide fields x and y to entity 132. During the second pass, entity 130 transmits message 185 to entity 132, as indicated by arrow 142. Entity 130 also stores data in memory or a data repository specifying that once field A is processed by entity 130, field A can be dropped from storage or otherwise deleted. Here, when the data flow graph shown in Figure 5 is executed (in real time) by the data processing system 4 (Figure 1A), the data processing system 4 (executing the data flow graph) can determine that field A should be dropped after processing by entity 130, for example, by retrieving data in memory (or the data repository) that specifies that field A should be dropped when data processing by entity 130 is complete.

[0140] In parallel with (or following) the transmission of message 170, entity 132 also transmits message 172 to entity 134 in the first pass, as indicated by arrow 150. In some examples, message 170 is the same message as message 172. In this example, message 172 specifies that fields x, y, and z are required by entity 132. Entity 134 receives message 172. In this example, entity 134 does not require any of its own fields. Therefore, entity 134 sends message 172 to entity 128 in the first pass, as indicated by arrow 152. Entity 128 sends message 172 to source 124, which in turn responds with message 184, which specifies that source 124 can provide fields y and z. Now entity 128 responds with message 184, which specifies that entity 128 can provide values ​​for fields y and z. In this example, entity 128 transmits message 184 to entity 134 during the second pass, as indicated by arrow 154. In this example, entity 134 transmits message 184 to entity 132 during the second pass, as indicated by arrow 156.

[0141] In this example, entity 132 receives two messages (i.e., messages 184 and 185), each specifying which fields required by entity 132 can be provided by the upstream entity. Specifically, message 184 specifies that fields x and y can be provided by entity 130. Message 184 specifies that fields y and z can be provided by entity 134. Since entity 132 can receive the value of field x from two different entities, entity 132 selects from these entities which it can request and receive the value of field x from. In some examples, entity 132 makes this selection randomly. In other examples, entity 132 may select the entity that responded first.

[0142] In this example, entity 132 selects entity 130 to receive field x. Therefore, during the third pass, entity 132 transmits message 179 to entity 130, as indicated by arrow 144. In this example, message 179 specifies that entity 132 has selected to receive fields x and y from entity 130. Entity 130 then generates message 178, specifying that entity 130 will receive the values ​​for fields x, y, and A from entity 126. During the third pass, message 178 is transmitted from entity 130 to entity 126, as indicated by arrow 146. During the third pass, entity 126 transmits message 178 to data source 122, as indicated by arrow 148. In this example, data source 122 compares the contents of message 178 to the fields it stores or otherwise accesses to determine, for example, which (if any) fields data source 122 can drop and delete from its storage. In this example, the data processing system determines, based on the content of message 178, that all fields of data source 122 are required by the downstream entity. Therefore, data source 122 is configured so that no fields are dropped.

[0143] In this example, entity 132 transmits message 176 to entity 134 during the third pass, as indicated by arrow 158, specifying that entity 132 will receive field z from entity 134. Entity 134 is then configured by the data processing system to transmit the value of field z to entity 132. Entity 134 also transmits message 176 to entity 128 during the third pass, as indicated by arrow 160. Thus, entity 128 is also configured to transmit the value of field z to entity 134. Entity 128 then transmits message 176 to data source 124 during the third pass, as indicated by arrow 162. In this example, data source 124 compares the contents of message 176 (i.e., specifying field z) with a field accessed by data source 124, in this case field y, z. Based on the comparison, the data processing system determines that field y is not included in message 176. Therefore, the data processing system determines that none of the downstream entities depend on data source 124 for the value of field y. Thus, data source 124 is configured to drop field y when the data flow graph shown in Figure 5 is executed. In this example, data source 124 is configured to drop a field by not storing a value for that field in memory.

[0144] In the explanation so far, we have stated that entities transmit and receive messages. In one example, an entity transmits and / or receives messages through executable logic or code representing an entity that is running and generating data, and the generated data is then analyzed or interpreted by other executable code representing other entities.

[0145] In the example above, an entity, for example entity 126, reads access to a field rather than storing the value for the field itself. In a variant, an entity may also store the value for a field. In this example, if an entity stores the value for a requested field, it does not need to go to an upstream entity and / or data source to determine whether that upstream entity and / or data source can provide the value for that field. Instead, the entity can respond to a downstream entity (which requests the field) that the upstream entity can provide the value for that field.

[0146] In the example above, entities know which fields to drop and when because each entity stores or can access data that specifies the input and output data formats for that entity. The output data format specifies the fields for which the entity should output values. In one example, the output data format (or output field or output format) specifies which fields required by one or more downstream (e.g., subsequently occurring) entities can be provided by a particular upstream entity. In this example, the upstream and downstream entities are part of the same data flow in the data flow graph. That is, the upstream entity is configured to provide these fields via the output format. The input data format specifies the fields for which the entity is configured to receive values. Generally, the input data format contains the fields that need to be accessed or processed for the entity. Therefore, to determine which fields to drop and when, the entity parses or otherwise analyzes the contents of each output data format.

[0147] In the variant, entity 132 is structured as follows: it receives the value of field y from entity 130 unless the value is zero, and if it is zero, it receives the value of field y from entity 134. In this variant, entities 126, 128, 130, and 134 each consist of field y as part of the output data format.

[0148] Referring to Figure 6, a graphical user interface 200 is displayed (e.g., by a client device or data processing system) that allows a user to configure a specification for an entity and for a data flow graph containing that entity. In this example, the graphical user interface 200 includes a section 202 for configuring one or more parts of a specification for an input dataset. The graphical user interface 200 also includes a section 204 for inputting values ​​that specify various actions to be performed on the input dataset (e.g., data joining). The graphical user interface 200 also includes a section 206 for inputting values ​​and data that specify an output dataset and the storage device for the output dataset.

[0149] Referring to Figure 7, the graphical user interface 209 is a variation of the graphical user interface 200. In this variation, an overlay 208 is displayed for configuring entities. In this example, an entity is configured by selecting one or more selectable parts of part 204. In this example, the overlay 208 includes configuration controls 210, 212, 214, and 216 for configuring actions to be performed as part of data processing (e.g., join operations, mapping operations, and filtering operations). In this example, the user selects configuration control 210 to specify that a join operation will be performed on data records.

[0150] Referring to Figure 8, the graphical user interface 220 displays an overlay 222 for configuring join operations to be included in entities. In this example, the overlay 222 is displayed when the configuration control 210 is selected. In this example, the overlay 222 displays controls for configuring join operations. In particular, the overlay 222 displays configuration controls 224, 226, 228, 229, and 230. In this example, configuration control 224 allows the user to specify the dataset to which the join will be performed. Configuration control 226 allows the user to enter a name for the join operation. Configuration control 228 allows the user to specify whether or not they want to remove duplicate data records. Configuration control 229 allows the user to specify the key (i.e., unique identifier) ​​on which the join will be performed. Configuration control 230 allows the user to specify the type of join to be performed.

[0151] Referring to Figure 9, the data processing system generates the data flow graph 240 based on a specification generated from, for example, specified values ​​(e.g., values ​​entered within the user interface shown in Figures 6-8, or otherwise specified). In this example, the specification is generated using the techniques described herein with respect to, for example, Figure 2E. Additionally, the data flow graph 240 is also generated from the specification using the techniques also described herein.

[0152] In this example, dataflow graph 240 includes input data sources 242, 244, 246, 248, 250, entities 252, 254, 256, 258, 260, 262, and output data source 264. In this example, each of entities 252, 254, 256, 258, 260, 262 includes a dataflow graph (for example, a dataflow graph that will be included within a parent dataflow graph such as dataflow graph 240).

[0153] Referring to Figure 10, schematic diagram 300 illustrates the process of converting a specification into entities for generating a data flow graph. In operation, the system (e.g., system 1 in Figure 1A as described herein) provides a user interface that includes a first part for inputting values ​​for at least a first parameter of the entity, and a second part for inputting values ​​for at least a second parameter of the entity. The system receives data representing the input values ​​for the first parameter and data representing the input values ​​for the second parameter (302). Optionally, the system may check whether the data representing the input values ​​conforms to predetermined criteria related to the corresponding parameters of the entity, and may output a notification to the user (e.g., via the user interface) indicating whether the input values ​​conform to the predetermined criteria. If the system identifies that some of the input values ​​do not conform to the predetermined criteria, the notification (e.g., a warning) may also include information to guide the user to input one or more new values ​​that conform to the predetermined criteria for the corresponding parameters of the entity. In other words, the system may optionally provide an guided user-machine interaction process that ensures the proper operation of the entity, for example, by ensuring that the final received values ​​conform to the predetermined criteria.

[0154] Based on the input values, the system generates a specification for the entity (304), which specifies several data operations for processing the data record according to the input values, at least one of which is specified by the portion of the received data that represents the input values, and optionally, at least one other data operation is identified to be included in the specification so that the data processing system can perform at least one of the data operations specified by the portion of the received data that represents the input values ​​(e.g., automatically and independently of user input, e.g.

[0155] The system generates entities having identified components for performing data operations (308), where the first component is configured to perform one or more operations according to the value of a first parameter, and the second component is configured to perform one or more operations according to the value of a second parameter. Additionally, based on the entities generated by each component identified from each specification, the system generates a computer program or data flow graph representing the data flow between entities (310).

[0156] Referring to Figure 11, the data processing system (for example, System 1 in Figure 1A, Data Processing System 4 in Figure 1A, or Conversion Environment 5 in Figure 1A) implements a process 320 that converts a specification into a persistent computer program. Generally, persistent refers to a characteristic of a state that remains even after the process that created it has terminated. Generally, a computer program contains a set of coded instructions that can be executed by the data processing system. In operation, the data processing system receives specification data that specifies the input values ​​of a first parameter of a first data operation implemented by a first entity, and further specifies the input values ​​of a second parameter of a second data operation implemented by a second entity (322).

[0157] Based on the input values ​​of the parameters of the first and second entities, the data processing system generates a specification for a computer program that performs several different data operations in processing one or more datasets (324), the specification specifying that the first entity is included in the computer program and that the first entity implements at least a first data operation for processing data according to at least the input values ​​of the first parameter, the specification further specifies that the second entity is included in the computer program and that the second entity implements a second data operation for processing data according to at least the input values ​​of the second parameter, and optionally, at least one data operation specified in the specification to be implemented by the first or second entity is selected by the data processing system to be included in that entity, regardless of whether it is specified as a data operation for that entity in the user interface.

[0158] Using the techniques described herein, a data processing system converts a specification into a computer program that performs several different data operations in processing one or more datasets (326), wherein the contents of one or more parts of the computer program are different from the input to the user interface, by choice. That is, the conversion of the specification yields a new computer program, which is stored in memory, for example, to be executed and / or modified later. The data processing system also stores the computer program in memory (328). This computer program is persistent and editable. Additionally, this computer program is stored, for example, to be executed at a later date.

[0159] For each entity, the data processing system stores input and output data formats for that entity (330). The input data format specifies one or more fields of one or more datasets accessed by that entity. The output data format specifies one or more fields accessed by one or more entities that occur after (i.e., downstream of) that entity in the computer program. In this example, the data processing system automatically identifies the input and output data formats using the techniques described herein, for example with respect to Figures 4 and 5. To identify the output data format, these techniques include identifying other entities that occur after (i.e., downstream of) that entity in the computer program, and that the other entities and that entity are part of the data flow in the computer program; identifying one or more fields accessed by the other entities; identifying which of the one or more fields accessed by the other entities are accessible by that entity; and setting the entity's output data format to include one or more fields accessed by and that can be accessed by the other entities.

[0160] In this example, the data processing system also determines, for each entity, whether one or more fields accessed by that entity can be dropped from the data record after the entity has processed the data (e.g., values) in those one or more fields (332). The data processing system does this by, for each entity in the computer program that is part of the same data flow as one or more other entities that occur later (i.e. downstream) in the computer program, by identifying one or more fields that are read by the entity in the computer program but not by any of the one or more other entities that occur later (i.e. downstream) in the entity, and by ensuring that those one or more fields are not subjected to further processing by the computer program after they have been read by the entity.

[0161] In one variant, the data processing system provides user interface data for selecting one or more entities from a group of entities to include in a computer program that performs multiple different data operations in processing one or more datasets, and for inputting parameter values ​​for the data operations implemented by each selected entity. In another variant, the data processing system provides a spreadsheet or rule editor for inputting data specifying which entities to include in the computer program and the parameter values ​​for those entities.

[0162] By allowing users to select one or more entities from a set of entities, data processing systems enable the reuse of software modules and code, improving flexibility and ease of use in generating computer programs. For example, once an entity is defined, that same entity can be reused in multiple different computer programs. This reduces errors in generating computer programs because computer programs can be developed through the selection of already defined computer programs and code (which are error-free), without having to start programming the computer program from scratch. Additionally, by allowing users to select an entity from a set of entities, data processing systems can show only a small (e.g., minimal) amount of complexity in the user interface required to configure the entity and generate the computer program. The data processing system does this by prompting the user only for the values ​​of parameters that require user input or specification. Other data operations that do not require user input are hidden from the view in the user interface and added to the specification by the data processing computer program as part of the specification generation.

[0163] In this example, the specification data (and / or user input) is a semantic specification of what is done (e.g., what the result is). Based on the semantic specification, the data processing system generates a logical specification (e.g., generally referred to herein as the specification) that specifies how it is done, for example, how the result is achieved. Through this difference between the semantic specification and the logical specification, the data processing system can perform end-to-end optimization when generating a computer program, for example, by omitting steps specified by the user in the semantic specification (via a user interface or flowchart). An exemplary optimization is when the user specifies that partitioning and sorting should be performed on a particular entity. However, if an entity preceding the particular entity is configured for partitioning and sorting, the computer program is optimized so as not to perform the partitioning and sorting operations twice in a single row. In this regard, the creation of the graph or computer program is carried out according to recommended best practices, which the user does not need to know. Thus, even someone with little expertise can create a professionally assembled computer program.

[0164] In some examples, an entity includes compute components corresponding to nodes connected by data flows corresponding to links. In this example, a computer program is a dataflow graph containing entities corresponding to nodes connected by data flows corresponding to links. In this example, memory includes volatile or non-volatile memory. Additionally, in some examples, an entity includes one or more other entities.

[0165] The graph and entity construction approaches described above can be implemented using software that runs on a computer. For example, the software may form a procedure within one or more computer programs that runs one or more computer-programmed or computer-programmable computer systems (which may be of various architectures, such as distributed, client / server, or grid), each including at least one processor, at least one data storage system (including volatile and non-volatile memory and / or memory elements), at least one input device or port, and at least one output device or port. The software may form one or more modules of a larger computer program that provides other services, for example, related to the design and construction of computation graphs. The nodes and elements of the graph can be implemented as data structures stored in a computer-readable medium or other organized data that conforms to a data model stored in a data repository.

[0166] The software may be provided on a storage medium such as a CD-ROM readable by a general-purpose or purpose-specific programmable computer, or it may be delivered (encoded in a propagation signal) over a network communication medium to a computer on which it is executed. All functions may be executed on a purpose-specific computer, or using purpose-specific hardware, such as a coprocessor. The software may be implemented in a distributed manner, in which case different parts of the computation specified by the software are executed by different computers. Each of such computer programs is preferably stored on or downloaded to a storage medium or device (e.g., solid-state memory or media, or magnetic or optical media) readable by a general-purpose or purpose-specific programmable computer, and the storage medium or device constitutes and operates the computer when read by a computer system and the procedures described herein are performed. The system of the present invention may be envisioned to be implemented as a computer-readable storage medium composed of computer programs, such a storage medium causing the computer system to operate in a special predetermined manner to perform the functions described herein.

[0167] Numerous embodiments have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the technology described herein. For example, some of the steps described above may be order-independent and can therefore be performed in a different order than those described. Furthermore, the technology described above in relation to data flow graphs can also be implemented and executed in relation to programs.

[0168] The above description is illustrative of the technology described herein and should not be understood as limiting its scope as defined by the accompanying claims. For example, the numerous functional steps described above may be performed in a different order without substantially affecting the overall process. Other embodiments are also included in the following claims.

Claims

1. A method that a computer system can perform in converting a specification into a computer program that processes one or more values ​​of one or more fields of a structured data item, The steps include accessing a specification that specifies a number of actions that will be associated by the computer program to process the one or more values ​​of the one or more fields of the structured data item, A step of converting the specification into a computer program using multiple modules, wherein the multiple modules are configurable to access fields of structured data items based on the operation, For each of the one or more first modules of the plurality of modules, Identifying one or more second modules of the plurality of modules, each of which receives an input based at least partially on the output of the first module, One or more fields, Identifying one or more fields from among the fields to be accessed by the first module that are dependent on the identified one or more second modules, wherein the one or more fields identified as dependent on the one or more second modules are fewer than the fields to be accessed by the first module. Specifying the output data format of the first module such that the output of the first module identifies only one or more values ​​of one or more fields that correspond to (i) the fields that will be accessed by the first module and (ii) the one or more fields that are dependent on by the one or more second modules. This is configured to include in the output data format of the first module. Steps including, A step of saving the computer program in persistent memory, wherein the saved computer program specifies the configured output data format for each of the one or more first modules. Methods that include...

2. The method according to claim 1, wherein the content of one or more parts of the computer program differs from the specifications and is automatically generated without requiring user input.

3. The method according to claim 1, wherein the computer program is persistent and editable.

4. The method according to claim 1, wherein the computer program is a data flow graph, each of the modules is a data flow graph component represented by nodes of the data flow graph, and the data flow graph components are connected by data flows represented by links between nodes.

5. A step in a data storage system to access a data structure that maps data representing a part of the specification to the data flow graph component, The steps include: identifying data items that represent a specific part of the specification within the aforementioned data structure; The steps include: identifying which data flow graph component in the data structure is mapped to the specific part of the specification; The steps include adding the identified data flow graph component to the data flow graph in accordance with the execution order of operations specified in the aforementioned specification, and The method of claim 4, further comprising:

6. The method of claim 5, further comprising the steps of identifying dependencies between the data flow graph components and specifying the output data format of the data flow graph components based at least in part on these identified dependencies.

7. For a specific first module, The steps include: determining that only one of the one or more second modules has access to a particular one or more fields accessible to the particular first module; Steps include identifying that only one of the one or more second modules accesses one or more of the specific fields from a third module of the plurality of modules, which is different from each of the specific first module and the one or more second modules, The steps of setting the output data format so as not to output one or more values ​​for one or more specific values ​​for one or more fields that are accessed by only one of the one or more second modules. The method of claim 1, further comprising:

8. Steps include providing data for displaying the user interface, The steps include receiving data through the user interface that represents the selection of one or more modules from among a plurality of modules to be included in the computer program, The steps include receiving, via the user interface and for each selected module, the values ​​of the parameters of the operation implemented by one or more of the selected modules. The method of claim 1, further comprising:

9. The steps include determining whether the value received via the user interface conforms to a predetermined criterion associated with the operation, A step of outputting a notification via the user interface indicating whether the received value conforms to the predetermined criteria, wherein if it is determined that the received value does not conform to the predetermined criteria, the notification includes guidance information on how the received value must be changed to conform to the predetermined criteria. The method of claim 8, further comprising:

10. The steps include configuring a module selected by the aforementioned values ​​of the parameters, The steps include: storing the configured module which is then included in and implemented by another computer program; The method of claim 8, further comprising:

11. Before executing the computer program, the steps further include, for each specific module of the plurality of modules, executing first, second, and third passes to determine and save the configured output data format for the specific module, which specifies which fields are required by downstream modules of the specific module, and determining and save the input data format for the specific module, which specifies which fields are accessible to the specific module, wherein the steps to be executed are, In the first pass, the particular module broadcasts one or more messages to a module upstream of the particular module, each message containing data representing one or more fields required by itself and by any module downstream of the particular module, wherein the particular module performing the broadcast is a broadcast module, and the upstream module receiving the broadcast is a receiving module. During the second pass, the receiving module sends one or more messages to the broadcast module specifying which receiving module can provide the values ​​for the required fields, During the third pass, the broadcast module analyzes the message received from the receiving module and, in response, sends back one or more messages to the receiving module specifying which receiving module is responsible for sending which fields to the broadcast module. The method of claim 1, further comprising:

12. The steps include receiving the structured data item containing one or more data records within the data stream, For at least one of the one or more data records, A step of executing the computer program to process at least one of the one or more data records, The steps of removing one or more fields from the at least one of the records that are not designated as input to the at least one of the at least one of the at least one of the first modules, based on the stored output data format of the at least one of the first modules, before submitting the at least one of the at least one of the at least one of the records to one of the at least one of the second modules, at least partially based on the output of the first module, The method of claim 1, further comprising:

13. The aforementioned specification includes a step of defining one or more new modules, The steps include saving at least one of the one or more new modules to a global palette that specifies entities that can be used for inclusion in other specifications, and The method of claim 1, further comprising:

14. The aforementioned specification includes a step of defining one or more new modules, The steps include saving at least one of the one or more new modules to a local palette that specifies entities that can only be used for inclusion in the aforementioned specification and cannot be included in other specifications, and The method of claim 1, further comprising:

15. For the purpose of defining the specifications, the steps involve accessing predefined modules included in the global palette, The steps include changing one or more attributes of the aforementioned predefined module, A step of performing an operation, wherein the operation is Storing the modified predefined module in a local palette that specifies entities that can only be used for inclusion in the aforementioned specification and cannot be included in other specifications, or To make the modified predefined module accessible to other specifications, save the modified predefined module in the global palette. Steps and The method of claim 1, further comprising:

16. The aforementioned specification includes the step of defining one or more new first modules, The aforementioned specification includes the step of defining one or more new second modules, The steps of configuring at least one of the one or more new first modules to serve as a data source for at least one of the one or more new second modules. The method of claim 1, further comprising:

17. The method according to claim 1, wherein the module includes one or more other modules.

18. Steps to configure the selected module based on the parameter values, A step of storing the configured module for later inclusion in other programs. The method of claim 1, further comprising:

19. The steps include determining that only one of the one or more second modules accesses a specific one of the one or more fields accessible to the first module, The step of determining that only one of the one or more second modules accesses one of the one or more fields from a third module that is different from the first module, The steps of setting the output data format so as not to output one or more values ​​for one or more specific values ​​for one or more fields that are accessed by only one of the one or more second modules. The method of claim 1, further comprising:

20. The method of claim 1, wherein the structured data item includes a data record.

21. A method that a computer system can perform in converting a specification into a computer program that processes one or more values ​​of one or more fields of a structured data item, A step of converting a specification that specifies a plurality of actions to a computer program so that it is associated with a plurality of modules for processing the one or more values ​​of the one or more fields of the structured data item, wherein the plurality of modules are configurable to access the fields of the structured data item based on the actions, For each of the one or more first modules of the plurality of modules, Identifying one or more second modules of the plurality of modules, each receiving an input that is at least partially based on the output of the first module, One or more values ​​of the fields are processed by the one or more second modules, For each of the one or more second modules, when the second module is configured to process one or more values ​​of one or more fields, by identifying one or more of the fields whose values ​​are processed by the second module, To identify, From among the one or more fields processed by the one or more second modules, one or more values ​​are used to identify one or more of the fields that are accessible to the first module. For each of the one or more fields accessible to the first module, Determining whether the value of the field is designated as an input to at least one of the one or more second modules based at least partially on the output of the first module, One or more fields, Identifying one or more fields from among the fields to be accessed by the first module that are dependent on the identified one or more second modules, wherein the one or more fields identified as dependent on the one or more second modules are fewer than the fields to be accessed by the first module. Specifying the output data format of the first module such that the output of the first module identifies only one or more values ​​of one or more fields that correspond to (i) the fields that will be accessed by the first module and (ii) the one or more fields that are dependent on by the one or more second modules. This specifies that it should be included in the output data format of the first module. Steps including, A step of storing the computer program in persistent memory, wherein the computer program specifies an output data format for each of the one or more first modules. Methods that include...

22. A computer system for converting a specification into a computer program that processes one or more values ​​of one or more fields of a structured data item, One or more processing devices, One or more machine-readable hardware storage devices that store instructions executable by the one or more processing devices, wherein the instructions are Accessing a specification that specifies a number of actions that will be associated by the computer program to process one or more values ​​of one or more fields of the structured data item, The process involves converting the specification into a computer program using multiple modules, wherein the multiple modules are configurable to access fields of structured data items based on the operation described above. For each of the one or more first modules of the plurality of modules, Identifying one or more second modules of the plurality of modules, each of which receives an input based at least partially on the output of the first module, One or more fields, Identifying one or more fields from among the fields to be accessed by the first module that are dependent on the identified one or more second modules, wherein the one or more fields identified as dependent on the one or more second modules are fewer than the fields to be accessed by the first module. Specifying the output data format of the first module such that the output of the first module identifies only one or more values ​​of one or more fields that correspond to (i) the fields that will be accessed by the first module and (ii) the one or more fields that are dependent on by the one or more second modules. This is configured to include in the output data format of the first module. This includes, Saving the computer program in persistent memory, wherein the saved computer program specifies the configured output data format for each of the one or more first modules. One or more machine-readable hardware storage devices are used to perform operations including the following: A computer system equipped with the following features.

23. One or more machine-readable hardware storage devices for converting a specification into a computer program that processes one or more values ​​of one or more fields of a structured data item, wherein the one or more machine-readable hardware storage devices store instructions that can be executed by one or more processing units, and the instructions are Accessing a specification that specifies a number of actions that will be associated by the computer program to process one or more values ​​of one or more fields of the structured data item, The process involves converting the specification into a computer program using multiple modules, wherein the multiple modules are configurable to access fields of structured data items based on the operation described above. For each of the one or more first modules of the plurality of modules, Identifying one or more second modules of the plurality of modules, each of which receives an input based at least partially on the output of the first module, One or more fields, Identifying one or more fields from among the fields to be accessed by the first module that are dependent on the identified one or more second modules, wherein the one or more fields identified as dependent on the one or more second modules are fewer than the fields to be accessed by the first module. Specifying the output data format of the first module such that the output of the first module identifies only one or more values ​​of one or more fields that correspond to (i) the fields that will be accessed by the first module and (ii) the one or more fields that are dependent on by the one or more second modules. This is configured to include in the output data format of the first module. This includes, Saving the computer program in persistent memory, wherein the saved computer program specifies the configured output data format for each of the one or more first modules. One or more machine-readable hardware storage devices for performing operations including the following.

Images