Transforming the behavior of a computer program for execution in a database

By translating computer program operations into database-specific languages using a transformation file, complex computations can be efficiently executed in databases, optimizing resource use and reducing computational load on the data processing system.

JP7591649B2Active Publication Date: 2024-11-28AB INITIO TECHNOLOGY LLC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023513773
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-09-04
Filing Date
2021-09-03
Publication Date
2024-11-28
Estimated Expiration
2041-09-03

AI Technical Summary

Technical Problem

Existing systems face challenges in efficiently executing complex computations represented as data flows through directed graphs in databases, as they often require significant computational resources and are limited by the capabilities of different database languages.

Method used

A method is employed to translate computer program operations into database-specific languages using a transformation file, allowing operations to be executed directly in the database, thereby reducing computational load on the data processing system and optimizing resource use.

Benefits of technology

This approach enables efficient execution of complex computations in databases by leveraging database capabilities, reducing processing requirements on the data processing system and enhancing overall computing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007591649000001
    Figure 0007591649000001
  • Figure 0007591649000002
    Figure 0007591649000002
  • Figure 0007591649000003
    Figure 0007591649000003
Patent Text Reader

Abstract

The method includes executing a program that specifies operations and accessing a translation file that contains instructions for converting the language of the program into the language of the database. The translation file specifies operations in the language of the program that are supported by the database and the semantic meaning of the operations that are supported in the language of the database. Operations of the program that are not supported by the database are processed by the program. Operations of the program that are supported by the database are determined from the translation file, and portions of the program that represent the supported operations are converted into the language of the database using the translation file and sent to the database. Data resulting from execution in the database of the converted portions of the program that represent the operations supported by the database is received by the program.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (Priority Claim) This application claims priority to U.S. Provisional Application No. 17 / 012,633, filed September 4, 2020, the entire contents of which are incorporated herein by reference.

[0002] FIELD OF THEINVENTION The present disclosure relates to techniques for transforming computer program operations for execution on a database. [Background technology]

[0003] Complex computations can often be represented as data flows through a directed graph, with components of the computation associated with the vertices of the graph and data flows between the components corresponding to the links (arcs, edges) of the graph. A system for implementing such graph-based computations is described in U.S. Patent No. 5,966,072, entitled "Executing Computations Expressed as Graphs," which is incorporated herein by reference in its entirety. In some cases, the computations associated with the vertices can be represented as queries to a data source. The queries can be provided to the data source (e.g., a database), and a data processing system (e.g., a database management system) associated with the data source can return the data specified by the query. Summary of the Invention

[0004] In general, in a first aspect, a method of converting operations of a computer program for execution in a database includes executing, by a data processing system, a program specifying operations, the program being written in a language; accessing a conversion file including one or more instructions for converting the language of the program to a language of the database, the conversion file specifying one or more operations in the language of the program supported by the database and further specifying a semantic meaning of the one or more supported operations in the language of the database; determining from the conversion file one or more of the program operations that are not supported by the database; processing by the program one or more of the operations that are not supported by the database; determining from the conversion file one or more of the program operations that are supported by the database; for one or more of the operations supported by the database, converting portions of the program representing one or more of the operations supported by the database into the language of the database using the conversion file; sending the converted portions of the program to the database; and receiving by the program data resulting from execution in the database of the converted portions of the program representing one or more of the operations supported by the database.

[0005] In general, in a second aspect, combinable with the first aspect, the method includes performing, by the program, one or more additional operations on the data received from the database.

[0006] In general, in a third aspect, combinable with the first and second aspects, the method includes programmatically performing one or more additional operations on data returned from processing one or more of the operations not supported by the database.

[0007] Generally, in a fourth aspect, combinable with any of the first to third aspects, a program includes an executable data flow graph having components representing operations connected by links representing flows of data between the operations.

[0008] Generally, in a fifth aspect, combinable with any of the first to fourth aspects, translating the portion of the program representing one or more of the operations supported by the database includes generating one or more queries to the database in a language of the database to perform one or more of the operations within the database.

[0009] Generally, in a sixth aspect, combinable with any of the first to fifth aspects, a method wherein at least one of the program operations includes a query operation, the method including removing from the program one or more operations supported by the database, and inserting into the query operation one or more queries to the database in a language of the database.

[0010] Generally, in a seventh aspect, combinable with any of the first to sixth aspects, determining one or more of the operations supported by the database includes applying one or more pattern matching rules to one or more operations in a transformation file specified in a language of the program.

[0011] In general, in an eighth aspect, combinable with any of the first to seventh aspects, the semantic meaning describes at least one of a data type, an operator meaning, a function meaning, or an intended result of an operation of one or more supported operations in the language of the database.

[0012] Generally, in the ninth aspect, which can be combined with any of the first to eighth aspects, the conversion file is not encoded within the program.

[0013] Generally, in a tenth aspect, combinable with any of the first to ninth aspects, a method includes receiving, from a program, an indication of a language of the database.

[0014] Generally, in an eleventh aspect, which can be combined with any of the first to tenth aspects, the program language includes a data manipulation language (DML) and the database language includes a structured query language (SQL).

[0015] Generally, in a twelfth aspect, which can be combined with any of the first to eleventh aspects, the conversion file includes one or more instructions for converting a language of the program into a plurality of languages ​​of the database, including the language of the database.

[0016] Generally, in a thirteenth aspect, which may be combined with any of the first to twelfth aspects, the system includes one or more processors and one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to execute, by a data processing system, a program specifying operations, the program being written in a language; accessing a conversion file including one or more instructions for converting the language of the program into a language of the database, the conversion file specifying one or more operations in a language of the program supported by the database and further specifying a semantic meaning of the one or more supported operations in the language of the database; and accessing from the conversion file, among the program operations not supported by the database. and one or more storage devices that cause the program to perform operations including: determining, by a program, one or more of the operations not supported by the database; processing one or more of the operations of the program that are supported by the database from the translation file; determining, for one or more of the operations supported by the database, using the translation file to convert portions of the program that represent one or more of the operations supported by the database into a language of the database; sending the converted portions of the program to the database; and receiving, by the program, data resulting from execution in the database of the converted portions of the program that represent one or more of the operations supported by the database.

[0017] In general, in a fourteenth aspect, combinable with any of the first to thirteenth aspects, a non-transitory computer readable storage medium stores instructions, the instructions causing a computing system to execute, by a data processing system, a program specifying operations, the program being written in a language, access a conversion file including one or more instructions for converting the language of the program to a language of a database, the conversion file specifying one or more operations in the language of the program supported by the database and further specifying a semantic meaning of the one or more supported operations in the language of the database, determine from the conversion file one or more of the program operations not supported by the database, process one or more of the operations not supported by the database, determine from the conversion file one or more of the program operations supported by the database, for one or more of the operations supported by the database, convert a portion of the program representing one or more of the operations supported by the database using the conversion file to the language of the database, transmit the converted portion of the program to the database, and receive, by the program, data resulting from execution in the database of the converted portion of the program representing one or more of the operations supported by the database.

[0018] In general, in a fifteenth aspect, which can be combined with any of the first to fourteenth aspects, a method includes: executing, by a data processing system, a computer program specifying operations, the computer program being written in a programming language; accessing a conversion file including one or more instructions for converting the programming language of the computer program into a language of a database, the conversion file specifying one or more operations in the language of the program supported by the database and further specifying a semantic meaning of the one or more supported operations in the language of the database; determining from the conversion file one or more of the operations of the program supported by the database; for one or more of the operations supported by the database, converting a portion of the program representing one or more of the operations supported by the database using the conversion file into the language of the database; transmitting the converted portion of the program to the database; and receiving, by the program, from the database, data resulting from execution in the database of the converted portion of the program representing one or more of the operations supported by the database.

[0019] Generally, in a sixteenth aspect, which may be combined with any of the first to fifteenth aspects, the method includes determining, from the conversion file, one or more of the program operations that are not supported by the database; and processing, by a computer program, one or more of the operations that are not supported by the database.

[0020] Generally, in a seventeenth aspect, which can be combined with any of the first to sixteenth aspects, the computer program does not process one or more of the program operations supported by the database.

[0021] Generally, in an eighteenth aspect, which can be combined with any of the first to seventeenth aspects, the computer program processes only one or more of the operations that are not supported by the database.

[0022] Generally, in a nineteenth aspect, which can be combined with any of the first to eighteenth aspects, the data processing system does not execute portions of a computer program that represent one or more of the operations supported by the database.

[0023] One or more of the above implementations may provide one or more of the following advantages.

[0024] The techniques described herein enable operations of a computer program (e.g., an executable data flow graph) to be translated for execution in a database (e.g., a relational database). In particular, the techniques described herein include a translation file that specifies instructions for converting operations specified in the language of the program into the language of the database, such that some or all of the operations of the program can be embedded in queries to the database. In this manner, the translation file facilitates "pushing down" the computational load of one or more operations from the data processing system executing the program to the database, thereby reducing processing requirements on the data processing system and enabling more efficient use of overall computing resources.

[0025] Because databases may use a variety of different languages ​​(e.g., SQL dialects) with different capabilities, program operations supported by one database using a particular language may not be supported by another database using a different language. The transformation files described herein specify which program operations are supported by each language in the set of languages ​​that may be used by the database. By doing so, the transformation files allow the database to effectively "negotiate" with the system that translates the program regarding the operations that will be pushed down to the database. In other words, if the system determines, based on the transformation files, that the language used by the database does not support a particular operation of the program, the system can decide not to push down the operation to the database 110 (or can reorder computational steps or take another action). Furthermore, because the transformation files are accessed at run time, they can be extended to specify operations supported by additional database languages, or to update operations supported by existing database languages, or both, without requiring modifications to the underlying computer program. In this manner, programs can be reused and their operations can be pushed down to virtually any database.

[0026] The transformations specified in the transformation file may also account for the semantic meaning of the operations specified in the program's language, thereby improving the accuracy of the transformation and supporting the transformation of additional operations, for example by allowing transformations even when there may not be a corresponding or equivalent operation in the database's language that is specified in the program's language.

[0027] By "pushing down" operations to one or more databases instead of performing the pushed-down operations by the data processing system executing the program, the built-in functionality of the databases can be efficiently used by any data processing system executing the program.

[0028] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter.

[0029] Details of one or more implementations are set forth in the accompanying drawings and the description below. Other features, objects, and advantages set forth herein will be apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 is a block diagram of a system for translating operations of a computer program executed on a database. [Figure 2A] FIG. 2 is a block diagram of an example of a translation operation of a computer program for execution in a database. [Figure 2B] FIG. 2 is a block diagram of an example of a translation operation of a computer program for execution in a database. [Figure 3A] FIG. 2 is a block diagram illustrating additional details of a system for translating computer program operations for execution on a database. [Figure 3B] FIG. 2 is a block diagram illustrating additional details of a system for translating computer program operations for execution on a database. [Figure 3C] FIG. 2 is a block diagram illustrating additional details of a system for translating computer program operations for execution on a database. [Figure 3D] FIG. 2 is a block diagram showing further details of the conversion engine. [Figure 3E]FIG. 2 is a block diagram showing further details of the conversion engine. [Figure 3F] FIG. 2 is a block diagram showing further details of the conversion engine. [Figure 3G] FIG. 2 is a block diagram showing further details of the conversion engine. [Figure 3H] FIG. 2 is a block diagram showing further details of the conversion engine. [Figure 3I] FIG. 2 is a block diagram showing further details of the conversion engine. [Figure 3J] FIG. 2 is a block diagram showing further details of the conversion engine. [Figure 3K] FIG. 2 is a block diagram showing further details of the conversion engine. [Figure 3L] FIG. 2 is a block diagram showing further details of the conversion engine. [Figure 3M] FIG. 2 is a block diagram showing further details of the conversion engine. [Figure 3N] FIG. 2 is a block diagram showing further details of the conversion engine. [Figure 3O] FIG. 2 is a block diagram showing further details of the conversion engine. [Figure 3P] FIG. 2 is a block diagram showing further details of the conversion engine. [Figure 3Q] FIG. 2 is a block diagram illustrating additional details of a system for translating computer program operations for execution on a database. [Figure 3R] FIG. 2 is a block diagram illustrating additional details of a system for translating computer program operations for execution on a database. [Figure 3S] FIG. 2 is a block diagram illustrating additional details of a system for translating computer program operations for execution on a database. [Figure 3T] FIG. 2 is a block diagram illustrating additional details of a system for translating computer program operations for execution on a database. [Figure 4A] FIG. 2 is a block diagram of an example of a translation operation of a computer program for execution in a database. [Figure 4B]FIG. 2 is a block diagram of an example of a translation operation of a computer program for execution in a database. [Diagram 5] FIG. 2 is a block diagram showing an example of extensibility of a conversion file. [Figure 6A] 3 shows an executable data flow graph at various stages of optimization. [Figure 6B] 3 shows an executable data flow graph at various stages of optimization. [Figure 6C] 3 shows an executable data flow graph at various stages of optimization. [Figure 7] 1 is a flowchart of a process for translating computer program operations for execution on a database. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] 1 illustrates a system 100 for translating computer program operations for execution on a database. The system 100 includes a development environment 102, an execution environment 104, storage systems 106, 108, and a number of candidate databases 110a, 110b, ..., 110N (collectively referred to as "databases 110"). In general, the optional development environment 102 provides a means for a user to create a computer program, such as a dataflow graph 112. For example, the development environment 102 provides a graphical development environment or other user interface that allows a user to specify calculations or rules that can be represented in the form of a dataflow graph 112. After generating the dataflow graph 112 configured to perform operations according to the user's specifications, the development environment 102 passes the dataflow graph 112 to the execution environment 104 for compilation and execution. In general, the dataflow graphs described herein are executable dataflow graphs, and the terms "dataflow graph" and "executable dataflow graph" may be used interchangeably. An executable dataflow graph has one or more components, and the term "component" may be used interchangeably with "node" or "vertex." A node, component, or vertex corresponds to program code, as also shown at the bottom of column 5 of the referenced U.S. Pat. No. 5,966,072. For example, a sort component corresponds to a particular program template. The same technique is used for SQL commands. Each command may have one or more associated templates. As is evident from the Abstract and columns 1-2, 5, and 9-11 of the referenced U.S. Pat. No. 5,966,072, those skilled in the art will understand that the executable dataflow graphs described herein may be used to directly invoke a computation.

[0032] The execution environment 104 includes a compiler engine 114 for compiling the dataflow graph 112 into an executable program (e.g., an executable dataflow graph) and an execution engine 116 for performing operations to execute the executable program. During operation, the compiler engine 114 receives data representing the dataflow graph 112 from the development environment 102 (or from another source, such as one of the storage systems 106, 108). In some cases, the compiler engine 114 determines that data (e.g., input data) to be processed by the executable version of the dataflow graph 112 is stored in one or more of the databases 110, which may be relational databases. For example, an input component of the executable dataflow graph 112 may specify that the input data is to be stored in one or more of the databases 110.

[0033] In other words, the computer system 100 includes a development environment 102 coupled to a data storage, such as a storage system 106, configured to build a data processing application associated with an executable data flow graph 112 that implements a graph-based computation to be performed on data flowing from one or more input datasets, through a graph of processing graph components to one or more output datasets, the data flow graph 112 being specified by a data structure in the data storage, the data flow graph 112 having a plurality of components specified by the data structure, the graph components being connected by one or more links, the links representing the graph components being linked by the data structure. a development environment, in which a data flow graph 112 is specified by a data structure representing data flows between graph components; and an execution environment 104 coupled to a storage system 106 and hosted on one or more computers, the execution environment 104 including a compiler engine 114 configured to read the stored data structures specifying the data flow graph 112 and to allocate and configure computing resources such as processes to perform computations of the graph components assigned by the data flow graph by the compiler engine 114, the execution environment 104 including an execution engine 116 that schedules and controls the execution of processes assigned to perform operations to execute the executable program.

[0034] In these and other cases, compiler engine 114 invokes transformation engine 118, which is configured to transform some or all of the operations of dataflow graph 112 so that the operations can be executed by one or more of databases 110 (or data processing systems associated with the respective databases). For example, transformation engine 118 generates queries (e.g., SQL queries) to database 110 that, when executed at the database, cause the database to execute one or more operations of dataflow graph 112 and return results. The queries are then inserted into dataflow graph 112 in place of one or more operations to generate transformed dataflow graph 122, effectively "pushing off" the computational burden of the one or more operations from execution environment 104 to database 110.

[0035] To convert some or all of the operations of the dataflow graph 112 into instructions executable by the database 110, the conversion engine 118 is configured to receive a conversion file 120 stored in the storage system 106. Typically, the conversion file 120 includes instructions for converting operations defined in the language of the dataflow graph 112 into the language of the database 110. For example, the dataflow graph 112 defines operations in a language such as Data Manipulation Language (DML). The database 110, on the other hand, is configured to interpret instructions (e.g., queries) in a different language such as SQL or a dialect thereof (e.g., DB2, PostgreSQL, Oracle, etc.). Thus, the conversion engine 118 uses the conversion file 120 to convert one or more operations defined in the language of the dataflow graph 112 into queries interpretable by the respective database 110.

[0036] Because databases 110 may use a variety of different languages ​​(e.g., SQL dialects) with different capabilities, an operation of executable dataflow graph 112 that is supported by one database 110 using a particular language may not be supported by another database 110 using a different language. Thus, transformation file 120 specifies which operations of dataflow graph 112 are supported by each language in a set of languages ​​that may be used by database 110. By doing so, transformation file 120 allows database 110 to effectively "negotiate" with transformation engine 118 regarding operations that will be pushed down to database 110. In other words, if transformation engine 118 determines, based on transformation file 120, that the language used by database 110 does not support a particular operation of dataflow graph 112, transformation engine 118 may decide not to push down the operation to database 110 (or may reorder computational steps or take another action, as described below). Furthermore, because the transformation file 120 is accessed at run time, it can be extended to specify operations supported by additional database languages, or to update operations supported by existing database languages, or both, without requiring modifications to the underlying dataflow graph 112. In this manner, the dataflow graph 112 can be reused, and the transformation engine 118 can push down its operations to virtually any database 110. Further details regarding the extensibility of the transformation file 120 are described below with reference to FIG.

[0037] In one example, an operation of a computer program, such as an executable dataflow graph, is "supported" by a database (or database language) if the operation can be expressed in the database language such that the operation (or an equivalent operation) can be executed by the database (or database management system). In one example, an operation of a computer program (e.g., an executable dataflow graph) is "not supported" by a database (or database language) if the operation cannot be expressed in the database language, or if the operation (or an equivalent operation) cannot be executed by the database (or database management system), or both. In one example, whether an operation of a computer program is supported by a particular database (or database language) is determined using a translation file 120.

[0038] In some examples, the transformations specified in the transformation file 120 describe the semantic meaning of the operations specified in the language of the dataflow graph 112. We use the term "semantic meaning" in this context to refer to a transformation that goes beyond a simple one-to-one translation from one language to another without associating a meaning to the transformation. For example, the transformations specified in the transformation file 120 may describe data types (e.g., type sensitivity), the meaning of an operator or function, the intended result of the operation, or a combination thereof, among other transformations that associate a meaning or go beyond mere syntactic translation. By considering the semantic meaning of the operations specified in the language of the dataflow graph 112 when translating the operations to the language of the database 110, the accuracy of the translation can be improved and the translation of additional operations can be supported (e.g., by allowing translation even when there may not be an operation in the language of the database 110 that corresponds to or is equivalent to the operation specified in the language of the dataflow graph 112).

[0039] After translating some or all of the operations of the dataflow graph 112 into queries to one or more of the databases 110, the transformation engine 118 passes the transformed dataflow graph 122 to the compiler engine 114 for compilation. The compiler engine 114 compiles the transformed dataflow graph 122 into an executable program and sends the executable program to the execution engine 116. The execution engine 116 performs operations to execute the executable program and stores the results in a storage system 108, which may be the same as or different from the storage system 106. To execute the executable program, the execution engine 116 invokes the query engine 124 to perform the operations specified by the portion of the program associated with the transformed portion of the dataflow graph. In doing so, the query engine 124 is adapted to send the queries generated by the transformation engine 118 to the appropriate databases 110. The databases 110 (or a data processing system associated with the respective databases, such as a database management system (DBMS)) execute each query received from the query engine 124 and return the resulting data. The query engine 124 provides the data received from the database 110 to the execution engine 116, which executes operations to process the data according to the portion of the program associated with the untransformed portion of the dataflow graph.

[0040] Referring to FIG. 2A, system 200 illustrates an example of translating computer program operations for execution on a database. In this example, a dataflow graph 202 is provided to compiler engine 114 for compilation. Dataflow graph 202 includes input components 203a, 203b that represent data inputs, action components 203c, 203d, 203e, 203f that represent operations performed on the data inputs or data flowing from another action component, and output component 203g that represents a destination (e.g., a write destination) for the processed data. In this example, input component 203a represents a data input from database 110a ("DB1"), which is a relational database. Meanwhile, input component 203b represents a data input that is not stored in a relational database or that cannot be otherwise queried.

[0041] The compiler engine 114 invokes the transformation engine 118 to generate a query to the relational database 110a to access the data specified by the input component 203a. The transformation engine 118 also determines whether the "operation 1" (or a portion thereof) represented by the component 203c can be pushed down to the database 110a via a query. To do so, the transformation engine 118 first determines that the language used by the database 110a is "dialect 1". This information is obtained, for example, from the input component 203a or a file associated with the input component 203a. The transformation engine 118 then reads the transformation file 204 to determine whether the "operation 1" represented by the component 203c can be converted from the language of the data flow graph 202 to "dialect 1", which is the language used by the database 110a.

[0042] For example, assume that the input data represented by input component 203a includes separate records for a customer's first name and last name, and that "action 1" represented by component 203c serves to combine the customer's first name and last name into a single string via the expression "First+Last" in the language of dataflow graph 202. By comparing the expression for "action 1" specified in the language of dataflow graph 202 to translation file 204 (e.g., using pattern matching techniques, as described below), translation engine 118 determines that the expression can be translated into the "dialect 1" language. In addition, because translation file 204 describes the semantic meaning of the expression in translation from the language of dataflow graph 202 to the language of database 110a, translation engine 118 determines that the expression "First+Last" should be translated into "First Name, Last Name" in the "Dialect 1" language (e.g., rather than Total(First,Last), which implies a numeric input).

[0043] After converting "operation 1" from the language of dataflow graph 202 to the language of database 110a, transformation engine 118 generates a query to database 110a to access the specified input data and execute the converted operation. Transformation engine 118 replaces input components 203a and 203c in dataflow graph 202 with query component 206 to generate a converted dataflow graph 208. Transformation engine 118 then passes converted dataflow graph 208 to compiler engine 114 for compilation. Compiler engine 114 compiles converted dataflow graph 208 into executable program 210 and sends program 210 to execution engine 116 (or to a storage system for storage). Execution engine 116 performs operations to execute executable program 210, including invoking query engine 124 to send query 212 to database 110a as specified by the portion of program 210 associated with query component 206. Database 110a executes query 212 and returns the resulting data to query engine 124. Query engine 124 provides the received data to execution engine 116, which executes operations to process the data according to the portions of program 210 associated with the untransformed portions of the dataflow graph (e.g., components 203b, 203d, 203e, 203f, 203g).

[0044] Referring to FIG. 2B, system 250 illustrates another example of translating the operations of a computer program for execution on a database. In this example, dataflow graph 252 is provided to compiler engine 114 for compilation. Dataflow graph 252 is the same as dataflow graph 202 illustrated in FIG. 2A, except that input data is received from database 110b ("DB2") as indicated by input component 253a. Database 110b, like database 110a, is a relational database. However, database 110b uses a different language ("Dialect 2") than database 110a.

[0045] In this example, compiler engine 114 invokes transformation engine 118 to generate a query to relational database 253a to access data specified by input component 110b. Transformation engine 118 also reads transformation file 204 to determine whether "Action 1" represented by component 203c can be transformed from the language of dataflow graph 252 to "Dialect 2", which is the language used by database 110b. As in the previous example, assume that "Action 1" represented by component 203c serves to combine a customer's first name and last name into a single string via the expression "First+Last" in the language of dataflow graph 252. By comparing the expression for "Action 1" in the language of dataflow graph 252 with the supported operations for "Dialect 2" specified in transformation file 204, transformation engine 118 determines that the expression can be transformed into the "Dialect 2" language. In addition, because the translation file 204 describes the semantic meaning of expressions in translation from the language of the dataflow graph 252 to the language of the database 110b, the translation engine 118 determines that the expression "First+Last" should be translated to "First Name||Last Name" in the "Dialect 2" language (rather than, for example, Sum(First,Last), which means a numeric input).

[0046] After converting "operation 1" from the language of dataflow graph 252 to the language of database 110b, transformation engine 118 generates a query to database 110b to access the specified input data and execute the converted operation. Transformation engine 118 replaces input component 253a and component 203c in dataflow graph 252 with query component 256 to generate a converted dataflow graph 258. Transformation engine 118 then passes converted dataflow graph 258 to compiler engine 114, which compiles converted dataflow graph 258 into executable program 260 and sends program 260 to execution engine 116. Execution engine 116 performs operations to execute executable program 260 as described above, including invoking query engine 124 to send query 262 to database 110b, as specified by the portion of program 260 associated with query component 256. As shown by the examples of Figures 2A and 2B, the techniques described herein allow the same dataflow graph to be optimized and run on different databases using different languages ​​without requiring modifications to the underlying dataflow graph.

[0047] 3A, system 300 illustrates additional details of system 100. In this example, compiler engine 114 of execution environment 104 receives data flow graph 302 (e.g., from development environment 102, from storage system 106 or 108). Data flow graph 302 includes input components 303a, 303b that represent data inputs, action components 303c, 303d, 303e, 303f that represent actions to be performed on the data inputs or data flowing from other action components, and output component 303g that represents a destination for the processed data. In this example, input component 303a represents data input from database 110a ("DB1") and input component 303b represents data input from database 110b ("DB2"), both of which are relational databases.

[0048] Because dataflow graph 302 is configured to access data from relational databases, compiler engine 114 invokes transformation engine 118 to generate queries to each of relational databases 110a, 110b to access the data specified by input components 303a, 303b. Transformation engine 118 also reads transformation file 120 to determine whether some or all of the operations of dataflow graph 302 (e.g., operations (or portions thereof) represented by components 303c, 303d, 303e, 303f) can be transformed into the language of one or more databases. In this example, transformation engine 118 can transform at least "operation 1" represented by component 303c into the language of database 110a. Additional details regarding the process for transforming operations of a dataflow graph are described below with reference to Figures 3D-3P.

[0049] After converting some or all of the operations from the language of the dataflow graph 302 to the language of one of the databases 110a, 110b, the conversion engine 118 generates one or more queries to the databases 110a, 110b to access the specified input data and execute the converted operations (if any). In this example, the conversion engine 118 replaces input component 303a and component 303c in the dataflow graph 302 with query component 304a in the converted dataflow graph 306. The conversion engine 118 also replaces input component 303b with query component 304b in the converted dataflow graph 306.

[0050] The transformation engine 118 then passes the transformed dataflow graph 306 to the compiler engine 114, which compiles the transformed dataflow graph 306 into an executable program 308 and sends the program 308 to the execution engine 116. The execution engine 116 then performs operations to execute the executable program 308. In general, the execution of the program 308 includes a remote execution process 310 and a local execution process 312, which may occur in series or in parallel depending on the implementation and / or the dataflow graph 302. During the remote execution process 310, the execution engine 116 executes the portion of the program 308 associated with the transformed portion of the dataflow graph. In particular, the execution engine 116 executes the portion of the program 308 including the machine code 309a, 309b representing instructions for queries to the databases 110a, 110b. In doing so, the execution engine 116 causes the query engine 124 to send queries 314a, 314b (e.g., as defined in the generated query components 304a, 304b) to the databases 110a, 110b. Each of the databases 110a, 110b executes its respective query 314a, 314b to generate query results 316a, 316b, which are returned to the query engine 124 (or the execution engine 116, or both).

[0051] In this example, the execution engine 116 uses the query results 316a, 316b as input data for the local execution process 312. During the local execution process 312, the execution engine 116 executes the portion of the program 308 associated with the untransformed portion of the dataflow graph. In particular, the execution engine 116 executes the portion of the program 308 that includes machine code 309c, 309d, 309e, 309f that represent operations of the dataflow graph that have not been transformed or pushed down to the database 110. Data resulting from the execution of the program 310 may be stored, displayed to a user, or both, among other things.

[0052] 3B and 3C show additional details of a system 300 for translating computer program operations for execution on a database. In this example, the development environment 102 provides a dataflow graph 302 to the compiler engine 114 for compilation. The dataflow graph 302 includes input components 303a, 303b representing data inputs, action components 303c, 303d, 303e, 303f representing operations performed on data flowing from the data inputs or other action components, and output component 303g representing destinations for processed data. In general, components 303c, 303d, 303e, 303f specify various operations to be performed on data flowing through the dataflow graph 302, including filter operations, aggregation operations (e.g., count, sum, min, max, average, etc.), and join operations, among others. In one example, each operation includes one or more expressions that define the entire operation. In some cases, expressions are nested within an operation, such as that represented by component 303d. It should be noted that the terms "action" and "representation" are sometimes used interchangeably in this disclosure, although in some instances, representations are more granular than actions.

[0053] The techniques described herein may be used to translate some or all of an operation (or a portion of an operation, such as one or more expressions that make up the operation) from the language of the dataflow graph 302 to the language of the database 110. In particular, the translation engine 118 utilizes a translation file 120 to translate an operation (or a portion thereof) from the language of the dataflow graph 302 to the language of the database 110 in a manner that is consistent with the semantic meaning of the operation. In some examples, the operation is translated into a query in the language of the database 110 that is inserted into the dataflow graph 302 in place of the translated operation, allowing the operation to be effectively pushed down to the database. Additional details regarding the translation file 120 are provided below.

[0054] A transformation file (sometimes called a "dialect file") contains a set of database language specifications (for example, a SQL dialect specification) such as: sql dialect "default" {...} sql dialect "db2" {...} sql dialect "db2eee" {...}

[0055] Each input data source in the dataflow graph specifies a database name (or DBMS name) or includes a file that provides the database name. This information can be used by the system to look up or otherwise obtain the database language or dialect. In one example, the "default" SQL dialect is used if no other dialect matching the database name is found.

[0056] Dialect inheritance: In one example, a dialect may specify a base dialect and then override some of its parameters or rules. sql dialect "greenplum" {base dialect="postgresql";} sql dialect "db2eee" {base dialect="db2";}

[0057] If no base dialect is specified, the "default" dialect can be used as the base.

[0058] Dialect Rules: In one example, an SQL dialect specification includes a number of pattern matching rules that embody the semantic meaning of an expression. In general, pattern matching is the act of checking a given expression or set of expressions for the presence of certain pattern components according to rules (e.g., pattern matching rules). These rules may have the following form: sql(...)=dml(...);

[0059] In one example, the DML expression is a function call or a unary or binary operator. sql(a * b) = dml(a * b); sql(ABS(num))=dml(math_abs(num));

[0060] In one example, arguments to a function or operator may be symbolic names as described above, or may be constants. sql(SIGN(num))=dml(compare(num,0));

[0061] A symbol can be matched against whatever sub-expression appears in that position in the actual DML. A constant can be matched if the sub-expression can be statically evaluated to be equal to that constant, so in some instances the form of the constant does not matter. For example, the DML expression compare(foo,(int)0) will match the above pattern in some instances.

[0062] In one example, patterns are matched in the order in which they appear, thus allowing more specific patterns to be combined with general patterns. sql(TRUNC(num))=dml(decimal_truncate(num,0)); sql(TRUNC(num,num_places)=dml(decimal_truncate(num,num_places));

[0063] If no patterns match a particular DML expression, it can be determined that the dialect does not support such an expression, and therefore the expression does not need to be transformed or pushed down.

[0064] On the DML side of the rule, the text is parsed as a DML expression and the match is done against the parsed DML expression. In one example, if the actual DML has key value arguments, they are first concatenated and then repositioned for matching.

[0065] On the SQL side of the rule, the text is tokenized using a SQL tokenizer that understands keywords from all common SQL dialects. Any non-keyword symbols that match names used on the DML side of the rule are rewritten to whatever the name was bound to when the pattern was matched.

[0066] Dialect parameters: In addition to the base dialect, a dialect may specify several other parameters to control the generation of SQL. For example, one or more parameters may be used to control, among other things, the generation of GROUP BY and ORDER BY clauses for voidable columns, how string concatenation is rendered, pushdown of record restriction behavior, the number of characters allowed in a column or table alias, pushdown of joins where no input is required, or a combination thereof.

[0067] 3D-3P, additional details of the transformation engine 118 are shown. For purposes of illustration, the transformation engine 118 is shown as including a process portion 320 and a data portion 322. In this example, the process portion 320 includes a flowchart of operations performed by the transformation 118 to transform the operations of the dataflow graph 302 for execution on the database 110. The data portion 322 includes the dataflow graph 302, dialect mappings 324 that specify a set of databases and their corresponding dialects, and a transformation file 326.

[0068] In this example, the process shown in process portion 320 begins with the transformation engine 118 receiving (328) the dataflow graph 302 (FIG. 3E). For example, the transformation engine 118 receives the dataflow graph 302 from the development environment 102 or the storage system 106, 108 via the compiler engine 114. The transformation engine then generates (330) query components 304a, 304b for the input components 303a, 303b that access data from a relational database (FIG. 3F). The query components 304a, 304b include queries in the language of the database (e.g., SELECT statements) to retrieve data from the database 110 specified by the input components 303a, 303b. As shown in data portion 322, the generated query components 304a, 304b replace the input components 303a, 303b in the dataflow graph 302.

[0069] The transformation engine 118 selects (332) an independent operation or sequence of operations in the dataflow graph 302 to consider for transformation (FIG. 3G). In one example, the transformation engine 118 processes the dataflow graph 302 in topological order, and therefore considers operations and sequences of operations from left to right. The transformation engine 118 then selects (334) an operation to consider for transformation (FIG. 3H). Because the selection in step 332 resulted in a single operation ("Operation 1," represented by component 303c), the transformation engine 118 selects that operation for consideration here.

[0070] After selecting an operation, the conversion engine 118 identifies (336) an expression in the operation, reads (338) the conversion file 326, and determines (340) whether the expression is supported by the database (e.g., whether the expression can be converted from the language of the dataflow graph to the language of the database). As shown in FIG. 3I, the conversion engine 118 identifies an expression "Expression A(In)" for "Operation 1". Since "Operation 1" is configured to process data received from database "DB1" (e.g., database 110a), it can only be logically pushed down to database "DB1". Thus, the conversion engine 118 considers whether the identified expression can be converted to the language of database "DB1". To do so, the conversion engine 118 uses dialect mapping 324 to determine that database "DB1" uses the "Dialect 1" language. The conversion engine 118 then accesses the conversion file 326 to determine whether the expression "Expression A(In)" is supported by database "DB1" using "Dialect 1". Because there is a match (indicated by the thick dashed line), the transformation engine 118 transforms the expression into a query in the "Dialect 1" language (342). In addition, the transformation engine 118 updates the dataflow graph 302 (shown at time T1) to insert the query into query component 304a and remove component 303c representing "Operation 1," generating a transformed dataflow graph 306 (at time T2), as shown in FIG.

[0071] Referring to FIG. 3K, the transformation engine 118 determines that the selected operation has no additional expressions (344), determines that the selected sequence has no additional operations (346), but determines that there are additional sequences to consider (348). Thus, the transformation engine 118 selects (332) a new sequence of operations in the transformed dataflow graph 303 to consider for transformation. In this example, the selected operations include "operation 2" represented by component 303d and "operation 3" represented by component 303e. The transformation engine 118 then selects (334) an operation to consider ("operation 2" shown in FIG. 3L) and identifies (336) expressions within the operation. Because "operation 2" contains nested expressions, the transformation engine 118 starts with the independent expression "expression C(In)" and reads (338) the transformation file 326 to determine whether the expression is supported by the database, as shown in FIG. 3M. Because "operation 2" is configured to process data received from database "DB2" (e.g., database 110b), conversion engine 118 considers whether the identified expression can be converted into the language of database "DB2". To do so, conversion engine 118 uses dialect mapping 324 to determine that database "DB2" uses the "dialect 2" language. Then, conversion engine 118 accesses conversion file 326 to determine whether expression "expression C(In)" is supported by database "DB2" using "dialect 2". Because there is a match (shown by the bold dashed line), conversion engine 118 converts the expression into a query in the "dialect 2" language (342). In addition, conversion engine 118 updates converted dataflow graph 306 (shown at time T1) to insert the query into query component 304b and remove the expression representing "operation 2" from component 303d, generating an updated dataflow graph 306 (at time T2) as shown in FIG. 3N.

[0072] After transforming "Expression C(In)" and updating the dataflow graph 306 accordingly, the transformation engine 118 determines that there are additional expressions for the selected operation (344). Thus, the transformation engine 118 identifies "Expression B(In)" (336), reads the transformation file 326 (338), and determines whether the expression is supported by the database "DB2" (340), as shown in FIG. 3O. Because there is no match for "Expression B(In)" in the transformation file 326, the transformation engine 118 determines that the expression is not supported by the "Dialect 2" language used by the database "DB2". As a result, the transformation engine 118 does not transform the expression or any dependent expressions (350). Because "OPERATION 3" represented by component 303e and "OPERATION 4" represented by component 303f each depend, in this example, at least in part, on "OPERATION 2" (and on "Representation B(In)"), transformation engine 118 does not attempt to transform or push down these operations or their corresponding representations. As a result, transformation engine 118 determines (348) that there are no additional sequences to consider and outputs (352) the transformed dataflow graph, as shown in FIG. 3P.

[0073] It should be noted that while transformation engine 118 did not consider the operations represented by components 303e and 303f in the above example, it may do so in other examples where the operations do not depend on operations that cannot be transformed or pushed down. Additionally, in some examples, transformation engine 118 may reorder the operations or their corresponding representations, including breaking down the operations or representations into separate components, to enable the operations or representations to be pushed down. For example, in an alternative version of the above example, transformation engine 118 may swap the order of "operation 2" and "operation 3" to allow "operation 3" to be pushed down if it does not affect the intended outcome of the dataflow graph.

[0074] 3Q-3T show additional details of a system 300 for transforming the operations of a computer program for execution on a database. After transforming the operations of the dataflow graph 302 to generate a transformed dataflow graph 306, the transformation engine 118 passes the transformed graph 306 to the compiler engine 114 for compilation. The compiler engine 114 compiles the transformed dataflow graph 306 into an executable program 308 having machine code 309a, 309b, 309c, 309d, 309e, 309f that represent instructions that can be executed to perform the operations of the transformed dataflow graph 306, as shown in FIG. 3R. The compiler engine 114 provides the program 308 to the execution engine 116 for execution. As described above, the execution engine 116 may begin by executing a portion of the program 308 associated with the transformed portion of the dataflow graph, as shown in FIG. 3S. In particular, the execution engine 116 executes a portion of the program 308, which includes machine code 309a, 309b representing instructions for querying the databases 110a, 110b. In doing so, the execution engine 116 causes the query engine 124 (or the execution engine itself) to establish communication with the databases 110a, 110b to be queried, and to send queries 314a, 314b (which may be the same as the queries inserted into the query components 304a, 304b by the transformation engine 118) to the respective databases 110a, 110b. Each of the databases 110a, 110b executes its respective query 314a, 314b to generate query results 316a, 316b, which are returned to the query engine 124 (or the execution engine 116, or both).

[0075] Upon receiving the query results 316a, 316b, the execution engine 116 executes the portion of the program 308 associated with the untransformed portion of the dataflow graph using the query results 316a, 316b as inputs. Specifically, the execution engine 116 executes the portion of the program 308 that includes machine code 309c, 309d, 309e, 309f that represent operations of the dataflow graph that were not transformed or pushed down to the database 110. Results 354 from the execution of the program 310 may be stored, for example, in the storage system 108, or displayed to a user, or both, among other things.

[0076] 4A, environment 400 illustrates a real-world example of transforming the behavior of a computer program for execution on a database. In this example, transformation engine 118 receives a dataflow graph 402 having an input query component 403a and an action component 403b. Note that in this example, only a portion of dataflow graph 402 is shown. Input query component 403a specifies a query in the language of a database (e.g., database 110) to retrieve all records from table "input1" in the database. Action component 403b specifies an action in the language of dataflow graph 402 configured to capitalize and concatenate the first name ("fname") and last name ("lname") fields (with a space in between) for each record in the input data.

[0077] To determine whether some or all of the expressions in the behavioral component 403b can be pushed down to the database, the transformation engine 118 first identifies the language used by the database (e.g., based on the input query component 403a, the file associated with the input component 403a, etc.). Assuming the transformation engine 118 determines that the database language is "Dialect 1", the transformation engine 118 reads the transformation file 404 to determine whether some or all of the expressions represented by the behavioral component 403b are supported by the "Dialect 1" language. Now that there is a match for each expression of the behavior, the transformation engine 118 can fully transform the behavior from the language of the dataflow graph 402 to the language of the database. The transformation engine 118 generates a transformed dataflow graph 406 in which the behavioral component 403b has been removed and the transformed behavior has been inserted into the input query component 403a, effectively pushing down the processing specified in the behavioral component 403b to the database.

[0078] Referring to FIG. 4B, environment 450 illustrates another real-world example of transforming computer program operations for execution on a database. The example illustrated in environment 450 is similar to the example illustrated in environment 400, except that input component 403a uses the "Dialect 2" language to access data from table "Input 2" in the database. As in the previous example, transformation engine 118 reads transformation file 404 to determine whether some or all of the expressions represented by operation component 403b are supported by the language of the database. In this example, transformation engine 118 again finds a match for each expression of the operation, albeit with a slightly different transformation. As a result, transformation engine 118 generates a transformed dataflow graph 408 in which the transformed operations are inserted into input query component 403a in place of operation 403b. As illustrated by the examples in environments 400 and 450, the techniques described herein allow the same dataflow graph to be optimized and executed on different databases using different languages ​​without requiring modifications to the underlying dataflow graph source code.

[0079] 5, an environment 500 illustrates an example of extensibility of a transformation file 502 according to the techniques described herein. In this example, the transformation engine 118 receives a dataflow graph 302 that is the same as the dataflow graph 302 described in FIGS. 3A-3T (after input query components 304a, 304b have been added). Recall that the transformation engine 118 was unable to transform or push down the “Expression B(In)” specified in component 303d because the transformation file did not indicate that such operations were within the capabilities of the “Dialect 2” language used by the “DB2” database (e.g., database 110b). As a result, the transformation engine was unable to transform or push down any of the operations represented by component 303e, despite the transformation file indicating that these operations are supported, because these operations depend on data output from component 303d.

[0080] In environment 500, transformation file 502 has been extended to include support for "representation B(In)" (shown in bold in FIG. 5). As a result, transformation engine 118 can now transform the same dataflow graph 302 into a highly optimized and efficient transformed dataflow graph 504 in which all of the operations specified in components 303d and 303e are pushed down (note that the operation represented by component 303f cannot be pushed down in this example because it depends on data from two different databases). Thus, transformation file 502 (in addition to other techniques described herein) provides great flexibility by allowing users to extend the capabilities of the system to any particular database they encounter (or update existing databases). Because the system picks up the transformation file at run time, the transformation is not encoded in binary, allowing the same source code (e.g., for a computer program such as a dataflow graph) to run efficiently on any database.

[0081] 6A-6C, a dataflow graph 600 is shown at various stages of optimization. In FIG. 6A, the dataflow graph 600 is not fully optimized, resulting in substantial inefficiencies and strain on computational resources. In FIG. 6B, the dataflow graph 600 is optimized to be faster, more efficient, and use fewer computational resources. The dataflow graph 600 is optimized using techniques such as removal of dead or redundant components, early filtering, or record refinement, or a combination thereof, among others. Additional optimization of the dataflow graph may include one or more of the optimizations described in U.S. Patent Application No. 15 / 993,284, entitled "Systems and Methods for Dataflow Graph Optimization," the entire contents of which are incorporated herein by reference. Finally, in FIG. 6C, the dataflow graph 600 is further optimized using the pushdown techniques described herein. Because operations previously performed locally (e.g., by the execution environment) have been pushed down to one or more databases, the dataflow graph 600 shown in FIG. 6C is faster, more efficient, and uses fewer computational resources than the dataflow graphs shown in FIGS. 6A and 6B.

[0082] 7 shows a flowchart of an example process 700 for transforming operations of a computer program for execution in a database. Process 700 can be implemented by one or more of the systems (e.g., systems 100, 200, 250, 300) and components (e.g., execution environment 104 or components thereof, such as transformation engine 118) described herein, including one or more computing systems configured to implement the techniques described with reference to FIGS.

[0083] The operations of process 700 include executing 702, by a data processing system, a program (e.g., a computer program) that specifies operations, the program being written in a language (e.g., a programming language). For example, the program is a computer program such as a data flow graph (e.g., data flow graphs 112, 202, 302, etc.) having nodes representing operations connected by links representing the flow of data between the operations. In one example, the program is written in a data manipulation language (DML).

[0084] The operations of the process 700 further include accessing (704) a translation file including one or more instructions for converting a language of the program to a language of the database. The translation file (e.g., translation file 120) specifies one or more operations in a language of the program supported by the database and further specifies a semantic meaning of the one or more supported operations in the language of the database. In one example, the database is a relational database and the language of the database is Structured Query Language (SQL) or an SQL dialect. In one example, the indication of the language of the database is received from the program. In one example, the translation file includes one or more instructions for converting the language of the program to a language of multiple databases including the language of the database. In one example, the semantic meaning describes at least one of the data types, operator meanings, function meanings, or intended results of the operations of the one or more supported operations in the language of the database. In one example, the translation file is extensible, accessed at run time, and not encoded within the program (e.g., not encoded in the binary or machine code for the program, but instead accessible, for example, as a separate file in storage).

[0085] Based on the translation file, one or more of the program's operations that are not supported by the database are determined (706). The one or more of the operations that are not supported by the database are processed by the program (708). In one example, operations 706 and 708 are optional. In one example, the program does not process one or more of the program's operations that are supported by the database (e.g., the program processes only one or more of the operations that are not supported by the database). In one example, the data processing system does not execute the portion of the program that represents one or more of the operations that are supported by the database. In one example, the one or more operations that are not supported by the database are processed by the program after receiving data resulting from the execution of one or more of the operations that are supported by the database, and the data resulting from such execution is used as input data for one or more of the operations that are not supported by the database.

[0086] Using the transformation file, one or more of the program's operations that are supported by the database are determined 710. In one example, determining one or more of the operations that are supported (and / or not supported) by the database includes applying one or more pattern matching rules to one or more operations in the transformation file specified in the language of the program.

[0087] For one or more of the operations supported by the database, the process operations include translating a portion of the program representing one or more of the operations supported by the database into a language of the database based on the translation file, sending the translated portion of the program to the database, and receiving by the program from the database data resulting from execution in the database of the translated portion of the program representing one or more of the operations supported by the database (712). In one example, translating the portion of the program representing one or more of the operations supported by the database includes generating one or more queries to the database in the language of the database. In one example, the queries to the database specify one or more operations (or equivalents of the one or more operations) in the language of the database. In one example, execution of the queries in the database generates data resulting from execution in the database of the translated portion of the program representing one or more of the operations supported by the database. In one example, at least one of the program operations includes a query operation, and the process includes removing one or more operations supported by the database from the program, and inserting one or more queries to the database in the language of the database into the query operation. In one example, one or more additional operations are performed by the program on data received from the database and returned from processing one or more of the operations not supported by the database.

[0088] Implementations of the subject matter and operations described herein, including the development environment 102, the execution environment 104, and components thereof, can be implemented in digital electronic circuitry, including the structures disclosed herein and their structural equivalents, or in computer software, firmware, or hardware, or in combinations of one or more of them. Implementations of the subject matter described herein can be implemented as one or more computer programs (also referred to as data processing programs) (i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by or for controlling the operation of a data processing apparatus). A computer storage medium may be or be included in a computer readable storage device, a computer readable storage substrate, a random access or serial access memory array or device, or a combination of one or more of them. A computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). The subject matter can be implemented in computer program instructions stored on a non-transitory computer storage medium.

[0089] The operations described herein may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0090] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system on a chip, or a plurality or combination of the foregoing. An apparatus may include special purpose logic circuitry (e.g., a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC)). In addition to hardware, an apparatus may also include code that provides an execution environment for the computer program in question (e.g., code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof). The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0091] A computer program (also known as a program, software, software application, script, or code) can be written in any type of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may or may not correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in several cooperating files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer, or on several computers located at one site or distributed across several sites and interconnected by a communication network.

[0092] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry (e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0093] Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer include a processor for performing actions in accordance with the instructions, and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from, transfer data to, or both, one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, although a computer need not have such devices. Furthermore, a computer can be incorporated in another device (e.g., a mobile phone, a Personal Digital Assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash drive)). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0094] Implementations of the subject matter described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the subject matter described herein), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0095] A computing system may include users and servers. Users and servers are generally remote from each other and typically interact through a communications network. The relationship of client and server may arise by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server transmits data (e.g., HTML pages) to a user device (e.g., for the purposes of displaying the data to a user interacting with the user device and receiving user input from a user interacting with the user device). Data generated at the user device (e.g., results of user interaction) can be received from the user device at the server.

[0096] Although this specification includes details of many specific implementations, these should not be interpreted as limitations on the scope of any implementation or what may be claimed, but as descriptions of features specific to the particular implementation. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single embodiment can also be implemented separately in multiple implementations or in any suitable subcombination. Furthermore, although features may be described above as acting in a certain combination and initially claimed as such, one or more features from the claimed combination may, in some cases, be removed from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0097] Similarly, although operations are depicted in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in any sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, it should be understood that the separation of various system components in the above implementations should not be understood as requiring such separation in all implementations, and that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0098] Other implementations are within the scope of the following claims.

Claims

1. A method of executing, by a data processing system, one or more portions of a program that specifies operations, said program being written in a first language, said method comprising: receiving from a device a translation file distinct from the program, the translation file including one or more instructions for translating the first language of the program to a second language of a database, the translation file specifying one or more operations in the first language of the program that are supported by the database and further specifying the one or more supported operations in the second language of the database; determining from the translation file one or more of the operations of the program that are not supported by the database; converting the determined one or more of the operations written in the first language of the program in preparation for execution, the one or more of the operations to be converted being unsupported by the database; determining from the transformation file one or more of the operations of the program that are supported by the database; For said one or more of said operations supported by said database, converting a portion of the program representing the one or more of the operations supported by the database into the second language of the database using the conversion file; transmitting the converted portion of the program from the data processing system to the database; receiving, by the data processing system, from the database, data resulting from execution within the database of the transformed portion of the program representing the one or more of the operations supported by the database; executing, by the data processing system, the one or more of the transformed operations of the program that are not supported by the database in accordance with the received data resulting from execution, within the database, of the transformed portions of the program that represent the one or more of the operations that are supported by the database; A method comprising:

2. The method of claim 1, comprising performing, by the data processing system, one or more additional operations on the data received from the database and on data resulting from execution of the one or more of the converted operations that are not supported by the database.

3. The method of claim 1, wherein the program includes an executable data flow graph having components representing the operations connected by links representing flow of data between the operations.

4. The method of claim 1, wherein converting the portion of the program that represents the one or more of the operations supported by the database includes generating one or more queries to the database in a language of the database to perform the one or more of the operations within the database.

5. At least one of the operations of the program includes a query operation, and the method further comprises: removing the one or more operations supported by the database from the program; and and inserting the one or more queries to the database in the language of the database into the query operation.

6. The method of claim 1, wherein determining one or more of the operations supported by the database includes applying one or more pattern matching rules to the one or more operations in the conversion file specified in the language of the program.

7. The method of claim 1, wherein the translation file describes at least one of a data type, a meaning of an operator, a meaning of a function, or an intended result of an operation of the one or more supported operations in the language of the database.

8. The method of claim 1, wherein the conversion file is not encoded in the program.

9. The method of claim 1, comprising receiving from the program an indication of the language of the database.

10. The method of claim 1, wherein the language of the program includes a data manipulation language and the language of the database includes a structured query language.

11. The method of claim 1, wherein the conversion file includes one or more instructions for converting the language of the program to languages ​​of a plurality of databases, including the language of the database.

12. A system comprising: one or more processors; One or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for executing one or more portions of a program that specifies operations, the program being written in a first language, the operations including: receiving from a device a translation file distinct from the program, the translation file including one or more instructions for translating the first language of the program to a second language of a database, the translation file specifying one or more operations in the first language of the program that are supported by the database and further specifying the one or more supported operations in the second language of the database; determining from the translation file one or more of the operations of the program that are not supported by the database; converting the determined one or more of the operations written in the first language of the program in preparation for execution, the one or more of the operations to be converted being unsupported by the database; determining from the transformation file one or more of the operations of the program that are supported by the database; For said one or more of said operations supported by said database, converting a portion of the program representing the one or more of the operations supported by the database into the second language of the database using the conversion file; sending the converted portion of the program to the database; receiving, from the database, data resulting from execution within the database of the transformed portion of the program representing the one or more of the operations supported by the database; executing the one or more of the transformed operations of the program that are not supported by the database in accordance with the received data resulting from execution, within the database, of the transformed portions of the program that represent the one or more of the operations that are supported by the database; Including, the system.

13. The system of claim 12, wherein the one or more storage devices store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including performing one or more additional operations on the data received from the database and on data resulting from execution of the one or more of the transformed operations not supported by the database.

14. The system of claim 12, wherein converting the portion of the program that represents the one or more of the operations supported by the database includes generating one or more queries to the database in a language of the database.

15. At least one of the operations of the program includes a query operation, the one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to: removing the one or more operations supported by the database from the program; and 15. The system of claim 14, further comprising: causing the query operation to perform an operation including: inserting the one or more queries to the database in the language of the database.

16. The system of claim 12, wherein the translation file describes at least one of a data type, a meaning of an operator, a meaning of a function, or an intended result of an operation of the one or more supported operations in the language of the database.

17. A non-transitory computer-readable storage medium storing instructions, the instructions causing a computing system to execute one or more portions of a program that specifies operations, the program being written in a first language, the instructions causing the computing system to: receiving from a device a translation file distinct from the program, the translation file including one or more instructions for translating the first language of the program into a second language of a database, the translation file specifying one or more operations in the first language of the program that are supported by the database and further specifying the one or more supported operations in the second language of the database; determining from the transformation file one or more of the operations of the program that are not supported by the database; converting the determined one or more of the operations written in the first language of the program in preparation for execution, the determined one or more of the operations to be converted being not supported by the database; determining from the transformation file one or more of the operations of the program that are supported by the database; For said one or more of said operations supported by said database, converting a portion of the program representing the one or more of the operations supported by the database into the second language of the database using the conversion file; causing said database to transmit said transformed portion of said program; receiving from the database data resulting from execution within the database of the transformed portion of the program representing the one or more of the operations supported by the database; A non-transitory computer-readable storage medium that causes one or more of the transformed operations of the program that are not supported by the database to be executed in accordance with the received data resulting from execution, in the database, of the transformed portions of the program that represent the one or more of the operations supported by the database.

18. The non-transitory computer-readable storage medium of claim 17, comprising instructions for the program to cause the computing system to perform one or more additional operations on the data received from the database and on data resulting from the execution of the one or more of the transformed operations that are not supported by the database.

19. A non-transitory computer-readable storage medium as described in claim 17, wherein the program includes an executable data flow graph having components representing the operations connected by links representing flows of data between the operations.

20. The non-transitory computer-readable storage medium of claim 17, wherein the translation file describes at least one of a data type, an operator meaning, a function meaning, or an intended result of an operation of the one or more supported operations in the language of the database.

21. The method of claim 1, wherein the device includes a computing device or a memory or both.

22. The method of claim 1, wherein receiving the conversion file includes accessing the conversion file.

23. The method of claim 1, wherein the conversion file is extensible.

24. The first language is an interpretable language, and converting comprises:

2. The method of claim 1, further comprising interpreting the one or more of the operations of the program written in the first language that are not supported by the database.

Citation Information

Patent Citations

  • Processing of Queries Containing Conjunctive Operations

    JP2017535869A

  • Translating functional graph traversal language to extended structured query language

    US20210034615A1

  • Translation techniques for distributed datasets

    US20220035611A1