A workflow formula pipeline fusion compiling and executing method and system in a data analysis platform

Abstract

This invention relates to the fields of computer software and data analysis technology, and particularly to a method and system for compiling and executing workflow formula pipelines in a data analysis platform. The technical solution includes: the system uses an Abstract Syntax Tree (AST) as the sole truth, covering formulas for multiple scenarios such as workflows, cells, and measures; the execution method equips pure DAG function nodes with `let` bindings and inline fusion, maximizing the generation of pushdown query statements and minimizing script engine switching frequency; function packages register pushdown hints through metadata decorators; the type system is divided into three layers: L0 native, L1 function package exported, and L2 user-defined, supplemented by Trait capability contracts to support cross-layer generic constraints. This invention, based on a unified AST, three-layer types, and Trait constraints, reduces data materialization and context switching through DAG purity partitioning, pipeline inline fusion, and SQL pushdown splitting, combined with metadata function management, thereby improving computational efficiency and balancing multi-scenario adaptability, scalability, and editing flexibility.

Description

Technical Field

[0001] This invention relates to the field of computer software and data analysis technology, and in particular to a method and system for compiling and executing workflow formula pipelines in a data analysis platform. Background Technology

[0002] Data analytics platforms typically allow users to perform data extraction, transformation, loading (ETL), and analysis tasks by building workflows. Each node in a workflow is usually defined by a formula expression (e.g., "node name = formula expression"), and nodes form a directed acyclic graph (DAG) through reference relationships. In existing technologies, workflow engines typically compile and execute each node independently, with intermediate results passed between nodes through temporary tables or memory buffers.

[0003] The existing technology has the following problems: When a workflow contains a large number of consecutive pure computational nodes (such as field renaming, filtering, computed columns, sorting, etc.), each node, when executed individually, will generate multiple data serialization, storage, and deserialization operations. Especially when data is transferred between nodes through temporary database tables or inter-process communication, the I / O and memory overhead is significant, resulting in low execution efficiency. Existing systems often only perform simple predicate pushdown on expressions within a single node, lacking overall optimization across nodes. The computational pipeline formed by multiple nodes is difficult to identify as a whole and convert into an efficient database query statement, forcing computations that could be completed within the database engine to be executed in an external scripting environment, further increasing the costs of context switching and data migration; Workflows often include various types of nodes, such as data source connections (IO-Read), data transformation (Pure), and data writing (IO-Write). Current technologies lack fine-grained classification of node purity, making it impossible to independently identify and optimize pure computational node regions. In systems that support user-defined functions (UDFs), optimization attributes such as function purity, determinism, and whether they can be pushed down to SQL are often maintained through hard coding or additional configuration files. The lack of a metadata declaration mechanism that is tightly bound to the function implementation itself makes it difficult for the optimizer to make automatic decisions.

[0004] To address this issue, we propose a method and system for integrating workflow formula pipelines into a data analysis platform for compilation and execution to solve existing problems. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the background technology by proposing a method and system for compiling and executing workflow formula pipelines in a data analysis platform.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a workflow formula pipeline fusion compilation and execution system in a data analysis platform, comprising: The lexical analysis module is used to decompose the formula text string into a sequence of tokens; The syntax analysis module is used to parse the marked sequence into an abstract syntax tree (AST), which serves as a unique and persistent truth storage for the formula; The type checking module is used to perform type inference on the abstract syntax tree. The type inference relies on a three-layer type system, which includes: the language native base type layer (L0), the function package exported type layer (L1), and the user-defined entity and enumeration type layer (L2). The function registration module includes a function registry, which stores function implementations annotated with metadata decoration. Each registration entry includes a function identifier, namespace, function type, purity flag, deterministic flag, and SQL pushdown capability flag. The function registry also stores the SQL and JavaScript implementations of each function, serving as a semantic bridge layer for the execution engine to make switching decisions between the SQL and JavaScript segments. The execution engine module receives the abstract syntax tree and the directed acyclic graph (DAG) of the workflow, generates an execution plan through a pipeline fusion mechanism, and schedules its execution. The formula language is uniformly applicable to workflow node transformation formulas, table cell formulas, calculated column expressions, measure aggregation formulas, conditional formatting rule Boolean expressions, data validation constraint expressions, and filter predicates. All scenarios share the same syntax, type system, and abstract syntax tree structure.

[0007] A method for integrating workflow formula pipelines into compilation and execution in a data analysis platform includes the following steps: S1: Receive the directed acyclic graph of the workflow. The directed acyclic graph consists of several nodes. Each node is represented as an assignment form of "node name = formula expression". Each node corresponds one-to-one with the LetExpr local binding node in the abstract syntax tree. The reference relationship between nodes is equivalent to the let / in local binding chain. S2: Perform purity partitioning on the directed acyclic graph, dividing the nodes into IO-Read segments, Pure segments, and IO-Write segments, wherein the Pure segment contains deterministic pure function nodes without external side effects; S3: Perform pipeline fusion on nodes within the Pure segment: Starting from the terminal node, backtrack along the reference chain, expand the formulas of intermediate nodes inline, and reduce and fuse the entire Pure segment into a single pipeline expression tree. This expression tree is a single AST structure that can be recursively pushed down. The node reference relationship is equivalent to let / in local binding. S4: Perform pushdown analysis on the merged pipeline expression tree, and divide the pipeline into SQL segments and JavaScript segments according to the SQL pushdown capability of each subtree, with the optimization goal of minimizing the number of SQL and JavaScript execution context switches; S5: The pushdown subtree is handed over to the SQL launcher to generate DuckDB query statements, and the non-pushdown subtree is compiled into JavaScript functions and scheduled for execution in the order of execution segments; S6: Collect the execution results of each segment and pass them to the subsequent IO-Write segment or return them to the caller.

[0008] Furthermore, in the three-layer type system: the L0 language native basic type layer includes null, bool, int, number, decimal, bigint, string, date, time, datetime, instant, zoned_datetime, duration, period, binary, json, and list. <t>, object and table <s>Total primitive types; The L1 function package export type layer is declared by the function package through metadata decorators. Each export type declares its basic primitive type, validation rules, formatting rules, and input cleaning strategy. The L2 user-defined type layer consists of entity classes and enumeration classes defined by the user in the resource manager. The underlying storage type of the entity class is a foreign key reference, and the underlying storage type of the enumeration class is a string. Both are dynamically registered at runtime.

[0009] Furthermore, the system also includes a Trait capability contract module, which is used for: Abstract classes serve as the carriers for Trait declarations. Trait identifiers and documentation are registered through Trait decorators. Capability contracts are defined through abstract methods. The `implements` keyword is used to declare the type's implementation of a particular Trait. Maintain a built-in set of Traits, including Traits with Numeric numerical operation capabilities, Comparable sorting and comparison capabilities, Equatable equality comparison capabilities, Hashable hash grouping capabilities, and Iterable... <t>Iterable container Trait, Serializable Trait, Validatable Trait with validation rules, Formattable Trait with formatting rules, and Aggregatable Trait with aggregateable metrics; In the generic constraints of function signatures, Trait is used as the type delimiter, enabling functions to work across type levels without depending on the specific class inheritance level. During the type inference process, the type checking module queries the Trait capability contract module based on the receiver type and parameter type when the function is called to resolve the most matching function overload, thereby realizing Trait-driven constraint solving and overload resolution. Custom traits can be defined and exported by function package developers, and custom traits use the same declaration mechanism as built-in traits.

[0010] Furthermore, the metadata structure of each entry in the function registry includes: The function identifier field is composed of a namespace and a function name, and is unique within the registry. The function type field can take the value of one of the following: scalar function, transformation function, aggregate function, extraction function, IO function, or constructor function. A list of function signatures, each signature containing the receiver type, parameter list, and return type, where the return type supports dynamic type transformation expressions that depend on the input type; The purity indicator has the values ​​"pure" or "impure". A deterministic flag, with values ​​of either deterministic or non-deterministic; The suggestion field has been optimized to include SQL pushdown capability flags and cache-friendly flags.

[0011] Furthermore, the set of node types of the abstract syntax tree includes: The LiteralExpr literal node carries the literal type and value; The FieldSelectorExpr field selector node uses square bracket syntax to reference field values ​​in the context of the current record; QualifiedFieldExpr qualifies a field node and references a named node or a column value of a table in the form of "node name.field name"; BinaryExpr is a binary expression node that covers arithmetic, comparison, logical, and null concatenation operations. The CallExpr function call node supports both positional parameter lists and named parameter dictionaries, and carries an optional receiver node to support chained calls; LetExpr locally bound nodes contain multiple bound items and bound bodies, and are semantically equivalent to node reference chains in a workflow DAG; IfExpr conditional expression nodes support multi-level branches without deep nesting; ForEachExpr iterates through the nodes and executes the binding body for each element in the container; ArrowFuncExpr is an anonymous function node used for passing parameters to higher-order functions.

[0012] Furthermore, the purity partitioning in S2 determines the node type according to the following rules: When a node's formula calls a data source connection function, file read function, or HTTP request function, it is identified as an IO-Read segment node; When a node's formula calls the workflow save function, file write function, or push function, it is determined to be an IO-Write segment node; A node is classified as a Pure segment node when its formula contains only function calls with a purity flag of "pure" and a deterministic flag of "deterministic", and does not directly reference external I / O resources. Pure segment nodes allow contiguous existence across IO-Read and IO-Write segments, forming a computational region that can be integrated as a whole.

[0013] Furthermore, in S3, when a node within the Pure segment is referenced by multiple downstream nodes, the method employs the following fan-out processing strategy: When the formula of the fan-out node is a simple table reference or an operation with low computational cost, an inline copy strategy is executed to expand the node expression in each downstream pipeline. When the formula of the fan-out node contains large join operations or computationally expensive transformations, the CTE materialization strategy is executed, and the calculation result of the node is defined in the generated SQL query with the WITH clause. Each downstream pipeline references the CTE name. When the calculation result of the fan-out node needs to be processed by the JavaScript segment, the node result is materialized into a DuckDB temporary table, and the downstream pipeline reads data from the temporary table; The selection of the fan-out processing strategy is further based on cost evaluation: when the computation cost of the fan-out node exceeds a preset threshold, the CTE materialization or temporary table strategy is preferred; otherwise, the inline replication strategy is adopted.

[0014] Furthermore, the pushdown analysis in S4 determines the SQL pushdown capability of the subtree according to the following rules: A subtree is considered a pushdown subtree when all function nodes in the subtree have the SQL pushdown capability flag set to pushdown and the row-level expression does not contain a custom Trait method or a constructor for a user-defined L1 or L2 class. When there are non-pushdown function nodes in the subtree, the non-pushdown nodes are used as switching points, and the pipeline is divided into several continuous SQL segments and JavaScript segments with the goal of minimizing the number of switching. The partitioning result satisfies the following: the number of switching operations depends on the number of non-pushdown operations rather than the total number of nodes, ensuring that the switching overhead does not increase when a large number of pushdownable nodes exist.

[0015] Furthermore, the type checking module also maintains schema state information, which includes: Given the structural state, represent a table. <s>All column names and column types have been fully determined; Partially known states, representing a table <s>The column names and types of some columns are already determined, while the remaining columns are inferred during execution. This is suitable for scenarios involving dynamic JSON responses or API return values. Unknown structural state, representing a table <s>The structure of a type cannot be statically inferred before execution; The formula editor and previewer allow for the presence of partially known states without interrupting the editing process, and do not require the schema to be fully determined during editing.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: By identifying continuous Pure segments through purity partitioning and inlining the entire Pure segment into a single pipeline expression tree, the independent materialization and read / write operations of intermediate node results are eliminated. In typical scenarios, the execution of 30 consecutive pushdown nodes can be merged from multiple independent queries into a single DuckDB query, significantly reducing data serialization and I / O operations. Based on the SQL pushdown capability flag in the function registry, a global pushdown analysis is performed on the fused pipeline. The pipeline is divided into SQL segment and JavaScript segment with the goal of minimizing the number of switching operations. All pushdownable pure computational operations are generated by the SQL emitter to generate DuckDB query statements. Only at unavoidable custom functions or IO operations is the switch to JavaScript execution. The overall number of switching operations is equal to the number of non-pushdownable operations rather than the total number of nodes, which greatly improves the execution efficiency. For fan-out nodes within the Pure segment that are referenced by multiple downstream nodes, the system automatically selects inline replication, CTE materialization, or temporary table strategy based on computational cost, avoiding redundant computation or repeated materialization and balancing execution time and memory usage. The three-tier type system, combined with Trait capability contracts, enables function signatures to carry generic constraints. The type checking module can use this to infer whether an expression is pushdown. Meanwhile, the function registry centrally manages the purity, determinism, and SQL pushdown capability flags of functions through metadata decoration, providing accurate decision-making basis for the pushdown parser. The type checking module maintains three states of the table structure: known, partially known, and unknown. This allows the editor to continue working even when the schema is incomplete. The execution engine dynamically adjusts the pushdown strategy based on the schema state, balancing flexibility and performance. Function packages declare optimized properties through metadata decorators, which are tightly coupled with function implementations. New functions or custom traits can be registered without modifying the core system. Pipeline fusion and pushdown analysis logic are independent of specific function implementations, and newly added pure functions automatically gain fusion optimization capabilities. Attached Figure Description

[0017] Figure 1 This is a diagram showing the overall architecture of the formula language processing system of the present invention; Figure 2 This is a schematic diagram of the three-layer structure of the formula of the present invention; Figure 3 This is a flowchart of the pipeline fusion execution method of the present invention; Figure 4 This is a schematic diagram of the workflow DAG inline fusion of the present invention; Figure 5 This is a diagram showing the relationship between the three-layer type system of this invention and the Trait capability contract; Figure 6 This is a diagram of the function registry metadata structure of the present invention.

[0018] Figure label: 101. Lexical Analysis Module; 102. Syntax Analysis Module; 103. Abstract Syntax Tree (AST) Storage; 104. Type Checking Module; 105. Function Registration Module / Function Registry; 106. Execution Engine Module; 107. DAG Analyzer; 108. Pipeline Merger; 109. Pushdown Analyzer; 110. SQL Emitter; 111. JavaScript Compiler; 112. Execution Scheduler; 201. L0 Primitive Type Layer; 202. L1 Function Package Export Type Layer; 203. L2 User-Defined Type Layer; 204. Trait Capability Contract Module; 301. IO-Read Section; 302. Pure Section; 303. IO-Write Section; 304. DuckDB Execution Environment; 305. V8 / Node.js Execution Environment. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1: Architecture of a Unified Equation Language Processing System This embodiment provides a workflow formula pipeline fusion compilation and execution system in a data analysis platform, the overall architecture of which is as follows: Figure 1 As shown.

[0021] The system includes: a lexical analysis module 101, a syntax analysis module 102, an abstract syntax tree (AST) storage 103, a type checking module 104, a function registration module / function registry 105, and an execution engine module 106.

[0022] Lexical analysis module 101 scans the formula text string into a sequence of tokens. The token types include: integer literals, floating-point literals, string literals, Boolean literals, null keywords, identifiers, field selectors enclosed in square brackets ([field name]), parameter references prefixed with @, operators, and punctuation marks.

[0023] The field selector [field name] is the core reference form of the formula language in this system. Its semantics are automatically inferred by the type checking module 104 based on the parameter role metadata of the function signature. In different calling contexts, it represents: the field value of the current row in the row-level filtering predicate, the field value of the current row in the derived column calculation, and a reference to all values ​​of the column in the aggregate expression.

[0024] The parsing module 102 parses the token sequence into an Abstract Syntax Tree (AST) and stores the AST in AST storage 103, which serves as the sole persistent truth store for the formulas. The surface formula text is only a text view of the AST, generated by the formatting module from the AST normalization, and is not persisted separately.

[0025] The node types in the Abstract Syntax Tree (AST) include: LiteralExpr (carrying literal types and values), FieldSelectorExpr (referring to field values ​​in the current record context using square bracket syntax), QualifiedFieldExpr (referring to named nodes or table column values ​​in the form of "node name.field name"), BinaryExpr (covering arithmetic, comparison, logical, and null concatenation operations), CallExpr (supporting both positional parameter lists and named parameter dictionaries, and carrying optional receiver nodes to support chained calls), LetExpr (containing multiple bind items and bind bodies, semantically equivalent to a chain of node references in a workflow DAG), IfExpr (conditional expression nodes), ForEachExpr (itemization nodes), and ArrowFuncExpr (anonymous function nodes).

[0026] The function call node CallExpr supports both positional parameter lists and named parameter dictionaries. It carries optional receiver nodes to support chained call syntax (e.g., table.filter(...).rename(...).group_by(...)), which are parsed in the function registry 105 by the execution engine module 106 according to the receiver type.

[0027] like Figure 2 As shown, the type checking module 104 performs type inference on the abstract syntax tree. The type inference relies on a three-layer type system, which includes: L0 Primitive Type Layer 201: Includes null, bool, int, number, decimal, bigint, string, date, time, datetime, instant, zoned_datetime, duration, period, binary, json, and list. <t>, object and table <s>Total primitive types. Among them, table <s>As the core analysis type, the schema parameter S describes the column names, column types, and nullability of the table.

[0028] L1 Package Export Type Layer 202: Declared by the package through metadata decorators, each exported type declares its underlying primitive type, validation rules, formatting rules, and input cleansing strategy. For example, the PhoneNumber class in the type package is based on the string primitive type, with additional phone number format validation rules and display formatting rules.

[0029] L2 User-Defined Type Layer 203: Consists of entity classes and enumeration classes defined by the user in the resource manager, which are dynamically registered to the type system at runtime. The underlying storage type of entity classes is a foreign key reference, and the underlying storage type of enumeration classes is a string.

[0030] This system adopts a nullable default type convention. The type symbol T indicates that the value of this type can be null; the type symbol T! indicates strictly non-null. This convention is consistent with the conventions in SQL and data analysis, making the formula type system naturally adaptable to datasets that may have missing values. The type checking module 104 performs nullability checks when function parameters are passed, column references are evaluated, and type conversions are performed according to the convention. A safe promotion direction is defined between different numeric types: int can be implicitly promoted to number, decimal, or bigint, and the reverse conversion must be explicitly called. Implicit conversions between number and decimal are prohibited to prevent accidental loss of precision.

[0031] The type checking module 104 also maintains schema state information, including: known structure state (table). <s>The formula editor and previewer allow for partially known states (where all column names and types are fully determined, and the remaining columns are inferred during execution, suitable for dynamic JSON responses or API return values) and unknown structural states (where the structure cannot be statically inferred before execution). The editor and previewer do not require the schema to be fully determined during editing.

[0032] like Figure 5 As shown in the diagram of the relationship between the three-tier type system and the Trait capability contract, the L0 native type layer 201, the L1 function package exported type layer 202, and the L2 user-defined type layer 203 can each declare and implement the built-in or custom Traits defined in the Trait capability contract module 204. When performing type inference, the type checking module 104 simultaneously queries the type definitions of the three-tier type system and the Trait capability contract module 204 to determine whether the function call satisfies the generic constraints and resolves the best-matching overload.

[0033] Trait Capability Contract Module 204 uses abstract classes as the carrier for trait declarations, registers trait identifiers and documentation descriptions through trait decorators (such as @trait), defines capability contracts through abstract methods, and declares the implementation of a trait by a type through the implements keyword.

[0034] The Trait capability contract module 204 maintains a built-in collection of Traits, including: Numeric (numerical computation capability), Comparable (sorting and comparison capability), Equatable (equality comparison capability), Hashable (hash grouping capability), and Iterable. <t>Traits (iterative containers), Serializable, Validatable (carrying validation rules), Formattable (carrying formatting rules), and Aggregatable (aggregatable metrics) are used as type delimiters in the generic constraints of function signatures, enabling functions to work across type levels without depending on the specific class inheritance hierarchy.

[0035] During type inference, the type checking module 104 queries the Trait capability contract module 204 based on the receiver type and parameter type at the time of function call to parse the most matching function overload, thus realizing Trait-driven constraint solving and overload parsing. It also supports custom Traits defined and exported by function package developers, and custom Traits use the same declaration mechanism as built-in Traits.

[0036] The function registration module / function registry 105 stores function implementations annotated with metadata decorators. Metadata decorators support three calling methods: direct invocation as a TypeScript function, attaching metadata properties to the function object in a non-enumerable and non-writable manner; invocation using the TC39 standard class method decorator, compatible with the TypeScript decorator proposal; and invocation using the legacy property descriptor decorator, backward compatible with older TypeScript compilers. These three calling methods produce semantically equivalent registration entries, uniformly identified by the function registration module.

[0037] The function registration module / function registry 105 stores function implementations annotated using metadata decoration. For example... Figure 6 As shown, function package developers call the metadata decorator func(metadata) to attach the _funcMetadata attribute to the function implementation, annotating metadata such as function identifier, namespace, function type, receiver type, parameter list, return type, purity, determinism, SQL pushdown capability, and cache friendliness.

[0038] Registry 105 maintains the following registry entry metadata structure: The function identifier field (consisting of a namespace and a function name, and is unique within the registry) Namespace Function type field (values ​​can be one of scalar function, transformation function, aggregate function, extraction function, I / O function, or constructor function). A list of function signatures (each signature includes the receiver type, parameter list, and return type; the return type supports dynamic type transformation expressions that depend on the input type). Purity markings (pure or impure) Deterministic or non-deterministic indicators Optimize the suggestion fields (including SQL pushdown capability flags and cache-friendly flags). The index structure of registry 105 includes: The items mapping table uses function identifiers as keys to point to the corresponding registered items, supporting O(1) exact lookup; Methods reverse mapping table: with function name as key, pointing to a list of registered items with the same name, used in chained call scenarios to parse the most matching overload based on the receiver type.

[0039] The registry also performs duplicate identifier detection: when the same function identifier is registered repeatedly, an explicit error is thrown to prevent identifier conflicts; it supports namespace recursive loading: when traversing function packages to export objects, if an object type attribute that does not directly carry metadata is encountered, its sub-attributes are recursively checked to support the unified export of multiple functions after organizing them into namespace objects.

[0040] The function registry stores both the SQL and JavaScript implementations of each function, serving as a semantic bridge layer for the execution engine to make switching decisions between the SQL and JavaScript segments. At runtime: The type checking module 104 queries the registry based on the function call node to verify the parameter type and deduce the return type; The execution engine module 106 matches the registered entries in the reverse mapping table of methods according to the receiver type, obtains the specific function implementation, and calls it. The pushdown analyzer 109 reads the SQL pushdown capability flag in the optimization hint field and decides whether to push the subtree down to the SQL transmitter 110; SQL transmitter 110 uses SQL implementation fragments from the registration items to generate DuckDB query statements; JavaScript compiler 111 uses the JavaScript implementation in the registry to generate fallback code.

[0041] The formula language is uniformly applicable to workflow node transformation formulas, table cell formulas, calculated column expressions, measure aggregation formulas, conditional formatting rule Boolean expressions, data validation constraint expressions, and filter predicates. All scenarios share the same syntax, type system, and abstract syntax tree structure.

[0042] Example 2: Pipeline Fusion Execution Method like Figure 3 As shown, this embodiment provides a workflow formula pipeline fusion compilation and execution method in a data analysis platform. This method is completed collaboratively by the execution engine module 106, DAG analyzer 107, pipeline fusion unit 108, pushdown analyzer 109, SQL launcher 110, JavaScript compiler 111, and execution scheduler 112. The following description uses a specific workflow DAG as an example.

[0043] Assume the workflow DAG contains the following nodes: Data source connection = mysql.connect(host="db.example.com", db="sales") Query = data source connection.sql("SELECT * FROM orders WHERE year=2025") Rename field = query.rename({"order number":"order_id","customer":"customer"}) Filter = rename field.filter([order_id] > 1000) Calculate tax amount = filter.derive("tax", [amount] * 0.06) Sort = Calculate tax amount.sort("amount", "desc") Output = workflow.save(sort) Step S1: Receive the directed acyclic graph of the workflow The execution engine module 106 receives the directed acyclic graph (DAG) of the above workflow. Each node is represented as an assignment form of "node name = formula expression", where each node corresponds one-to-one with a LetExpr local binding node in the abstract syntax tree, and the reference relationship between nodes is equivalent to a let / in local binding chain.

[0044] Step S2: Purity Zoning The DAG analyzer 107 performs purity partitioning on the directed acyclic graph, dividing the nodes into IO-Read segments, Pure segments, and IO-Write segments.

[0045] When a node's formula calls a data source connection function, file read function, or HTTP request function, it is identified as an IO-Read segment 301 node. In this example, the data source connection and query call IO functions, which must be executed in JavaScript, and the output data is written to DuckDB, belonging to the IO-Read segment.

[0046] When a node's formula contains only function calls with a purity flag of "pure" and a deterministic flag of "deterministic", and does not directly reference external I / O resources, it is determined to be a Pure segment 302 node. In this example, renaming fields, filtering, calculating tax amounts, and sorting are all deterministic pure function nodes, belonging to the Pure segment.

[0047] When a node's formula calls the workflow save function, file write function, or push function, it is identified as an IO-Write segment 303 node. In this example, the output calls the workflow save function, which belongs to the IO-Write segment.

[0048] Pure segment nodes allow contiguous existence across IO-Read and IO-Write segments, forming a computational region that can be merged as a whole. For example... Figure 3 As shown, Pure segment 302 is located between IO-Read segment 301 and IO-Write segment 303.

[0049] Step S3: Pipeline Fusion Pipeline merger 108 performs pipeline fusion on nodes within the Pure segment: starting from the terminal node (sorted), it backtracks along the reference chain, inlines and expands the formulas of intermediate nodes, and reduces and merges the entire Pure segment into a single pipeline expression tree. This expression tree is a single AST structure that can be recursively pushed down, and the node reference relationship is equivalent to let / in local binding.

[0050] like Figure 4 As shown, the merged single pipeline expression tree is as follows: Query .rename({"order number":"order_id","customer":"customer"}) .filter([order_id] > 1000) .derive("tax", [amount] * 0.06) .sort("amount", "desc") This merging process is equivalent to expanding the four let bindings as a whole, eliminating the four independent queries or function calls corresponding to the four independent nodes.

[0051] Fan-out Node Processing: When a node within a Pure segment is referenced by multiple downstream nodes, the pipeline merging unit 108 employs the following fan-out processing strategy: If the formula of the fan-out node is a simple table reference or a low-cost operation, an inline replication strategy is executed, expanding the node expression in each downstream pipeline; if the formula of the fan-out node contains large join operations or high-cost transformations, a CTE materialization strategy is executed, defining the node's calculation result with a WITH clause in the generated SQL query, and each downstream pipeline references the CTE name; if the calculation result of the fan-out node requires processing by a JavaScript segment, the node result is materialized into a DuckDB temporary table, and the downstream pipeline reads data from the temporary table. The selection of the fan-out processing strategy is further based on cost evaluation: when the computational cost of the fan-out node exceeds a preset threshold, the CTE materialization or temporary table strategy is preferentially adopted; otherwise, the inline replication strategy is adopted.

[0052] Step S4: Pushdown Analysis The pushdown analyzer 109 performs pushdown analysis on the merged pipeline expression tree, and divides the pipeline into SQL segments and JavaScript segments according to the SQL pushdown capability of each subtree, with the optimization goal of minimizing the number of SQL and JavaScript execution context switches.

[0053] The pushdown analysis criteria are as follows: A subtree is considered pushdown-capable if all function nodes in the subtree have the SQL pushdown capability flag set to "pushdownable," and the row-level expression does not contain custom Trait methods or constructors for user-defined L1 or L2 classes. If a subtree contains non-pushdown function nodes, these nodes are used as switching points, and the pipeline is divided into several consecutive SQL and JavaScript segments with the minimum number of switches required. The segmentation results satisfy the following: the number of switches depends on the number of non-pushdown operations, not the total number of nodes, ensuring that the presence of a large number of pushdownable nodes does not increase switching overhead.

[0054] In this example, rename, filter, derive (row-level arithmetic expression), and sort are all pushdown operations, so the entire pipeline belongs to a single pushdown subtree and no switching is required.

[0055] Step S5: Code Generation and Scheduling The pushdown subtree is passed to the SQL transmitter 110 to generate DuckDB query statements, which are then executed by the DuckDB execution environment 304.

[0056] The non-pulldown subtree is compiled into a JavaScript function, the code is generated by the JavaScript compiler 111, and then executed by the V8 / Node.js execution environment 305.

[0057] For this example, SQL transmitter 110 generates the following DuckDB query statement: SELECT "order_id", "customer", "amount" * 0.06 AS "tax" FROM source_table WHERE "order_id" > 1000 ORDER BY "amount" DESC The execution scheduler 112 schedules execution according to the order of execution segments. In this example, there is only one SQL segment and no JavaScript segment switching. During the execution process, there is no data exchange between DuckDB and V8.

[0058] Step S6: Result Collection and Transmission The execution scheduler 112 collects the execution results of each segment and passes them to the subsequent IO-Write segment or returns them to the caller. In this example, the query results of the SQL segment are passed to the IO-Write segment (output node), and the workflow save function writes them to the target location.

[0059] When there are non-pushdown operations in the pipeline (such as calling a custom prediction model function), the system splits the pipeline with the fewest switching operations: the pushdownable parts before and after the non-pushdown operation are generated as independent SQL segments, and the non-pushdown operation is generated as a JavaScript segment. The total number of switching operations is equal to the number of non-pushdown operations, not the total number of nodes. The execution scheduler 112 calls the DuckDB execution environment 304 and the V8 / Node.js execution environment 305 in the order of "SQL segment → JavaScript segment → SQL segment".

[0060] Example 3: Function Package Metadata Decoration and Loading This embodiment describes a function package metadata decoration and loading mechanism, which provides the foundation for function registration, type inference, and SQL pushdown capability flags for the pipeline fusion execution method described in Embodiment 2.

[0061] Function package developers call the `func()` decorator factory, passing in a metadata object containing function identifiers, namespaces, function types, signature lists, purity, and optimization hints, to obtain a decorator function. The decorator appends the metadata to the function object as non-enumerable properties, producing the decorated function.

[0062] As described in Example 1, this decorator supports three forms: direct invocation, class method decorator, and legacy descriptor decorator. Developers can choose any form according to the project configuration, and all forms produce semantically equivalent registration items.

[0063] Taking the `filter` function in the `table` package as an example, its registration metadata includes: function identifier `table.filter`, namespace `table`, function type `transform`, and receiver type `table`. <any>, signature description receiver table <s>Add predicate parameters and return type table <s>.

[0064] The function registration module 105 calls the `load()` method to receive the exported object of the function package and iterates through all exported attributes: exported values ​​that directly carry metadata attributes are registered directly; for object type attributes that do not carry metadata, their sub-attributes are recursively checked to support the unified export of functions after they are organized into namespace objects. After loading, the registry maintains the main mapping from function identifiers to registration items (items mapping table) and the reverse mapping from function names to the list of registration items (methods reverse mapping table), and performs duplicate identifier detection to prevent function identifier conflicts. This loading protocol uniformly processes ES module exported objects, ensuring that namespace structures of any nesting depth can be correctly identified and registered. The cache-friendly flag is used to indicate whether the function calculation results can be cached and reused, providing an optimization basis for the execution engine's cache scheduling.

[0065] When evaluating the CallExpr node, the execution engine module 106 queries the registry with the called function name (callee) of the function call node and the existence of a receiver node as parameters: if no receiver is found, an exact match is performed; if a receiver is found, the registry with a receiver type declaration is searched in the list of registry entries with the same name. The function implementation in the found registry is directly invoked with the expanded parameter list. The pushdown parser 109 can read the SQL pushdown capability flag from the registry to determine whether to push down the subtree containing the function call to the SQL launcher 110.

[0066] The above specific embodiments are merely several further embodiments of the present invention. Based on the technical solutions of the present invention and the relevant teachings of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.

[0067] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.< / s> < / s> < / any> < / t> < / s> < / s> < / s> < / t> < / s> < / s> < / s> < / t> < / s> < / t>

Claims

1. A workflow formula pipeline integrated compilation and execution system in a data analysis platform, characterized in that, include: The lexical analysis module is used to decompose the formula text string into a sequence of tokens; The syntax analysis module is used to parse the marked sequence into an abstract syntax tree (AST), which serves as a unique and persistent truth storage for the formula; The type checking module is used to perform type inference on the abstract syntax tree. The type inference relies on a three-layer type system, which includes: the language native base type layer (L0), the function package exported type layer (L1), and the user-defined entity and enumeration type layer (L2). The function registration module includes a function registry, which stores function implementations annotated with metadata decoration. Each registration entry includes a function identifier, namespace, function type, purity flag, deterministic flag, and SQL pushdown capability flag. The function registry also stores the SQL and JavaScript implementations of each function, serving as a semantic bridge layer for the execution engine to make switching decisions between the SQL and JavaScript segments. The execution engine module receives the abstract syntax tree and the directed acyclic graph (DAG) of the workflow, generates an execution plan through a pipeline fusion mechanism, and schedules its execution. The formula language is uniformly applicable to workflow node transformation formulas, table cell formulas, calculated column expressions, measure aggregation formulas, conditional formatting rule Boolean expressions, data validation constraint expressions, and filter predicates. All scenarios share the same syntax, type system, and abstract syntax tree structure.

2. A method for integrating workflow formulas and pipelines for compilation and execution in a data analysis platform, characterized in that, Includes the following steps: S1: Receive the directed acyclic graph of the workflow. The directed acyclic graph consists of several nodes. Each node is represented as an assignment form of "node name = formula expression". Each node corresponds one-to-one with the LetExpr local binding node in the abstract syntax tree. The reference relationship between nodes is equivalent to the let / in local binding chain. S2: Perform purity partitioning on the directed acyclic graph, dividing the nodes into IO-Read segments, Pure segments, and IO-Write segments, wherein the Pure segment contains deterministic pure function nodes without external side effects; S3: Perform pipeline fusion on nodes within the Pure segment: Starting from the terminal node, backtrack along the reference chain, expand the formulas of intermediate nodes inline, and reduce and fuse the entire Pure segment into a single pipeline expression tree. This expression tree is a single AST structure that can be recursively pushed down. The node reference relationship is equivalent to let / in local binding. S4: Perform pushdown analysis on the merged pipeline expression tree, and divide the pipeline into SQL segments and JavaScript segments according to the SQL pushdown capability of each subtree, with the optimization goal of minimizing the number of SQL and JavaScript execution context switches; S5: The pushdown subtree is handed over to the SQL launcher to generate DuckDB query statements, and the non-pushdown subtree is compiled into JavaScript functions and scheduled for execution in the order of execution segments; S6: Collect the execution results of each segment and pass them to the subsequent IO-Write segment or return them to the caller.

3. The workflow formula pipeline fusion compilation and execution system in a data analysis platform according to claim 1, characterized in that: The three-layer type system includes the L0 language native basic type layer, which contains null, bool, int, number, decimal, bigint, string, date, time, datetime, instant, zoned_datetime, duration, period, binary, json, and list. <t>, object and table <s> Total primitive types;< / s> < / t> <s> The L1 function package export type layer is declared by the function package through metadata decorators. Each export type declares its basic primitive type, validation rules, formatting rules, and input cleaning strategy. The L2 user-defined type layer consists of entity classes and enumeration classes defined by the user in the resource manager. The underlying storage type of the entity class is a foreign key reference, and the underlying storage type of the enumeration class is a string. Both are dynamically registered at runtime.

4. The workflow formula pipeline fusion compilation and execution system in a data analysis platform according to claim 1, characterized in that: The system also includes a Trait capability contract module, which is used for: Abstract classes serve as the carriers for Trait declarations. Trait identifiers and documentation are registered through Trait decorators. Capability contracts are defined through abstract methods. The `implements` keyword is used to declare the type's implementation of a particular Trait. Maintain a built-in set of Traits, including Traits with Numeric numerical operation capabilities, Comparable sorting and comparison capabilities, Equatable equality comparison capabilities, Hashable hash grouping capabilities, and Iterable... <t> Iterable container Trait, Serializable Trait, Validatable Trait with validation rules, Formattable Trait with formatting rules, and Aggregatable Trait with aggregateable metrics;< / t> In the generic constraints of function signatures, Trait is used as the type delimiter, enabling functions to work across type levels without depending on the specific class inheritance level. During the type inference process, the type checking module queries the Trait capability contract module based on the receiver type and parameter type when the function is called to resolve the most matching function overload, thereby realizing Trait-driven constraint solving and overload resolution. Custom traits can be defined and exported by function package developers, and custom traits use the same declaration mechanism as built-in traits.

5. The workflow formula pipeline fusion compilation and execution system in a data analysis platform according to claim 1, characterized in that: The metadata structure of each entry in the function registry includes: The function identifier field is composed of a namespace and a function name, and is unique within the registry. The function type field can take the value of one of the following: scalar function, transformation function, aggregate function, extraction function, IO function, or constructor function. A list of function signatures, each signature containing the receiver type, parameter list, and return type, where the return type supports dynamic type transformation expressions that depend on the input type; Purity indicator, with values ​​of pure or impure; A deterministic flag, with values ​​of either deterministic or non-deterministic; The suggestion field has been optimized to include SQL pushdown capability flags and cache-friendly flags.

6. The workflow formula pipeline fusion compilation and execution system in a data analysis platform according to claim 1, characterized in that: The set of node types of the abstract syntax tree includes: The LiteralExpr literal node carries the literal type and value; The FieldSelectorExpr field selector node uses square bracket syntax to reference field values ​​in the context of the current record; QualifiedFieldExpr qualifies a field node and references a named node or a column value of a table in the form of "node name.field name"; BinaryExpr is a binary expression node that covers arithmetic, comparison, logical, and null concatenation operations. The CallExpr function call node supports both positional parameter lists and named parameter dictionaries, and carries an optional receiver node to support chained calls; LetExpr locally bound nodes contain multiple bound items and bound bodies, and are semantically equivalent to node reference chains in a workflow DAG; IfExpr conditional expression nodes support multi-level branches without deep nesting; ForEachExpr iterates through the nodes and executes the binding body for each element in the container; ArrowFuncExpr is an anonymous function node used for passing parameters to higher-order functions.

7. The workflow formula pipeline fusion compilation and execution method in a data analysis platform according to claim 2, characterized in that: The purity partitioning in S2 determines the node type according to the following rules: When a node's formula calls a data source connection function, file read function, or HTTP request function, it is identified as an IO-Read segment node; When a node's formula calls the workflow save function, file write function, or push function, it is determined to be an IO-Write segment node; A node is classified as a Pure segment node when its formula contains only function calls with a purity flag of "pure" and a deterministic flag of "deterministic", and does not directly reference external I / O resources. Pure segment nodes allow contiguous existence across IO-Read and IO-Write segments, forming a computational region that can be integrated as a whole.

8. The workflow formula pipeline fusion compilation and execution method in a data analysis platform according to claim 2, characterized in that: In step S3, when a node within a Pure segment is referenced by multiple downstream nodes, the method employs the following fan-out processing strategy: When the formula of the fan-out node is a simple table reference or an operation with low computational cost, an inline copy strategy is executed to expand the node expression in each downstream pipeline. When the formula of the fan-out node contains large join operations or computationally expensive transformations, the CTE materialization strategy is executed, and the calculation result of the node is defined in the generated SQL query with a WITH clause. Each downstream pipeline references the CTE name. When the calculation result of the fan-out node needs to be processed by the JavaScript segment, the node result is materialized into a DuckDB temporary table, and the downstream pipeline reads data from the temporary table; The selection of the fan-out processing strategy is further based on cost evaluation: when the computation cost of the fan-out node exceeds a preset threshold, the CTE materialization or temporary table strategy is preferred; otherwise, the inline replication strategy is adopted.

9. The workflow formula pipeline fusion compilation and execution method in a data analysis platform according to claim 2, characterized in that: The pushdown analysis in S4 determines the SQL pushdown capability of the subtree according to the following rules: A subtree is considered a pushdown subtree when all function nodes in the subtree have the SQL pushdown capability flag set to pushdown and the row-level expression does not contain a custom Trait method or a constructor for a user-defined L1 or L2 class. When there are non-pushdown function nodes in the subtree, the non-pushdown nodes are used as switching points, and the pipeline is divided into several continuous SQL segments and JavaScript segments with the goal of minimizing the number of switching. The partitioning result satisfies the following: the number of switching operations depends on the number of non-pushdown operations rather than the total number of nodes, ensuring that the switching overhead does not increase when a large number of pushdownable nodes exist.

10. The workflow formula pipeline fusion compilation and execution system in a data analysis platform according to claim 1, characterized in that: The type checking module also maintains schema status information, which includes: Given the structural state, represent a table. <s> All column names and column types have been fully determined;< / s> <s> Partially known states, representing a table <s>The column names and types of some columns are already determined, while the remaining columns are inferred during execution. This is suitable for scenarios involving dynamic JSON responses or API return values.< / s> <s> Unknown structural state, representing a table <s> The structure of a type cannot be statically inferred before execution;< / s> <s> The formula editor and previewer allow for the presence of partially known states without interrupting the editing process, and do not require the schema to be fully determined during editing. < / s> < / s> < / s> < / s>

Images