ETL workflow arrangement and execution method and system based on natural language driving

Abstract

The invention discloses an ETL workflow arrangement and execution method and system based on natural language driving, and particularly relates to the crossing field of data processing and artificial intelligence. A large model fusing domain knowledge is accurately analyzed into a structured ETL intention triple, and a data source, a table and field metadata are automatically matched based on a preset business term mapping library; then, the system intelligently arranges and generates a standardized ETL node sequence and a process configuration list according to the intention and the data source information; a plurality of visual correction nodes are embedded in the automatically generated process, and a user is allowed to carry out graphical adjustment on key configurations such as a data source, an association rule and a conversion formula, so that the process optimization of man-machine cooperation is realized; and finally, based on the corrected configuration list, the system completes an ETL process through analysis, scheduling and execution of an engine, and outputs a visual report and a multi-format file or stores the visual report and the multi-format file to a specified target.

Description

Technical Field

[0001] This invention relates to the field of data processing and artificial intelligence, and more specifically, to a method and system for orchestrating and executing ETL workflows based on natural language. Background Technology

[0002] In enterprise data governance and analytics, ETL is a crucial link connecting raw data with data applications. Traditional ETL tools (such as Kettle and Informatica) typically require users to have a technical background and understand concepts such as data pipelines, transformation rules, and job dependencies, resulting in a high barrier to entry. While visual drag-and-drop ETL tools have lowered the barrier to entry somewhat in recent years, they still require business personnel to understand abstract concepts such as "nodes" and "connections," meaning there is still a learning curve. This leads to business personnel's data processing needs being highly dependent on technical personnel, resulting in slow response times and high communication costs.

[0003] With the development of Large Language Model (LLM) technology, tools have emerged that convert natural language into data queries (such as SQL). These tools allow users to ask questions in natural language and automatically generate query statements. However, these solutions have significant limitations: First, they typically only support simple, single-step query operations and cannot handle complete ETL processes involving multiple data source associations, complex condition transformations, and multi-step cleansing; second, the generated results (such as SQL) are "black boxes," meaning that when they do not meet expectations, users can only redescribe their requirements and cannot make intuitive adjustments based on the generated process, making the process uncontrollable; third, they lack intelligent connection to data sources, requiring users to manually specify which database or table to query, and cannot automatically match based on business intent.

[0004] Therefore, existing technologies have failed to provide a fully intelligent and controllable ETL solution that allows non-technical business personnel to complete the entire process from requirement description, automatic process construction, visual adjustment to final execution in one stop using natural language they are familiar with. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a method and system for orchestrating and executing ETL workflows based on natural language, in order to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: S1: Receives natural language commands input by the user; S2: Perform intent recognition on the natural language instruction and parse out the ETL intent triplet, which includes the data processing target, operation type and output requirements; S3: Based on the ETL intent triple, the corresponding data source, data table and field metadata are automatically matched by querying the preset mapping relationship between business terms and data sources; S4: Based on the ETL intent triple and the matching data source information, automatically organize and generate a process configuration list containing ETL nodes; S5: Embed a visual correction node in the generated process, wherein the visual correction node is used to receive visual adjustment operations from the user on at least one configuration in the process configuration list; S6: Based on the process configuration list adjusted by the aforementioned visualization correction node, execute the ETL process and output the data processing results.

[0007] Preferably, in step S2, the intent recognition process is implemented based on a large model architecture that integrates a domain knowledge base. The domain knowledge base pre-contains core business terms, common operation expressions, and output format specifications for ETL scenarios. Through transfer learning and fine-tuning training of the large model, it acquires the ability to accurately understand the semantics of non-standard natural language instructions. After receiving natural language instructions input by business personnel, the key information elements in the instructions are first extracted through word segmentation, part-of-speech tagging, and entity recognition technologies. Then, semantic mapping and intent classification are performed based on the domain knowledge base, and finally, the instructions are decomposed into structured ETL intent triples. The analysis of data processing objectives is based on core business requirements and is determined by identifying key elements such as the main subject, filtering conditions, and time range in the instructions.

[0008] Preferably, in step S3, the system pre-constructs a multi-dimensional business terminology-data source mapping library. This mapping library is classified with business scenarios as the core dimension, integrates various internal structured data sources, semi-structured data sources, and authorized external data sources, and establishes a five-level association mapping relationship of "business terminology-data source type-database instance-data table-field metadata". The field metadata includes key attributes such as field name, data type, field definition, data format, and business module to which it belongs. During the matching process, key business terms are first extracted from the elements of the ETL intent triple. These include the main objects in the data processing target (such as "sales revenue," "attendance records," and "customer information"), business entities associated with operation types (such as "return orders" and "user registration data"), and field details in the output requirements (such as "product name," "attendance days," and "consumption amount"). Through word segmentation and deduplication, synonym normalization (such as mapping "sales revenue," "sales amount," and "revenue" to the core term "sales revenue"), and business scenario categorization, a standardized set of terms to be matched is formed. Subsequently, the system calls the retrieval engine of the mapping library, employing a combined retrieval strategy of "exact matching + fuzzy matching + scenario-related matching." Exact matching quickly locates the completely corresponding data source and data table. For terms that do not match precisely, fuzzy matching is performed based on field definitions and business unit relevance (e.g., "customer contact information" can match related fields such as "mobile phone number," "email address," and "contact number" in the "CRM customer relationship database - customer basic information table"). This process is then combined with ETL... The weights of the business scenarios corresponding to the intent (such as prioritizing financial data sources for financial statistics scenarios and prioritizing HR data sources for human resource management scenarios) are adjusted to improve matching accuracy.

[0009] Preferably, in step S4, a standardized ETL node system is constructed based on the operation type sequence in the ETL intent triple and the data source matching result. Then, through the logical association rules and parameter mapping mechanism between nodes, a structured process configuration that can be directly executed is formed. In the ETL node generation stage, a core ETL node library covering all scenarios is pre-defined, including seven basic types: data extraction nodes, data cleaning nodes, data transformation nodes, data association nodes, data aggregation nodes, data filtering nodes, and data output nodes. Each node has a built-in standardized configuration template (including required parameters, optional parameters, and default rules) and supports adaptive adaptation to the matching data source type (structured / semi-structured / external interface). The generation logic is as follows: First, the corresponding core node is mapped according to the operation type in the ETL intent triplet; second, the node configuration parameters are supplemented based on the matching data source information. For example, data extraction nodes automatically fill in data source connection information, data table name, field filtering conditions, and data filtering conditions, while data transformation nodes automatically configure field transformation rules. For multi-step combined operations (such as "extraction-association-aggregation-output"), the system generates an ordered sequence of nodes according to the business logic order. For complex scenarios involving multiple data sources and multiple data table associations, the system ensures process continuity through a mechanism of "automatic identification of node association keys + logical dependency verification." For example, when the ETL intent is "to calculate the customer repurchase rate for each product line," it matches three data sources: "Product Information Table," "Customer Order Table," and "Customer Basic Data Table." The system first generates independent data extraction nodes for each data table, then generates data association nodes based on the preset table association rules in the mapping library, automatically configures the association method, and verifies the data type consistency of the association keys. If intermediate data processing is required during the association process, the system automatically inserts data cleaning nodes, forming a complete node chain of "Extraction Node 1 → Extraction Node 2 → Extraction Node 3 → Data Cleaning Node → Data Association Node → Data Summary Node → Output Node." The structured design of the process configuration list uses standardized JSON format storage. The list contains three units: First, basic process information, including process ID, business scenario tag, creation time, associated ETL intent triplet text, and standardized parsing results; second, a set of node configurations, each node including node ID, node type, sorting number, input parameters (data source connection information, field list, filter conditions, association keys, transformation rules, etc.), output parameters (output data format, temporary storage address, field mapping relationship, etc.), and exception handling strategy; third, process execution rules, including node execution dependencies, parallel / serial execution identifiers, execution priority, and result storage path.

[0010] Preferably, in step S5, the embedding position and type of the visual correction node match the core steps of the ETL process: embedding data source and filter condition correction nodes after the data extraction node, embedding association rule correction nodes before and after the data association node, embedding transformation rule and formula correction nodes next to the data transformation node, embedding summary dimension and statistical rule correction nodes after the data summary node, and embedding global configuration verification and batch correction nodes at the end of the process, ensuring that each core configuration step has a visual adjustment entry point; each correction node adopts a design mode of "graphical display + form-style editing + real-time preview"; The core functions and operation logic of each type of visual correction node are as follows: Data Source and Filter Condition Adjustment Node: The system visually displays the automatically matched data sources, databases, tables, and field lists in a tree structure. Users can adjust the field filtering range by checking / unchecking boxes, reset time filtering conditions using a date picker, switch data sources using dropdown menus, and manually upload external data source files to supplement data. Filter condition adjustment adopts a dual mode of "visual rule configuration + custom SQL input." Non-technical users can generate filtering rules through a combination form of "field + operator + value," while technical users can directly edit SQL fragments to replace automatically generated filtering conditions, verify the correctness of SQL syntax in real time, and preview the amount of filtered data.

[0011] Preferably, in step S6, after the visualization correction node completes the configuration adjustment, it automatically triggers the ETL process execution mechanism. Through the full-link closed loop of "configuration parsing - task scheduling - step-by-step execution - result verification - multi-dimensional output", the structured process configuration list is transformed into actual data processing actions, ensuring the stability of the execution process and the accuracy of the data results. The core execution of the process follows a progressive logic of "parsing-scheduling-execution": First, the configuration parsing engine receives the adjusted process configuration list and performs structured parsing on the core configurations in the list. This includes breaking down node execution dependencies, extracting input and output parameters for each node, verifying configuration integrity, and converting the parsing results into a machine-readable set of execution instructions. The instruction set is serialized and stored according to node order and dependencies to ensure execution order and logical consistency. Second, the task scheduling engine dynamically allocates execution resources based on the execution instruction set and system resource usage, employing a hybrid "parallel + serial" scheduling strategy: parallel execution threads are started for nodes without dependencies, and the number of parallel threads is dynamically calculated using a formula. (Where A represents the number of parallel threads, C represents the number of available CPU cores, and D represents the number of nodes to be executed without dependencies) This is determined to avoid resource overload; nodes with dependencies are executed sequentially, and a status monitoring mechanism is used to ensure that the preceding node has completed execution and returned a "successful execution" status before the subsequent node is triggered to prevent data flow interruption; The ETL process is executed step-by-step, with each node type corresponding to its specific steps. Each step incorporates built-in data quality verification logic: In the data extraction phase, the execution engine connects to the target data source via JDBC, API interfaces, file reading, and other adaptation methods according to the extraction node instructions. It extracts target field data based on filtering conditions and verifies the completeness and accuracy of the extracted data in real time. If data is missing or formatted incorrectly, it triggers processing according to preset strategies (e.g., filling in default values ​​when missing data is ≤5%, pausing execution and issuing a pop-up alert when missing data exceeds 5%). In the data association phase, the execution engine performs join operations on multiple tables according to association rules, simultaneously verifying the reasonableness of the data volume in the association results. In the data transformation phase, the execution engine processes fields according to transformation rules and formulas. In the data aggregation phase, the execution engine completes grouped statistics according to aggregation dimensions and statistical rules, generating structured aggregation results while verifying the logical consistency of the aggregated data. All log information during the execution process is written to the execution log database in real time.

[0012] The technical effects and advantages of this invention are as follows: High accuracy and controllability of intent understanding: By combining large models and error correction mechanisms, the accuracy of identifying complex business intents is significantly improved; through visual correction nodes, users are given the ability to "fine-tune" the automatically generated process, solving the core pain point of uncontrollable processes in pure natural language solutions; Highly intelligent: It realizes automatic matching from business terms to data sources, reduces tedious manual searching and data source specification operations, and improves the level of intelligence in process construction; High efficiency and flexibility: Visualized corrections avoid the inefficient cycle of having to re-describe the entire requirement due to minor adjustments, significantly improving process adjustment efficiency. At the same time, the system can be integrated with existing ETL systems, protecting existing investments and reducing integration costs. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0014] Figure 2 This is a schematic diagram of the module links of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Please see Figure 1As shown, this invention provides a natural language-driven ETL workflow orchestration and execution method, including: S1: Receives natural language commands input by the user; S2: Perform intent recognition on the natural language instruction and parse out the ETL intent triplet, which includes the data processing target, operation type and output requirements; In S2, the intent recognition process is implemented based on a large model architecture that integrates a domain knowledge base. The domain knowledge base pre-contains core business terms, common operation expressions, and output format specifications for ETL scenarios. Through transfer learning and fine-tuning training of the large model, it is equipped with the ability to accurately understand the semantics of non-standard natural language instructions. After receiving natural language instructions input by business personnel, the key information elements in the instructions are first extracted through word segmentation, part-of-speech tagging, and entity recognition technologies. Then, semantic mapping and intent classification are performed based on the domain knowledge base, and finally, the instructions are decomposed into structured ETL intent triples. Among them, the analysis of data processing objectives is based on core business requirements and is determined by identifying key elements such as the main object, filtering conditions, and time range in the instructions; For example, in response to the instruction "Statistics on sales revenue of each product line in East China in the first quarter of 2024, excluding returned orders", the system will automatically extract the subject as "sales revenue of each product line in East China", the filter condition as "excluding returned orders", and the time range as "first quarter of 2024", and integrate them to form the clear data processing objective "Statistics on sales revenue of each product line in East China in the first quarter of 2024 for non-returned orders". If there are ambiguous expressions in the instruction (such as "recently" or "major customers"), the system will trigger interactive confirmation through preset ambiguity handling rules to ensure the accuracy of the processing objective.

[0017] The parsing of operation types is based on the mapping between the action description in the instruction and the business scenario. The system maps the operation descriptions in natural language (such as "get", "calculate", "merge", "filter", "compare") to the core ETL operation type set, including data extraction, data cleaning, data transformation, data association, data aggregation, data filtering, etc. For complex multi-step instructions, the system will break them down into multiple operation types in logical order to form an ordered operation sequence, providing a basis for subsequent process orchestration. For example, "extracting user registration data from the past three months and categorizing it by province" corresponds to the "extraction + categorization" combined operation, while "linking the sales table and customer table by customer ID and calculating the cumulative consumption amount for each customer" corresponds to the "linking + calculation" combined operation. The parsing of output requirements focuses on the descriptions in the instructions regarding the result presentation format, field range, storage method, etc., and extracts the output carrier, field details, format specifications, and storage / export requirements; For example, the command "Generate attendance reports for each department in March 2024, including employee names, attendance days, and number of leave requests, displayed in groups by department and supporting Excel export" can have its output requirements specified as "Output medium: attendance report; Field range: employee name, attendance days, number of leave requests; Display rules: grouped by department; Export requirements: support Excel format export". Through the above structured parsing, non-standardized natural language instructions are transformed into structured intent information that can be recognized and executed by machines.

[0018] S3: Based on the ETL intent triple, the corresponding data source, data table and field metadata are automatically matched by querying the preset mapping relationship between business terms and data sources; In S3, the system pre-builds a multi-dimensional business term-data source mapping library. This mapping library is classified by business scenario as the core dimension, integrates various internal structured data sources, semi-structured data sources and authorized external data sources, and establishes a five-level association mapping relationship of "business term-data source type-database instance-data table-field metadata". The field metadata includes key attributes such as field name, data type, field definition, data format and business module to which it belongs. During the matching process, key business terms are first extracted from the elements of the ETL intent triple. These include the main objects in the data processing target (such as "sales revenue," "attendance records," and "customer information"), business entities associated with operation types (such as "return orders" and "user registration data"), and field details in the output requirements (such as "product name," "attendance days," and "consumption amount"). Through word segmentation and deduplication, synonym normalization (such as mapping "sales revenue," "sales amount," and "revenue" to the core term "sales revenue"), and business scenario categorization, a standardized set of terms to be matched is formed. Subsequently, the system calls the retrieval engine of the mapping library, employing a combined retrieval strategy of "exact matching + fuzzy matching + scenario-related matching." Exact matching quickly locates the completely corresponding data source and data table. For terms that do not match precisely, fuzzy matching is performed based on field definitions and business unit relevance (e.g., "customer contact information" can match related fields such as "mobile phone number," "email address," and "contact number" in the "CRM customer relationship database - customer basic information table"). This process is then combined with ETL... The weights of the business scenarios corresponding to the intents (such as prioritizing financial data sources for financial statistics scenarios and prioritizing HR data sources for human resource management scenarios) are adjusted to improve matching accuracy. In exact matching, let the data source and data table set be... For each term t in the set of terms to be matched, if there exists Make t and If the field names are exactly the same, then an exact match is achieved, denoted as . For example, the core term "sales revenue" is precisely matched to "sales management system database - sales order table"; In fuzzy matching, for a term t that does not match an exact match, the similarity is based on the field's semantic similarity. The method for calculating the comprehensive similarity by performing fuzzy matching based on the relevance to business units is as follows: ; in, Represented as overall similarity, Represented as the degree of relevance of business units. , Represented as weighting factors; Select the data source and table with the highest score as the matching result; For complex scenarios where data processing objectives involve multiple business entity relationships (such as "statistics on customer repurchase rates for each product line," which requires association with three types of entities: "product information," "customer orders," and "customer basic data"), the system matches multiple corresponding data sources and tables based on the logical relationships between terms. It also automatically identifies the association keys and logic between tables using preset table association rules in the mapping library, forming a data table association chain. In the field metadata matching stage, the system uses the output field details of the ETL intent triple as a benchmark, filtering corresponding field metadata from the matched data tables. If the output requirements explicitly specify the field name, the field metadata is directly matched; if only the business meaning of the field is described (such as "employee's department"), the optimal field is matched based on the semantic similarity calculation of the field definition, and metadata information such as the field's data type and format constraints is extracted simultaneously. The system has a built-in matching result verification mechanism. Verification logic is triggered in the following situations: First, if a key business term does not match a corresponding data source (e.g., "market activity conversion rate" has no associated mapping relationship), the system will generate a "list of unmatched terms" and prompt the user to supplement the mapping rules or manually specify a data source. Second, if the same term matches multiple data sources (e.g., "customer mobile number" exists in both the CRM database and the membership management database), the system will comprehensively score based on the data source priority configuration, data update timestamp, and field completeness, recommending the optimal matching result and allowing the user to manually switch. Third, if the matched field metadata has data type conflicts (e.g., the output requirement of numeric "sales amount" matches a character field) or missing fields (e.g., the output requirement of "order date" does not exist in the matching data table), the system will mark the abnormal fields and provide field replacement suggestions. Through the above automated matching and verification process, a precise mapping from business intent to data source and metadata is achieved.

[0019] S4: Based on the ETL intent triple and the matching data source information, automatically organize and generate a process configuration list containing ETL nodes; In S4, a standardized ETL node system is constructed based on the operation type sequence in the ETL intent triple and the data source matching results. Then, through the logical association rules and parameter mapping mechanism between nodes, a structured process configuration that can be directly executed is formed. In the ETL node generation stage, a core ETL node library covering all scenarios is pre-defined, including seven basic types: data extraction nodes, data cleaning nodes, data transformation nodes, data association nodes, data aggregation nodes, data filtering nodes, and data output nodes. Each node has a built-in standardized configuration template (including required parameters, optional parameters, and default rules) and supports adaptive adaptation to the matching data source type (structured / semi-structured / external interface). The generation logic is as follows: First, the corresponding core node is mapped according to the operation type in the ETL intent triplet; second, the node configuration parameters are supplemented based on the matching data source information. For example, data extraction nodes automatically fill in data source connection information, data table name, field filtering conditions, and data filtering conditions, while data transformation nodes automatically configure field transformation rules. For multi-step combined operations (such as "extraction-association-aggregation-output"), the system generates an ordered sequence of nodes according to the business logic order. For complex scenarios involving multiple data sources and multiple data table associations, the system ensures process continuity through a mechanism of "automatic identification of node association keys + logical dependency verification." For example, when the ETL intent is "to calculate the customer repurchase rate for each product line," it matches three data sources: "Product Information Table," "Customer Order Table," and "Customer Basic Data Table." The system first generates independent data extraction nodes for each data table, then generates data association nodes based on the preset table association rules in the mapping library, automatically configures the association method, and verifies the data type consistency of the association keys. If intermediate data processing is required during the association process, the system automatically inserts data cleaning nodes, forming a complete node chain of "Extraction Node 1 → Extraction Node 2 → Extraction Node 3 → Data Cleaning Node → Data Association Node → Data Summary Node → Output Node." The structured design of the process configuration list uses standardized JSON format storage. The list contains three units: First, basic process information, including process ID, business scenario tag, creation time, associated ETL intent triplet text, and standardized parsing results; second, a set of node configurations, each node containing node ID, node type, sorting number, input parameters (data source connection information, field list, filter conditions, association keys, transformation rules, etc.), output parameters (output data format, temporary storage address, field mapping relationships, etc.), and exception handling strategies; third, process execution rules, including node execution dependencies, parallel / serial execution identifiers, execution priority, and result storage path. For example, for the intent to "generate a sales report of non-returned orders for each product line in East China in the first quarter of 2024", the process configuration list will include "East China product data extraction node", "non-returned order data extraction node" (executed in parallel), "data association node" (associating product IDs of the two tables), "sales summary node" (grouped and statistically analyzed by product line), and "report output node" (configuring report fields, sorting rules, and Excel export format). The input and output parameters of each node are automatically associated through field mapping relationships.

[0020] S5: Embed a visual correction node in the generated process, wherein the visual correction node is used to receive visual adjustment operations from the user on at least one configuration in the process configuration list; In S5, the embedding position and type of the visual correction node match the core links of the ETL process: embedding data source and filter condition correction nodes after the data extraction node, embedding association rule correction nodes before and after the data association node, embedding transformation rule and formula correction nodes next to the data transformation node, embedding summary dimension and statistical rule correction nodes after the data summary node, and embedding global configuration verification and batch correction nodes at the end of the process to ensure that each core configuration link has a visual adjustment entry point; each correction node adopts a design mode of "graphical display + form-style editing + real-time preview"; The core functions and operation logic of each type of visual correction node are as follows: Data source and filter condition adjustment node: The system visually displays the automatically matched data sources, databases, tables, and field lists in a tree structure. Users can adjust the field filtering range by checking / unchecking boxes, reset the time filtering conditions using a date picker, switch data sources using drop-down lists, and manually upload external data source files to supplement data. The filter condition adjustment adopts a dual mode of "visual rule configuration + custom SQL input". Non-technical users can generate filter rules through a combination form of "field + operator + value", while technical users can directly edit SQL fragments to replace the automatically generated filter conditions, verify the correctness of SQL syntax in real time, and preview the amount of filtered data. Association rule correction node: The system displays the relationships between data tables in a graphical topology diagram. Association keys are connected by highlighted lines, and users can drag the lines to adjust the association keys and select the association method through drop-down boxes. It also supports adding new association tables (by dragging a new data table to the topology diagram, it automatically identifies the fields that can be associated and recommends association rules). For multi-table association scenarios, it provides an association order adjustment function. Users can adjust the execution order of data table associations by dragging nodes up and down. The system verifies the legality of the association logic in real time and prompts the estimated data volume after the data association. Conversion Rules and Formula Correction Node: The interface combines a "rule list + formula editor" to visually display the automatically generated field conversion rules. Users can click the "Edit" button after the rule item to modify the parameters and redefine the calculation logic through the formula editor. The editor has a built-in library of commonly used functions and supports syntax highlighting, auto-completion, and formula validation. Users can directly modify existing formulas and add custom conversion rules. After editing, click "Preview" to view the converted sample data. Summary Dimensions and Statistical Rule Correction Nodes: The automatically generated summary results are previewed with visual charts. Users can adjust grouping dimensions by dragging dimension fields, switch statistical methods via drop-down menus, and set summary sorting rules and TOPN filtering. For multi-level summary needs, dimension collapse / expansion functions are provided, and users can flexibly configure the summary level. The system updates the summary result preview in real time. Global configuration validation and batch node correction: Integrates the core configuration parameters of all nodes, displaying process ID, node type, key configuration items, current value, and adjustment status in a form matrix. Users can filter parameters to be adjusted by node type and configuration type, and batch modify configurations of the same type (e.g., uniformly adjust the "timeout threshold" of all nodes to 120 seconds). The built-in configuration validation engine automatically detects whether there are conflicts in the adjusted configuration, marks abnormal items in red, and provides correction suggestions. Users can quickly fix the problem by clicking on the suggestions. The linkage between the visual correction node and the process configuration list adopts a "real-time operation synchronization + historical version traceability" mechanism: every adjustment operation completed by the user in the correction node will be instantly mapped to the underlying JSON format process configuration list; for example, after modifying the filter condition, the "filterCondition" field of the corresponding node in the configuration list will be automatically updated; after adding a conversion formula, the new rule will be synchronously appended to the "transformRule" array; after switching the association method, the "joinType" parameter will be changed instantly. The system automatically records the operation log for each adjustment (including the operator, operation time, configuration before adjustment, and configuration after adjustment), allowing users to revert to any historical version. If the effect after adjustment does not meet expectations, the previous valid configuration can be restored with one click, ensuring the security and traceability of the process configuration. In addition, the correction node also supports the saving and reuse of configuration templates, allowing users to save the adjusted personalized configuration as a custom template.

[0021] S6: Based on the process configuration list adjusted by the aforementioned visualization correction node, execute the ETL process and output the data processing results.

[0022] In S6, after the visualization correction node completes the configuration adjustment, it automatically triggers the ETL process execution mechanism. Through the full-link closed loop of "configuration parsing - task scheduling - step-by-step execution - result verification - multi-dimensional output", the structured process configuration list is transformed into actual data processing actions, ensuring the stability of the execution process and the accuracy of the data results. The core execution of the process follows a progressive logic of "parsing-scheduling-execution": First, the configuration parsing engine receives the adjusted process configuration list and performs structured parsing on the core configurations in the list. This includes breaking down node execution dependencies, extracting input and output parameters for each node, verifying configuration integrity, and converting the parsing results into a machine-readable set of execution instructions. The instruction set is serialized and stored according to node order and dependencies to ensure execution order and logical consistency. Second, the task scheduling engine dynamically allocates execution resources based on the execution instruction set and system resource usage, employing a hybrid "parallel + serial" scheduling strategy: parallel execution threads are started for nodes without dependencies, and the number of parallel threads is dynamically calculated using a formula. (Where A represents the number of parallel threads, C represents the number of available CPU cores, and D represents the number of nodes to be executed without dependencies) This is determined to avoid resource overload; nodes with dependencies are executed sequentially, and a status monitoring mechanism is used to ensure that the preceding node has completed execution and returned a "successful execution" status before the subsequent node is triggered to prevent data flow interruption; The ETL process is executed step-by-step, with each node type corresponding to its specific steps. Each step incorporates built-in data quality verification logic: In the data extraction phase, the execution engine connects to the target data source via JDBC, API interfaces, file reading, and other adaptation methods according to the extraction node instructions. It extracts target field data based on filtering conditions and verifies the completeness and accuracy of the extracted data in real time. If data is missing or formatted incorrectly, it triggers processing according to preset strategies (e.g., filling in default values ​​when missing data is ≤5%, pausing execution and issuing a pop-up alert when missing data exceeds 5%). In the data association phase, the execution engine performs join operations on multiple tables according to association rules, simultaneously verifying the reasonableness of the data volume in the association results. In the data transformation phase, the execution engine processes fields according to transformation rules and formulas. In the data aggregation phase, the execution engine completes grouped statistics according to aggregation dimensions and statistical rules, generating structured aggregation results while verifying the logical consistency of the aggregated data. All log information during the execution process is written to the execution log database in real time. After data processing, the system enters the result output stage, providing a three-in-one output format of "visualization + multi-format export + data storage" to meet the needs of different users: First, visualization: the result output unit automatically generates diverse visualization charts according to the output requirements, including basic statistical charts, detailed tables, cross-analysis tables, etc. The charts support interactive operation, and the detailed tables support field sorting, fuzzy search, and pagination. All visualization content is linked in real time with the adjusted configuration. Second, multi-format export: users can export the processing results to common data formats, including Excel, CSV, PDF, JSON, etc. The exported files are automatically named according to "business scenario - processing date - process ID". The export process supports batch export of multiple node results, custom export fields and sorting rules, and large file exports support breakpoint resume and email notification functions to avoid export interruption. Third, data storage: the processing results are directly stored to the specified target location, including the enterprise internal database, the data platform ODS / DWD layer, data warehouse, etc. The target database table structure is automatically matched during storage, supporting both incremental storage and full coverage storage modes. After storage is completed, a storage success indicator and data storage path are returned. The system has a built-in result verification and exception handling mechanism: After execution, it automatically compares the processing result with the output requirements of the ETL intent triple to verify the integrity of fields and the accuracy of data. If there are any mismatches, it generates a result verification exception report and marks the reason for the exception. If an exception occurs during execution, the execution engine immediately suspends the execution of the current node, records the exception information and triggers the fault tolerance mechanism. If the fault tolerance mechanism still cannot solve the problem, the system automatically interrupts the process and sends an alarm notification to the user, while providing exception repair suggestions. After the user repairs the problem, they can resume the process execution through the "resuming from breakpoint" function without having to re-execute all nodes.

[0023] Please see Figure 2 As shown, this invention also provides a natural language-driven ETL workflow orchestration and execution system, including: Receiving module: Receives natural language commands input by the user; Parsing module: performs intent recognition on natural language commands and parses out ETL intent triples, which include data processing target, operation type and output requirements; Matching module: Based on the ETL intent triple, it automatically matches the corresponding data source, data table and field metadata by querying the preset mapping relationship between business terms and data sources; Configuration list module: Based on the ETL intent triple and the matching data source information, automatically organize and generate a process configuration list containing ETL nodes; Embedded module: Embeds a visual correction node in the generated process, the visual correction node being used to receive visual adjustment operations from the user on at least one configuration in the process configuration list; Execution module: Based on the process configuration list adjusted by the visualization correction node, execute the ETL process and output the data processing results.

[0024] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A natural language-driven ETL workflow orchestration and execution method, characterized in that, include: S1: Receives natural language commands input by the user; S2: Perform intent recognition on the natural language instruction and parse out the ETL intent triplet, which includes the data processing target, operation type and output requirements; S3: Based on the ETL intent triple, the corresponding data source, data table and field metadata are automatically matched by querying the preset mapping relationship between business terms and data sources; S4: Based on the ETL intent triple and the matching data source information, automatically organize and generate a process configuration list containing ETL nodes; S5: Embed a visual correction node in the generated process, wherein the visual correction node is used to receive visual adjustment operations from the user on at least one configuration in the process configuration list; S6: Based on the process configuration list adjusted by the aforementioned visualization correction node, execute the ETL process and output the data processing results.

2. The method for orchestrating and executing ETL workflows based on natural language driving according to claim 1, characterized in that: In S2, the intent recognition process is implemented based on a large model architecture that integrates a domain knowledge base. After receiving natural language instructions input by business personnel, key information elements in the instructions are first extracted through word segmentation, part-of-speech tagging, and entity recognition technologies. Then, semantic mapping and intent classification are performed based on the domain knowledge base, and finally, the instructions are decomposed into structured ETL intent triples. The parsing of operation types is based on the mapping between the action description in the instruction and the business scenario. The system maps the operation description in natural language to the ETL core operation type set. For complex multi-step instructions, the system splits them into multiple operation types in logical order to form an ordered operation sequence.

3. The method for orchestrating and executing ETL workflows based on natural language driving according to claim 1, characterized in that: In S3, the system pre-builds a multi-dimensional business terminology-data source mapping library. During the matching process, key business terms are first extracted from the elements of the ETL intent triple. Through word segmentation and deduplication, synonym normalization, and business scenario classification, a standardized set of terms to be matched is formed. Subsequently, the system calls the retrieval engine of the mapping library and adopts a combination of exact matching, fuzzy matching, and scenario-related matching. Exact matching quickly locates the fully corresponding data source and data table. For terms that do not match exactly, fuzzy matching is performed based on field definitions and business unit relevance. At the same time, the weight is adjusted in combination with the business scenario corresponding to the ETL intent.

4. The natural language-driven ETL workflow orchestration and execution method according to claim 3, characterized in that: In exact matching, let the data source and data table set be... For each term t in the set of terms to be matched, if there exists Make t and If the field names are exactly the same, then an exact match is achieved, denoted as . ; In fuzzy matching, for a term t that does not match an exact match, the similarity is based on the field's semantic similarity. The method for calculating the comprehensive similarity by performing fuzzy matching based on the relevance to business units is as follows: ; in, Represented as overall similarity, Represented as the degree of relevance of business units. , Represented as weighting factors; The data source and table with the highest score are selected as the matching results.

5. The method for orchestrating and executing ETL workflows based on natural language driving according to claim 3, characterized in that: For complex scenarios where the data processing target involves the association of multiple business entities, the system matches multiple corresponding data sources and data tables according to the logical relationship between terms, and automatically identifies the association keys and association logic between data tables through the preset table association rules in the mapping library, forming a data table association relationship chain. In the field metadata matching stage, the system uses the output requirement field details of the ETL intent triple as a benchmark, and filters the corresponding field metadata from the matched data table. If the output requirement explicitly specifies the field name, the field metadata is directly matched; if it only describes the business meaning of the field, the optimal field is matched by calculating the semantic similarity of the field definition, and the data type and format constraint metadata information of the field is extracted simultaneously.

6. The method for orchestrating and executing ETL workflows based on natural language driving according to claim 1, characterized in that: In S4, a standardized ETL node system is constructed based on the operation type sequence in the ETL intent triple and the data source matching results. Then, through the logical association rules and parameter mapping mechanism between nodes, a structured process configuration that can be directly executed is formed. In the ETL node generation stage, a core ETL node library covering all scenarios is preset. The generation logic is as follows: First, the corresponding core node is mapped according to the operation type in the ETL intent triple. Second, the node configuration parameters are supplemented by the matching data source information. For multi-step combined operations, the system generates an ordered sequence of nodes according to the business logic order.

7. The method for orchestrating and executing ETL workflows based on natural language driving according to claim 1, characterized in that: In S5, the embedding position and type of the visualization correction node match the core links of the ETL process: embed data source and filter condition correction nodes after the data extraction node, embed association rule correction nodes before and after the data association node, embed transformation rule and formula correction nodes next to the data transformation node, embed summary dimension and statistical rule correction nodes after the data summary node, and embed global configuration verification and batch correction nodes at the end of the process.

8. The method for orchestrating and executing ETL workflows based on natural language driving according to claim 1, characterized in that: In S6, after the visual correction node completes the configuration adjustment, it automatically triggers the ETL process execution mechanism to transform the structured process configuration list into data processing actions. The core execution of the process follows a progressive logic of "parsing-scheduling-execution": First, the configuration parsing engine receives the adjusted process configuration list, performs structured parsing on the core configurations in the list, and transforms the parsing results into a machine-recognizable set of execution instructions. The instruction set is serialized and stored according to the node sequence number and dependencies. Second, the task scheduling engine dynamically allocates execution resources based on the execution instruction set and the system resource usage, adopting a hybrid parallel and serial scheduling strategy: parallel execution threads are started for nodes without dependencies, and the number of parallel threads is dynamically calculated using a formula. Decision: For nodes with dependencies, execute them sequentially. A status monitoring mechanism ensures that the preceding node has completed execution and returned a "successful execution" status before triggering the execution of the subsequent node.

9. The method for orchestrating and executing ETL workflows based on natural language driving according to claim 1, characterized in that: The step-by-step execution process of the ETL process corresponds one-to-one with the node type, and each step has built-in data quality verification logic: In the data extraction stage, the execution engine connects to the target data source through JDBC according to the extraction node instruction, extracts the target field data according to the filtering conditions, and verifies the integrity and accuracy of the extracted data in real time. If data is missing or the format is abnormal, the preset strategy will be triggered for processing. During the data association phase, the execution engine performs join operations on data from multiple tables according to the association rules, and simultaneously verifies the reasonableness of the data volume of the association results. During the data transformation phase, the execution engine processes the fields according to the transformation rules and formulas; During the data aggregation phase, the execution engine completes grouped statistics according to aggregation dimensions and statistical rules, generates structured aggregation results, and verifies the logical consistency of the aggregated data. All log information during the execution process is written to the execution log database in real time.

10. A natural language-driven ETL workflow orchestration and execution system, using any one of the natural language-driven ETL workflow orchestration and execution methods as described in claims 1-9, characterized in that: Receiving module: Receives natural language commands input by the user; Parsing module: performs intent recognition on natural language commands and parses out ETL intent triples, which include data processing target, operation type and output requirements; Matching module: Based on the ETL intent triple, it automatically matches the corresponding data source, data table and field metadata by querying the preset mapping relationship between business terms and data sources; Configuration list module: Based on the ETL intent triple and the matching data source information, automatically organize and generate a process configuration list containing ETL nodes; Embedded module: Embeds a visual correction node in the generated process, the visual correction node being used to receive visual adjustment operations from the user on at least one configuration in the process configuration list; Execution module: Based on the process configuration list adjusted by the visualization correction node, execute the ETL process and output the data processing results.

Images