A system and method for generating a data processing workflow

Abstract

The application relates to the technical field of data processing, and provides a data processing workflow generation system and method, which comprises the following steps: a semantic analysis model is used to analyze a received natural language requirement, and a structured task topology containing execution dependency is generated; business metadata of a target business platform is combined, the structured task topology is mapped into a platform interface calling sequence in sequence, the platform interface calling sequence is triggered in sequence to build a data processing workflow, and data checking rules are automatically embedded in nodes of the data processing workflow based on the business metadata. The application realizes intelligent generation and quality built-in of the data processing workflow, and improves development efficiency and compliance.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing workflow generation system and method. Background Technology

[0002] In the creation and management of data processing workflows (ETL) on data platforms, existing technologies typically employ either visual drag-and-drop development or script-based development. Visual drag-and-drop requires users to manually drag and drop components, configure parameters, and define dependencies in a graphical interface, making the process cumbersome and time-consuming. Script-based development, on the other hand, demands higher technical skills from developers and incurs significant code maintenance costs.

[0003] Furthermore, in existing solutions, data quality verification is typically configured as a separate module after the process is created, resulting in a separation between verification and processing logic. Data quality issues are often only discovered after the process has run, increasing rework costs. Meanwhile, while natural language-generated code lowers the barrier to entry, directly generated code may bypass the platform's control system, posing compliance risks and making it difficult to reuse existing platform capabilities.

[0004] Therefore, there is an urgent need for a data processing workflow generation solution that can balance development efficiency, platform compliance, and built-in quality. Summary of the Invention

[0005] To address the problems of low efficiency in creating data processing workflows, disconnect between quality verification and process flow, and difficulty in adapting natural language generation results to platform management systems in existing technologies, this application proposes a data processing workflow generation system and method that enables the automatic generation of compliant data processing workflows with embedded quality verification based on natural language requirements.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A data processing workflow generation system includes: a structured task topology generation module, used to parse received natural language requirements through a semantic parsing model to generate a structured task topology containing execution dependencies; an interface call sequence mapping module, used to combine the business metadata of the target business platform to map the structured task topology into a sequentially invoked platform interface call sequence; and a data processing workflow construction module, used to trigger the platform interface call sequence sequentially to construct the data processing workflow, and automatically embed data validation rules in the nodes of the data processing workflow based on the business metadata.

[0008] In this way, unstructured natural language requirements are transformed into structured task topologies through semantic parsing models, and then business metadata is used to map them into the platform's native interface call sequence instead of directly generating code. This ensures that the generated workflow fully complies with the target business platform's specifications and control system. At the same time, data verification rules are automatically embedded based on business metadata when building the workflow, realizing the synchronous generation and integrated deployment of data processing logic and quality assurance logic, avoiding rework issues caused by post-verification.

[0009] In some embodiments, the structured task topology generation module includes: an intent classification result generation unit, used to classify the natural language requirement through the semantic parsing model to obtain an intent classification result; a slot extraction result acquisition unit, used to extract source data identifier, target data identifier and conversion logic from the natural language requirement as slot extraction results; and an execution dependency relationship construction unit, used to construct the execution dependency relationship based on the data flow logic between the slot extraction results.

[0010] In this way, by decoupling the natural language parsing process into three sub-steps—intent classification, slot extraction, and dependency construction—the semantic parsing model can more accurately identify key elements in user needs and automatically deduce the sequence of tasks based on data flow logic, thereby improving the accuracy and completeness of structured task topology generation.

[0011] In some embodiments, the structured task topology generation module further includes a topology anomaly detection and repair unit, which performs circular dependency detection on the structured task topology. When a circular dependency is detected, the circular dependency is resolved based on a predefined subtask type stage sequence until there are no cycles in the structured task topology.

[0012] In this way, by introducing a topology anomaly detection and automatic repair mechanism, and using the predefined subtask type stage order as the priority basis for resolving circular dependencies, logical errors can be automatically corrected while ensuring semantic rationality, thus ensuring that the generated structured task topology is executable.

[0013] In some embodiments, the interface call sequence mapping module includes: an interface path matching unit, used to match the intent classification result and the slot extraction result as an interface path and request parameters based on the interface registry of the target business platform; and a missing parameter completion unit, used to infer a default value based on the business metadata or trigger multiple rounds of natural language interaction to complete the missing required parameters when it is determined that there are missing required parameters in the request parameters.

[0014] In this way, the interface registry achieves a precise mapping from natural language semantics to platform interfaces. Combined with business metadata inference or multiple rounds of interaction to complete missing parameters, it not only ensures the effectiveness of interface calls, but also improves the user experience and avoids process generation failure due to missing parameters.

[0015] In some embodiments, the data processing workflow construction module includes: a data verification rule determination unit, used to determine the data verification rule type based on the data classification features in the business metadata; and a verification node embedding unit, used to embed the data verification rule corresponding to the data verification rule type as an independent verification node after the corresponding data processing node and before the data writing node, according to the execution dependency relationship.

[0016] In this way, the verification rule type is intelligently matched according to the data classification characteristics, and the verification rule is embedded as an independent node into a specific topological position, realizing a differentiated quality assurance strategy and ensuring that key data processing links are subject to appropriate quality monitoring.

[0017] In some embodiments, the verification node embedding unit is configured to: synchronously mount the corresponding data verification rule configuration when generating each data processing node, and insert the data verification rule as an independent verification node into the data processing workflow when generating the data processing workflow, so that the data verification rule is synchronously embedded when the data processing workflow is generated.

[0018] In this way, by adopting a synchronous mounting and synchronous insertion mechanism, it is ensured that data verification rules and data processing nodes are created in the same generation transaction, which fundamentally eliminates the production of semi-finished products without verification and achieves built-in atomicity of quality.

[0019] In some embodiments, the business metadata includes at least a data dictionary and model definition information; the data processing workflow construction module further includes a semantic alignment unit, which is used to generate a mapping relationship between fields from different data sources to a unified encoding based on the unified encoding system in the data dictionary when the structured task topology contains multiple heterogeneous data sources, and to align fields with different names but the same semantics based on a semantic recognition model.

[0020] In this way, for multi-source heterogeneous data fusion scenarios, the data dictionary and semantic recognition model are used to automatically complete the encoding mapping and field alignment, which solves the problem of cumbersome and error-prone manual configuration of mapping rules in cross-source data fusion and improves the intelligence level of data fusion.

[0021] In some embodiments, the system further includes an interactive correction module, configured to generate natural language prompts containing the location of the error node and correction suggestions when the data verification rules determine that there is a logical error in the data processing workflow; receive correction instructions based on the natural language prompts, and update only the interface call parameters of the affected nodes without regenerating the entire structured task topology.

[0022] This provides a natural language-based differential correction mechanism that updates only the parameters of the affected nodes when an error is detected, rather than rebuilding the entire topology. This significantly reduces the computational overhead of iterative correction and user waiting time, and improves the efficiency of human-computer interaction.

[0023] Furthermore, the present invention also provides a method for generating a data processing workflow, comprising: parsing the received natural language requirements through a semantic parsing model to generate a structured task topology containing execution dependencies; mapping the structured task topology to a sequentially invoked platform interface call sequence in conjunction with the business metadata of the target business platform; triggering the platform interface call sequence in sequence to construct the data processing workflow, and automatically embedding data validation rules in the nodes of the data processing workflow based on the business metadata.

[0024] In this way, the workflow is built by using natural language to drive platform interface calls, and data verification rules are embedded synchronously during the construction process, realizing the intelligent, compliant, and built-in quality generation of the data processing workflow.

[0025] In some embodiments, the method further includes: when the data verification rule determines that there is a logical error in the data processing workflow, generating natural language prompt information containing the location of the error node and correction suggestions; receiving a correction instruction input based on the natural language prompt information, and updating only the interface call parameters of the affected node without regenerating the entire structured task topology.

[0026] In this way, by implementing a local correction mechanism at the method level, users can efficiently fix workflow logic errors in natural language interaction, avoiding the waste of resources caused by full regeneration.

[0027] Compared with the prior art, this application has the following technical effects:

[0028] 1. This application transforms natural language requirements into structured task topologies through a semantic parsing model and maps them into platform interface call sequences in conjunction with business metadata. Compared with the method of directly generating code, this mechanism reuses the existing functional modules and control capabilities of the target business platform, ensuring the compliance of the generated data processing workflow in terms of syntax, permissions and business specifications, while avoiding efficiency bottlenecks caused by manual configuration or code writing.

[0029] 2. This application achieves the integrated generation of quality assurance logic and data processing logic by automatically embedding data verification rules based on business metadata when constructing the data processing workflow, and synchronously inserting the verification rules as independent nodes into the workflow. This solves the problems of delayed problem discovery and high rework costs caused by the separation of quality verification and ETL process in the prior art.

[0030] 3. This application enhances the system's adaptability to the ambiguity of natural language and complex business scenarios through mechanisms such as topology anomaly detection and repair, missing parameter completion, and interactive correction, significantly improving the success rate of data processing workflow generation and user experience. Attached Figure Description

[0031] Figure 1 This is a structural block diagram of a data processing workflow generation system provided in one embodiment of this application;

[0032] Figure 2 This is an execution flowchart of the structured task topology generation module provided in one embodiment of this application;

[0033] Figure 3 This is a flowchart of a data processing workflow generation method provided in one embodiment of this application.

[0034] In the picture:

[0035] 10. Structured task topology generation module; 20. Interface call sequence mapping module; 30. Data processing workflow construction module; 40. Interactive correction module; 50. Process template learning and recommendation module. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0037] The technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of a person skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this application.

[0038] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0039] Example 1:

[0040] like Figure 1As shown, this embodiment provides a data processing workflow generation system. The system includes a structured task topology generation module 10, an interface call sequence mapping module 20, and a data processing workflow construction module 30. Furthermore, the system also includes a natural language interaction interface and a process deployment and scheduling interface, used respectively to receive user natural language requests and return system feedback in natural language form, and a scheduling engine to deploy validated data processing workflows to the target business platform. It should be understood that... Figure 1 This only illustrates one exemplary connection relationship between the modules. In actual deployment, each module can be an independent software service, a microservice component, or a different functional unit integrated within the same computing device, as long as it can achieve the following data flow and processing logic.

[0041] The structured task topology generation module 10 is used to parse the received natural language requirements through a semantic parsing model and generate a structured task topology containing execution dependencies. Specifically, this module is the system's semantic understanding entry point, and its core function is to transform unstructured user intents into intermediate logical expressions that can be processed by a computer. The structured task topology here is not directly executable program code or scripts, but rather an abstract data structure describing data processing logic, data flow, and task sequence, such as a directed acyclic graph (DAG) or task planning tree. The semantic parsing model can employ a Large Language Model (LLM) or other Natural Language Understanding (NLU) models. Through intent recognition and slot filling techniques, it extracts key elements such as source data objects, target data objects, transformation logic, and quality constraints from the user's natural language description, and constructs execution dependencies based on the inherent logic between these elements. For example, when a user inputs "clean external macro data and write it to the model layer," the structured task topology generated by this module will include three logical nodes: "data extraction," "data cleaning," and "data writing," as well as directed edges for "extraction → cleaning → writing," clearly expressing the task's execution sequence and data transfer path. This design allows the system to focus on the correct expression of business logic, while decoupling specific platform adaptation details to subsequent stages, avoiding syntax errors and platform incompatibility issues that might arise from directly generating code.

[0042] The interface call sequence mapping module 20 is used to map the structured task topology into a sequential sequence of platform interface calls, based on the business metadata of the target business platform. Specifically, this module acts as a translator between abstract business logic and concrete platform capabilities. Business metadata is the core basis for the mapping process, and it includes at least the target business platform's interface registry, data dictionary, model definition information, supplier information, and data lineage information. The interface registry records in detail the paths, request methods, parameter definitions, and constraints of all functional interfaces exposed by the platform; the data dictionary defines a unified data standard and encoding system; and the data lineage information records the upstream and downstream dependencies and data flow paths of existing data processing workflows, enabling the system to perceive the relationships between existing data assets when generating new workflows, avoiding isolated nodes or broken data links. During the mapping process, this module traverses each logical node in the structured task topology, performs semantic matching based on the node type and the interface description in the business metadata, determines the corresponding platform interface path, and converts the slot parameters in the topology into request parameters that conform to the interface specifications. For example, for the "data cleaning" node in the topology, this module queries the interface registry, matches the " / api / clean / config" interface, and converts the user-described "remove null values" into the JSON-formatted configuration parameters required by the interface based on the field definitions in the data dictionary. The final output platform interface call sequence is an ordered set of API request instructions conforming to the platform specifications, rather than code in general programming languages ​​such as Python or SQL. This NL2API (Natural Language to API) approach ensures that the generated workflow fully reuses the existing functional modules and control capabilities of the target business platform, fundamentally guaranteeing the compliance and security of the data processing process, while also avoiding security vulnerabilities and maintenance difficulties introduced by directly generating code.

[0043] The data processing workflow construction module 30 is used to sequentially trigger platform interface call sequences to build data processing workflows and automatically embed data validation rules into the nodes of the data processing workflow based on business metadata. Specifically, this module is responsible for transforming static interface call sequences into dynamically executable workflow instances and simultaneously building a quality assurance system during the construction process. When triggering the platform interface call sequence, this module sends requests to the target business platform in the order defined by the sequence, creates data synchronization tasks, configures cleaning rules, sets scheduling strategies, etc., thereby assembling a complete data processing workflow. More importantly, the embedding of data validation rules is not added separately after the workflow is built, but is executed synchronously in the same transaction of building data processing nodes. This module automatically determines the type of validation rule (such as integrity validation, consistency validation, timeliness validation, etc.) applicable to the current node based on the data classification characteristics (such as data type, sensitivity level, and layer) in the business metadata, and inserts these rules as independent functional nodes or configuration items after the corresponding data processing node and before the data writing node. For example, when constructing the "financial indicator cleaning" node, the system identifies the data as "core financial data" based on business metadata. It then automatically attaches "non-empty rate check" and "account reconciliation verification" rules, and simultaneously generates the corresponding quality verification API call while generating the cleaning task API call. This atomic mechanism of "generation as embedding" ensures that every online data processing workflow inherently possesses quality protection capabilities that match its business characteristics, completely resolving the problems of delayed problem discovery and high rework costs caused by the separation of data processing and quality verification in traditional solutions.

[0044] Through the aforementioned system architecture, this embodiment achieves end-to-end automated generation from natural language requirements to compliant data processing workflows. Compared with existing technologies, this application uses a structured task topology as an intermediate state, combined with platform interface mapping driven by business metadata. This retains the convenience of natural language interaction while ensuring strict compliance of the generated results with the target platform's control system. Simultaneously, by embedding data verification rules synchronously during the construction phase, quality assurance is shifted to the development stage, significantly improving the reliability of data processing and operational efficiency.

[0045] As one implementation method, the structured task topology generation module 10 includes an intent classification result generation unit, a slot extraction result acquisition unit, and an execution dependency construction unit. These three units constitute a complete parsing link from natural language to structured logical expression. Their workflow is not a simple linear sequence, but involves close data interaction and verification feedback.

[0046] Specifically, the intent classification result generation unit is used to classify natural language requests using a semantic parsing model to obtain intent classification results. This intent classification is not open-ended text understanding, but rather maps user input to a predefined set of closed intent categories. This category set includes at least creation of new data processing workflows, modification of existing data processing workflows, querying workflow running status, querying data asset information, and creating data validation rules. For example, when a user inputs "clean external macro data and write it to the model layer," the semantic parsing model classifies it as the intent "create a new data processing workflow"; while when a user inputs "view the error information of the previous task," it is classified as the intent "query workflow running status." This classification mechanism based on predefined categories provides a clear contextual index for subsequent slot extraction and interface mapping, avoiding the illusion problem that easily arises in open-domain understanding of general large models, and ensuring that system responses are always limited to the platform's controllable functional range.

[0047] The slot extraction result acquisition unit is used to extract source data identifiers, target data identifiers, and transformation logic from natural language requirements, as the slot extraction results. After determining the intent category, the system uses natural language processing techniques such as Named Entity Recognition (NER) or sequence labeling to accurately extract key parameters from unstructured text based on the slot template corresponding to that category. For example, for the intent "create a new data processing workflow," the system will selectively extract "external macroeconomic indicators source table" as the source data identifier, "unified model layer - macroeconomic indicators table" as the target data identifier, and "cleaning," "fusion," and "encoding mapping" as the transformation logic. It should be noted that the slot extraction results include not only explicitly mentioned parameters but also implicit parameters obtained through contextual reasoning. If the user only mentions "macroeconomic data" without specifying a particular supplier, the system can complete the slot by combining historical dialogues or default configurations. This structured slot expression transforms ambiguous natural language into key-value pair data that computers can accurately process, laying the data foundation for subsequent task orchestration.

[0048] Furthermore, such as Figure 2 As shown, the structured task topology generation module 10 is based on a three-layer mechanism of "predefined subtask type template library + large language model semantic decomposition + semantic alignment verification" to identify and decompose multiple subtask nodes from a single sentence of natural language. The specific process is as follows:

[0049] The first layer consists of a predefined subtask type template library: the system has a built-in set of subtask type templates covering the entire data processing chain. Each template includes a subtask type identifier, input / output data contracts, a set of callable interfaces, default parameters, and quality verification configurations. The subtask types include at least: data extraction (data source connection, incremental synchronization, full synchronization), data cleaning (null value handling, outlier filtering, deduplication, format standardization, time zone alignment), encoding mapping (unified mapping of multi-vendor encodings, dictionary conversion), data fusion (multi-source merging, field alignment, conflict resolution), derived calculations (metric calculation, aggregation, window functions), data writing (writing to the source layer, standard layer, model layer, and data mart layer), quality verification (completeness, consistency, timeliness, business logic verification), and scheduling configuration (periodic scheduling, dependency triggering, alarm notification). Each template corresponds to a set of callable interfaces on the target business platform, enabling the subtasks generated by the semantic parsing model to be directly implemented as platform interface calls, rather than abstract descriptions.

[0050] The second layer involves semantic decomposition of the large language model: After receiving the user's natural language description, the semantic parsing model decomposes it according to a four-step process: "intent classification → verb recognition → action classification → subtask segmentation". First, the intent classifier determines that the user's need belongs to the "create data processing workflow" category. Second, it identifies all action verbs and business terms in the sentence, such as "cleaning," "fusion," "writing," "encoding mapping," and "alarms." Third, it semantically matches each identified action with a predefined subtask type template library, mapping each action to one or more candidate subtask types. Fourth, it segments the natural language description into several subtask nodes according to the principle of "one subtask corresponding to one core data processing action," and fills each subtask with slot parameters (source object, target object, processing rules, quality constraints). For example, for the task of "cleaning and merging macroeconomic indicator data from the first and second data sources and writing it into the model layer", the model will sequentially identify five sub-task nodes: "data extraction (macroeconomic data from the first data source + macroeconomic data from the second data source)", "data cleaning (there is no explicit cleaning requirement in this example, so it will be ignored after asking the user for confirmation)", "encoding mapping (determining mapping rules by combining the encoding dictionary)", "data fusion (the model determines the fusion primary key according to the data dictionary)" and "data writing (model layer, the final target library is determined by the user's destination data source definition)".

[0051] The third layer, semantic alignment verification: After the subtasks are split, the system performs alignment verification on the split results: ① Checks whether all action verbs in the user description are covered; if any are missing, the corresponding subtasks are added; ② Checks whether the subtask sequence forms a complete "source → processing → target" data link; if necessary intermediate links are missing (e.g., only extraction and writing without cleaning), default subtask nodes (e.g., default cleaning nodes) are automatically added based on the data classification features in the business metadata; ③ Checks whether the slot parameters of each subtask are complete; missing parameters are automatically filled through business metadata retrieval or confirmed with the user through multiple rounds of dialogue. Through the above three-layer mechanism, the system utilizes predefined templates to ensure feasibility and leverages the semantic understanding capabilities of the semantic parsing model to address the diversity and ambiguity of natural language, achieving accurate decomposition from single-sentence natural language to structured multi-subtask topology.

[0052] The execution dependency construction unit is used to construct execution dependencies based on the data flow logic between slot extraction results. In this embodiment, the construction of execution dependencies is not based solely on automatic inference of data flow, but rather on a comprehensive determination using a triple mechanism of "data flow inference + template dependency rules + business semantic constraints". First, the system matches the input and output data contracts of each subtask. If the output object of the previous task is semantically consistent with the input object of the next task, a preliminary dependency edge is established. Second, the system introduces predefined template dependency rules as strong constraints. For example, it stipulates that the mandatory precondition for the "data cleaning" node is "data extraction", and the mandatory postcondition is "data fusion" or "data writing". When the data flow inference result conflicts with the template rules, the template rules prevail to prevent logical gaps caused by omissions in user descriptions. Finally, semantic validation is performed based on the layered specifications in the business metadata. For example, it mandates that tasks writing high-level data must depend on low-level data processing tasks, and direct cross-layer connections are strictly prohibited. Through these three mechanisms, the system can automatically generate robust topological structures that conform to both user semantics and platform specifications, effectively making up for the shortcomings in logical rigor of simply relying on large model reasoning.

[0053] As one implementation, the structured task topology generation module 10 also includes a topology anomaly detection and repair unit, used to perform circular dependency detection on the structured task topology. When a circular dependency is detected, it resolves the circular dependency based on a predefined subtask type stage order until there are no cycles in the structured task topology. Due to the arbitrariness of natural language descriptions or deviations in semantic parsing, the generated initial topology may contain logical loops (such as task A depending on task B, and task B depending on task A), which will cause the workflow to fail to be scheduled and executed. This embodiment solves this problem by introducing a deterministic resolution algorithm based on stage priority.

[0054] Specifically, the predefined subtask type stage order refers to the standard data processing pipeline order established by the system at the domain knowledge level, for example, defined as: Data Extraction Stage < Data Cleaning Stage < Encoding and Mapping Stage < Data Fusion Stage < Derivative Computation Stage < Data Writing Stage. When the topology anomaly detection and repair unit detects a loop through depth-first traversal, it compares the stage priorities of each node in the loop. The system retains the dependency edges from lower stages to higher stages and deletes the reverse edges from higher stages to lower stages, thereby breaking the loop. For example, suppose there is an incorrect dependency in the initial topology where "data cleaning node" points to "data extraction node," and at the same time there is a correct dependency where "data extraction node" points to "data cleaning node," forming a loop. Since the priority of the "data extraction" stage is lower than that of the "data cleaning" stage, the system will automatically delete the reverse edge of the former and retain the latter, restoring the topology to a valid directed acyclic graph. If the nodes in the loop are in the same stage or their priorities cannot be directly determined, the system further deletes the edge with the lowest confidence based on the confidence of the data flow matching and feeds back the corrected topology structure to the user for confirmation.

[0055] Furthermore, the topology anomaly detection and repair unit also includes a parallel executable subtask identification and optimization function. This function identifies parallel executable subtasks based on the dependencies between the final subtasks input by the user. Specifically, in the directed acyclic graph (DAG) of the structured task topology, if two or more subtask nodes have no direct or indirect dependencies (i.e., no common ancestor node causing execution order constraints), these subtasks are marked as parallel executable. For example, the nodes "first data source data extraction" and "second data source data extraction" depend on different source data objects and have no data transfer relationship with each other; therefore, they can be identified as parallel tasks. The system passes this parallel task information to the subsequent scheduling module, which then implements concurrent execution of the tasks, thereby improving the overall execution efficiency of the data processing workflow.

[0056] As one implementation method, the interface call sequence mapping module 20 includes an interface path matching unit and a missing parameter completion unit. These two units together constitute the core of the conversion from abstract structured task topology to specific platform executable instructions, resolving the contradiction between the ambiguity of natural language description and the strictness of platform interface definition, and ensuring the feasibility of the NL2API technology route.

[0057] Specifically, the interface path matching unit is used to match the intent classification results and slot extraction results into interface paths and request parameters based on the interface registry of the target business platform. This interface registry is not a simple list of URLs, but a structured knowledge base containing rich semantic information. In this embodiment, the interface registry is built using the extended OpenAPI specification. Each interface entry not only records the API path, HTTP request method, and basic parameter definitions, but also includes functional semantic description tags, parameter business meaning annotations, and constraints between parameters. For example, for the data synchronization interface, the registry not only defines the path / api / sync / create, but also labels its semantic tags as "data extraction" and "incremental synchronization," and explicitly defines the business meaning of the parameter source_id as "unique identifier of the source data object," while the optional enumeration values ​​for the parameter sync_mode are "full" or "incremental." During the matching process, the interface path matching unit first performs semantic retrieval in the interface registry based on the intent classification results (such as the "data extraction" action in "creating a new data processing workflow") to filter out a set of candidate APIs. Then, it aligns the entity values ​​from the slot extraction results (such as the "external macroeconomic indicator source table") with the parameter definitions of the candidate APIs. Through semantic similarity calculation or pre-defined parameter mapping rules, it determines the final API path and specific request parameter values. This matching mechanism based on a semantically enhanced registry enables the system to understand the business intent behind the user's natural language, rather than simply performing keyword matching, thereby significantly improving the accuracy of interface mapping.

[0058] The missing parameter completion unit is used to infer default values ​​based on business metadata or trigger multiple rounds of natural language interaction to complete the missing parameters when a required parameter is determined to be missing in the request parameters. Due to the inherent omission nature of natural language interaction, users often do not provide all the complete parameters required by the API in a single input. To avoid process generation failure due to missing parameters and to reduce unnecessary interaction disturbances, this embodiment designs a hierarchical completion strategy. The system pre-sets a confidence threshold for the required parameters of each API. When a required parameter is detected to be missing, the missing parameter completion unit first attempts to infer it based on business metadata. If the confidence of the inference result is higher than the preset threshold, the inferred value is directly used as the default value and informed to the user in subsequent feedback for confirmation; if the confidence is lower than the threshold or cannot be inferred, a targeted natural language follow-up question is generated, triggering multiple rounds of interaction to obtain the accurate value.

[0059] To illustrate the parameter completion mechanism more clearly, a specific inference chain example is used below. Suppose a user only mentions "synchronize external macro data" in their natural language request, but does not specify the required parameter "target storage level." After identifying the missing parameter, the missing parameter completion unit queries the data layering specifications and supplier information in the business metadata. The inference process is as follows: First, based on the source data identifier "external macro data," supplier information is retrieved to determine that the data belongs to externally procured raw market data; second, the data layering specifications are queried, revealing that the standard landing level for external raw data is "source-attached layer"; finally, combined with historical operation statistics, it is found that 98% of similar data synchronization tasks choose "source-attached layer" as the target. Based on the above multiple pieces of evidence, the system calculates the confidence level of "source-attached layer" as the default value to be 0.98, far exceeding the preset threshold of 0.8. Therefore, the system automatically fills the target storage level parameter with "source-attached layer" without interrupting the user's inquiry. Conversely, if the user does not specify a "primary key field" and the data table has multiple candidate primary keys (such as "securities code" and "supplier code"), the system cannot uniquely determine it through metadata, and the confidence level is lower than the threshold. In this case, the system will generate a prompt message: "Multiple identifier fields have been detected in this table. Please confirm whether you want to use 'securities code' or 'supplier code' as the synchronization primary key?", guiding the user to complete the completion. This intelligent completion mechanism ensures the integrity of API calls while maintaining a smooth experience of natural language interaction to the greatest extent. Furthermore, when the interface call sequence mapping module 20 transforms the structured task topology into a platform interface call sequence, it generates the interface call sequence in the following ways: (a) Mapping each node in the structured task topology to a predefined parameterized template, with each template corresponding to a standard interface call structure for a subtask type; (b) Filling the template parameters with specific information (table name, field list, data type) from the business metadata, making the abstract template instantiated into an interface call instruction for a specific data object; (c) Translating the user-described conversion logic into the parameter expressions required by the interface, such as translating "remove null values" into a filtering condition. JSON expression; (d) Generate scheduling configuration information based on the dependencies between nodes, and define the execution order and concurrency strategy of interface calls; (e) In NL2API mode, convert the above mapping results into a sequence of call requests to existing interfaces of the target business platform: each data processing node corresponds to a set of interface calls (such as creating a data synchronization task interface, configuring cleaning rules interface, creating a quality verification task interface, and creating a scheduling task interface). The dependencies between nodes are realized through the interface call order and parameter passing. The parameters of the interface calls are automatically filled by the slot extraction results and business metadata, without requiring the user to manually fill in any interface parameters.

[0060] As one implementation method, the aforementioned interface path matching and parameter completion process manifests as an ordered API call sequence generation flow in actual execution. Specifically, the user's natural language input and the data platform API registry serve as the input context for the large language model (NL2API parsing). The model first outputs an intent parsing result containing the intent type, slot value, and logical parameters. Subsequently, based on this parsing result, the system generates a series of specific HTTP request instructions in sequence. For example, for the requirement of "cleaning and integrating macro indicators of the first and second data sources," the system generates the following in sequence: ①POST / api / dict / query (query the encoded dictionary), ②GET / api / meta / table / schema (get the table structure), ③POST / api / sync / create (create the first data source synchronization task), ④POST / api / sync / create (create the second data source synchronization task), ⑤POST / api / clean / config (configure cleaning rules), etc. Each API call box contains a complete JSON request body after path matching and parameter completion. These API call sequences are ultimately sent to the data platform functional modules for execution, thus completing the construction of the workflow from natural language to actual data processing. It should be understood that in practical applications, the system will dynamically adjust the types, number, and parameter content of API calls according to the specific task topology and business metadata. As long as the mapping and completion logic described in this application is followed, it falls within the protection scope of this application.

[0061] As one implementation method, the data processing workflow construction module 30 includes a data verification rule determination unit and a verification node embedding unit. These two units together constitute the built-in quality core of the system, which transforms abstract data quality assurance requirements into concrete, executable logical entities in the workflow. This transformation process is entirely driven by business metadata, rather than relying on manual post-configuration.

[0062] Specifically, the data verification rule determination unit is used to determine the data verification rule type based on the data classification features in the business metadata. Here, the data classification features serve as a bridge connecting business semantics and technical verification; they are not single-dimensional labels but rather a combination of multi-dimensional attributes, including data layer identifiers, data business categories, and data security sensitivity. In this embodiment, the system pre-defines a mapping table between data classification features and verification rule types. This mapping table defines the set of verification strategies applicable under different combinations of business attributes. For example, when the layer identifier in the data classification feature is "Source Layer" and the business category is "External Market Data," the mapping table points to a combination of "Timeliness Verification" and "Integrity Verification," focusing on monitoring source update delays and the proportion of extracted rows. Conversely, when the layer identifier is "Model Layer" and the business category is "Financial Indicators," the mapping table points to a combination of "Business Logic Verification" and "Consistency Verification," forcibly executing account reconciliation checks and cross-source data comparisons. This multi-dimensional feature-based differentiated matching mechanism enables the system to automatically adapt the most appropriate quality protection strategy for each type of data. This avoids the waste of computational resources caused by generic verification rules and prevents data risks due to missing verification, ensuring the accuracy and effectiveness of quality assurance. Furthermore, the data verification rule determination unit also grades the verification intensity based on data sensitivity. The system adjusts the strictness of verification according to the data classification identifier (ordinary / important / core): ordinary data executes the default verification rule set; important data has increased verification frequency and higher sensitivity for anomaly alarm thresholds; core data, based on important data, further incorporates cross-verification and deep business logic verification, and adds strong verification measures such as double-write comparison. Through the above three-layer selection logic (data layering, data classification, data sensitivity), the system ensures that the quality verification rules are comprehensive yet non-redundant, guaranteeing data quality while avoiding unnecessary computational overhead.

[0063] Furthermore, the verification node embedding unit is used to embed data verification rules corresponding to the data verification rule type as independent verification nodes, based on execution dependencies, after the corresponding data processing node and before the data writing node. It is important to emphasize that the "independent verification node" here refers to a functional object within the directed acyclic graph (DAG) topology of the data processing workflow, where the verification rule is instantiated as a functional object with independent input / output interfaces, independent scheduling status, and independent execution logs, rather than merely an assertion statement or auxiliary logic embedded within the data processing code. In terms of specific embedding locations, the system strictly adheres to the principle of "verification after processing and verification before writing," inserting independent verification nodes downstream of the data-generating processing node and upstream of the data-consuming writing node. For example, a "cleaning quality verification" node is automatically inserted between the "data cleaning" node and the "data fusion" node; a "fusion consistency verification" node is inserted between the "data fusion" node and the "target table writing" node. This topology-level independent node design makes the verification process visible, controllable, and traceable at the workflow scheduling level. When verification fails, it can accurately block subsequent links and trigger alarms without polluting the data results already generated upstream. This achieves decoupling and collaboration between data processing logic and quality assurance logic in the architecture.

[0064] As a preferred implementation, the verification node embedding unit is configured to: synchronously mount the corresponding data verification rule configuration when generating each data processing node, and insert the data verification rule as an independent verification node into the data processing workflow when generating the data processing workflow, so that the data verification rule is synchronously embedded when the data processing workflow is generated. This configuration defines the specific technical connotation of "synchronization" and aims to solve the compliance loopholes caused by the separation of process creation and quality configuration in the prior art. Specifically, "synchronous mounting" occurs in the logic generation stage, that is, when the data processing workflow building module 30 calls the component template to generate a configuration object of a data processing node (such as a data synchronization task), the corresponding verification rule configuration object is instantiated in the same function call or memory context according to the aforementioned mapping table, and the two are bound together. This means that the verification configuration is no longer a patch added after the fact, but an inherent attribute that comes with the data processing node when it is created.

[0065] Building upon this, "synchronous insertion" occurs during the physical construction phase. When the system translates the logical topology into an actual sequence of platform interface calls to construct the data processing workflow, verification nodes and data processing nodes are considered equal members within the same transaction. The system sends the instruction to create a verification task simultaneously with the API request to create a data processing task, or within the same workflow orchestration API request body. For example, within the same transaction block that calls the ` / api / sync / create` interface to create a data synchronization node, the system automatically calls the ` / api / quality / rule / bind` interface to register the corresponding integrity verification rule and associates it with the ID of the newly created synchronization node. The transaction is only committed and effective after all associated verification nodes have been successfully created and bound; if any verification node creation fails, the entire workflow construction transaction is rolled back. This atomic synchronization mechanism fundamentally eliminates the possibility of deploying a half-finished workflow with "processing without verification," truly achieving the built-in quality goal of "no deployment without verification," and significantly improving the compliance and reliability of the data processing system. To ensure that quality verification nodes are embedded simultaneously with process generation, the system employs a "three-synchronization" guarantee mechanism: ① Template binding synchronization: Each subtask type template has an embedded reference to the associated quality verification rule template (associated through template identifiers) during definition, eliminating the possibility of processing templates without quality configuration. The binding relationship between subtask templates and quality templates is stored in the metadata of the template library as part of the template definition, ensuring that quality rules are determined the instant the subtask is generated. ② Generation process synchronization: Both the interface call sequence mapping module 20 and the data processing workflow construction module 30 have built-in quality processing logic—quality templates are matched synchronously during interface matching, verification nodes are inserted synchronously during workflow construction, and quality API calls are generated synchronously during API mapping. Each step is executed sequentially in the same execution pipeline without external intervention, ensuring that the generation of processing nodes and verification nodes is atomic. ③ Deployment transaction synchronization: When deploying the data processing workflow, the process deployment and scheduling interface registers the processing task and the quality verification task as the same data process. If the processing task is successfully deployed but the quality verification task fails, the entire process deployment is rolled back, ensuring that there is no unfinished process with "only processing and no verification". The transaction is committed and the process officially takes effect only when all processing nodes and all verification nodes have been successfully deployed.

[0066] As one implementation, the business metadata includes at least a data dictionary and model definition information; the data processing workflow construction module 30 also includes a semantic alignment unit, which is used to generate a mapping relationship between fields from different data sources to a unified encoding based on the unified encoding system in the data dictionary when the structured task topology contains multiple heterogeneous data sources, and to align fields with different names but the same semantics based on the semantic recognition model.

[0067] Specifically, in complex data processing scenarios such as financial investment research, structured task topologies often involve heterogeneous data sources from multiple vendors or business systems. These data sources differ significantly in field naming conventions, encoding standards, and data formats, and direct fusion can lead to semantic conflicts or association failures. This embodiment solves this technical challenge through a semantic alignment unit. The data dictionary is not simply a list of field names, but an authoritative knowledge base containing an enterprise-level or platform-level unified indicator encoding system. This system defines universal semantic identifiers across data sources and their correspondence with each vendor's proprietary encoding. Model definition information describes the standard data structure constraints of the target data layer (such as the standard layer and model layer). When performing alignment operations, the semantic alignment unit first uses the unified encoding system in the data dictionary as a semantic anchor to map fields representing the same business entity from different data sources (such as stock codes and industry classification codes from different vendors) to a unified internal encoding space, thereby eliminating heterogeneity at the encoding level.

[0068] Furthermore, to address the semantic ambiguity caused by non-standard field naming, the semantic alignment unit employs an intelligent alignment strategy based on a semantic recognition model, specifically handling fields with different names but identical semantics. This strategy goes beyond traditional keyword matching or regular expression rules, instead using a dual verification mechanism of "vector similarity calculation + contextual feature verification." The specific execution process is as follows: The semantic recognition model first converts the source and target field names to be aligned into high-dimensional semantic vectors, calculates their cosine similarity in the semantic space, and filters out candidate matching pairs with similarity higher than a preset threshold. Subsequently, the system introduces contextual features for secondary verification. These contextual features include, but are not limited to, the data type of the field, the statistical distribution characteristics of historical data samples, the upstream and downstream position of the field in the data lineage, and the business description text of the data table to which it belongs. Only when the candidate matching pair meets the consistency condition in both the semantic vector and contextual feature dimensions is the field mapping relationship finally established. This dual verification mechanism effectively avoids incorrect alignment caused by coincidental field naming (such as fields with the same name but different meanings in different business domains), significantly improving the accuracy and automation level of heterogeneous data fusion.

[0069] To illustrate the semantic alignment mechanism more clearly, a specific cross-source macroeconomic indicator data fusion scenario is used as an example. Assume the structured task topology includes macroeconomic indicator data from a first data source and similar indicator data from a second data source, and both need to be merged and written into a unified model layer. In this scenario, the date field of the first data source is named "trade_date," while the corresponding field of the second data source is named "transaction date." Simultaneously, the first data source uses the "first industry classification code" to identify the industry classification, and the second data source uses the "second industry classification code."

[0070] The semantic alignment unit's processing flow is as follows: First, for the date field, the semantic recognition model encodes "trade_date" and "transaction date" into semantic vectors, calculating a similarity of 0.96, exceeding the preset threshold of 0.85. Next, the system reads the contextual features of the two fields, finding that both are of data type "DATE" and serve as primary key timestamps in their respective data tables, with a 99% overlap in the time range of the sample data. Based on the dual verification of vector similarity and context, the system automatically establishes a mapping relationship between "trade_date" and "transaction date" to the unified model layer standard field "stat_date". Second, for the industry classification field, since "first industry classification code" and "second industry classification code" have no direct similarity in literal and numerical terms, the semantic alignment unit queries the unified coding system in the data dictionary. The system retrieves the "unified industry classification standard" mapping table in the data dictionary, which clearly records that both the first industry classification code "A010101" and the second industry classification code "430019" correspond to the unified code "IND_AGR_FORESTRY". Accordingly, the system automatically generates the conversion logic from two source encodings to a unified encoding and encapsulates it as the encoding mapping node configuration parameters in the data processing workflow. Through the above process, the system completes the semantic understanding and standardized conversion of cross-source fields without manual intervention, enabling subsequent data fusion nodes to correctly perform association and aggregation operations based on a unified semantic space, fundamentally ensuring the quality and efficiency of multi-source heterogeneous data fusion. Furthermore, the data processing workflow construction module 30 also includes a data conflict resolution unit, which automatically resolves conflicts based on predefined priority rules or user-specified strategies when multiple source data have different values ​​for the same indicator. Specifically, the data conflict resolution unit pre-configures various conflict resolution strategies, including but not limited to: source data priority strategies (such as specifying data from a certain data source as priority), timestamp priority strategies (taking the latest updated value), and numerical aggregation strategies (taking the average, maximum, or minimum value), etc. When the system detects conflicting values ​​for the same indicator across different data sources, it first queries predefined priority rules. If no priority rules are configured, it queries the user for a resolution strategy via natural language interaction. For example, it generates a prompt message: "The indicator 'GDP growth rate' has a value of 6.5% in the first data source and 6.3% in the second data source. Please confirm which data source's value to use, or specify another resolution strategy." The user can reply in natural language with "Use the value from the first data source" or "Take the average of the two values," and the system will then perform the corresponding conflict resolution operation. This intelligent conflict resolution mechanism effectively solves the common data inconsistency problem in multi-source data fusion, avoiding the inefficient operation of manual item-by-item verification and adjustment.

[0071] As one implementation, the system also includes an interactive correction module 40, which generates natural language prompts containing the location of the error node and correction suggestions when the data verification rules determine that there is a logical error in the data processing workflow; receives correction instructions based on the natural language prompts, and only updates the interface call parameters of the affected nodes without regenerating the entire structured task topology. This mechanism constitutes the system's closed-loop optimization capability after the process is built. Its core value lies in compressing the lengthy chain of "error reporting - manual investigation - code / configuration modification - resubmission" in traditional ETL development into an agile interactive mode of "natural language prompts - conversational correction - immediate effect". At the same time, the partial update strategy avoids the waste of resources and the risk of state loss caused by triggering a full rebuild due to minor adjustments.

[0072] Specifically, the process of generating natural language prompts containing error node locations and correction suggestions is not a simple transmission of error codes, but a structured translation based on pre-defined error semantic templates. The interactive correction module 40 maintains an error description template library. Each template defines a natural language expression paradigm for a specific error type, including the location identifier of the error, a simplified explanation of the error cause, the names of affected parameters, and the system's recommended correction scheme. When the data validation rule returns a validation failure result, the interactive correction module 40 first parses the structured error object in the validation result, extracting the error node ID, error type code, and abnormal parameter values. Then, it matches the corresponding template in the error description template library and fills the extracted structured information into the template slots, ultimately generating natural language text that conforms to human reading habits. For example, when the validation rule detects a missing target field in the "Code Mapping" node, the system will not directly throw technical exceptions such as "NullPointerException at Node_3" or "FieldMappingError: target _col not found." Instead, it will generate the following message: "In step 3, the 'Code Mapping' node, it was detected that the source field 'industry_code' did not find a corresponding column in the target model table. According to the data dictionary recommendation, it is suggested to map it to the 'industry classification code' field, or you can specify other target fields." This expression not only accurately locates the topological position of the problem (step 3 node) but also provides actionable correction suggestions based on business metadata, significantly reducing the cognitive load for users to understand errors and devise solutions.

[0073] Furthermore, the key technical feature that distinguishes this embodiment from the full regeneration scheme in the prior art is that it receives correction instructions based on natural language prompts and only updates the interface call parameters of the affected nodes without regenerating the entire structured task topology. In specific implementation, after receiving the user's correction instruction, the interactive correction module 40 first uses a semantic parsing model to perform intent recognition and slot extraction on the instruction, transforming it into a parameter update request for a specific node. Since a complete structured task topology already exists in the system, the processing of the correction instruction no longer goes through the topology generation stage but directly enters the topology editing stage. The system locates the target node in the existing topology based on the node identifier carried in the correction instruction (explicitly specified or implicitly determined through semantic matching), and then only performs incremental update operations (such as parameter replacement or appending under HTTP PATCH semantics) on the interface call parameters associated with that node, while keeping the definitions of other nodes in the topology, the execution dependencies between nodes, and the global configuration completely unchanged. For example, in response to the aforementioned mapping error of "industry classification code", the user replied "change to mapping to 'second industry classification code'", the system parsed out the target node as "encoding mapping node", updated the parameter to "target_field=second industry classification code", and then only regenerated the API call parameters corresponding to the node and triggered local deployment. The DAG structure of the entire structured task topology and the configuration of other nodes remained unchanged.

[0074] This partial update mechanism offers significant technical advantages over full regeneration. From a computational resource perspective, full regeneration requires re-executing the entire process, including semantic parsing, topology construction, dependency derivation, and end-to-end verification, consuming substantial computing power and time. Partial updates, on the other hand, only need to handle parameter changes and local verifications for a single node, reducing computational overhead by more than an order of magnitude. From a state consistency perspective, full regeneration may experience unexpected drift in unmodified parts due to the ambiguity of natural language or the randomness of model inference (e.g., fine-tuning of previously correct cleaning rules), disrupting the intermediate state confirmed by the user. Partial updates, however, strictly limit the scope of modification, ensuring the absolute stability of the remaining parts of the topology, meeting the safety requirements of the "minimum change principle" in engineering practice. From an interactive experience perspective, partial updates provide millisecond-level response feedback, enabling users to quickly iterate and optimize workflows in continuous dialogue, rather than experiencing lengthy waiting and reconfirmation processes after each modification. This truly achieves an agile data development experience driven by natural language. It should be understood that although this embodiment uses parameter update as an example to illustrate the implementation of local correction, in other embodiments, if the correction instruction involves adding or deleting nodes or adjusting dependencies, the system can also reconstruct only the affected local subgraphs while maintaining the stability of the overall topology framework. As long as its core logic still follows the principle of "avoiding full regeneration", it falls within the protection scope of this application.

[0075] Furthermore, the interactive correction module 40 also includes a scheme comparison unit, used to display the key differences between different schemes for the user to choose from when multiple feasible correction schemes exist. Specifically, when the system detects that there are multiple feasible correction paths for a certain error, it generates comparison information containing multiple candidate schemes. Each candidate scheme includes key dimensions such as scheme description, expected scope of impact, implementation complexity, and risk assessment. For example, when a field type mismatch error is detected in the data fusion node, the system may generate the following comparison information: "Scheme 1: Add a format conversion step after the source data extraction node, with a small scope of impact but increased processing delay; Scheme 2: Modify the field type definition of the target model table, with a large scope of impact but no additional processing node required; Scheme 3: Add type adaptation logic in the fusion node, with a moderate scope of impact and reusable existing components." The user can select any scheme by replying in natural language, such as "Adopt Scheme 1," and the system will then perform the corresponding correction operation. This scheme comparison mechanism provides transparent decision support for users, enabling them to make the optimal choice based on a full understanding of the advantages and disadvantages of each scheme, rather than passively accepting a single correction suggestion from the system.

[0076] As one implementation, the system also includes a process template learning and recommendation module 50, connected to a natural language interaction interface and interface call sequence mapping module 20, comprising a historical process learning unit and an intelligent recommendation unit. The historical process learning unit performs pattern mining on created and deployed data processing workflows, extracts high-frequency processing patterns, and abstracts them into reusable process templates stored in a predefined subtask type template library. The intelligent recommendation unit, when a user inputs a natural language requirement, matches historical process templates based on the requirement characteristics and recommends the template with the highest matching degree to the user as a starting point. The user can then customize the recommended template using natural language descriptions. This recommendation mechanism based on historical process pattern mining achieves continuous accumulation and reuse of data processing knowledge, avoids redundant configuration of similar data processing scenarios, and significantly improves the efficiency and consistency of workflow creation.

[0077] Example 2:

[0078] like Figure 3 As shown, this embodiment provides a method for generating a data processing workflow. This method is based on the collaborative execution of the modules in the aforementioned system embodiment, focusing on explaining the temporal logic and data state transitions between each processing stage from a time perspective. Specifically, the method includes the following steps.

[0079] In step S100, the received natural language requirements are parsed using a semantic parsing model to generate a structured task topology containing execution dependencies. In this step, the system first receives a data processing requirement description input by the user in natural language, such as "clean and merge the macroeconomic indicator data from the first and second data sources and write it into the model layer." The semantic parsing model processes this text, identifying the user's intent category (e.g., creating a new workflow), key entity slots (source data identifier, target data identifier, transformation logic, etc.), and implicit quality constraints. Subsequently, based on a predefined subtask type template library and a data flow inference mechanism, the system transforms the unstructured text into a computer-processable structured task topology. This structured task topology is typically represented as a directed acyclic graph (DAG) or task planning tree, where nodes represent abstract data processing subtasks (e.g., data extraction, data cleaning, encoding mapping), and edges represent execution dependencies and data transfer paths between subtasks. It is important to emphasize that this stage only generates an intermediate topology describing the business logic and does not contain any platform-specific code syntax or specific interface parameters, thus decoupling the expression of business intent from the underlying technical implementation.

[0080] In step S200, by combining the business metadata of the target business platform, the structured task topology is mapped into a sequential sequence of platform interface calls. In this step, the system reads the target business platform's business metadata, which includes at least an interface registry, a data dictionary, and model definition information. For each logical node in the structured task topology, the system retrieves the platform interface path that matches its semantics from the interface registry and, based on the field definitions and constraints in the data dictionary, converts the abstract slot parameters in the topology into specific request parameters that conform to the interface specifications. For example, for the "data cleaning" node in the topology, the system matches the cleaning configuration API provided by the platform and converts the abstract logic of "removing null values" into the JSON-formatted filtering expression required by the API. Finally, the system orchestrates the interface calls of all nodes into an ordered request sequence according to the execution dependencies in the topology. This mapping mechanism based on business metadata ensures that the generated interface call sequence fully complies with the target platform's control specifications and capability boundaries, avoiding the compliance risks and maintenance difficulties that might arise from directly generating code.

[0081] S300 sequentially triggers a series of platform interface calls to construct a data processing workflow, automatically embedding data validation rules into the nodes of the workflow based on business metadata. In this step, the system sends requests in the interface call sequence to the target business platform in sequence, actually creating data synchronization tasks, configuring transformation rules, and setting scheduling strategies, thereby assembling an executable data processing workflow. Simultaneously, during the construction process, the system reads data classification characteristics (such as data layer identifiers, business categories, and sensitivity levels) from the business metadata in real time, determining the appropriate data validation rule type for the currently constructed data processing node. Subsequently, the system instantiates the determined data validation rule into an independent validation node and inserts it after the corresponding data processing node and before the data writing node in the data processing workflow. For example, in the same transaction where an API is called to create a "financial indicator cleaning" task, the system automatically calls the quality validation API to register a "subject reconciliation verification" rule and associates this verification task downstream of the newly created cleaning task. This synchronous embedding mechanism ensures that data processing logic and quality assurance logic are completed in the same construction process, fundamentally eliminating the possibility of deploying a workflow without validation.

[0082] S400 performs syntax and logic validation on the completed data processing workflow. The system checks whether the generated interface call parameters are complete, whether the data types are compatible, and whether upstream and downstream dependencies are connected. If the validation passes, proceed to S600; if errors are found, proceed to S500.

[0083] The S500 executes an interactive correction process. When verification fails or the user is dissatisfied with the generated results, the system generates natural language prompts containing the location of the faulty node and correction suggestions, and receives correction commands input by the user. The system only updates the interface call parameters of the affected nodes, without regenerating the entire structured task topology, thus achieving rapid local iteration.

[0084] The S600 deploys validated data processing workflows to the scheduling engine of the target business platform. The system automatically configures the execution frequency, dependency triggering conditions, and alarm notification policies, and registers the workflow metadata with the platform's data asset management system, formally integrating it into the production environment's monitoring and governance system.

[0085] The S700 responds to user natural language management commands and performs full lifecycle maintenance on deployed data processing workflows. Users can query workflow running status, modify scheduling parameters, or add new data sources using natural language. The system parses these commands and reuses the aforementioned mapping and construction logic to achieve continuous optimization and management of existing workflows.

[0086] Through the above-described method and process, this embodiment achieves end-to-end automated generation of a workflow from natural language requirements to compliant data processing. Compared to existing technologies, this method uses a structured task topology as an intermediate state, combined with platform interface mapping driven by business metadata. This retains the convenience of natural language interaction while ensuring strict adherence of the generated results to the target platform's control system. Simultaneously, by embedding data verification rules synchronously during the construction phase, quality assurance is shifted to the development stage, significantly improving the reliability of data processing and operational efficiency. It should be understood that although this embodiment describes the method and process in a specific order of steps, in practical applications, some steps (such as verification and embedding) may be executed in parallel or iteratively. As long as their core logic still follows the parsing, mapping, construction, and embedding mechanisms described in this application, they all fall within the scope of protection of this application.

[0087] As one implementation method, in S400, when the data verification rules determine that there is a logical error in the data processing workflow, a natural language prompt message containing the location of the error node and correction suggestions is generated; a correction instruction based on the natural language prompt message is received, and only the interface call parameters of the affected nodes are updated, without regenerating the entire structured task topology. This step constitutes an iterative optimization closed loop at the method level, enabling the system to complete the repair and adjustment of the workflow with minimal computational cost when encountering logical errors or user changes in requirements, avoiding the resource waste of full-link reconstruction triggered by local errors in traditional methods.

[0088] Specifically, the process of generating natural language prompts containing error node location and correction suggestions manifests as a structured exception translation step in the method execution flow. When the verification operation in S400 returns a failure result, the system does not directly throw the underlying technical exception stack. Instead, it first parses the verification result object, extracting the node identifier where the error occurred, the error type code, and the relevant exception parameter values. Subsequently, the system queries a pre-built error semantic template library and fills the above structured information into the corresponding natural language template slots. For example, if the verification rule detects a mismatch in the input field type of a data fusion node, the system will generate prompt text such as "In the 'Macroeconomic Indicator Fusion' node, the source field 'trade_date' is a string type, but the target model requires a date type. It is recommended to add a format conversion step or change the source field." This processing method transforms abstract topology errors into business language that users can understand and clearly indicates the topology location where the error occurred, providing necessary contextual anchors for subsequent accurate correction.

[0089] Furthermore, the core improvement of this embodiment's method flow lies in receiving correction instructions based on natural language prompts, updating only the interface call parameters of the affected nodes, without regenerating the entire structured task topology. In specific execution, when the system receives the user's correction instruction, it first identifies the target node identifier and the new parameter value in the instruction using a semantic parsing model. Since a complete structured task topology instance already resides in memory, the system directly locates the target node on the existing topology object and performs incremental update operations on its associated interface call parameters. Here, "incremental update" refers to replacing or appending only the key-value pairs of specific parameters, while keeping the directed edge connections between nodes in the topology graph, the configuration attributes of other nodes, and the global scheduling strategy completely unchanged. For example, regarding the aforementioned field type mismatch issue, if the user replies "add a date formatting step," the system inserts a new format conversion node configuration before the fusion node and updates the input parameters of the fusion node to point to this new node, without re-executing the topology generation steps such as intent classification, slot extraction, and dependency building.

[0090] The feasibility of this local update mechanism, which avoids reverting to the topology generation step, stems from the fact that this application decouples the workflow's "logical structure" from its "operational parameters" in a layered manner. The structured task topology defines the skeleton and flow of data processing, exhibiting relative stability; while the interface call parameters are the flesh and blood attached to the skeleton, possessing high-frequency variability. When the correction instruction only involves parameter adjustment or minor node insertion / removal on the established skeleton, the system determines that it does not touch the core dependency logic of the topology, thus skipping the time-consuming topology reconstruction stage and directly entering the parameter injection and local verification stage. Compared to the full regeneration method, this branch process reduces the computational complexity of the correction operation from O(N) to O(1) or O(k) (where k is the number of affected nodes), significantly shortening the interaction response time. At the same time, by maintaining the stability of the main topology, it effectively avoids the risk of unexpected drift that may be introduced by re-inference of large models, ensuring that the business logic confirmed by the user in multiple rounds of dialogue is not tampered with, thereby achieving efficient, secure, and controllable iterative optimization of the data processing workflow at the method level.

[0091] Example 3:

[0092] To more clearly illustrate the application of the technical solution of this application in actual business scenarios, this embodiment takes the fusion of multi-source macroeconomic indicator data, which is common in the field of financial investment research, as an example to fully demonstrate the generation and iteration process of the data processing workflow. In this scenario, the user inputs the following natural language requirement through a natural language interaction interface: "Clean and fuse the macroeconomic indicator data from the first data source and the second data source and write it into the model layer." This requirement describes a typical cross-source heterogeneous data ETL task, involving multiple data providers, cleaning and transformation logic, and a clear target storage layer, which can fully verify the collaborative working capability of the various modules of this application.

[0093] First, the structured task topology generation module 10 receives the aforementioned natural language request and parses it using a semantic parsing model. Specifically, the intent classification result generation unit identifies the request as the intent to "create a new data processing workflow"; the slot extraction result acquisition unit extracts the source data, identified as "first data source macroeconomic indicator source table" and "second data source macroeconomic indicator source table," and the target data, identified as "unified model layer - macroeconomic indicator table." The transformation logic includes "cleaning," "fusion," and the implicit "encoding mapping." Subsequently, the execution dependency construction unit, based on the data flow logic between the slot extraction results, constructs a structured task topology containing six logical nodes: "first data source data extraction," "second data source data extraction," "data cleaning," "encoding mapping," "data fusion," and "model layer writing," and establishes the execution dependency relationship of "dual-path extraction parallel → cleaning → mapping → fusion → writing." During this process, the topology anomaly detection and repair unit automatically performs circular dependency detection on the structured task topology, confirming that there are no loops in the graph, thereby ensuring the executability of the topology. It should be understood that although the loop resolution operation was not triggered in this example, this detection mechanism, as the system's default security barrier, will be forcibly executed every time a topology is generated to prevent logical deadlocks caused by natural language ambiguity.

[0094] Next, the interface call sequence mapping module 20, in conjunction with the business metadata of the target business platform, maps the aforementioned structured task topology into a sequential sequence of platform interface calls. During this mapping process, the interface path matching unit queries the interface registry, matching the "first data source data extraction" node as the / api / sync / create interface, and the "data cleaning" node as the / api / clean / config interface, etc. For the "model layer write" node, the missing parameter completion unit detects that the user has not explicitly specified the required "write mode" parameter. Based on the model definition information in the business metadata, it infers that since the target table is a macroeconomic indicator table and belongs to the core model layer, the system infers the default write mode as "upsert (primary key update)" according to the preset data layering specifications, with a confidence level of 0.95, which is higher than the preset threshold. Therefore, the parameter is automatically completed without interrupting user interaction. This metadata-based intelligent completion mechanism ensures the integrity of API calls while maintaining a smooth natural language dialogue experience.

[0095] During the construction of the data processing workflow, the semantic alignment unit in the data processing workflow construction module 30 automatically performed semantic alignment operations to address the field differences between the first and second heterogeneous data sources. Specifically, based on the unified coding system in the data dictionary, the system generated a mapping relationship from the "first industry classification code" and "second industry classification code" to the platform's unified industry code. Simultaneously, based on the semantic recognition model, it was found that the semantic vector similarity between the first data source field "trade_date" and the second data source field "transaction date" was 0.96, and both were of data type DATE with a 99% overlap in their sample time ranges. Therefore, it was determined that they were semantically identical, and a mapping relationship to the unified model layer standard field "stat_date" was automatically established. This process eliminates the need for manual configuration item by item, effectively solving the problem of association failure caused by non-standard naming in cross-source data fusion.

[0096] Furthermore, when the data processing workflow construction module 30 triggers the platform interface call sequence in sequence to construct the data processing workflow, it automatically embeds data verification rules into the nodes based on business metadata. Specifically, the data verification rule determination unit determines two rule types, "integrity verification" and "consistency verification," based on the data classification characteristic of "macro indicators." The verification node embedding unit synchronously mounts the corresponding verification configuration in the same transaction that generates the "data fusion" node, and inserts "fusion consistency verification" as an independent verification node after the "data fusion" node and before the "model layer write" node. This means that when the data processing workflow is generated, the quality verification rules are synchronously embedded and take effect, ensuring that subsequent data write operations must undergo consistency verification, achieving atomic integration of quality assurance and process construction.

[0097] Finally, assuming that during the validation phase before workflow deployment, the system detects a logical error in the "Encoding Mapping" node where a target field is missing, the interactive correction module 40 immediately generates a natural language prompt message containing the location of the error node and correction suggestions: "In step 3, the 'Encoding Mapping' node, the source field 'industry_code' was found to have no corresponding column in the target model table. It is recommended to map it to the 'Industry Classification Code' field." Based on this prompt, the user inputs a correction instruction: "Change to mapping to 'Second Industry Classification Code'." After receiving this instruction, the system only locates the "Encoding Mapping" node in the existing structured task topology, updates the target_field value in its interface call parameter to "Second Industry Classification Code", while keeping the definitions and execution dependencies of the other five nodes in the topology completely unchanged. Subsequently, it only regenerates the API call corresponding to this node and completes the partial deployment. Compared to regenerating the entire workflow, this partial update mechanism reduces the correction response time from tens of seconds to milliseconds and avoids the risk of unexpected drift that may be introduced by re-inference of large models, significantly improving the iteration efficiency and operational safety of users in complex data development scenarios.

[0098] As demonstrated by the complete process described above, the technical solution provided in this application can transform a simple natural language requirement into a compliant data processing workflow that includes heterogeneous data alignment, built-in quality control, and interactive correction capabilities. This application example not only verifies the specific implementation methods of each module's functions but also reflects the technical advantages of this application in solving the multi-source data fusion problem in the financial investment research field. Specifically, through the dual drive of structured task topology and business metadata, it achieves an organic unity between the convenience of natural language interaction and the rigor of enterprise-level data management.

[0099] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A data processing workflow generation system, characterized in that, include: The structured task topology generation module is used to parse the received natural language requirements through a semantic parsing model and generate a structured task topology containing execution dependencies. The interface call sequence mapping module is used to combine the business metadata of the target business platform to map the structured task topology into a sequential platform interface call sequence; The data processing workflow construction module is used to trigger the platform interface call sequence in sequence to construct the data processing workflow, and automatically embed data verification rules in the nodes of the data processing workflow based on the business metadata.

2. The system according to claim 1, characterized in that, The structured task topology generation module includes: The intent classification result generation unit is used to classify the natural language request through the semantic parsing model to obtain the intent classification result. The slot extraction result acquisition unit is used to extract source data identifier, target data identifier and conversion logic from the natural language requirements as slot extraction results. An execution dependency construction unit is used to construct the execution dependency based on the data flow logic between the slot extraction results.

3. The system according to claim 2, characterized in that, The structured task topology generation module also includes a topology anomaly detection and repair unit, which performs circular dependency detection on the structured task topology. When a circular dependency is detected, the circular dependency is resolved based on a predefined subtask type stage sequence until there are no cycles in the structured task topology.

4. The system according to claim 2, characterized in that, The interface call sequence mapping module includes: The interface path matching unit is used to match the intent classification result and the slot extraction result into an interface path and request parameters based on the interface registry of the target business platform. The missing parameter completion unit is used to infer a default value based on the business metadata or to trigger multiple rounds of natural language interaction to complete the missing required parameter when it is determined that there is a missing required parameter in the request parameters.

5. The system according to claim 1, characterized in that, The data processing workflow construction module includes: The data verification rule determination unit is used to determine the data verification rule type based on the data classification characteristics in the business metadata. The verification node embedding unit is used to embed the data verification rule corresponding to the data verification rule type as an independent verification node after the corresponding data processing node and before the data writing node, according to the execution dependency relationship.

6. The system according to claim 5, characterized in that, The verification node embedding unit is configured to: synchronously mount the corresponding data verification rule configuration when generating each data processing node, and insert the data verification rule as an independent verification node into the data processing workflow when generating the data processing workflow, so that the data verification rule is synchronously embedded when the data processing workflow is generated.

7. The system according to claim 5, characterized in that, The business metadata includes at least a data dictionary and model definition information; The data processing workflow construction module also includes a semantic alignment unit, which is used to generate a mapping relationship between fields from different data sources to a unified encoding based on the unified encoding system in the data dictionary when the structured task topology contains multiple heterogeneous data sources, and to align fields with different names but the same semantics based on a semantic recognition model.

8. The system according to claim 1, characterized in that, The system also includes an interactive correction module, which generates natural language prompts containing error node location and correction suggestions when the data verification rules determine that there is a logical error in the data processing workflow. Upon receiving a correction instruction based on the natural language prompt information, only the interface call parameters of the affected nodes are updated, without regenerating the entire structured task topology.

9. A method for generating a data processing workflow, characterized in that, include: The received natural language requirements are parsed using a semantic parsing model to generate a structured task topology containing execution dependencies. By combining the business metadata of the target business platform, the structured task topology is mapped into a sequential sequence of platform interface calls; The platform interface call sequence is triggered sequentially to build a data processing workflow, and data verification rules are automatically embedded in the nodes of the data processing workflow based on the business metadata.

10. The method according to claim 9, characterized in that, The method further includes: When the data verification rule determines that there is a logical error in the data processing workflow, a natural language prompt message containing the location of the error node and correction suggestions is generated; Upon receiving a correction instruction based on the natural language prompt information, only the interface call parameters of the affected nodes are updated, without regenerating the entire structured task topology.

Images