A method, device and equipment for scheduling workflow automatic orchestration

Abstract

The application provides a scheduling workflow automatic arrangement method, device and equipment, the method comprises: obtaining the full amount of task nodes of the scheduling platform and its metadata, and extracting information therefrom, and generating an initial business semantic description of the task nodes; forming a business semantic cluster of the task nodes according to the colloquial expression mode mined from the historical workflow data of the task nodes and the initial business semantic description, and generating a structured mapping table in combination with the business semantic cluster, the node ID active technician data template path generation structure; retrieving a matching similar historical case and enterprise specification fragment according to the arrangement requirement, and generating an enhanced prompt therefrom; based on the generated data, the first layer CoT instruction and the second layer CoT instruction, using a large language model to determine the candidate nodes related to the arrangement requirement and the dependency relationship between the candidate nodes in the full amount of task nodes; and generating a structured DAG model object based on the candidate nodes and the dependency relationship between the candidate nodes.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus and equipment for automatically orchestrating scheduling workflows. Background Technology

[0002] The current workflow scheduling and orchestration implementation relies entirely on manual operation by technical personnel. The specific process is as follows: technical personnel manually receive verbal business requirements, manually drag and drop various task nodes visually within the scheduling platform, manually fill in the technical configuration parameters for each node, manually build upstream and downstream dependencies, branch logic, and trigger preconditions for each node, and after completing the workflow DAG construction, manually verify the configuration and logic for errors. Only after repeated debugging and modifications can it be released online. In actual enterprise scheduling scenarios, in addition to general standard task nodes, there are also a large number of customized non-standard heterogeneous nodes such as branch judgments, cross-flow dependencies, pre-triggered conditions, and third-party data exchange. These types of nodes lack a unified business semantic description and can only be configured and used by technical personnel.

[0003] The existing technology has the following core shortcomings: First, workflow orchestration heavily relies on technical professionals. Business personnel cannot directly express requirements and generate workflows through natural language, leading to a disconnect between business and technology, significant deviations in requirement transmission, and extremely low orchestration iteration efficiency. Second, the general-purpose large model only has basic text generation capabilities and cannot identify various customized non-standard heterogeneous nodes of enterprises. It does not understand node configuration rules and orchestration constraints, resulting in garbled workflow parameters and logical failures, making it only formally compliant but unusable in practice. Third, after the large model generates workflows, there is no dedicated pre-processing error correction and verification step. It relies entirely on manual verification of configuration defects and topology logic errors, which is labor-intensive and prone to omissions, leading to deployment failures. Fourth, it cannot adapt to different scheduling system architectures. Some scheduling platforms lack node registration mechanisms, and traditional AI orchestration methods cannot automatically identify all nodes, resulting in poor adaptability and high implementation difficulty. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of the present invention provide an automatic scheduling workflow orchestration method, comprising: A dual-mode fallback detection mechanism is adopted to detect all task nodes and their metadata on different types of scheduling platforms; Based on the detected data, specific information is extracted, and based on the extracted information, an initial business semantic description for each task node is generated. Based on the historical workflow DAG definition file, historical business requirements, and work order text of the task node, multiple colloquial expressions of the task node when it is referenced in the real business context are automatically mined. Based on the colloquial expression and initial business semantic description, a corresponding business semantic cluster is formed for the task node, and a structured mapping table is generated by combining the business semantic cluster, node ID, and active technician data template path. In response to the user's input of an orchestration request, similar historical cases and enterprise specification fragments that match the orchestration request are retrieved, and enhanced prompts are generated based on them; Based on the orchestration requirements, enhanced hints, structured mapping tables, first-layer CoT instructions, and second-layer CoT instructions, a large language model is used to determine candidate nodes related to the orchestration requirements and the dependencies between candidate nodes in all task nodes. The first-layer CoT instructions are used to extract requirement information from the enhanced hints, and the second-layer CoT instructions are used to determine the dependencies between candidate nodes. A complete DAG model object is generated based on the candidate nodes and the dependencies between them.

[0005] In one embodiment, the method further includes: For each task node in the DAG model object, a verification function that matches the orchestration specification constraints is used to perform the verification. Static graph analysis is performed on the DAG model object to automatically identify and intercept specified types of errors.

[0006] In one embodiment, the method further includes: The DAG model object is driven to run in an isolated shadow environment using Mock operators to simulate state transitions and data lineage transmission, and to verify branch jump logic and upstream and downstream field compatibility.

[0007] In one embodiment, the step of determining candidate nodes related to the orchestration requirements and the dependencies between candidate nodes in all task nodes based on the orchestration requirements, enhanced hints, structured mapping tables, first-layer CoT instructions, and second-layer large language models includes: The orchestration requirements and enhanced prompts are input into the large language model, along with the constructed first-layer CoT instructions, which guide the large language model to identify and extract key elements from the orchestration requirements in a structured manner. The key elements include at least one of the following: target data entity list, data operation type, pre-trigger condition, and post-action. Based on the key elements, semantic similarity matching is performed in the structured mapping table to determine candidate nodes related to the orchestration requirements; The key elements, candidate nodes, enhanced hints, and pre-built second-layer CoT instructions are input into the large language model to guide the large language model in determining the dependencies between candidate nodes.

[0008] In one embodiment, generating a structurally complete DAG model object based on the candidate nodes and the dependencies between them includes: The key elements are matched and filled into the pre-selected node metadata template; By combining the node metadata template filled with the key elements, candidate nodes, and the dependencies between candidate nodes, a structurally complete DAG model object is generated.

[0009] In one embodiment, the dual-mode fallback detection mechanism for detecting all task nodes and their metadata on different types of scheduling platforms includes: For scheduling platforms with node registration and aggregation capabilities, all registered and active task nodes and their metadata can be read directly through the platform API. For scheduling platforms that do not have the ability to register and aggregate nodes, the system automatically traverses all the definition files of the online workflows of the scheduling platform and parses them to obtain task node configuration samples. Density-based spatial clustering algorithms cluster vectorized data of task node configuration samples to determine node types; Based on the node types and task node configuration samples, a full set of task nodes and their metadata are determined.

[0010] In one embodiment, the step of extracting specified information based on the detected data and generating initial business semantic descriptions corresponding to each task node based on the extracted information includes: The detected metadata is used to extract specified information, which includes at least one of the following: node name, development notes, and historical change records; The large language model is used to generate initial business semantic descriptions for each task node based on the extracted information.

[0011] In one embodiment, responding to user-inputted orchestration requirements, retrieving similar historical cases and enterprise specification fragments matching the orchestration requirements, and generating enhanced prompts based on them, includes: A dedicated knowledge base is created by compiling historical compliance workflow definitions, orchestration specifications, and node configuration management records. The document data in the dedicated knowledge base is segmented and vectorized, and an index of the vectorized paragraphs is established; In response to an orchestration request input by the user, the orchestration request is converted into a query vector; The query vector is used to search the dedicated knowledge base to obtain similar historical cases and enterprise specification fragments that match the orchestration requirements; The enhanced prompt is generated based on the aforementioned change request and the retrieved similar historical cases and fragments of enterprise specifications.

[0012] Another embodiment of the present invention also provides an automatic scheduling workflow orchestration device, comprising: The detection module is used to detect all task nodes and their metadata on different types of scheduling platforms using a dual-mode fallback detection mechanism. The first generation module is used to extract specified information based on the detected data and generate initial business semantic descriptions for each task node based on the extracted information. The mining module is used to automatically mine various colloquial expressions of the task node when it is referenced in the real business context, based on the historical workflow DAG definition file, historical business requirements and work order text of the task node. The second generation module is used to form a business semantic cluster corresponding to the task node based on the colloquial expression and the initial business semantic description, and to generate a structured mapping table by combining the business semantic cluster, node ID, and active technician data template path. The retrieval module is used to respond to the user's input of the arrangement requirements, retrieve similar historical cases and enterprise specification fragments that match the arrangement requirements, and generate enhanced prompts based on them; The determination module is used to determine, based on the orchestration requirements, enhancement prompts, structured mapping tables, first-layer CoT instructions, and second-layer CoT instructions, the candidate nodes related to the orchestration requirements and the dependencies between the candidate nodes in the full set of task nodes using a large language model. The first-layer CoT instructions are used to extract the requirement information from the enhancement prompts, and the second-layer CoT instructions are used to determine the dependencies between the candidate nodes. The third generation module is used to generate a structurally complete DAG model object based on the candidate nodes and the dependencies between them.

[0013] Another embodiment of the present invention also provides an electronic device, comprising: One or more processors; Memory, configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the automatic orchestration method for scheduling workflows as described above.

[0014] Based on the content disclosed in the above embodiments, the beneficial effects of the solution in this application include: business personnel can directly generate compliant workflows using natural language, without relying on professional technicians to manually drag and drop configurations, significantly reducing the orchestration threshold and shortening the requirement iteration cycle; it can fully cover and identify all standard and non-standard heterogeneous nodes, and can adapt to any scheduling platform regardless of whether it has a node registration mechanism; relying on enterprise-specific knowledge base retrieval enhancement (RAG) as a mandatory constraint, combined with thought chain (CoT) step-by-step reasoning to generate candidate nodes and dependencies between candidate nodes, it can effectively suppress the illusion of large models, ensure accurate parameters and compliant dependencies, and can be used without secondary manual modification, thus completely realizing the automation of task orchestration and improving orchestration efficiency and accuracy.

[0015] Other features and advantages of this application will be set forth in the following description. The objectives and other advantages of this application can be realized and obtained through the structures particularly pointed out in the written description and drawings.

[0016] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the automatic orchestration method for scheduling workflows in an embodiment of the present invention.

[0019] Figure 2 This is a flowchart illustrating an automatic scheduling workflow orchestration method according to another embodiment of the present invention.

[0020] Figure 3 This is a flowchart illustrating the automatic orchestration method for scheduling workflows in another embodiment of the present invention.

[0021] Figure 4 This is a structural block diagram of the automatic scheduling workflow orchestration device in an embodiment of the present invention. Detailed Implementation

[0022] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but these are not intended to limit the scope of the invention.

[0023] It should be understood that various modifications can be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope of this disclosure will be apparent to those skilled in the art.

[0024] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present disclosure and, together with the general description of the disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.

[0025] These and other features of the invention will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0026] It should also be understood that although the invention has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of the invention.

[0027] The above and other aspects, features and advantages of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0028] Specific embodiments of the present disclosure are described thereafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure and can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the present disclosure. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but are merely representative of the present disclosure and are intended to teach those skilled in the art to use the present disclosure in a variety of substantially any suitable detailed structures.

[0029] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in still another embodiment,” all of which may refer to one or more of the same or different embodiments according to this disclosure.

[0030] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0031] To overcome the technical shortcomings of existing manual orchestration, such as low efficiency, non-compliant large model generation workflows, and inability to adapt to non-standard nodes, Figure 1 As shown, this embodiment of the invention provides an automatic orchestration method for scheduling workflows, including: S1: A dual-mode fallback detection mechanism is used to detect all task nodes and their metadata on different types of scheduling platforms; S2: Extract specified information based on the detected data, and generate initial business semantic descriptions for each task node based on the extracted information; S3: Based on the historical workflow DAG definition file, historical business requirements and work order text of the task node, automatically mine multiple colloquial expressions when the task node is referenced in the real business context; S4: Based on the colloquial expression and initial business semantic description, form the corresponding business semantic cluster of the task node, and generate a structured mapping table by combining the business semantic cluster, node ID, and active technician data template path; S5: In response to the user's input orchestration requirements, retrieve similar historical cases and enterprise specification fragments that match the orchestration requirements, and generate enhanced prompts based on them; S6: Based on the orchestration requirements, enhanced prompts, structured mapping table, first-layer CoT instructions, and second-layer CoT instructions, using a large language model, candidate nodes related to the orchestration requirements and the dependencies between candidate nodes are determined in all task nodes. The first-layer CoT instructions are used to extract the requirement information from the enhanced prompts, and the second-layer CoT instructions are used to determine the dependencies between candidate nodes. S7: Generate a structurally complete DAG model object based on the candidate nodes and the dependencies between them.

[0032] The solution described in this embodiment is mainly applied to distributed workflow scheduling and orchestration scenarios such as enterprise big data data warehouse processing, cross-system data exchange, and business collaborative scheduling. Its core function is to quickly transform the natural language requirements of the business side into a directed acyclic graph (DAG model) of the workflow that the scheduling system can recognize and execute.

[0033] The solution in this embodiment adopts a "logical decoupling and step-by-step implementation" technical approach. The original scheduling platform kernel code and data structure are not modified throughout the process. Node adaptation, semantic conversion, and DAG generation are completed only through an external independent logic layer. This includes first identifying all available workflow nodes for various scheduling platforms by probing different modes for different types of scheduling platforms. Then, based on a large model and the company's existing fragmented knowledge, a many-to-one mapping between nodes and business semantics is constructed. Content is generated in accordance with the company's actual orchestration specifications. Finally, a compliant DAG model is produced through step-by-step reasoning using the CoT (Cooperation of Thought) method, thus realizing workflow orchestration.

[0034] Based on the disclosed content of the above embodiments, the beneficial effects of the solution include: business personnel can directly generate compliant workflows using natural language, without relying on professional technicians to manually drag and drop configurations, significantly reducing the orchestration threshold and shortening the requirement iteration cycle; it can fully cover and identify all standard and non-standard heterogeneous nodes, adapting to any scheduling platform regardless of whether it has a node registration mechanism; relying on enterprise-specific knowledge base retrieval enhancement (RAG) as a mandatory constraint, combined with thought chain (CoT) step-by-step reasoning to generate candidate nodes and dependencies between candidate nodes, it can effectively suppress the illusion of large models, ensure accurate parameters and compliant dependencies, and can be used without secondary manual modification, thus completely realizing the automation of task orchestration and improving orchestration efficiency and accuracy.

[0035] In one embodiment, such as Figure 2 As shown, the dual-mode fallback detection mechanism for detecting all task nodes and their metadata on different types of scheduling platforms includes: S101: For scheduling platforms with node registration and aggregation capabilities, the platform API can be used to directly read all registered and active task nodes and their metadata. S102: For scheduling platforms that do not have the ability to register and aggregate nodes, automatically traverse all the definition files of the online workflows of the scheduling platform and parse them to obtain task node configuration samples; S103: A density-based spatial clustering algorithm clusters the vectorized data of task node configuration samples to determine the node type; S104: Determine all task nodes and their metadata by combining the node types and task node configuration samples.

[0036] For example, a dual-mode fallback detection mechanism can be adopted, which is independent of whether the scheduling platform has a node registration mechanism and adapts to all scheduling scenarios. This mechanism includes two modes: Mode 1 is for scheduling platforms with node registration and aggregation capabilities, which directly reads all registered and effective task nodes and their definition metadata through the platform API; Mode 2 is for scheduling platforms without a registration mechanism, which automatically traverses and collects the definition files of all historically deployed workflows on the platform, parses task node configuration samples from them, and uses a density-based spatial clustering algorithm to automatically cluster these samples to distinguish different node types. In application, the configuration parameter structure of each node can be vectorized, and clustering can be performed based on the vector distance between nodes. This method does not require a preset total number of node types and can automatically identify and remove noise samples caused by configuration errors. In a specific embodiment, this clustering algorithm can be implemented using the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm.

[0037] Furthermore, for each type of identified task node, five inherent attributes can be extracted from its definition: essential configuration parameter set, optional configuration parameter set, execution constraints (such as maximum concurrency and allowed rerun strategies), parameter validation rules (such as data type, value range, and regular expression matching rules), and node version number. This generates a node-specific structured metadata template, and the metadata generated through Mode 2 conforms to the requirements of this template. The template is stored in JSON Schema format for subsequent generation and validation. Similarly, the metadata generated through Mode 1 can also be stored in JSON Schema format.

[0038] In one embodiment, the step of extracting specified information based on the detected data and generating initial business semantic descriptions corresponding to each task node based on the extracted information includes: S201: Extract specified information from the detected metadata, wherein the execution information includes at least one of the following: node name, development notes, and historical change records; S202: Utilize a large language model to generate initial business semantic descriptions for each task node based on the extracted information.

[0039] For example, extract readable text fields such as "node name," "development notes," and "historical change records" from the metadata of each task node. These text fields, along with a pre-defined first prompt, can then be input into the large language model. An example of the first prompt is: "You are a data warehouse expert. Based on the following technical node names and notes, infer its core business function and describe it in a single sentence of general business language." The large model then generates an initial business semantic description for that task node. For example, the initial semantic description of the technical node dwd_order_full_load is "full synchronization task of order details."

[0040] Furthermore, retrieve the workflow DAG definition files containing this task node that were previously deployed in the enterprise's history, along with their corresponding original business requirement documents or work order texts. Then, using the large language model and the second prompt instruction, perform semantic comparison analysis on the retrieved historical files for the "DAG-requirement" relationship. The second prompt instruction constrains the large model to automatically extract from the requirement documents multiple colloquial expressions used when this task node is referenced in the real business context. For example, the aforementioned full synchronization node might be described in an actual work order as "full pull," "initial extraction," or "first full table import." These colloquial expressions, along with the previously obtained initial business semantics, together form the business semantic cluster for this task node.

[0041] Furthermore, the generated business semantic clusters, the unique identifiers (IDs) of task nodes, and the technical metadata template paths of the nodes are associated and stored to form a structured mapping table of <semantic cluster, node ID, metadata template path>. This mapping table is used to achieve precise routing from various fuzzy natural language expressions to uniquely determined task nodes and their configuration templates; that is, one task node can correspond to multiple business semantic descriptions, but any business semantic description can lock a unique task node. When a new task node is detected and added to the database in the first step, the system automatically completes the semantic mapping of the new node.

[0042] In one embodiment, such as Figure 3 As shown, in response to user-inputted orchestration requirements, similar historical cases and enterprise specification fragments matching the orchestration requirements are retrieved, and enhanced prompts are generated based on them, including: S401: Compile historical compliance workflow definitions, orchestration specification documents, and node configuration management records into a dedicated knowledge base; S402: Segment and vectorize the document data in the dedicated knowledge base, and establish an index for the vectorized paragraphs; S403: In response to the orchestration request input by the user, the orchestration request is converted into a query vector; S404: Use the query vector to search the dedicated knowledge base to obtain similar historical cases and enterprise specification fragments that match the arrangement requirements; S405: Generate the enhanced prompt based on the aforementioned change request and the retrieved similar historical cases and enterprise specification fragments.

[0043] For example, a dedicated knowledge base is created by compiling compliance workflow definitions, orchestration specification documents, and node configuration convention records from the company's historical period. These documents are then segmented, vectorized, and indexed with semantic features. When a user initiates an orchestration request, the user's natural language request is converted into a query vector, and similar historical cases and company specification fragments are prioritized for retrieval from the dedicated knowledge base. Next, the top-K similar workflow fragments and orchestration specification entries returned by the retrieval are used as mandatory constraint contexts, along with the user's original request, to form an augmented prompt.

[0044] Furthermore, based on the orchestration requirements, enhanced hints, structured mapping tables, first-layer CoT instructions, and second-layer instructions, using a large language model, the process of determining candidate nodes related to the orchestration requirements and the dependencies between these candidate nodes across all task nodes includes: S601: Input the orchestration requirements and enhanced prompts into the large language model, and input the constructed first layer CoT instructions to guide the large language model to identify and extract key elements in the orchestration requirements in a structured manner. The key elements include at least one of the following: target data entity list, data operation type, pre-trigger condition, and post-action. S602: Based on the key elements, perform semantic similarity matching in the structured mapping table to determine candidate nodes related to the orchestration requirements; S603: Input the key elements, candidate nodes, enhanced hints, and pre-built second-layer CoT instructions into the large language model to guide the large language model to determine the dependencies between candidate nodes.

[0045] For example, predefined chain-of-thought (CoT) instructions guide large models through step-by-step reasoning, avoiding logical confusion that can occur when generating the entire DAG at once. Specifically, this includes: The first layer of CoT instructions is constructed to guide the large model to identify and extract key elements in the orchestration requirements in a structured manner, requiring the output to strictly follow the predefined JSON Schema structured data. The JSON includes fields such as: a list of target data entities, data operation type (e.g., synchronization, cleaning, aggregation, branch judgment), pre-trigger conditions (e.g., timed triggering, triggering upon completion of a parent task), and post-actions (e.g., notification, export, starting a downstream stream).

[0046] The operation type field in the extracted data is semantically similar to the constructed structured mapping table. Several candidate nodes that meet the matching requirements are retrieved and then trimmed according to the commonly used nodes of this type of operation in the enterprise orchestration specifications to form a candidate node list containing node ID and metadata template.

[0047] Next, a second-layer CoT instruction is constructed, taking the pre-triggered conditions and post-action elements from the extracted data, the candidate node list, and previously retrieved similar historical workflow fragments as context, and inputting them into the large language model. The second-layer CoT instruction guides the large model to explicitly output the dependencies between nodes in the form of a directed edge list, such as A→C; B→C, and includes the reasoning behind each dependency in the output, such as "because parent task A is order data synchronization, child task C is order report generation, and the requirement is to generate the report after data synchronization is completed."

[0048] Furthermore, the step of generating a structurally complete DAG model object based on the candidate nodes and the dependencies between them includes: S701: Based on the key elements, fill them into the pre-selected node metadata template; S702: Combine the node metadata template filled with the key elements, candidate nodes, and the dependencies between candidate nodes to generate a structurally complete DAG model object.

[0049] For example, based on the topological relationships generated in the previous step, the system loads the selected node metadata template and fills the extracted data entities, time windows, and other details into the mandatory parameter slots of the nodes according to the template definition. Finally, all node instances and their dependencies are assembled into a complete DAG model object in strict accordance with the native DAG definition specifications of the target scheduling platform.

[0050] In one embodiment, the method further includes: S8: For each task node in the DAG model object, perform verification using predefined parameter verification rules and verification functions that match the orchestration specification constraints; S9: Perform static graph analysis on the DAG model object to automatically identify and intercept specified types of errors.

[0051] Furthermore, the method also includes: S10: Drive the DAG model object to run in an isolated shadow environment using Mock operators to simulate state transitions and data lineage transmission, and verify branch jump logic and upstream and downstream field compatibility.

[0052] For example, in this embodiment, only static verification and simulation are performed after generation; no real task execution is performed, and the online task status is not changed. The specific verification and simulation are divided into three layers of interception logic: First layer: Rigid validation of node configuration parameters by the rule engine.

[0053] Based on the parameter validation rules defined in the pre-built node metadata template, each node instance in the generated DAG is checked to ensure that all required parameters are complete and that the parameter value types and formats conform to regular expression matching rules. Simultaneously, constraints from the enterprise orchestration specification knowledge base are converted into executable validation functions. For example, for nodes that only support full synchronization, the function checks whether incremental filtering condition expressions have been incorrectly entered in their configuration; and it checks whether the partition field of the target table is compatible with the partition definition of its upstream source table.

[0054] Second layer: Workflow topology logical verification.

[0055] Static graph analysis is performed on the generated DAG model to automatically identify and intercept the following structural errors, such as: cycles with cyclic dependencies, non-starting nodes with an in-degree of 0 (dangling or isolated nodes), non-ending nodes with an out-degree of 0, and ghost dependencies that depend on non-existent nodes. Simultaneously, it checks whether the pre-triggered conditions are correctly associated with the starting node of the DAG model.

[0056] The third layer: Deterministic verification of business logic based on sandbox simulation.

[0057] This layer adopts a deterministic sandbox operation mode to replace the large model's self-verification: Pre-execution environment construction: A set of isolated "shadow" task instances is temporarily created for the DAG to be released. The task bodies in the instances do not perform real data reading and writing, but are replaced by Mock operators to simulate state transitions and metadata transmission.

[0058] State and data flow simulation: The shadow DAG is driven by the verification engine to check whether the state transitions meet expectations, such as whether branch nodes jump to the correct downstream nodes according to the configured conditional expressions. At the same time, data lineage parsing is performed on upstream and downstream node pairs, and the transmission of input and output field schemas is statically simulated to check whether the output fields of the source end are correctly referenced by the inputs of the downstream nodes, and whether the field names and types are compatible.

[0059] Interception logic: Only after the shadow DAG state transition and data flow simulation have all passed will it be determined that it has passed the business logic verification.

[0060] If any of the above three layers of verification fails, the release will be blocked immediately and a failure report will be generated; only workflows that pass all verifications can proceed to the subsequent release stage.

[0061] After completing the above steps, the validated workflow can be converted into the native format required by the target scheduling platform via a pre-built platform connector before being deployed online. The platform connector is an adaptation layer abstracted for different scheduling platforms (such as Apache DolphinScheduler, Apache Airflow, and self-developed scheduling systems). It achieves bidirectional conversion between the internally unified DAG description object and the specific APIs and formats of each platform. Upon successful deployment, the generated compliant workflow logic, node selection, and parameter configuration records are automatically extracted and stored in the database, updating the knowledge base and semantic mapping rules.

[0062] The following explanation uses specific application examples: Taking a securities company's data warehouse actual scheduling scenario as an example, this paper will explain the entire process from business-side natural language requirements to the generation of a publishable scheduling workflow DAG step by step.

[0063] Business Context: A securities firm needs to calculate risk control indicators after the market closes each day. Risk control personnel have submitted the following requirements using natural language: “Every day at 5 p.m., we first import the day’s over-the-counter transaction data and customer position data. Then, based on these two sets of data, we calculate the net capital risk control indicators for each business line. Finally, we push the risk control calculation results to the compliance reporting system. If the risk control calculation task fails, we immediately notify the on-duty personnel of the risk control post through WeChat.” The method of this invention will be applied step by step to address this requirement.

[0064] The target scheduling platform in this example has a node registration mechanism, so Mode 1 is used for detection. The full list of registered task nodes is obtained through the scheduling platform API, and three types of technical nodes relevant to this requirement are identified: Node T1 (data synchronization class): CounterToHive_BatchSync. The required parameters extracted from the metadata are source table name, target table name, and data source connection ID; the optional parameters are concurrency and timeout; the execution constraint is "only supports full batch synchronization, incremental conditions cannot be filled in".

[0065] Node T2 (Calculation Class): Hive_RiskMetric_Calculation, required parameters are risk control script path and entry parameters; optional parameters are execution queue and memory allocation.

[0066] Node T3 (Notification Class): WeCom_Notification, required parameters are recipient and notification template ID; execution constraint is "trigger only when the upstream task status is failed".

[0067] The system extracts configuration parameters, constraints, and validation rules from the above nodes and generates corresponding JSON Schema metadata templates.

[0068] Extract the technical field "CounterToHive_BatchSync" and the development note "Full synchronization of counter transaction and holding data" from node T1. Input the large model and the large model outputs the initial business semantic description: "Full synchronization task of counter transaction flow or customer holdings".

[0069] Searching historical work orders revealed colloquial expressions related to this node, such as "synchronize counter data," "retrieve daily transaction data and open positions," and "import all transaction records." These phrases were merged with the initial description to form the business semantic cluster for node T1. The same operations were performed on T2 and T3, resulting in the semantic clusters "calculate net capital risk control indicators / run risk control scripts" and "send enterprise WeChat alerts / notify risk control personnel," respectively.

[0070] Generate a many-to-one structured mapping table.

[0071] After the user's requirements are vectorized, they are retrieved from a dedicated knowledge base and a similar historical specification is found: "In the risk control indicator calculation chain, the calculation task must be triggered strictly after all source data synchronization tasks have been successfully completed, and the alarm node is only attached to the 'failure' output branch of the calculation task node." This specification item serves as a mandatory constraint context and is concatenated with the user's original requirements to form an augmented prompt, which is then input into the large model.

[0072] Intent parsing and feature extraction: The first-layer CoT instructions guide the extraction of structured features from the large model as follows: Target entities: Over-the-counter transaction records, customer holdings records, net capital risk control indicator results, and compliance reporting system.

[0073] Data operations: full synchronization, risk control indicator calculation, and enterprise WeChat notification.

[0074] Prerequisite: Time: 5 PM every day.

[0075] Post-processing: Push to the compliance reporting system; if it fails, notify the risk control personnel.

[0076] Node retrieval and candidate set generation: The entire operation is synchronized and risk control indicators are calculated in the mapping table for semantic matching, accurately matching nodes T1 (counter full synchronization) and T2 (risk control indicator calculation). Based on the extracted "notify if it fails", the notification node T3 (WeChat notification) is matched.

[0077] Topological relation reasoning generation: The second-level CoT instruction inputs the elements, candidate nodes (T1, T2, T3), and the retrieved enterprise specification context into the large model. The large model outputs dependencies as a list of directed edges according to the instruction, along with the reasoning basis: Counter transaction history_T1 -> Net capital risk control calculation_T2 (Based on the requirement to first import transaction history, and then calculate risk control indicators based on it); Client holdings_T1 -> Net capital risk control calculation_T2 (Based on: Same as above, risk control calculation also depends on holdings data); Net Capital Risk Control Calculation_T2 -> Compliance Reporting System (Based on: Requirement requirements and results pushed to the Compliance Reporting System); Net Capital Risk Control Calculation_T2 --[Failure]-->Enterprise WeChat Notification_T3 (Based on: Enterprise regulations require that on-duty personnel must be notified when risk control fails).

[0078] The system loads the T1 metadata template, fills in the source table name slots with the entities "Counter Transaction Flow Sheet" and "Customer Position Sheet", and generates two T1 node instances; loads the T2 template and fills in the risk control script path; loads the T3 template and fills in the risk control personnel on duty. Finally, all node instances and the topology relationships generated in 4.3 are assembled into a complete DAG model object.

[0079] Output Interpretation: The risk control personnel only input a colloquial request: "Every day at 5 PM, first import the day's over-the-counter transaction records and customer position data..." This method automatically completed the following: it identified the two source tables in the request, generated two parallel synchronous task instances, `sync_trade` and `sync_position`; set the risk control calculation task `calc_risk_metric` as a dual dependency according to enterprise specifications; and precisely mounted the WeChat notification task `wecom_alert` only on the "failure" output branch of `calc_risk_metric`, not the "success" path. This model is the direct object for subsequent validation and deployment.

[0080] Multi-level pre-processing static validation and error interception in the generated workflow: The first layer (rule engine verification) verifies the two generated T1 node instances. If the parameters are complete, the format is compliant, and the constraint of "incremental conditions cannot be filled" is not violated, the process passes.

[0081] Second layer (topology verification): The graph analysis did not detect any circular dependencies or isolated nodes, and all dependent edges point to existing nodes, so it passed.

[0082] Layer 3 (Sandbox Simulation Verification): Driving the shadow DAG within the sandbox. It was found that the "wecom_alert" node was correctly configured on the "failure" output port of calc_risk_metric, not the "success" port, consistent with the expected business logic of "triggered only when the upstream task fails." Data lineage analysis confirmed that the input fields trade_amount and position_volume of calc_risk_metric can be provided by the two outputs of T1, and the field types are compatible; therefore, this step passed.

[0083] With all three layers of verification passing, the workflow was deemed compliant.

[0084] The system loaded the scheduling system adapter connector, converted the compliant DAG object into the platform's API request format, and successfully launched it online. Simultaneously, the generated workflow pattern of "counter data dual-path synchronization -> risk control calculation -> failure enterprise notification" and its parameter configuration were automatically extracted and stored in the database, updating the knowledge base and semantic mapping rules for use in the generation and verification of similar requirements in the future.

[0085] As can be seen from the above embodiments, the beneficial effects of the present invention include: The orchestration threshold has been significantly lowered, and business personnel can directly generate scheduling workflows through natural language without relying on professional technicians to manually drag and drop configurations. This greatly improves the efficiency of business and technology collaboration and significantly shortens the requirement iteration cycle.

[0086] It fully adapts to various standard nodes and enterprise-customized non-standard heterogeneous nodes. Through dual-mode detection as a fallback and construction based on historical work order mapping, it can completely identify nodes regardless of whether the scheduling platform has a registration mechanism and establish a many-to-one semantic mapping. No manual configuration of adaptation rules is required. It is compatible with all types of scheduling platforms, with strong deployment compatibility, non-intrusive modification, and no risk.

[0087] By leveraging the enterprise-specific knowledge base retrieval enhancement (RAG) as a mandatory constraint and combining it with step-by-step reasoning generation via the thought chain, the illusion of a large model is effectively suppressed. The generated workflow parameters are accurate and logically compliant, and can be used without secondary manual modification.

[0088] A dedicated pre-processing error interception and verification feature has been added after the workflow is generated. Through an executable rule engine and deterministic sandbox simulation, it automatically identifies configuration defects, topology logic errors, and deep business field mismatches, intercepting unqualified workflows in advance and preventing workflow failures from the source, thus significantly improving orchestration accuracy.

[0089] It has the ability to learn and accumulate knowledge independently. Each compliant release feeds back the results into the knowledge base and mapping table. The more the system is used, the higher the accuracy of its generation. It is adapted to enterprise-specific orchestration scenarios in the long term, and the intelligent orchestration effect is continuously optimized.

[0090] like Figure 4 As shown, another embodiment of the present invention also provides an automatic scheduling workflow orchestration device, comprising: The detection module is used to detect all task nodes and their metadata on different types of scheduling platforms using a dual-mode fallback detection mechanism. The first generation module is used to extract specified information based on the detected data and generate initial business semantic descriptions for each task node based on the extracted information. The mining module is used to automatically mine various colloquial expressions of the task node when it is referenced in the real business context, based on the historical workflow DAG definition file, historical business requirements and work order text of the task node. The second generation module is used to form a business semantic cluster corresponding to the task node based on the colloquial expression and the initial business semantic description, and to generate a structured mapping table by combining the business semantic cluster, node ID, and active technician data template path. The retrieval module is used to respond to the user's input of the arrangement requirements, retrieve similar historical cases and enterprise specification fragments that match the arrangement requirements, and generate enhanced prompts based on them; The determination module is used to determine, based on the orchestration requirements, enhancement prompts, structured mapping tables, first-layer CoT instructions, and second-layer CoT instructions, the candidate nodes related to the orchestration requirements and the dependencies between the candidate nodes in the full set of task nodes using a large language model. The first-layer CoT instructions are used to extract the requirement information from the enhancement prompts, and the second-layer CoT instructions are used to determine the dependencies between the candidate nodes. The third generation module is used to generate a structurally complete DAG model object based on the candidate nodes and the dependencies between them.

[0091] In one embodiment, the device further includes: The verification module is used to verify each task node in the DAG model object using predefined parameter verification rules and verification functions that match the orchestration specification constraints. The identification module is used to perform static graph analysis on the DAG model object, automatically identify and intercept specified types of errors.

[0092] In one embodiment, the device further includes: The verification module is used to drive the DAG model object to run in an isolated shadow environment with Mock operators, simulate state transitions and data lineage transmission, and verify branch jump logic and upstream and downstream field compatibility.

[0093] In one embodiment, the step of determining candidate nodes related to the orchestration requirements and the dependencies between candidate nodes in all task nodes based on the orchestration requirements, enhanced hints, structured mapping tables, first-layer CoT instructions, and second-layer large language models includes: The orchestration requirements and enhanced prompts are input into the large language model, along with the constructed first-layer CoT instructions, which guide the large language model to identify and extract key elements from the orchestration requirements in a structured manner. The key elements include at least one of the following: target data entity list, data operation type, pre-trigger condition, and post-action. Based on the key elements, semantic similarity matching is performed in the structured mapping table to determine candidate nodes related to the orchestration requirements; The key elements, candidate nodes, enhanced hints, and pre-built second-layer CoT instructions are input into the large language model to guide the large language model in determining the dependencies between candidate nodes.

[0094] In one embodiment, generating a structurally complete DAG model object based on the candidate nodes and the dependencies between them includes: The key elements are matched and filled into the pre-selected node metadata template; By combining the node metadata template filled with the key elements, candidate nodes, and the dependencies between candidate nodes, a structurally complete DAG model object is generated.

[0095] In one embodiment, the dual-mode fallback detection mechanism for detecting all task nodes and their metadata on different types of scheduling platforms includes: For scheduling platforms with node registration and aggregation capabilities, all registered and active task nodes and their metadata can be read directly through the platform API. For scheduling platforms that do not have the ability to register and aggregate nodes, the system automatically traverses all the definition files of the online workflows of the scheduling platform and parses them to obtain task node configuration samples. Density-based spatial clustering algorithms cluster vectorized data of task node configuration samples to determine node types; Based on the node types and task node configuration samples, a full set of task nodes and their metadata are determined.

[0096] In one embodiment, the step of extracting specified information based on the detected data and generating initial business semantic descriptions corresponding to each task node based on the extracted information includes: The detected metadata is used to extract specified information, which includes at least one of the following: node name, development notes, and historical change records; The large language model is used to generate initial business semantic descriptions for each task node based on the extracted information.

[0097] In one embodiment, responding to user-inputted orchestration requirements, retrieving similar historical cases and enterprise specification fragments matching the orchestration requirements, and generating enhanced prompts based on them, includes: A dedicated knowledge base is created by compiling historical compliance workflow definitions, orchestration specifications, and node configuration management records. The document data in the dedicated knowledge base is segmented and vectorized, and an index of the vectorized paragraphs is established; In response to an orchestration request input by the user, the orchestration request is converted into a query vector; The query vector is used to search the dedicated knowledge base to obtain similar historical cases and enterprise specification fragments that match the orchestration requirements; The enhanced prompt is generated based on the aforementioned change request and the retrieved similar historical cases and fragments of enterprise specifications.

[0098] Another embodiment of the present invention also provides an electronic device, comprising: One or more processors; Memory, configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the automatic orchestration method for scheduling workflows as described above.

[0099] Furthermore, one embodiment of the present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the automatic scheduling workflow orchestration method described above. It should be understood that the various solutions in this embodiment have the corresponding technical effects in the above-described method embodiments, and will not be repeated here.

[0100] Furthermore, embodiments of the present invention also provide a computer program product, which is tangibly stored on a computer-readable medium and includes computer-readable instructions that, when executed, cause at least one processor to perform an automatic scheduling workflow orchestration method such as those described in the embodiments above.

[0101] It should be noted that the computer storage medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program configured for use by or in connection with an instruction execution system, system, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, antenna, optical fiber, RF, etc., or any suitable combination thereof.

[0102] Furthermore, those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0103] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0105] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

Claims

1. A method for automatically orchestrating scheduling workflows, characterized in that, include: A dual-mode fallback detection mechanism is adopted to detect all task nodes and their metadata on different types of scheduling platforms; Based on the detected data, specific information is extracted, and based on the extracted information, an initial business semantic description for each task node is generated. Based on the historical workflow DAG definition file, historical business requirements, and work order text of the task node, multiple colloquial expressions of the task node when it is referenced in the real business context are automatically mined. Based on the colloquial expression and initial business semantic description, a corresponding business semantic cluster is formed for the task node, and a structured mapping table is generated by combining the business semantic cluster, node ID, and active technician data template path. In response to the user's input of an orchestration request, similar historical cases and enterprise specification fragments that match the orchestration request are retrieved, and enhanced prompts are generated based on them; Based on the orchestration requirements, enhanced hints, structured mapping tables, first-layer CoT instructions, and second-layer CoT instructions, a large language model is used to determine candidate nodes related to the orchestration requirements and the dependencies between candidate nodes in all task nodes. The first-layer CoT instructions are used to extract requirement information from the enhanced hints, and the second-layer CoT instructions are used to determine the dependencies between candidate nodes. A complete DAG model object is generated based on the candidate nodes and the dependencies between them.

2. The automatic scheduling workflow orchestration method according to claim 1, characterized in that, The method further includes: For each task node in the DAG model object, a verification function that matches the orchestration specification constraints is used to perform the verification. Static graph analysis is performed on the DAG model object to automatically identify and intercept specified types of errors.

3. The automatic scheduling workflow orchestration method according to claim 2, characterized in that, The method further includes: The DAG model object is driven to run in an isolated shadow environment using Mock operators to simulate state transitions and data lineage transmission, and to verify branch jump logic and compatibility between upstream and downstream fields.

4. The automatic scheduling workflow orchestration method according to claim 1, characterized in that, The process of determining candidate nodes related to the orchestration requirements and the dependencies between candidate nodes in all task nodes, based on the orchestration requirements, enhanced hints, structured mapping tables, first-layer CoT instructions, and second-layer instructions, using a large language model, includes: The orchestration requirements and enhanced prompts are input into the large language model, along with the constructed first-layer CoT instructions, which guide the large language model to identify and extract key elements from the orchestration requirements in a structured manner. The key elements include at least one of the following: target data entity list, data operation type, pre-trigger condition, and post-action. Based on the key elements, semantic similarity matching is performed in the structured mapping table to determine candidate nodes related to the orchestration requirements; The key elements, candidate nodes, enhanced hints, and pre-built second-layer CoT instructions are input into the large language model to guide the large language model in determining the dependencies between candidate nodes.

5. The automatic scheduling workflow orchestration method according to claim 4, characterized in that, The process of generating a structurally complete DAG model object based on the candidate nodes and the dependencies between them includes: The key elements are matched and filled into the pre-selected node metadata template; By combining the node metadata template filled with the key elements, candidate nodes, and the dependencies between candidate nodes, a structurally complete DAG model object is generated.

6. The automatic scheduling workflow orchestration method according to claim 1, characterized in that, The dual-mode fallback detection mechanism is used to detect all task nodes and their metadata on different types of scheduling platforms, including: For scheduling platforms with node registration and aggregation capabilities, all registered and active task nodes and their metadata can be read directly through the platform API. For scheduling platforms that do not have the ability to register and aggregate nodes, the system automatically traverses all the definition files of the online workflows of the scheduling platform and parses them to obtain task node configuration samples. Density-based spatial clustering algorithms cluster vectorized data of task node configuration samples to determine node types; Based on the node types and task node configuration samples, a full set of task nodes and their metadata are determined.

7. The automatic scheduling workflow orchestration method according to claim 1, characterized in that, The extraction of specified information based on the detected data, and the generation of initial business semantic descriptions for each task node based on the extracted information, include: The detected metadata is used to extract specified information, which includes at least one of the following: node name, development notes, and historical change records; The large language model is used to generate initial business semantic descriptions for each task node based on the extracted information.

8. The automatic scheduling workflow orchestration method according to claim 1, characterized in that, In response to user-inputted orchestration requirements, the system retrieves similar historical cases and enterprise specification fragments that match the orchestration requirements, and generates enhanced prompts based on these, including: A dedicated knowledge base is created by compiling compliance workflow definitions, orchestration specifications, and node configuration management records from historical periods. The document data in the dedicated knowledge base is segmented and vectorized, and an index of the vectorized paragraphs is established; In response to an orchestration request input by the user, the orchestration request is converted into a query vector; The query vector is used to search the dedicated knowledge base to obtain similar historical cases and enterprise specification fragments that match the orchestration requirements; The enhanced prompt is generated based on the aforementioned change request and the retrieved similar historical cases and fragments of enterprise specifications.

9. An automatic scheduling device for workflows, characterized in that, include: The detection module is used to detect all task nodes and their metadata on different types of scheduling platforms using a dual-mode fallback detection mechanism. The first generation module is used to extract specified information based on the detected data and generate initial business semantic descriptions for each task node based on the extracted information. The mining module is used to automatically mine various colloquial expressions of the task node when it is referenced in the real business context, based on the historical workflow DAG definition file, historical business requirements and work order text of the task node. The second generation module is used to form a business semantic cluster corresponding to the task node based on the colloquial expression and the initial business semantic description, and to generate a structured mapping table by combining the business semantic cluster, node ID, and active technician data template path. The retrieval module is used to respond to the user's input of the arrangement requirements, retrieve similar historical cases and enterprise specification fragments that match the arrangement requirements, and generate enhanced prompts based on them; The determination module is used to determine, based on the orchestration requirements, enhancement prompts, structured mapping tables, first-layer CoT instructions, and second-layer CoT instructions, the candidate nodes related to the orchestration requirements and the dependencies between the candidate nodes in the full set of task nodes using a large language model. The first-layer CoT instructions are used to extract the requirement information from the enhancement prompts, and the second-layer CoT instructions are used to determine the dependencies between the candidate nodes. The third generation module is used to generate a structurally complete DAG model object based on the candidate nodes and the dependencies between them.

10. An electronic device, comprising: One or more processors; Memory, configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the scheduling workflow automatic orchestration method as described in any one of claims 1-8.

Images