Process arrangement method and device for big data scene, equipment and medium

Abstract

The application discloses a process arrangement method and device for a big data scene, equipment and a medium, relates to the technical field of big data processing, and comprises the following steps: acquiring a natural language task description input by a user; performing intention analysis on the natural language task description based on a preset artificial intelligence extension system, determining a target task type and corresponding process configuration parameters; according to the process configuration parameters, matching corresponding execution units from a dynamically registered component set, and constructing a business process comprising a plurality of processing nodes; rendering the business process through a visual interface, and scheduling and executing the business process based on the execution units. The application breaks the limitation of hard-coded extension of a traditional ETL tool, realizes a full-link closed loop from natural language interaction to process execution, and enables a user to complete big data task arrangement through natural language without paying attention to underlying node implementation details, thereby reducing the use threshold and shortening the business online period.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to process orchestration methods, apparatus, equipment and media for big data scenarios. Background Technology

[0002] Currently, most big data ETL tools use hard-coded predefined nodes. Adding new data sources or processing logic requires modifying the source code, recompiling and re-deploying, resulting in high expansion costs and long deployment cycles.

[0003] Existing technologies have three limitations: First, nodes are merely configuration layer abstractions and do not support dynamic expansion of execution capabilities during runtime through JAR uploads and script writing. Second, AI capabilities only cover natural language processing, failing to integrate tool calls, protocol adaptation, sub-agent collaboration, skill reuse, and other dimensions, thus hindering deep involvement in the entire data processing process. Third, front-end component libraries are hard-coded for maintenance, requiring simultaneous modifications to the front-end code when adding new back-end nodes, violating the hot-swappable design goal. Some existing technologies focus only on optimizing a single aspect, such as optimizing only the latency of MCP tool calls or implementing only data source layer plug-ins, failing to establish an end-to-end closed loop of "front-end canvas - back-end execution - AI interaction," and also failing to cover full-link node orchestration and comprehensive data lineage governance, making it difficult to meet the agile iteration and intelligent operation and maintenance needs of big data scenarios. Summary of the Invention

[0004] The main purpose of this application is to provide a process orchestration method, apparatus, equipment and medium for big data scenarios, aiming to solve the technical problems of difficult expansion and lack of intelligent closed loop in big data process orchestration.

[0005] To achieve the above objectives, this application proposes a process orchestration method for big data scenarios, the method comprising: Obtain a natural language task description from user input; Based on a pre-defined artificial intelligence extension system, the intent of the natural language task description is parsed to determine the target task type and the corresponding process configuration parameters. Based on the process configuration parameters, the corresponding execution unit is matched from the dynamically registered component set, and a business process containing several processing nodes is constructed. The business process is rendered through a visual interface, and the business process is scheduled and executed based on the execution unit.

[0006] In one embodiment, the step of performing intent parsing on a natural language task description based on a preset artificial intelligence extension system to determine the target task type and corresponding process configuration parameters includes: The pre-defined intent parsing model in the artificial intelligence extension system is invoked to perform semantic recognition on the natural language task description and obtain the task intent label; Match the corresponding extended dimension executor based on the task intent label. The extended dimension executor includes one or more of the following: atomic tool executor, protocol service executor, sub-agent executor, and skill pack executor. By extending the dimensional executor, process configuration parameters are generated by combining dynamically registered component metadata information.

[0007] In one embodiment, the step of matching the corresponding execution unit from the dynamically registered component set according to the process configuration parameters and constructing a business process containing several processing nodes includes: Traverse the node type identifier in the process configuration parameters; Query the preset node registry to obtain the loading strategy and component definition information corresponding to the node type identifier; Based on the loading strategy, load the corresponding execution unit instance from the component repository; The execution unit instances are connected according to the topological relationship in the process configuration parameters to generate the business process.

[0008] In one embodiment, the step of loading the corresponding execution unit instance from the component repository according to the loading strategy includes: If the loading strategy is the first loading strategy, then a pre-built native execution unit instance is obtained from the runtime container; If the loading strategy is the second loading strategy, the script code is dynamically compiled to generate a script execution unit instance, and the calling permissions of the script code are security-verified. If the loading strategy is the third loading strategy, the plugin file is loaded through an isolated class loader, a plugin execution unit instance is generated, and lifecycle isolation is performed on different versions of the plugin execution unit instance.

[0009] In one embodiment, after the steps of rendering the business process through a visual interface and scheduling the execution of the business process based on the execution unit, the method further includes: During the execution of the business process, real-time collection of operational metrics data and field-level data lineage of each processing node is performed. The operational metrics data are mapped to the corresponding nodes on the visualization interface for status display; In response to a user's bloodline query request, a graph traversal is performed based on field-level bloodline relationships to generate an impact range analysis report.

[0010] In one embodiment, the step of real-time collection of operational metric data and field-level data lineage of each processing node includes: For relational database nodes, the field mapping relationships are extracted by parsing the abstract syntax tree of the structured query statement; For non-relational data source nodes, extract field input / output mappings are defined by intercepting operation logs or using aggregation pipelines; For message queue nodes, a field mapping chain between the producer and consumer is established by parsing the metadata information in the message header; For application interface nodes, establish field mapping relationships between upstream and downstream interfaces by intercepting request and response data; For file data source nodes, a mapping identifier between file paths and fields is established by parsing the file structure.

[0011] In one embodiment, the step of obtaining a natural language task description input by the user further includes: In response to the user's selection of a preset template, load the corresponding preset dialogue configuration; The dialogue preset configuration includes basic model configuration, system prompts, and a list of enabled extended dimensions; The AI ​​dialogue context is initialized based on the preset dialogue configuration.

[0012] Furthermore, to achieve the above objectives, this application also proposes a workflow orchestration device for big data scenarios, which includes: The task acquisition module is used to acquire the natural language task description input by the user; The intent parsing module is used to parse the intent of natural language task descriptions based on a preset artificial intelligence extension system, and to determine the target task type and the corresponding process configuration parameters. The process construction module is used to match the corresponding execution unit from the dynamically registered component set according to the process configuration parameters, and construct a business process containing several processing nodes. The execution rendering module is used to render business processes through a visual interface and to schedule and execute business processes based on execution units.

[0013] In addition, to achieve the above objectives, this application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the workflow orchestration method for big data scenarios as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the workflow orchestration method for big data scenarios described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: 1. By employing a pre-defined AI extension system to parse the user's input natural language task description to determine the task type and process configuration parameters, this approach solves the high barrier to entry in traditional orchestration, which requires manual understanding and mapping of complex business semantics. This allows the system to automatically transform ambiguous business requirements into precise engineering parameters. Furthermore, by matching execution units from a dynamically registered component set based on these parameters and constructing processes, this approach addresses the poor scalability of statically hard-coded component libraries that cannot adapt to the rapid iteration of big data technology stacks. It enables hot-swappable integration of new components without modifying the core scheduling logic. Finally, by using a visual interface rendering combined with execution unit scheduling, this approach solves the problems of the traditional black-box execution state and lack of feedback optimization mechanisms. It achieves visualized monitoring and data closure of business processes, thus achieving the overall effect of lowering the orchestration threshold, improving system scalability and intelligent operation and maintenance.

[0016] 2. It solves the problem that in existing solutions, AI can only generate processes but cannot participate deeply in data processing; the four types of extensions have clear boundaries and do not overlap with each other, supporting not only single atomic operations and external protocol interfaces, but also complex task decomposition and domain experience reuse; at the same time, parameter generation and dynamically registered component metadata are linked in real time, and newly added hot-swappable components can be automatically identified by AI and directly included in the process generation candidate, avoiding the disconnect between AI capabilities and backend components.

[0017] 3. Breaking through the limitations of traditional lineage solutions that only support relational databases and rely on pre-registered metadata, it covers all types of data sources such as NoSQL, message queues, APIs, and file systems, reducing lineage collection latency from hours to seconds, and truly achieving full-domain data lineage tracking. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the workflow orchestration method for big data scenarios in this application; Figure 2 This is a flowchart illustrating an embodiment of the workflow orchestration method for big data scenarios in this application; Figure 3This is a flowchart illustrating an embodiment of the workflow orchestration method for big data scenarios in this application; Figure 4 This is a schematic diagram of the module structure of the workflow orchestration device for big data scenarios in this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0023] This application provides a workflow orchestration method for big data scenarios. Specifically, refer to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the workflow orchestration method for big data scenarios according to this application. In this embodiment, the workflow orchestration method for big data scenarios includes steps S10 to S40:

[0024] Step S10: Obtain the natural language task description input by the user.

[0025] Natural language task description refers to the business requirements entered by the user in the AI ​​dialogue interface, such as "Help me create a synchronization process from MySQL to ClickHouse, and perform data cleaning in the middle", which serves as the trigger entry point for the entire process.

[0026] Step S20: Based on the preset artificial intelligence extension system, perform intent parsing on the natural language task description to determine the target task type and corresponding process configuration parameters.

[0027] The pre-defined AI extension system, namely the AI ​​Extension Four-Dimensional Matrix, decomposes AI capabilities into four orthogonal dimensions: TOOL (atomic tools), MCP (protocol services), SUBAGENT (sub-agents), and SKILL (skill packs), replacing the traditional single-point AI capabilities that can only perform intent recognition. Process configuration parameters include process name, source / target node type, transformation node, and category ID, derived from dynamically registered component metadata in etl_node_registry. For example, if the task is identified as "ETL process construction," it needs to include three types of parameters: input source, transformation node, and output target. The parameter range comes entirely from dynamically registered component metadata, rather than being hard-coded.

[0028] Step S30: Based on the process configuration parameters, match the corresponding execution unit from the dynamically registered component set and construct a business process containing several processing nodes.

[0029] The dynamically registered component set refers to all process nodes uniformly managed through the etl_node_registry table. It supports adding, updating, and disabling at runtime, unlike the hard-coded nodes in traditional ETL tools. All components (data source, transformation logic, output target) are registered in the etl_node_registry table, eliminating the need for hard-coded mapping. New components can be recognized simply by registering them.

[0030] Step S40: Render the business process through a visual interface and schedule its execution based on the execution unit.

[0031] The front-end uses Vue Flow to implement a process canvas, and the back-end uses NodeExecutorFactory to dynamically load executors according to node type, which can run without restarting the service.

[0032] In this embodiment, by employing a technique based on a preset artificial intelligence extension system to parse the user-input natural language task description to determine the task type and process configuration parameters, the technical problem of high barriers to entry in traditional orchestration, which requires manual understanding and mapping of complex business semantics, is solved. This allows the system to automatically transform ambiguous business requirements into precise engineering parameters. Furthermore, by matching execution units and constructing processes based on these parameters in a dynamically registered set of components, the technical problem of poor scalability caused by static hard-coded component libraries being unable to adapt to the rapid iteration of big data technology stacks is solved, enabling the hot-swappable integration of new components without modifying the core scheduling logic. Finally, by using a visual interface rendering combined with execution unit scheduling and execution, the technical problems of black-box execution status and lack of feedback optimization mechanisms in traditional processes are solved, achieving visualized monitoring and data closure of business processes. Thus, the overall effect is to reduce the orchestration threshold, improve system scalability and intelligent operation and maintenance.

[0033] In one implementation, reference is made to Figure 2 Step S20 includes: Step S201: Call the preset intent parsing model in the artificial intelligence extension system to perform semantic recognition on the natural language task description and obtain the task intent label.

[0034] Based on the large model configured by AiChatPreset (dialogue preset template), semantic understanding is performed on user input. For example, querying "ods.user table data quality" is identified as the intent data_quality_check.

[0035] Step S202: Match the corresponding extended dimension executor according to the task intent label. The extended dimension executor includes one or more of the following: atomic tool executor, protocol service executor, sub-agent executor, and skill package executor.

[0036] It should be noted that the extended dimension executor refers to the execution carrier corresponding to each type of AI extension, specifically including TOOL: flow_builder, sql_executor, metadata_query; MCP: McpToolExecutor; SUBAGENT: SubagentExecutor; SKILL: SkillPromptInjector.

[0037] TOOL (Tools): Atomic callable capabilities for specific big data business operations, characterized by "single-call, no side effects, idempotent retryability". Examples include flow_builder (automatically builds ETL processes), sql_executor (executes SQL and returns results), metadata_query (queries metadata), cdc_task_query (queries CDC task status), and quality_checker (checks data quality). Each tool implements the AiToolExecutor interface, is automatically registered in AiToolRegistry when the application starts, and is adapted to the Function Calling specification recognizable by the large language model through the LangChain4j ToolProvider interface. The criterion for classifying a capability as a TOOL is: the capability can be completed in a single FunctionCall, does not depend on the dialogue context, and does not maintain session state.

[0038] MCP (Model Context Protocol Service): Following the MCP protocol standard proposed by Anthropic, it encapsulates external data sources, API services, file systems, etc., into an MCP Server. The AI ​​assistant communicates with the MCP Server through McpToolExecutor in a standardized manner, automatically handling tool discovery, parameter filling, execution calls, and result feedback. The fundamental difference from existing technologies is that MCP is not a simple HTTP tool call, but a protocol layer with standardized lifecycle management—including connection handshake, capability negotiation, dynamic schema discovery, session state maintenance, and graceful shutdown. In the ETL orchestration system, the MCP Server is registered as a special hot-swappable jar_class node and incorporated into the unified scheduling of the NodeExecutorFactory. Therefore, MCP tools also enjoy the scheduling, retry, alerting, and monitoring capabilities of the ETL engine. Specifically, when an MCP node fails to execute, FlowScheduler retryes it according to the retry strategy (fixed interval / exponential backoff) configured at the process level. When the retry is exhausted and the node still fails, an alarm notification is triggered (supporting email, DingTalk, and WeChat Work) and the error information is sent back to the AI ​​dialogue layer, where the AI ​​assistant explains the reason for the failure to the user and provides repair suggestions.

[0039] SUBAGENT: For complex big data tasks, a "divide and conquer" architecture is used to distribute work among specialized sub-agents. Its characteristics include "lifecycle, parallel execution, and support for state rollback." For example, in the scenario of "migrating MySQL data to ClickHouse and performing data cleaning," the main agent can break down the task into "schema migration sub-agent," "data extraction sub-agent," "cleaning rule sub-agent," and "data loading sub-agent." Each sub-agent executes in parallel, and communication and result aggregation are coordinated through SubagentExecutor. The task splitting rules for sub-agents are as follows:

[0040] 1. Data domain splitting: Data is split according to the data processing stage, with each sub-Agent responsible for one data domain (extraction / transformation / loading / validation), ensuring clear input and output interfaces between sub-Agents; 2. Granularity determination: When the main agent evaluates the task complexity to exceed the threshold (e.g., the number of data sources involved is greater than 3, the number of transformation rules is greater than 10, and the expected amount of data to be processed is greater than 10 million rows), the splitting strategy is automatically triggered; 3. Failure rollback mechanism: Before execution, each sub-Agent registers a checkpoint with the CompensationService (containing the log of executed operations and reversible operation instructions). When any sub-Agent fails, the main agent notifies the CompensationService to execute the compensation operation in reverse order, and the successful sub-Agents are rolled back to ensure data consistency. 4. Result Aggregation: After all sub-Agents succeed, SubagentExecutor aggregates the output results of each sub-Agent according to a predefined merging strategy (such as Union, Join, Append) into a final result and returns it to the main Agent.

[0041] SKILL: A skill package encapsulates specialized capabilities for specific big data business scenarios into a reusable skill set, characterized by "multi-step combinations, stateful context, and domain specialization." Examples include "data cleaning skills" (including sub-capabilities such as missing value handling, deduplication, outlier detection, and format standardization) and "ETL process construction skills" (including sub-capabilities such as data source identification, transformation rule recommendation, and target mapping). The criteria for classifying a capability as a SKILL are: the capability is composed of multiple TOOLs, requires maintenance of domain context (such as the current data source schema and identified field types), and possesses cross-process reusability. SkillPromptInjector injects skill descriptions into the system prompt context before dialogue, enabling large models to quickly acquire domain-specific expert capabilities without retraining.

[0042] A capability is defined as a single atomic operation with no side effects and no state maintenance. A capability that requires long-term collaboration with external systems through standardized protocols and has a dynamic discovery mechanism is defined as an MCP. A capability that requires autonomous execution, supports parallelism and rollback, and has an independent lifecycle is defined as a SUBAGENT. A capability that involves multiple steps, has a domain context, and can be reused across processes is defined as a SKILL. For example, TOOL corresponds to the AiToolExecutor interface implementation class, and MCP corresponds to McpToolExecutor, both of which are uniformly registered in AiToolRegistry. Four types of executors are selected based on intent: for example, TOOL (such as sql_executor) for single queries, MCP for interfacing with external services, SUBAGENT for complex migrations, and SKILL for multi-step scenarios. The boundaries between these four types are strictly defined to avoid logical confusion.

[0043] Step S203: Generate process configuration parameters by extending the dimension executor and combining the dynamically registered component metadata information.

[0044] Component metadata information refers to the node description information stored in the etl_node_registry table, including node type, category, icon, configuration schema, loading strategy, etc. Before generating parameters, the flow_builder tool dynamically queries the etl_node_registry table to obtain all currently enabled node types, ensuring that the sourceType, targetType, and transformNodes in the parameters are all components actually supported by the system, and avoiding the problem of "AI generating non-existent nodes".

[0045] In this embodiment, through the division of labor and collaboration in four-dimensional expansion, AI can not only understand user needs, but also generate accurate and usable configuration parameters by combining real-time node metadata, thus avoiding the shortcomings of traditional solutions where AI can only perform simple intent recognition and cannot connect with actual execution capabilities.

[0046] Specifically, AiToolRegistry is adapted to LangChain4j's ToolProvider interface, allowing all registered tools to be automatically invoked and reused by LangChain4j's AiServices tools. Each tool executor (AiToolExecutor) is automatically registered to the registry via @PostConstruct when the application starts, enabling hot-plugging of tools.

[0047] In this embodiment, the key to the flow_builder tool lies in the fact that the node type enum in its parameter Schema is not hard-coded, but dynamically queried from the etl_node_registry table. This means that when an administrator adds a custom node via a hot-plug mechanism (e.g., by uploading a JAR or writing a Groovy script) and enables it, the AI ​​model can automatically detect the existence of the node in the next dialogue and include it in the candidate scope of flow construction. This achieves a closed loop of "adding a component → automatic AI detection → direct citation in natural language".

[0048] In one implementation, reference is made to Figure 3 Step S30 includes: Step S301: Traverse the node type identifiers in the process configuration parameters.

[0049] For example, it can parse out three nodes with nodeType as db-input, cleaner, and db-output.

[0050] Step S302: Query the preset node registry to obtain the loading strategy and component definition information corresponding to the node type identifier.

[0051] The node registry, or etl_node_registry table, is the "central register" for all hot-swappable components, replacing the hard-coded node mappings of traditional ETL. Information such as load_type (spring_bean / groovy_script / jar_class), config_schema, and default_config for each node are read from the etl_node_registry table.

[0052] Step S303: Load the corresponding execution unit instance from the component repository according to the loading strategy.

[0053] The three-tier loading mechanism of NodeExecutorFactory handles the loading, prioritizing retrieval from the cache, and loading according to a strategy if no cache is available, thus avoiding duplicate parsing.

[0054] Step S304: Connect the execution unit instances according to the topological relationship in the process configuration parameters to generate the business process.

[0055] Topological relationships refer to the sequential dependencies between nodes in a process, such as the requirement to extract, transform, and load data beforehand. These relationships are adjusted by the user on the front-end canvas or automatically generated by AI. An executable directed acyclic flow is generated based on the upstream and downstream relationships of nodes in the configuration parameters (e.g., db-input→cleaner→db-output).

[0056] In this embodiment, the process construction and the specific execution logic are completely decoupled, and all hard coding is eliminated. All node capabilities come from the dynamic registry. Adding or upgrading components does not require modifying the process engine code, thus solving the problem of traditional ETL extensions having to be recompiled and deployed from the root.

[0057] Specifically, NodeExecutorFactory completely removes the hard-coded mapping tables found in traditional ETL tools; all node executors are dynamically resolved through the etl_node_registry table. Built-in nodes are automatically synchronized to the registry by BuiltinNodeRegistrySyncRunner at application startup and are uniformly loaded via DynamicNodeLoaders. The loading priority is: memory cache → registry query → load strategy selected by loadType → fallback to DefaultNodeExecutor. Timeout protection is added during Groovy script execution.

[0058] In one implementation, step S303 includes: If the loading strategy is the first loading strategy, then a pre-built native execution unit instance is obtained from the runtime container.

[0059] The first loading strategy, also known as the `spring_bean` strategy (highest priority, used for system-built-in stable nodes), directly references existing executor beans in the Spring container, such as `databaseInputExecutor` and `messageQueueExecutor`, achieving zero-cost integration through Spring's dependency injection. This strategy is suitable for well-tested, long-term stable official nodes, offering the highest execution performance and lowest startup latency.

[0060] If the loading strategy is the second loading strategy, the script code is dynamically compiled to generate a script execution unit instance, and the calling permissions of the script code are security-verified.

[0061] The second loading strategy, the groovy_script strategy (medium priority, used for rapidly iterative business customization logic): For logic requiring rapid iteration or business customization, administrators are allowed to directly write Groovy scripts in the web interface and save them to the source_code field of the etl_node_registry table. The system dynamically compiles and loads these scripts at runtime using GroovyClassLoader, working with SecureASTCustomizer to implement compile-time security interception (prohibiting dangerous behaviors such as Runtime, ProcessBuilder, reflection calls, network access, and file system operations), and setting execution timeout protection (default 30 seconds) through ExecutorService to prevent malicious or erroneous scripts from crashing threads. This strategy is suitable for scenarios with frequently changing business logic and no complex dependencies, shortening the development cycle from "days" to "minutes".

[0062] If the loading strategy is the third loading strategy, the plugin file is loaded through an isolated class loader, a plugin execution unit instance is generated, and lifecycle isolation is performed on different versions of the plugin execution unit instance.

[0063] The third loading strategy, the jar_class strategy (lowest priority, used for complex third-party plugins): For complex plugins developed by third-party developers or within an enterprise, it supports uploading independent JAR files. The system loads plugin classes through a custom PluginClassLoader (inheriting from URLClassLoader, using a parent-first + package blacklist strategy), prohibiting plugins from accessing sensitive packages within the application (such as authentication modules, tenant contexts, etc.), achieving class loading-level security isolation. During hot replacement, unload is first called to close the old ClassLoader, clear cached instances in executorCache, and then the new version of the JAR is loaded, ensuring that component upgrades are completed without restarting the service. This strategy is suitable for scenarios that require introducing external dependency libraries, implementing complex algorithms, or interfacing with proprietary protocols.

[0064] In this embodiment, the three strategies cover all scenarios of "stable built-in capabilities, rapid business customization, and complex third-party plugins", which not only ensures the security of the core system, but also leaves enough flexible expansion space for the business side, and all expansions can be implemented without downtime.

[0065] It's important to note that the priority determination rules for the three loading strategies are as follows: During initialization, NodeExecutorFactory first attempts to find a Bean (spring_bean) matching the nodeType in the Spring container. If not found, it queries the `load_type` field of the node in the `etl_node_registry` table. If it's `groovy_script`, it loads it via `GroovyClassLoader`; if it's `jar_class`, it loads it via `PluginClassLoader`. This priority design ensures that built-in system nodes are unaffected by plugins, while custom logic and third-party plugins can coexist without interference.

[0066] It should be noted that the context adaptation logic between the plugin and the ETL engine is as follows: multi-tenant isolation is a core security requirement of big data platforms. This application ensures that the plugin does not violate multi-tenant isolation during execution through the following mechanisms: Tenant context pass-through: TenantContext is automatically inherited by InheritableThreadLocal when a thread is created in NodeExecutorFactory. The plugin code loaded by PluginClassLoader can safely read the current tenant ID during execution, but cannot modify or forge the tenant context; Data source connection pool isolation: The plugin obtains data source connections through a uniformly encapsulated DataSourceManager. DataSourceManager returns the data source connection pool of the corresponding tenant based on the current tenant ID. The plugin cannot directly access the underlying connection pool configuration; Distributed lock interface: The plugin requests a distributed lock through the LockService interface. LockService internally uses prefix isolation for the lock key based on the tenant ID to prevent plugins from competing for lock resources across tenants; Execution resource quota: The execution thread pool of each plugin node adopts tenant-level isolation. The execution timeout or resource exhaustion of a single tenant will not affect other tenants. Hot-swap task compatibility handling: When an administrator uploads a new version of the JAR plugin, the system executes the following compatibility assurance process: Version marking: The instance in the ClassLoader and executorCache corresponding to the old version JAR is marked as "legacy," and currently executing tasks continue to use the old version instance until completion; New task routing: Newly arriving execution requests are automatically routed to the new version of the ClassLoader and instance; Smooth unloading: After the old version instance's reference count reaches zero (i.e., all tasks depending on this instance have completed), it is automatically reclaimed by the JVM garbage collection mechanism. The old ClassLoader calls close() to release the file handle after confirming that there are no active instances, avoiding JAR file locking issues on the Windows platform; Class conflict avoidance: PluginClassLoader uses independent URL paths and class namespaces. Classes with the same name in different versions are completely isolated in different ClassLoaders, preventing class conflicts or NoClassDefFoundError.

[0067] In one implementation, after step S40, the method further includes: During the execution of the business process, real-time data collection is performed on the operational metrics and field-level data lineage of each processing node.

[0068] Operational metrics include node status, input / output data volume, latency, and error rate, which are uniformly reported by the AbstractNodeExecutor base class; lineage is collected in real time during execution and does not rely on pre-registered metadata.

[0069] The operational metrics data are mapped to the corresponding nodes on the visualization interface for status display.

[0070] The front-end canvas updates the node status (blue = running, green = success, red = failure) based on the metrics pushed by WebSocket. Hovering the mouse over the node displays details such as throughput and latency.

[0071] In response to a user's bloodline query request, a graph traversal is performed based on field-level bloodline relationships to generate an impact range analysis report.

[0072] When a user initiates a query (such as "which reports will be affected by modifying a certain field"), the system uses BFS / DFS to traverse the lineage graph and AI to organize it into a structured report that includes direct impact, indirect impact, and regression suggestions.

[0073] It's important to note that the process canvas is not only a design tool but also a visual representation of the execution status. As each node executes, the process engine pushes real-time status change events (node ​​running / successful / failed / skipped) to the designer via WebSocket or Server-Sent Events. The designer updates the node border color based on the status (blue for running, green for success, and red for failure) and overlays a progress ring animation on the node. Simultaneously, the system collects runtime metrics for each node, such as input / output volume, processing latency, and error rate, displaying them as floating tooltips on the canvas. When a user clicks on a running node, a real-time log panel expands to view the node's execution log. This "configuration as monitoring, canvas as dashboard" design philosophy enables operations personnel and data engineers to complete process design, debugging, operation monitoring, and troubleshooting within a single interface, significantly reducing the cognitive burden of operating big data processes.

[0074] In this implementation, process design, operation monitoring, fault diagnosis, and data governance are integrated into a single interface, solving the fragmentation problem of traditional ETL where design and monitoring are separate and lineage is lagging, and transforming data governance from passive to proactive.

[0075] In one implementation, the steps of real-time collection of operational metric data and field-level data lineage of each processing node include: For relational database nodes, the field mapping relationships are extracted by parsing the abstract syntax tree of the structured query statement.

[0076] An Abstract Syntax Tree (AST) is a structured representation of an SQL statement, facilitating program identification of field origins and destinations. For relational database nodes, during database input / output execution, the SQL parser (JSqlParser) parses the AST of the SQL statement, extracting the field list from the SELECT clause, table names from the FROM clause, and join conditions and field mappings from the JOIN clause. For INSERTINTO ...SELECT statements, a direct mapping of "source table fields → target table fields" is established; for complex subqueries, the AST of the subquery is recursively parsed, and a multi-level lineage chain is established.

[0077] For non-relational data source nodes, extract field input / output mappings by intercepting operation logs or defining aggregation pipelines.

[0078] It's important to note that for non-relational data source nodes (taking MongoDB as an example), during execution at the mongo-input / mongo-output nodes, the MongoTemplate's operation methods are intercepted to extract the definitions of each stage of the aggregation pipeline. For the $project stage, the mapping relationship between output and input fields is recorded; for the $lookup stage, field associations across collections are recorded. MongoDB's Schema validation rules (ValidationSchema) are used to infer field types and establish type-aware lineage relationships.

[0079] For message queue nodes, a field mapping chain between the producer and consumer is established by parsing the metadata information in the message header.

[0080] It should be noted that, for message queue nodes (taking Kafka as an example), when executing on the kafka-input / kafka-output nodes, the KafkaConsumer interceptor reads the lineageMeta field of the message header (containing the producer's traceId, topic, partition, and field schema), and combines this with the message body parsing results from the consumer to establish a lineage chain of "producer Topic fields → consumer processing fields → downstream output fields." Automatic retrieval of message field definitions from the Avro Schema Registry is supported.

[0081] For application interface nodes, field mapping relationships between upstream and downstream interfaces are established by intercepting request and response data.

[0082] During execution at the api-input / api-output nodes, the Swagger / OpenAPI documentation of the interface is parsed using the OpenAPI Parser to extract the field structure of request parameters and response bodies. During execution, the HttpClient interceptor records the actual request and response field values, establishing a mapping of "upstream API response fields → fields in this process → downstream API request fields." For RESTful interfaces, JSONPath expressions are used to precisely identify the position of nested fields.

[0083] For file data source nodes, a mapping identifier between file paths and fields is established by parsing the file structure.

[0084] When executing at nodes such as excel-input / csv-input / json-input, the file parser records the mapping between the file path, sheet name / column name / JSON key name, and target field during the reading phase. For nested JSON, it records the complete key path (e.g., user.address.city); for multi-sheet scenarios in Excel, it records a three-level lineage identifier of "filename#sheet name#column name".

[0085] In this implementation, lineage is collected in real time through execution-phase interception, with lineage establishment latency at the second level; it does not rely on external metadata platforms and automatically identifies field structures through schema inference; it covers all types of data sources such as relational databases, NoSQL, message queues, APIs, and file systems, achieving true full-domain lineage tracking.

[0086] In one implementation, the method further includes the following step before step S10: In response to the user's selection of a preset template, load the corresponding preset dialogue configuration.

[0087] Users can choose different templates such as "ETL Process Building Assistant" and "Data Quality Inspection Assistant", which correspond to different combinations of models, prompts, and extended capabilities.

[0088] The dialogue preset configuration includes basic model configuration, system prompts, and a list of enabled extended dimensions.

[0089] The dialogue preset configuration consists of three parts: basic model configuration (connecting to any model such as OpenAI, Wenxin, Xinghuo, and local Ollam); system prompt words (role settings customized for big data scenarios, such as "You are a senior data engineer"); and a list of enabled extended dimensions (specifying which TOOL / MCP / SKILL / SUBAGENT are available for this dialogue).

[0090] The AI ​​dialogue context is initialized based on the preset dialogue configuration.

[0091] During initialization, skill descriptions are injected into system prompts, tools are converted into FunctionCalling specifications that the model can recognize, and MCP connections are established, all without requiring manual configuration by the user.

[0092] In this implementation, the AI ​​assistant can be quickly adapted to different business scenarios, and business personnel do not need to worry about underlying details such as model selection and tool configuration. It is ready to use out of the box. At the same time, it is also convenient for enterprises to customize differentiated AI capability boundaries for different roles (data development, data analyst, operation and maintenance).

[0093] It should be noted that the tools such as flow_builder and sql_executor in the AI ​​extension matrix must rely on NodeExecutorFactory to be actually executed. The existence of the hot-swappable bus enables the AI-generated process to have a closed-loop capability of "generating and executing immediately". Dynamic form rendering (Schema-Driven UI) depends on the config_schema provided by the hot-swappable node. If the node does not register the schema through etl_node_registry, the front end will not be able to render the configuration interface. Therefore, the hot-swappable mechanism is a prerequisite for front-end and back-end decoupling. Digital twin monitoring depends on the metrics reported by the hot-swappable executor in the AbstractNodeExecutor.execute() method. If the executor does not uniformly implement this base class, monitoring data will not be collected.

[0094] When building a workflow, the flow_builder tool dynamically queries the etl_node_registry to obtain the currently available node types and translates new user requirements described in natural language (such as "connect to YashanDB") into explicit instructions for adding hot-swappable components, guiding developers on which type of node to prioritize. The "ETL workflow building skills" in SKILL identify frequently used node combination patterns by accumulating historical workflow data and feeds this information back to the NodeExecutorFactory to optimize the executorCache preloading strategy (preloading frequently used nodes into memory). When executing complex tasks, SUBAGENT evaluates the performance consumption of each sub-Agent on the executor and feeds back performance bottlenecks to the hot-swappable bus, triggering parameter tuning of the groovy_script or jar_class strategy (such as adjusting timeout and thread pool size).

[0095] The user's natural language request is first parsed by the AI ​​expansion matrix to determine the intent and select TOOL / SKILL / SUBAGENT. The generated process data must be correctly rendered by the front-end process canvas. The prerequisite for the front-end canvas to render the data is that the hot-swappable node has been registered in etl_node_registry and has provided complete metadata (icon, color, config_schema). When the user manually modifies the node configuration on the front-end canvas, the modified configuration data (conforming to the JSON Schema specification) is written back to the default_config field of etl_node_registry, becoming the default reference for the AI ​​when generating the process next time, forming a positive feedback loop of "human modification → machine learning → more accurate recommendations next time".

[0096] Dynamic forms ensure that new hot-swappable components can be displayed in the configuration interface without front-end modifications, lowering the barrier to entry for hot-swappable components; digital twins provide real-time feedback on the execution status of hot-swappable components to users, enabling them to intuitively perceive the health status of the components and promptly identify performance anomalies or data quality issues, thereby driving the optimization or replacement of hot-swappable components.

[0097] Lineage data is collected in real time by hot-swappable actuators during execution. Without unified data points for hot-swappable actuators, lineage relationships cannot be established. The results of lineage analysis (such as the scope of impact and regression test checklist) are returned to the user through AI dialogue. The user decides whether to adjust the process configuration based on the analysis results. The adjusted configuration takes effect immediately through the hot-swappable mechanism, forming a governance closed loop of "execution → analysis → decision → adjustment → re-execution".

[0098] Based on the same inventive concept, such as Figure 4 As shown, this application also provides a workflow orchestration device for big data scenarios, which includes: The task acquisition module 10 is used to acquire the natural language task description input by the user; The intent parsing module 20 is used to parse the natural language task description based on the preset artificial intelligence extension system, and determine the target task type and the corresponding process configuration parameters. The process construction module 30 is used to match the corresponding execution unit from the dynamically registered component set according to the process configuration parameters, and construct a business process containing several processing nodes. The execution rendering module 40 is used to render the business process through a visual interface and schedule the execution of the business process based on the execution unit.

[0099] It should be noted that the workflow orchestration device for big data scenarios provided in this application embodiment and the workflow orchestration method for big data scenarios provided in this application embodiment are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned workflow orchestration method for big data scenarios, and the repeated parts will not be described again.

[0100] In some embodiments, an electronic device provided in this application includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the above-described process orchestration method for big data scenarios.

[0101] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0102] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0103] This application also provides a storage medium storing a computer program that, when executed by a processor, implements the aforementioned workflow orchestration method for big data scenarios. This storage medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not be assembled into that device / apparatus / system. The storage medium carries one or more programs, which, when executed, implement the methods described in the embodiments or implementations of this application.

[0104] According to embodiments of this application, the storage medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. A computer-readable signal medium may also be any storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0105] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0106] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A process orchestration method for big data scenarios, characterized in that, The method includes: Obtain a natural language task description from user input; Based on a pre-defined artificial intelligence extension system, the intent of the natural language task description is parsed to determine the target task type and the corresponding process configuration parameters. Based on the process configuration parameters, the corresponding execution unit is matched from the dynamically registered component set, and a business process containing several processing nodes is constructed. The business process is rendered through a visual interface, and the execution of the business process is scheduled based on the execution unit.

2. The method as described in claim 1, characterized in that, The steps of performing intent parsing on the natural language task description based on a preset artificial intelligence extension system to determine the target task type and corresponding process configuration parameters include: The pre-set intent parsing model in the artificial intelligence extension system is invoked to perform semantic recognition on the natural language task description to obtain the task intent label; The corresponding extended dimension executor is matched according to the task intent label. The extended dimension executor includes one or more of the following: atomic tool executor, protocol service executor, sub-agent executor, and skill pack executor. The process configuration parameters are generated by the extended dimension executor in conjunction with the dynamically registered component metadata information.

3. The method as described in claim 2, characterized in that, The step of matching the corresponding execution unit from the dynamically registered component set according to the process configuration parameters and constructing a business process containing several processing nodes includes: Iterate through the node type identifiers in the process configuration parameters; Query the preset node registry to obtain the loading strategy and component definition information corresponding to the node type identifier; According to the loading strategy, the corresponding execution unit instance is loaded from the component repository; The execution unit instances are connected according to the topological relationship in the process configuration parameters to generate the business process.

4. The method as described in claim 3, characterized in that, The step of loading the corresponding execution unit instance from the component repository according to the loading strategy includes: If the loading strategy is the first loading strategy, then a pre-set native execution unit instance is obtained from the runtime container; If the loading strategy is the second loading strategy, the script code is dynamically compiled to generate a script execution unit instance, and the calling permissions of the script code are security-verified. If the loading strategy is the third loading strategy, then the plugin file is loaded through an isolated class loader, a plugin execution unit instance is generated, and lifecycle isolation is performed on different versions of the plugin execution unit instance.

5. The method as described in claim 1, characterized in that, After the step of rendering the business process through a visual interface and scheduling the execution of the business process based on the execution unit, the method further includes: During the execution of the business process, the operational metrics data and field-level data lineage of each processing node are collected in real time; The operational indicator data is mapped to the corresponding nodes on the visualization interface for status display; In response to a user's bloodline query request, a graph traversal is performed based on the bloodline relationship data at the field level to generate an impact range analysis report.

6. The method as described in claim 5, characterized in that, The steps for real-time collection of operational indicator data and field-level data lineage of each processing node include: For relational database nodes, the field mapping relationships are extracted by parsing the abstract syntax tree of the structured query statement; For non-relational data source nodes, extract field input / output mappings are defined by intercepting operation logs or using aggregation pipelines; For message queue nodes, a field mapping chain between the producer and consumer is established by parsing the metadata information in the message header; For application interface nodes, establish field mapping relationships between upstream and downstream interfaces by intercepting request and response data; For file data source nodes, a mapping identifier between file paths and fields is established by parsing the file structure.

7. The method as described in claim 1, characterized in that, The step of obtaining the natural language task description input by the user also includes: In response to the user's selection of a preset template, load the corresponding preset dialogue configuration; The preset dialogue configuration includes basic model configuration, system prompts, and a list of enabled extended dimensions; The AI ​​dialogue context is initialized based on the preset dialogue configuration.

8. A workflow orchestration device for big data scenarios, characterized in that, The process orchestration device for big data scenarios includes: The task acquisition module is used to acquire the natural language task description input by the user; The intent parsing module is used to parse the natural language task description based on a preset artificial intelligence extension system to determine the target task type and the corresponding process configuration parameters. The process construction module is used to match the corresponding execution unit from the dynamically registered component set according to the process configuration parameters, and construct a business process containing several processing nodes. The execution rendering module is used to render the business process through a visual interface and to schedule and execute the business process based on the execution unit.

9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the workflow orchestration method for big data scenarios as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the workflow orchestration method for big data scenarios as described in any one of claims 1 to 7.

Images