An ETL driving-based no-code data service generation method and system

Abstract

This invention discloses a no-code data service generation method based on ETL (Extract, Translate, and Load) driven technology. It automatically parses ETL files, converts them into data service interfaces or registers them as scheduled tasks, and combines AI assistance and internal / external network mapping technology to achieve "design as a service" for the data flow. The method includes the following steps: S1 receiving and parsing the ETL file; S2 automatically generating standardized data interfaces; S3 generating machine-executable schedulable strategies; S4 dynamically mapping internal / external network database connections; S5 intelligent task scheduling and execution; S6 real-time monitoring and log push; and S7 AI-assisted support. This invention also discloses the system, including a file receiving and parsing module, a no-code service generation module, an internal / external network mapping module, a task scheduling and execution engine module, a real-time monitoring and logging module, and an AI intelligent assistant module. This invention solves the technical gap between ETL data flow and API service-oriented architecture, and has advantages such as deep integration and enhanced decoupling.

Description

Technical Field

[0001] This invention relates to the field of data processing and service technology, and in particular to a method and system for generating, scheduling and executing, and assisting in the creation of code-free data services based on ETL (Extract, Transform, Load) technology. Background Technology

[0002] In today's enterprise digital transformation process, the ability to rapidly deliver data processing and services has become a core competitive advantage. Extraction, transformation, and loading (ETL), as a core technology for data integration and processing, plays a crucial role in enterprise data architecture. ETL tools (such as a mainstream visual ETL tool) allow users to design data processing workflows through a visual drag-and-drop interface, generating executable transformation files (e.g., .ktr files) and job files (e.g., .kjb files), greatly simplifying the development of complex data processes. On the other hand, RESTful APIs have become the de facto standard for data interaction between modern application systems. However, within the existing technology framework, there is a significant technological gap between the data processing capabilities of ETL tools and API service-oriented technologies.

[0003] Based on traditional data processing and development models, the industry currently mainly offers the following technical solutions: 1. Purely manual coding mode: Developers implemented data querying, cleaning, and transformation logic entirely through coding and encapsulated it as an API. While this approach offers high flexibility, it inherently suffers from long development cycles, poor code reusability, and high maintenance costs associated with changes to business logic. Each data processing requirement necessitates development from scratch, leading to a significant waste of human resources. Furthermore, the process exhibits the following pain points: Low development efficiency: Each data processing requirement requires repetitive coding, resulting in a long cycle, often taking 1-2 weeks from requirement submission to deployment.

[0004] High communication costs: Business logic is prone to deviations during transmission, leading to rework.

[0005] Poor flexibility: Business logic changes require developers to modify code, retest, and redeploy, resulting in slow response times.

[0006] High technical threshold: Complex scheduled task configurations (such as Cron expressions) and business data processing logic can only be configured and written by developers, and non-developers cannot participate.

[0007] 2. Using ETL tools: Business personnel use ETL tools to design data processing logic and generate final data processing results. Developers write bridging code to capture the execution results, interface with the result data, and then encapsulate it into an API service when necessary. Several ETL integration solutions already exist on the market; however, these solutions generally suffer from limited functionality and low integration. They typically only implement the most basic "execution triggering" function, providing a general HTTP endpoint to start a predefined ETL task, but lack the ability to build a complete and usable data service API. These are common practices, but they have the following technical limitations: No callable result API provided: Unable to automatically generate a RESTful API interface with a canonical path and request / response model based on the input parameters and output structure defined in the ETL task.

[0008] The output is non-standardized: the raw data generated after an ETL task is executed (such as file, database table changes, or command-line output) cannot be automatically packaged into a standardized HTTP response. Developers must write additional adaptation code to convert the processing results into structured data formats such as JSON and XML that clients expect, and to handle success and failure status codes.

[0009] Weak support for multi-task scheduling: Existing solutions are usually limited to single-task triggering and lack the ability to schedule multi-tasks for complex business processes. For scenarios that need to be executed sequentially according to dependencies, triggered by events, or that need to handle loop execution logic, these solutions cannot provide a visual and configurable solution and still need to rely on external scripts or scheduling systems, resulting in a complex and fragmented system architecture.

[0010] Current technical solutions have failed to solve the problem of automatically converting mature ETL processes designed by business personnel into ready-to-use data services. There is a technical gap between ETL capabilities and service-oriented requirements, and the "design as a service" concept of data processing logic has not been realized. They are also lacking in terms of making the entire workflow lifecycle more intelligent and adapting to complex network environments (such as the internal and external network isolation of government networks). Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides a no-code data service generation method and system based on ETL-driven architecture, which aims to solve the technical problems of technical gaps between ETL data flow and API service, low development and delivery efficiency, low level of intelligence, and difficulty in adapting to complex network environments in the prior art. This invention breaks down the barriers between business and development, realizes "design as a service", adapts to complex network environments, and deeply integrates large-scale intelligent assistance.

[0012] To achieve the above objectives, the present invention adopts the following technical solution: A no-code data service generation method based on ETL automatically parses ETL files, converts them into data service interfaces or registers them as scheduled tasks, and combines AI assistance and internal / external network mapping technology to achieve "design as a service" and intelligent operation and maintenance of data flows. Specifically, it includes the following steps: S1 receives and parses ETL files: It receives ETL job files or transformation files uploaded by users, performs in-depth parsing on them, and extracts data processing logic, predefined parameters and data source connection information.

[0013] S2 Automatically Generates Standardized Data Interfaces: Based on the data processing logic, predefined parameters, and data source connection information extracted in step S1, standardized data service interfaces are automatically generated and published through the no-code service generation module.

[0014] Preferably, in step S2, the automatic generation and publication of standardized data service interfaces specifically includes: S21. Automatically map the predefined parameters in the ETL file to the HTTP request parameters of the data service interface; S22. Capture and map the final output data of the ETL process into a standard JSON response body of the data service interface.

[0015] S3 generates a machine-executable schedulable strategy: Based on the ETL file context information parsed in step S1, the system receives the user's scheduled task requirements input in natural language, performs semantic analysis on the natural language description through an integrated large language model, and automatically generates the corresponding machine-executable scheduled strategy.

[0016] Furthermore, in step S3, the semantic analysis involves converting the user-input natural language description into a standard Cron expression.

[0017] S4 Dynamic mapping of internal and external network database connections: Based on the data source connection information extracted in step S1, before executing the ETL file task, check the internal and external network database mapping switch; if enabled, dynamically map the database connection address in the ETL file according to the preset mapping rules.

[0018] S5 Intelligent Task Scheduling and Execution: Based on the timed scheduling strategy generated in step S3 or triggered by the data service interface generated in step S2, and using the data source connection information mapped in step S4, the corresponding ETL file is automatically executed.

[0019] Furthermore, the automated execution includes support for configuring intelligent retry strategies, including linear incremental retries or exponential backoff retries.

[0020] S6 Real-time Monitoring and Log Push: When executing the ETL file in step S5, the task execution status is monitored in real time, and the execution log is pushed to the front-end user in real time via WebSocket technology.

[0021] S7 AI Assistance: Provides parallel AI assistance throughout the entire lifecycle of the ETL process in steps S1-S6; specifically, it includes: receiving natural language questions from users regarding ETL process design, error diagnosis, or performance optimization; analyzing the natural language questions by combining a large language model of the ETL domain knowledge base, and generating and outputting professional assistance suggestions.

[0022] An ETL-driven no-code data service generation system is disclosed. The system includes a file receiving and parsing module, a no-code service generation module, an internal and external network mapping module, a task scheduling and execution engine module, a real-time monitoring and logging module, and an AI intelligent assistant module.

[0023] The file receiving and parsing module is used to receive and parse user-uploaded ETL jobs or conversion files, extract key information, and output the parsing results to the no-code service generation module and the task scheduling and execution engine module.

[0024] The no-code service generation module is used to automatically generate and publish standardized data service interfaces for external systems to call, based on the key information extracted by the file receiving and parsing module and through the built-in interface generation unit.

[0025] Preferably, the no-code service generation module is further configured to: automatically map predefined parameters in the ETL file to HTTP request parameters of the data service interface; and capture and map the final output data of the ETL process to the standard JSON response body of the data service interface.

[0026] The internal and external network mapping module is called before the task scheduling and execution engine module executes tasks to determine and perform dynamic mapping of the database connection address in the ETL file.

[0027] Furthermore, the dynamic mapping includes the conversion of service addresses, port numbers, database names, usernames, passwords, and authentication information on both internal and external networks, in order to adapt to database access under different network environments.

[0028] The task scheduling and execution engine module is used to receive scheduling strategies generated by the AI ​​intelligent assistant module or trigger interface calls from the no-code service generation module, and automatically execute the corresponding ETL files.

[0029] Furthermore, the automated execution of the corresponding ETL file is achieved by calling the system's built-in ETL runtime environment and loading the execution configuration processed by the internal and external network mapping module.

[0030] Furthermore, the task scheduling and execution engine module is further configured to support intelligent retry strategies, including linear incremental retry or exponential backoff retry.

[0031] The real-time monitoring and logging module is used to receive execution logs from the task scheduling and execution engine module, monitor task status, and push them to the front-end user in real time via WebSocket technology.

[0032] The AI ​​intelligent assistant module is used to integrate a large language model and combine it with an ETL domain knowledge base to provide intelligent assistance throughout the entire lifecycle of the ETL process.

[0033] Preferably, the AI ​​intelligent assistant module includes: (1) Scheduling strategy generation unit: used to convert the user's natural language scheduling requirements into machine-executable timed scheduling strategies and output them to the task scheduling and execution engine module; (2) Intelligent Assist Unit: Used to respond to user inquiries and provide auxiliary suggestions for ETL process design, error diagnosis and performance optimization.

[0034] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements any of the above method steps (S1 to S7).

[0035] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above method steps (S1 to S7).

[0036] The beneficial effects of this invention are as follows: 1. Improve efficiency and realize "design as a service": This invention breaks down the barriers between business design and technical implementation by automatically converting ETL files into standard APIs, enabling ETL processes designed by business personnel to be "designed as a service", shortening the traditional data requirement delivery cycle of several weeks to hours or even minutes, and greatly improving delivery efficiency.

[0037] 2. Deep integration of intelligence and automation: This invention integrates an AI intelligent assistant, which allows users to configure complex scheduled tasks and obtain process diagnosis and performance optimization suggestions using natural language, greatly reducing the threshold for use; at the same time, it combines an enterprise-level scheduling engine and dynamic mapping capabilities between internal and external networks to achieve intelligent, automated and highly available data process throughout its entire lifecycle.

[0038] 3. Enhanced Decoupling and Environmental Adaptability: This invention separates data processing logic from business systems and manages it uniformly through ETL files and APIs, thereby decoupling data logic from business logic and improving reusability. Furthermore, through dynamic mapping between internal and external networks and plug-in design, it efficiently adapts to complex network environments (such as internal and external network isolation) and specific database environments, demonstrating strong technological foresight and scalability. Attached Figure Description

[0039] Figure 1 This is a diagram illustrating the overall system architecture and collaborative workflow provided in this embodiment of the invention.

[0040] Figure 2 This is a path diagram for ETL file processing and service-oriented architecture of modules (110 / 115 / 120 / 140) provided in the embodiments of the present invention.

[0041] Figure 3 This is a diagram of the intelligent scheduling function architecture of modules (130 / 140) provided in the embodiments of the present invention.

[0042] Figure 4 This is a functional architecture diagram of module (130) AI intelligent assistant provided in the embodiment of the present invention.

[0043] Figure 5 This is a system integration call architecture diagram of modules (115 / 120 / 140) provided in an embodiment of the present invention.

[0044] Figure 6 This is a flowchart of the method execution provided in the embodiments of the present invention. Detailed Implementation

[0045] The following will refer to the accompanying drawings in the embodiments of the present invention ( Figures 1 to 6 This document provides a clear and complete description of the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0046] Figure 1 This is a system architecture diagram provided by an embodiment of the present invention. The interaction and data flow of the system clearly demonstrate the complete closed loop from design to execution. The system can be built on a server to provide data services to enterprises. Logically, the system includes the following modules: file receiving and parsing module (110), internal and external network mapping module (115), no-code service generation module (120), AI intelligent assistant module (130), task scheduling and execution engine (140), and real-time monitoring and logging module (150).

[0047] like Figure 1 As shown, the collaborative workflow of the system is as follows: Business personnel or maintenance personnel interact with the system (100) of this invention through a visual operation interface.

[0048] Input: Business personnel use external visual ETL tools (such as a mainstream visual ETL tool) to design data flows and generate ETL files (such as .ktr or .kjb files). Subsequently, business personnel upload the ETL file through the system's visual interface, and the file is received by the file receiving and parsing module (110).

[0049] Parsing and Distribution: The file receiving and parsing module (110) parses the received file, extracts key information, and then distributes it to two paths: (2.1) Output to the no-code service generation module (120) to generate API. (2.2) Output to the task scheduling and execution engine module (140) for registering schedulable tasks.

[0050] (3) Execution Trigger: The execution of the task scheduling and execution engine (140) is triggered in two ways: (3.1) Scheduling strategy trigger: After receiving natural language input, the AI ​​intelligent assistant module (130) generates a scheduling strategy and sends it to the engine (140).

[0051] (3.2) Interface call trigger: The no-code service generation module (120) generates an API for developers / business systems to call. When the API is called externally, the module (120) sends an interface call trigger signal to the engine (140).

[0052] (4) Unified execution and mapping: After receiving any of the above trigger methods, the task scheduling and execution engine (140) calls the internal and external network mapping module (115). The module (115) is responsible for handling the dynamic mapping of database addresses and returning the mapped configuration to the engine (140). The engine (140) continues to execute the ETL task according to the configuration content.

[0053] (5) Output and monitoring: During the execution process, the engine (140) sends the execution log to the real-time monitoring and logging module (150). The module (150) is responsible for monitoring the task status and feeding back the real-time logs or alarms to business personnel / operations personnel through technologies such as WebSocket.

[0054] (6) AI Parallel Assistance: In the above process, the AI ​​intelligent assistant module (130) also provides intelligent assistance support to business personnel / operation and maintenance personnel in parallel, such as error diagnosis and performance optimization.

[0055] Through this architecture, the system (100) of this invention integrates the parsing, service-oriented architecture, intelligent scheduling, unified execution, dynamic mapping, real-time monitoring and intelligent assistance of ETL files, forming an efficient, intelligent and automated data service generation and management platform.

[0056] The following will combine Figures 2 to 5 This section provides a more detailed explanation of the functions of each critical path and module.

[0057] Figure 2 This is a diagram illustrating the ETL file processing and service-oriented path of modules (110 / 115 / 120 / 140) provided in this embodiment of the invention. The diagram details the two main definition paths and the unified execution path after the ETL file is received by the system.

[0058] File receiving and parsing module (110): As the starting point of the process, business personnel design the process using a visual ETL tool and generate an ETL file. The file can be uploaded through the system of this invention (100) or automatically obtained by connecting to a resource library. This module is responsible for reading the contents of the ETL file (usually in ktr or kjb format), deeply parsing its structure, and extracting all key information, such as data source connection, data processing logic steps, and "named parameters" or "variables" defined in the file.

[0059] After parsing, different execution paths will be used depending on the specific requirements: (2.1) Path 1: API Service: Module (110) will read and parse the file, and at the same time, it will be standardized through the no-code service generation module (120). Module (120) will automatically generate standardized RESTful API interfaces and publish the interfaces. This path realizes "design as a service".

[0060] For example, suppose a business user uploads a business data processing ETL file named get_sales_report.ktr through module (110). This file defines two named parameters in the ETL visualization tool: ${param_start_date} and ${param_region}. Its final output step (e.g., "JSONOutput") contains three fields: order_id, amount, and customer_name.

[0061] Based on the parsed ETL file content, the no-code service generation module (120) will perform a mapping of its input and output parameters, as shown in the table below:

[0062] This module (120) generates and publishes a standardized RESTful API interface and provides a unique API endpoint, such as / api / service / transform / get_sales_report. External business systems (such as...) Figure 5 As shown, the ETL business logic process can be invoked in real time via an HTTP POST request, and standardized JSON response data can be obtained. External business systems only need to define the standardized response data structure during the initial access, and then they can automatically access the API interface generated by the system (100) of this invention. At the same time, modifications to the business logic made by business personnel or operations personnel on the ETL visualization tool can be directly reflected in the business system without the intervention of developers. This path realizes "design as a service".

[0063] (2.2) Path 2: Scheduled task registration. Module (110) registers the parsed result as an executable task, waiting for the timing scheduling strategy (the Cron expression generated in S3) to be triggered.

[0064] (3) Unified execution path: Whether the API generated through path one is called by an external system or the task registered through path two is triggered by the scheduler, both methods will go through the internal and external network mapping module (115).

[0065] (3.1) Specifically, network mapping will be performed: If the internal / external network mapping switch is on, the task first enters the internal / external network mapping module (115). In this module, the system will perform internal / external network database mapping, specifically including finding the registered mapping rules and handling the dynamic replacement of database addresses in the ETL file within the program. If the internal / external network mapping switch is off, the task is directly sent to the task scheduling and execution engine (140). After processing, the execution context adapted to the internal network environment is sent to the task scheduling and execution engine (140).

[0066] (4) Execution and Access: The task scheduling and execution engine (140) ultimately receives and executes the tasks from the upper layer. The execution engine (140) accesses the corresponding data source according to the definition in the ETL file, such as relational databases (MySQL, SQL Server, DM and domestic databases, etc.), file data sources (including Excel, CSV files, etc.) or other data sources.

[0067] (5) Output results: The execution results of the ETL execution engine (140) are output in the form of execution logs and results, and are captured by the real-time monitoring and logging module (150).

[0068] Figure 3This is an architecture diagram of the intelligent scheduling function of modules (130 / 140) provided in this embodiment of the invention. This function is implemented collaboratively by the AI ​​intelligent assistant module (130) and the task scheduling and execution engine (140). The starting point of scheduling is the policy configuration center, where users initiate scheduling configuration requests. The request will enter the Cron expression generation engine (130), which provides two expression generation methods: manual configuration mode and AI intelligent generation mode.

[0069] AI-powered intelligent generation mode: This is a more intelligent and user-friendly mode. Users enter everyday phrases in the natural language input box (as shown in the example: "for example: 2 AM every day"). This input is submitted to the AI ​​big data model for semantic parsing. After understanding the user's intent, the big data model directly generates the corresponding Cron expression.

[0070] Manual Configuration Mode: A traditional mode for more specialized users. Users can define Cron expressions through a visual configuration interface, quick configuration templates, or by directly entering advanced manual input. The results of all manual configuration methods are displayed in a real-time expression preview for user confirmation.

[0071] Regardless of the mode used, the generated expression will be validated by a Cron expression validator to check its syntactic correctness, ensure it is in a machine-executable format, and output a standard Cron expression. The table below shows several examples of AI-generated modes:

[0072] This Cron expression is used to generate and preview execution plans, allowing users to visually preview the trigger times of future tasks (such as "2025-11-02 23:00:00", "2025-11-03 23:00:00", etc.).

[0073] Ultimately, these execution plans are submitted to the engine (140), which stores the plan in the scheduling queue and is responsible for precisely waking up and executing the corresponding ETL task at the specified time.

[0074] Figure 4 This is a functional architecture diagram of module (130) AI intelligent assistant provided in an embodiment of the present invention. This module (130) is designed to provide deep intelligent support for the entire lifecycle of ETL data flow (including design, configuration, execution, monitoring and troubleshooting).

[0075] This module is logically divided into three core layers: Bottom Layer: Core Technology Framework Layer. This layer forms the technological foundation and execution engine for implementing intelligent assistants. It includes: AI application development framework: As the brain of AIAgent and the business process orchestrator, it is responsible for parsing user intent and calling other components to work together.

[0076] Large Model Integration Layer: As an abstraction layer of model capabilities, it is configured to integrate and manage one or more large language models, and supports flexible switching between online large models (such as APIs provided by cloud vendors) and local large models (such as models deployed in private environments) to adapt to the security and compliance needs of different enterprises.

[0077] Context manager and message history management work together to achieve context-aware capabilities. They are responsible for maintaining multi-turn dialogue states, temporarily storing user-uploaded files (such as ETL files and log files), and ensuring that AI responses are closely related to the current session.

[0078] Streaming output engine: Configured to progressively transmit responses generated from large models to the front end in a streaming manner, providing an instant and smooth interactive experience.

[0079] Middle Layer: Domain Knowledge Enhancement. This layer is the core source of the AI ​​assistant's professional capabilities. Through Retrieval-Augmented Generation (RAG) technology, it injects ETL domain expertise into the core technology framework layer. It includes: ETL-specific knowledge base: A vectorized knowledge base that serves as an external expert memory bank in addition to the basic knowledge of large AI models.

[0080] System prompt word engineering: Defines the role, responsibilities, and response specifications of the AI ​​assistant (e.g., making it act as an "ETL performance optimization expert").

[0081] The knowledge base includes subsets such as the ETL best practices library, common error code library, and performance optimization technique library, which together constitute the original data source for the dedicated knowledge base.

[0082] Top Layer: Core Intelligent Assistive Functions. This layer represents the final value delivered by module (130) to the user, and is the product of the synergy between the technical framework and domain knowledge. These functions are presented to the user through external system integration (e.g., the front-end UI interface of the system of this invention), and mainly include: ETL process design assistance: Provides suggestions for users to design ETL processes.

[0083] Performance optimization recommendations: Diagnose performance bottlenecks in ETL files and provide optimization solutions.

[0084] Intelligent SQL statement generation: Automatically generates or optimizes SQL based on natural language descriptions.

[0085] Error diagnosis and resolution: Parse error logs and provide solutions.

[0086] Intelligent scheduling strategy generation: i.e. Figure 3 As shown, natural language is converted into Cron expressions.

[0087] The following specific example illustrates how the various modules of the AI ​​intelligent assistant (130) in this invention work together: User Input and Context Building: A business user uploaded an ETL transformation file named extract_sales_data.ktr through the front-end interface and asked, "This transformation file is executing very slowly. Please help me analyze the reason and provide optimization suggestions." This request was received by the AI ​​application development framework. The context manager associated the uploaded ETL file with the user's natural language question and stored it in the message history management.

[0088] Intent Recognition and Knowledge Retrieval: The AI ​​application development framework identifies the user's intent as performance optimization suggestions. Simultaneously, it performs preliminary parsing of the ETL file, extracting configurations for key steps. The framework then initiates a retrieval request to the domain knowledge enhancement layer. This request searches the performance optimization technique library and the ETL best practice library for knowledge fragments related to "slow table input," "SQL optimization," and "table output commit size."

[0089] Enhanced Prompt and Large Model Inference: The framework assembles the original question, key configurations from the parsed ETL file, and knowledge fragments retrieved from the knowledge base to form a rich, context-complete prompt. This prompt is then sent to the large model integration layer for inference.

[0090] Response Generation and Streaming Output: Based on the enhanced hints, the large model generates professional optimization suggestions that incorporate the specific content of the ETL file. This response is pushed to the front-end interface in real time through the streaming output engine, and the user will see a reply like the following: "Hello, I analyzed the extract_sales_data.ktr file you uploaded and found the following potential performance bottlenecks: SQL query optimization: In the 'Table Input' step, you used SELECT * FROM sales_raw. This reads all columns, increasing the I / O burden. It is recommended that you only select the fields required by your business, such as: SELECT order_id, customer_name, amount FROM sales_raw. Batch commit optimization: In the 'Table Output' step, your 'Commit Size' is set to 100. This value is too small, causing the database to commit transactions frequently. It is recommended that you increase this value to 5000 or 10000, depending on your database performance, to improve write efficiency. Index check: If your sales_raw table has a large amount of data, please ensure that the fields involved in the query conditions (WHERE clause) have database indexes." Through the above process, the AI ​​intelligent assistant (130) not only provides general optimization theories, but also provides professional guidance that can be executed immediately based on the specific content of the user's uploaded files, deeply empowering the entire lifecycle management of ETL.

[0091] Figure 5 This is a system integration call architecture diagram of module 120 / 140 provided in an embodiment of the present invention. The diagram illustrates how the system of the present invention can be seamlessly integrated with external business application systems.

[0092] The specific integration process is as follows: Caller Configuration: The application system (such as an enterprise's front-end business framework or other business systems) acts as the API consumer. In order to call the services of this invention, developers need to configure the API call on the caller (i.e., the API consumer), including defining the API address of the target service, token authentication for identity verification, and finally encapsulating it into a standard HTTP request.

[0093] The called party processes the standard HTTP request received by the system from the caller. This request first reaches the system's gateway layer, namely the no-code service interface, which is generated and exposed by the no-code service generation module (120). Upon receiving the request, the system triggers the execution flow, first passing the task to the internal / external network mapping module (115).

[0094] Mapping and Execution: The internal and external network mapping module (115) acts as a "preprocessor" before execution. It is responsible for checking and applying the internal and external network mapping rules to ensure that the data source connection defined in the ETL file is adapted to the current production network environment. After mapping, the module (115) submits the adapted execution configuration and task parameters to the ETL execution engine, which is handled by the engine (140).

[0095] Data Access and Response: After receiving the task, the ETL execution engine begins the actual data processing work. It accesses the data source (such as database, file, etc.) according to the definition of the ETL file. After execution, the engine (140) encapsulates the final output of the ETL process into a standardized JSON response. This response is returned in reverse along the call chain. First, it returns the HTTP request handle to the API caller, and finally it is received by the business application system, thus completing a real-time data call of "design as a service".

[0096] Figure 6 This is a flowchart illustrating the execution steps of the method of the present invention provided in an embodiment of the present invention. This method can be based on the above-described system architecture (such as...). Figure 1 (As shown in the diagram) to execute. This process clearly demonstrates the separation and unity of "definition" and "execution," and how AI assistance is integrated throughout the entire process.

[0097] S1 accepts and parses the ETL file. The process begins when a user (e.g., a business user) uploads a pre-designed job file (e.g., a .kjb file) or conversion file (e.g., a .ktr file) through the visual interface of the system (100). The system's file receiving and parsing module (110) receives the file and immediately performs structured parsing of its contents, extracting the complete logic of data processing, including data source connection information, all processing steps (e.g., "table input," "table output," "JSON output"), and named parameters or variables defined in the file.

[0098] After step S1 is completed, the process is logically divided into two parallel "definition" phases (S2 and S3) to prepare for two different triggering methods (API call or timed scheduling): S2 automatically generates standardized data interfaces (service-oriented paths). This path corresponds to... Figure 2 The first path is "API service-oriented". Based on S1, the no-code service generation module (120) automatically executes the interface generation logic.

[0099] (1) Input parameter mapping: Module (120) maps the named parameters defined in the ETL file to JSON fields in the HTTP request body of the API.

[0100] (2) Output parameter mapping: Module (120) encapsulates the data structure of the final output step in the ETL process into a JSON response body (ResponseBody) of the standard API.

[0101] (3) After generation, the API is published and activated, obtaining a unique API endpoint. For example... Figure 5 As shown, external business application systems can call this interface in real time through API call configurations (such as token authentication and HTTP requests), realizing "design as a service".

[0102] S3 generates a scheduling policy (scheduling path) that can be executed by the machine. This path corresponds to... Figure 2 The second path is "Scheduled Task Registration". For ETL tasks that need to be executed periodically, users can configure them through the AI ​​intelligent assistant module (130) in the system's scheduled task configuration interface.

[0103] (1) As Figure 3 As shown, users can select "AI Intelligent Generation Mode". Enter everyday phrases (such as "10:30 pm every weekday") in the natural language input box.

[0104] (2) In module (130), the AI ​​big model will analyze the received natural language text for semantic parsing and convert it into a standard Cron expression (e.g., 03022?*MON-FRI).

[0105] (3) These strategies are saved by the system and associated with the ETL file uploaded in S1, and registered in the task scheduling and execution engine (140) to wait for the timed scheduling strategy to be triggered.

[0106] Whether the API generated by S2 is called by an external system, or the scheduling policy generated by S3 is triggered by the system on time, both execution paths will converge to S4 and enter a unified execution flow: S4 checks and executes the mapping between internal and external network databases. This step is a necessary prerequisite for all execution tasks (whether it is a real-time API call or a scheduled task), and is designed to solve deployment problems in complex network environments (such as the isolation between internal and external networks of government networks).

[0107] (1) As Figure 2 and Figure 5 As shown, the system checks the internal and external network mapping switches before the task is about to be executed.

[0108] (2) If the switch is on, the task is first routed to the internal / external network mapping module (115). The module (115) dynamically replaces the ETL file database connection address parsed by S1 according to the preset mapping rules (for example, mapping the external network test database 192.168.1.10:3306 / test_db to the internal network production database 10.0.0.50:3306 / prod_db).

[0109] (3) If the switch is off, skip this step.

[0110] S5 automated execution. The system's task scheduling and execution engine (140) receives the task after S4 mapping processing and immediately (for API calls) or according to plan (for scheduled tasks) creates and executes the corresponding ETL job or transformation instance on the server. The engine (140) is responsible for managing the thread pool, concurrency control, and resource allocation, and accesses the corresponding data source (such as...) according to the ETL definition. Figure 2 (Databases, files, etc. in the database).

[0111] S6 Real-time Monitoring and Log Push. This step is tightly coupled with the execution process of S5. During the execution of the S5 ETL file, the real-time monitoring and logging module (150) remains active. It captures the standard output (stdout) and error output (stderr) of the ETL engine and continuously pushes the execution log (including startup, execution steps, number of lines read / written, error messages, etc.) to the front-end visualization interface via WebSocket, ensuring that users are aware of the task status in real time, forming a complete management loop. After S5 is completed, the process execution is finished.

[0112] S7 AI Assistance Support. AI assistance support is a continuous, parallel process provided by the AI ​​Intelligent Assistant module (130), such as... Figure 4 As shown.

[0113] Architecture: This module is built upon an AI application development framework and a domain knowledge enhancement layer. The knowledge base includes ETL best practices, common error codes, and performance optimization techniques.

[0114] Example: If S5 fails, the user can obtain the error log pushed by S7, upload the error information to the AI ​​intelligent assistant (130), and ask: "This ETL task failed. The log is attached. Please help me diagnose the cause." Processing: The AI ​​framework of module (130) receives the problem and log file, first calls the knowledge base to retrieve the error code, and then combines the large model and system prompt words to perform contextual analysis on the error log.

[0115] Output: Finally, the AI ​​assistant (130) provides specific diagnoses and solutions, and outputs them through a streaming output engine (e.g., "Hello, according to log analysis, error code ORA-00942 indicates 'table or view does not exist'. Please check if the table name you configured in the 'Table Input' step is correct, or confirm whether the database user has read permissions for the table.").

[0116] Through the collaborative work of steps S1 to S7 above, this embodiment of the invention establishes a complete closed loop from ETL process design to API service provisioning (S1->S2->S4->S5->S6) and from ETL process design to automated scheduling (S1->S3->S4->S5->S6). Furthermore, through the AI-assisted capabilities of S7, it greatly improves the delivery efficiency of data services and the level of intelligent operation and maintenance.

[0117] This invention also provides an electronic device, which may include: a processor, a memory, and a computer program stored in the memory and executable on the processor. The memory is used to store program code, ETL files, generated API configurations, scheduling policies, etc. When the processor executes the computer program, it implements the above-described method embodiments (such as...). Figure 6 The steps in S1 to S7 (as shown) are as follows. The electronic device can be a server, a personal computer (PC), a workstation, etc.

[0118] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method embodiments (e.g., ...). Figure 6 The steps in S1 to S7 (as shown) are as follows. The computer-readable storage medium can be a volatile or non-volatile storage medium, such as a read-only memory (ROM), a random access memory (RAM), a hard disk, or an optical disk.

[0119] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "preferred embodiment," and "illustrative embodiment" mean that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0120] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An ETL driver-based no-code data service generation method, characterized by: By automating the parsing of ETL files and converting them into data service interfaces or registering them as scheduling tasks, and combining AI assistance with internal and external network mapping technology, the "design as a service" and intelligent operation and maintenance of data processes are realized. Specifically, this includes the following steps: S1 receives and parses ETL files: It receives ETL job files or transformation files uploaded by users, performs in-depth parsing on them, and extracts data processing logic, predefined parameters and data source connection information. S2 Automatically Generates Standardized Data Interfaces: Based on the data processing logic, predefined parameters, and data source connection information extracted in step S1, standardized data service interfaces are automatically generated and published through the no-code service generation module. The automatically generated and published standardized data service interface specifically includes: S21. Automatically map the predefined parameters in the ETL file to the HTTP request parameters of the data service interface; S22. Capture and map the final output data of the ETL process into the standard JSON response body of the data service interface; S3 generates a machine-executable schedulable strategy: Based on the ETL file context information parsed in step S1, the system receives the user's scheduled task requirements in natural language input, performs semantic analysis on the natural language description through an integrated large language model, and automatically generates the corresponding machine-executable scheduled strategy. S4 Dynamic mapping of internal and external network database connections: Based on the data source connection information extracted in step S1, before executing the ETL file task, check the internal and external network database mapping switch; if enabled, dynamically map the database connection address in the ETL file according to the preset mapping rules. S5 Intelligent Task Scheduling and Execution: Based on the timed scheduling strategy generated in step S3 or triggered by the data service interface generated in step S2, and using the data source connection information mapped in step S4, the corresponding ETL file is automatically executed. S6 Real-time Monitoring and Log Push: When executing the ETL file in step S5, the task execution status is monitored in real time, and the execution log is pushed to the front-end user in real time via WebSocket technology; S7 AI Assistance: Provides parallel AI assistance throughout the entire lifecycle of the ETL process in steps S1-S6; specifically, it includes: receiving natural language questions from users regarding ETL process design, error diagnosis, or performance optimization; analyzing the natural language questions by combining a large language model of the ETL domain knowledge base, and generating and outputting professional assistance suggestions.

2. The ETL driver-based no-code data service generation method of claim 1, wherein In step S3, the semantic analysis involves converting the natural language description input by the user into a standard Cron expression.

3. The ETL driver-based no-code data service generation method of claim 1, wherein: In step S5, the automated execution includes support for configuring intelligent retry strategies, including linear incremental retry or exponential backoff retry.

4. An ETL-driven no-code data service generation system, characterized by: The system includes a file receiving and parsing module, a no-code service generation module, an internal and external network mapping module, a task scheduling and execution engine module, a real-time monitoring and logging module, and an AI intelligent assistant module; The file receiving and parsing module is used to receive and parse user-uploaded ETL jobs or transformation files, extract key information, and output the parsing results to the no-code service generation module and the task scheduling and execution engine module. The no-code service generation module is used to automatically generate and publish standardized data service interfaces for external systems to call, based on the key information extracted by the file receiving and parsing module and through the built-in interface generation unit. The no-code service generation module is further used to automatically map the predefined parameters in the ETL file to the HTTP request parameters of the data service interface; and to capture and map the final output data of the ETL process to the standard JSON response body of the data service interface. The internal and external network mapping module is used to be called before the task scheduling and execution engine module executes tasks, and to determine and perform dynamic mapping of the database connection address in the ETL file. The task scheduling and execution engine module is used to receive scheduling strategies generated by the AI ​​intelligent assistant module or trigger interface calls from the no-code service generation module, and automatically execute the corresponding ETL files. The real-time monitoring and logging module is used to receive the execution logs from the task scheduling and execution engine module, monitor the task status, and push them to the front-end user in real time via WebSocket technology. The AI ​​intelligent assistant module is used to integrate a large language model and combine it with an ETL domain knowledge base to provide intelligent assistance throughout the entire lifecycle of the ETL process.

5. The ETL-driven no-code data service generation system according to claim 4, characterized in that: The dynamic mapping includes the conversion of service addresses, port numbers, database names, usernames, passwords, and authentication information on both internal and external networks, in order to adapt to database access under different network environments.

6. The ETL-driven no-code data service generation system according to claim 4, characterized in that: The automated execution of the corresponding ETL file is achieved by calling the system's built-in ETL runtime environment and loading the execution configuration processed by the internal and external network mapping module.

7. The ETL-driven no-code data service generation system according to claim 4, characterized in that: The task scheduling and execution engine module is further used to support the configuration of intelligent retry strategies, including linear incremental retry or exponential backoff retry.

8. The ETL-driven no-code data service generation system according to claim 4, characterized in that: The AI ​​intelligent assistant module includes: (1) Scheduling strategy generation unit: used to convert the user's natural language scheduling requirements into machine-executable timed scheduling strategies and output them to the task scheduling and execution engine module; (2) Intelligent Assist Unit: Used to respond to user inquiries and provide auxiliary suggestions for ETL process design, error diagnosis and performance optimization.

Images