Low-code platform-based automated data pipeline generation method, device and medium

Abstract

The application discloses a kind of based on low code platform's automated data pipeline generation method, equipment and medium, belong to data processing technical field.The method is parsed business requirement text by natural language processing technology, extracts structured demand information and is converted into graphical pipeline model;Optimal base template is selected from template library using semantic-driven template matching algorithm, and intermediate template is generated in combination with data type automatic mapping mechanism;Then, the target code is generated and deployed by the engine automatically generated by the large model conversion rule.The application realizes the full-process automation from business requirement to data pipeline, reduces the dependence on professionals, improves the efficiency and adaptability of data pipeline construction, and can be widely applied to heterogeneous data source integration scenarios.

Description

Technical Field

[0001] This document relates to the field of data processing technology, and in particular to an automated data pipeline generation method, device and medium based on a low-code platform. Background Technology

[0002] Currently, enterprise data integration mainly relies on professional developers manually writing ETL code to build data pipelines. This approach is inefficient and costly to maintain. Although some visual orchestration tools and low-code platforms exist in the market, many technical bottlenecks remain.

[0003] Existing solutions perform poorly in understanding requirements, struggling to accurately parse complex business rules described in natural language, particularly exhibiting weak ability to identify implicit processing logic and special constraints. Traditional template matching methods primarily rely on simple label classification, failing to achieve semantic-level intelligent matching, resulting in unsatisfactory template reuse performance.

[0004] At the technical implementation level, existing tools have relatively rigid type conversion mechanisms, making it difficult to handle the processing needs of non-standard data formats. The generated code often lacks robust exception handling mechanisms, failing to meet the stability requirements of production environments. Operational monitoring functions are also relatively weak, mainly remaining at the level of basic operational status monitoring.

[0005] As enterprise data environments become increasingly complex, traditional development models are no longer sufficient to meet the demands of business agility. There is an urgent need for an innovative solution that can automate the entire process from business requirements to a workable pipeline, addressing the shortcomings of existing technologies in areas such as intelligent parsing, semantic matching, and code generation. Summary of the Invention

[0006] According to embodiments of the present invention, an automated data pipeline generation method, device, and medium based on a low-code platform are provided to address the aforementioned problems.

[0007] According to an embodiment of the present invention, an automated data pipeline generation method based on a low-code platform is provided, comprising:

[0008] S1. Parse the user's input business requirement text using natural language processing technology, and extract source end information, target end information, data processing rules and flow constraints of the data pipeline to form structured requirement information;

[0009] S2. Based on a preset visual data pipeline modeling language, the parsed requirement information is converted into a graphical pipeline model. The visual data pipeline modeling language includes custom data flow node symbols, data transformation operator symbols, and node connection rules.

[0010] S3. A semantically driven pipeline template matching algorithm is adopted to select the basic template with the highest matching degree with the graphical pipeline model from the template library. The matching algorithm performs a comprehensive score by calculating the cosine similarity between the requirement semantic vector and the template semantic vector and combining the matching degree of the data processing node type.

[0011] S4. Based on the automatic data type mapping mechanism, the source data type and target data type in the basic template are matched and converted to generate an intermediate template containing type conversion logic;

[0012] S5. Through an automatic generation engine based on large model transformation rules, target code is generated according to intermediate templates and special processing rules in business requirements, and the target code is deployed to the specified runtime environment to complete the automatic generation of data pipeline.

[0013] According to an embodiment of the present invention, an electronic device is provided, comprising:

[0014] Processor; and,

[0015] A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the automated data pipeline generation method based on a low-code platform as described above.

[0016] According to an embodiment of the present invention, a storage medium is provided for storing computer-executable instructions, which, when executed, implement the steps of the automated data pipeline generation method based on a low-code platform as described above.

[0017] By employing embodiments of the present invention, and through the deep integration of natural language processing technology and visual modeling technology, business personnel can directly describe data processing requirements using natural language. The system automatically converts these requirements into executable data pipelines, effectively lowering the technical threshold in the field of data engineering. The semantic understanding-based intelligent template matching mechanism can accurately identify key business requirements, automatically select the optimal pipeline template, and generate adaptable code, significantly improving the development efficiency of data pipelines.

[0018] By organically combining the automatic data type mapping mechanism with the code generation engine, the system can intelligently generate type conversion logic when dealing with heterogeneous data sources, significantly improving its ability to handle complex data scenarios. The built-in automated monitoring code generation function provides comprehensive operational status monitoring support for the data pipeline, making operations and maintenance more efficient and reliable.

[0019] This application also establishes a continuously optimized template knowledge base mechanism, enabling the system to continuously accumulate best practices for various data processing scenarios, forming a self-evolving technical system. This design not only enhances the reusability of templates but also builds sustainable data assets for enterprises, achieving a dual improvement in development efficiency and data processing quality. The entire solution effectively reduces the development and maintenance costs of data pipelines while ensuring code quality through intelligent and automated technical means. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of an automated data pipeline generation method based on a low-code platform, according to an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0023] Method Implementation Examples

[0024] According to embodiments of the present invention, an automated data pipeline generation method based on a low-code platform is provided. Figure 1 This is a flowchart of an automated data pipeline generation method based on a low-code platform according to an embodiment of the present invention. Figure 1 As shown, the automated data pipeline generation method based on a low-code platform according to an embodiment of the present invention specifically includes:

[0025] S1. Parse the user's input business requirement text using natural language processing technology, and extract source end information, target end information, data processing rules and flow constraints of the data pipeline to form structured requirement information;

[0026] The process of parsing user-inputted business requirement text using natural language processing technology specifically includes:

[0027] A pre-trained language model is used to segment, identify entities, and extract relationships from business requirement texts, identifying the source system name, target system name, data entities, data fields, and conversion relationships between fields in the data pipeline. The pre-trained language model is fine-tuned using data domain-specific corpora, and special word vectors are constructed for professional terms such as "distributed database," "data warehouse," and "time series data." At the same time, a bidirectional long short-term memory network (BiLSTM) is introduced to deeply mine complex conversion relationships between fields. For example, it can accurately identify conversion relationships containing computational logic, such as "convert the 'transaction amount (cents)' of system A to the 'transaction amount (yuan)' of system B and keep two decimal places."

[0028] The intent recognition model determines the user's requirements for the data pipeline's operating frequency, fault tolerance, and data consistency. The model incorporates a database of requirements for typical industry scenarios. When a user inputs "critical business data synchronization," it automatically associates high-priority fault tolerance requirements (such as at least three retries and immediate alerts upon failure) and strong data consistency requirements (such as using transaction mechanisms). Simultaneously, the model assigns a confidence score to the recognition results. When the score falls below a preset threshold, a manual confirmation process is automatically triggered to ensure accurate understanding of the requirements.

[0029] S2. Based on a preset visual data pipeline modeling language, the parsed requirement information is converted into a graphical pipeline model. The visual data pipeline modeling language includes custom data flow node symbols, data transformation operator symbols, and node connection rules.

[0030] The visual data pipeline modeling language also includes:

[0031] Dynamic arrow symbols are used to represent real-time data streams, while static arrow symbols are used to represent batch data streams. Dynamic arrow symbols use different flashing frequencies to distinguish the priority of real-time data streams, with higher priority real-time data streams flashing more frequently. Static arrow symbols use color depth to indicate the size of the batch data, with larger data volumes having darker colors.

[0032] This is an operator attribute panel used for labeling data cleaning, filtering, and aggregation operations. The panel allows users to configure operator parameters via dropdown selection or parameter input. It also features parameter recommendation functionality; for example, when configuring threshold parameters for data cleaning, it recommends a reasonable threshold range based on historical configuration data and data distribution characteristics. Furthermore, it provides a real-time preview of parameter configurations, allowing users to intuitively understand the impact of parameter settings on the data processing results.

[0033] S3. A semantically driven pipeline template matching algorithm is adopted to select the basic template with the highest matching degree with the graphical pipeline model from the template library. The matching algorithm performs a comprehensive score by calculating the cosine similarity between the requirement semantic vector and the template semantic vector and combining the matching degree of the data processing node type.

[0034] The semantically driven pipeline template matching algorithm specifically includes:

[0035] The graphical pipeline model is semantically encoded to generate a requirement semantic vector containing node type sequences, data flow characteristics, and constraint characteristics. During the encoding process, the node type sequences are weighted, with core node types having higher weights in the sequence to highlight their importance in the data pipeline.

[0036] Each basic template in the template library is semantically encoded to generate a template semantic vector. The template library is updated regularly to include high-quality templates for new business scenarios, while outdated or low-usage templates are cleaned up to ensure the effectiveness and efficiency of the template library.

[0037] The cosine similarity between the requirement semantic vector and the semantic vectors of each template is calculated. At the same time, the proportion of the same type of nodes in the graphical pipeline model and the basic template is counted to obtain the node matching degree. When calculating the cosine similarity, the features of each dimension in the vector are normalized to ensure that the influence of different feature dimensions on the similarity calculation is within a reasonable range.

[0038] The cosine similarity and node matching scores are weighted and summed according to preset weights, and the base template with the highest score is selected as the matching result. The preset weights are adjusted according to different business domains. For example, in business domains with high real-time requirements, the features related to real-time processing will have higher weights in the cosine similarity.

[0039] S4. Based on the automatic data type mapping mechanism, the source data type and target data type in the basic template are matched and converted to generate an intermediate template containing type conversion logic;

[0040] The automatic data type mapping mechanism includes:

[0041] A cross-database type mapping table is established, which contains the correspondence between data types in relational databases, non-relational databases, and file formats. The mapping table will contain precision loss information for data type conversion. When precision loss occurs, prompts will be given during the conversion process, and optional precision compensation schemes will be provided.

[0042] When there is a direct mapping relationship between the source data type and the target data type, the conversion rules in the mapping table are directly adopted; at the same time, the conversion history is recorded to facilitate subsequent traceability and auditing of the data type conversion process.

[0043] When there is no direct mapping between the source and target data types, adapter code containing data type conversion functions is generated. These conversion functions perform type conversion based on data precision requirements and business rules. The adapter code undergoes pre-compilation checks to ensure its syntactic correctness and execution efficiency. For complex type conversions, interfaces for code preview and manual adjustment are also provided.

[0044] S5. Through an automatic generation engine based on large model transformation rules, target code is generated according to intermediate templates and special processing rules in business requirements, and the target code is deployed to the specified runtime environment to complete the automatic generation of data pipeline.

[0045] The automatic conversion rule generation engine based on the large model includes: inputting intermediate templates, special processing rules in business requirements, and target runtime environment information into a pre-trained code generation large model to generate initial target code; performing syntax verification and logical consistency checks on the initial target code; if errors are found, feeding back the error information to the large model for secondary generation until the correct target code is generated.

[0046] The automated data pipeline generation method based on a low-code platform also includes: visualizing the generated data pipeline, allowing users to adjust the data pipeline by dragging and dropping nodes and modifying node attributes; the visualization interface provides multiple view modes, including a global view (showing the architecture of the entire data pipeline), a detailed view (focusing on the detailed configuration of a single node), and a dependency view (showing the data dependencies between nodes); it also supports node search and location functions, allowing users to quickly find target nodes by entering node names. For large and complex data pipelines, it also provides an automatic layout optimization function, automatically adjusting the layout according to the relationships between nodes to reduce crossovers and improve view clarity.

[0047] The system responds to user adjustments in real time, synchronously updating the graphical model of the data pipeline and the corresponding target code. User adjustments are recorded in the operation log, including the operation time, operation content, and operator information, and support operation rollback and undo functions. When user adjustments may cause logical errors in the data pipeline (such as deleting critical nodes or establishing circular dependencies), the system will immediately pop up an early warning prompt, explaining the cause of the error and its possible impact, and providing correction suggestions to help users quickly correct the error.

[0048] The target code includes data extraction code, data transformation code, data loading code, and monitoring code. The monitoring code is used to collect real-time operational status indicators of the data pipeline, including data throughput, conversion efficiency, and latency. The data extraction code supports both incremental and full extraction modes and can automatically select the optimal extraction method based on the update frequency and data volume of the source data. Incremental extraction uses an incremental identification mechanism based on timestamps or logs to ensure the accuracy and completeness of the extracted data. The data transformation code includes detailed logging functions, recording the transformation process and results of each data entry. For data that fails to be transformed, the error reason is recorded and stored in an error data buffer, supporting subsequent manual intervention and reprocessing. In addition to collecting basic operational status indicators, the monitoring code can also monitor the resource usage of the data pipeline (such as CPU utilization and memory usage). When resource usage exceeds a preset threshold, a resource expansion warning is automatically triggered. The monitoring code also supports custom indicator alarm functions, allowing users to set alarm thresholds and alarm methods for specific indicators (such as email, SMS, and system pop-ups) to promptly grasp abnormal situations in the data pipeline.

[0049] The embodiments of the present invention have the following beneficial effects:

[0050] By deeply integrating natural language processing (NLP) and visual modeling technologies, business users can directly describe their data processing needs using natural language. The system automatically converts these requirements into executable data pipelines, effectively lowering the technical barrier in the field of data engineering. The semantic understanding-based intelligent template matching mechanism accurately identifies key business requirements, automatically selects the optimal pipeline template, and generates adaptable code, significantly improving the development efficiency of data pipelines.

[0051] By organically combining the automatic data type mapping mechanism with the code generation engine, the system can intelligently generate type conversion logic when dealing with heterogeneous data sources, significantly improving its ability to handle complex data scenarios. The built-in automated monitoring code generation function provides comprehensive operational status monitoring support for the data pipeline, making operations and maintenance more efficient and reliable.

[0052] This application also establishes a continuously optimized template knowledge base mechanism, enabling the system to continuously accumulate best practices for various data processing scenarios, forming a self-evolving technical system. This design not only enhances the reusability of templates but also builds sustainable data assets for enterprises, achieving a dual improvement in development efficiency and data processing quality. The entire solution effectively reduces the development and maintenance costs of data pipelines while ensuring code quality through intelligent and automated technical means.

[0053] Device Example 1

[0054] According to an embodiment of the present invention, an electronic device is provided, comprising:

[0055] Processor; and,

[0056] A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the method embodiments described above.

[0057] Device Example 2

[0058] According to an embodiment of the present invention, a storage medium is provided for storing computer-executable instructions, which, when executed, implement the steps of the method embodiments described above.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automated data pipeline generation method based on a low-code platform, characterized in that... include: S1. Parse the user's input business requirement text using natural language processing technology, and extract source end information, target end information, data processing rules and flow constraints of the data pipeline to form structured requirement information; S2. Based on a preset visual data pipeline modeling language, the parsed requirement information is converted into a graphical pipeline model. The visual data pipeline modeling language includes custom data flow node symbols, data transformation operator symbols, and node connection rules. S3. A semantically driven pipeline template matching algorithm is adopted to select the basic template with the highest matching degree with the graphical pipeline model from the template library. The matching algorithm performs a comprehensive score by calculating the cosine similarity between the requirement semantic vector and the template semantic vector and combining the matching degree of the data processing node type. S4. Based on the automatic data type mapping mechanism, the source data type and target data type in the basic template are matched and converted to generate an intermediate template containing type conversion logic; S5. Through the automatic generation engine of transformation rules based on the large model, target code is generated according to the intermediate template and special processing rules in business requirements, and the target code is deployed to the specified runtime environment to complete the automatic generation of data pipeline; The visual data pipeline modeling language also includes: Dynamic arrow symbols used to represent real-time data streams and static arrow symbols used to represent batch data streams; An operator attribute panel for labeling data cleaning, data filtering, and data aggregation operations. The attribute panel allows users to configure operator parameters through drop-down selection or parameter input. The semantically driven pipeline template matching algorithm specifically includes: Semantic encoding is performed on the graphical pipeline model to generate a demand semantic vector containing node type sequences, data flow characteristics, and constraint characteristics; Semantic encoding is performed on each basic template in the template library to generate a template semantic vector; Calculate the cosine similarity between the demand semantic vector and the semantic vector of each template, and at the same time, count the proportion of the same type of nodes in the graphical pipeline model and the basic template to obtain the node matching degree. The cosine similarity and node matching degree are weighted and summed according to preset weights, and the basic template with the highest score is selected as the matching result.

2. The method according to claim 1, characterized in that, The process of parsing user-inputted business requirement text using natural language processing technology specifically includes: A pre-trained language model is used to segment business requirement text, identify entities, and extract relationships, identifying the source system name, target system name, data entities, data fields, and the conversion relationships between fields in the data pipeline; The intent recognition model is used to determine the user's requirements for the data pipeline's operating frequency, fault tolerance, and data consistency.

3. The method according to claim 1, characterized in that, The automatic data type mapping mechanism includes: Establish a cross-database type mapping table, which contains the correspondence between data types in relational databases, non-relational databases, and file formats; When there is a direct mapping relationship between the source data type and the target data type, the conversion rules in the mapping table are directly used. When there is no direct mapping relationship between the source data type and the target data type, adaptation code containing data type conversion functions is generated. These conversion functions perform type conversion based on data precision requirements and business rules.

4. The method according to claim 1, characterized in that, The automatic generation engine for conversion rules based on a large model includes: inputting intermediate templates, special processing rules in business requirements, and target runtime environment information into a pre-trained code generation large model to generate initial target code; performing syntax verification and logical consistency checks on the initial target code; if errors are found, feeding back the error information to the large model for secondary generation until the correct target code is generated.

5. The method according to claim 1, characterized in that, Also includes: The generated data pipeline is visualized, and users can adjust the data pipeline by dragging and dropping nodes and modifying node properties; It responds to user adjustments in real time and updates the graphical model of the data pipeline and the corresponding target code synchronously.

6. The method according to claim 1, characterized in that, The target code includes data extraction code, data transformation code, data loading code, and monitoring code. The monitoring code is used to collect real-time operating status indicators of the data pipeline, including data throughput, conversion power, and latency.

7. An electronic device, comprising: processor; as well as, A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the automated data pipeline generation method based on a low-code platform as described in any one of claims 1-6.

8. A storage medium for storing computer-executable instructions, which, when executed, implement the steps of the automated data pipeline generation method based on a low-code platform as described in any one of claims 1-6.

Images