Data cleaning and arranging template system and method based on Flink

Abstract

The invention discloses a data cleaning and arranging template system and method based on Flink, and relates to the field of computer information management application. The system comprises a highly abstracted general data cleaning template design module, a dynamic parameterization configuration and intelligent adaptation mechanism module, an end-to-end intelligent automatic filling and precision consistency verification module, an Flink seamless integration and efficient resource optimization module and an omnibearing task management and real-time intelligent monitoring module. According to the system, the flexibility and reusability of the data cleaning task are remarkably improved, the execution efficiency and the processing capacity of the data cleaning task are improved, the task management and monitoring capacity is enhanced, and the operation and maintenance experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer information management applications, and in particular to a Flink-based data cleaning and orchestration template system and method. Background Art

[0002] With the continuous advancement of big data technology, data cleaning, as a core link in the data processing process, has become increasingly important. The main goal of data cleaning is to ensure the quality and accuracy of data and provide a reliable foundation for subsequent data analysis and decision-making. However, when performing data cleaning tasks, current technical solutions often face a series of challenges such as fixed data sources, excessive task customization, and poor reusability. Specifically, traditional data cleaning tasks are often tailored to specific data sources and cleaning requirements. Although this approach can address the needs of specific business scenarios, it also brings some significant problems:

[0003] (1) Limited flexibility: The hard coding of data sources and cleaning rules often requires manual code modification or repeated development when the data source or cleaning requirements change, resulting in high deployment and maintenance costs.

[0004] (2) Insufficient reusability: Although there are certain similarities between different data cleaning tasks, the lack of universal templates and intelligent automation tools makes it difficult to reuse and share these tasks efficiently, thereby reducing overall development efficiency.

[0005] (3) Inefficiency: Data cleaning tasks based on customized development usually require tedious coding, testing, and verification processes, which not only prolongs the development cycle but may also lead to a significant decrease in task execution efficiency, especially when facing large-scale data.

[0006] To address these challenges, the industry has gradually begun exploring template-based intelligent data cleaning task development methods. By defining flexible, universal template formats and highly parameterized configurations, this approach abstracts and encapsulates data cleaning tasks, enabling rapid deployment and efficient reuse. However, existing template-based data cleaning development methods still face numerous shortcomings: limited flexibility in template formats, low automation levels, and difficulty meeting complex and ever-changing data cleaning needs.

[0007] Therefore, a new solution is urgently needed: a universal template system for intelligent data cleaning tasks based on Flink. This system not only overcomes the limitations of existing technologies but also provides a flexible, scalable, and efficient development platform. Through simple configuration and parameterization, users can easily generate and deploy highly customized data cleaning tasks, achieving automated data cleaning and efficient processing, greatly improving task execution efficiency and system adaptability. Summary of the Invention

[0008] The purpose of this invention is to address the above shortcomings and propose a Flink-based data cleaning orchestration template system and method with a flexible and over-defined universal template format and a highly parameterized configuration method, which can abstract and encapsulate data cleaning tasks, thereby achieving rapid deployment and efficient reuse.

[0009] The present invention specifically adopts the following technical solutions:

[0010] A Flink-based data cleaning and orchestration template system, including a highly abstract general data cleaning template design module, a dynamic parameter configuration and intelligent adaptation mechanism module, an end-to-end intelligent automatic filling and precise consistency verification module, a Flink seamless integration and efficient resource optimization module, and a comprehensive task management and real-time intelligent monitoring module.

[0011] A domain-oriented template modeling method is introduced into the highly abstract general data cleaning template design module to build a flexible and scalable general template for data cleaning tasks. Through multi-level task abstraction, the core structure, input and output parameters, and rule strategies of the data cleaning process are highly parameterized and encapsulated.

[0012] The dynamic parameterized configuration and intelligent adaptation mechanism module proposes a dynamic parameterized configuration method based on knowledge graph, combined with an intelligent rule reasoning engine to achieve intelligent matching of templates, data sources, and cleaning rules;

[0013] The end-to-end intelligent auto-population and precise consistency verification module uses historical pattern-based intelligent population to automatically identify the structural patterns and cleaning rules of the data source, infer and populate key parameters through deep semantic analysis, and simultaneously uses a dynamic consistency verification method based on constraint learning to perform high-precision detection of potential anomalies in cleaning rules and data streams.

[0014] The Flink seamless integration and efficient resource optimization module deeply integrates Apache Flink's data stream programming paradigm. Leveraging its event-driven computing architecture, it enables real-time and efficient cleaning of large-scale data streams. Through reinforcement learning-based resource scheduling optimization, it adaptively adjusts computing resource allocation and parallelism configuration, combined with load-aware elastic scaling strategies, to improve computing node utilization and fault recovery capabilities.

[0015] The all-round task management and real-time intelligent monitoring module is based on a panoramic task management platform based on stream computing, providing end-to-end management capabilities from task definition, scheduling, monitoring to optimization. With the help of adaptive anomaly detection and multi-dimensional performance analysis, the system can track task execution status in real time, automatically identify abnormal patterns, and provide AI-based intelligent alarms and repair suggestions.

[0016] Preferably, the highly abstract universal data cleaning template design module loads data from the source data table, cleans the data, and then writes the data into the target data table.

[0017] Preferably, in a highly abstract universal data cleaning template design module, the highly abstract universal data cleaning template only needs to define the input, cleaning rules and output, and expand the input. In addition to the structured data source, the input type will also have the API type. Since different interfaces return different data contents, dynamic parameterized configuration and intelligent adaptation mechanism are required.

[0018] Preferably, in the end-to-end intelligent automatic filling and precise consistency verification module, when the user selects a template and fills in the parameters, the information is stored in the database. When the user selects a template again, the historical records in the database will be retrieved, and the parameters previously filled in by the user will be automatically retrieved and displayed on the interface, that is, the data sources and cleaning rules that the user may fill in are recommended to the user.

[0019] Preferably, in Flink's seamless integration and efficient resource optimization module, each time a data cleaning task is submitted, the server cluster needs to allocate certain resources. When many tasks are submitted, some server resources will be occupied more and some server resources will be occupied less. In this case, if the submitted task is judged to require more computing resources, a relatively idle server will be assigned to this task; in addition, if too many tasks are submitted and the entire server cluster is already fully loaded, the server cluster will be automatically expanded, that is, new servers will be added to this server cluster.

[0020] Preferably, in the all-round task management and real-time intelligent monitoring module, adaptive anomaly detection and multi-dimensional performance analysis mean that after submitting a task, the system will collect various types of task information. If it is found that the task is running for too long and the server resources are basically occupied, it will automatically issue an abnormality notification and the multi-dimensional performance status of the server; these data are sent to the big model and prompt words are added, and the big model returns relevant intelligent alarms and repair suggestions.

[0021] A Flink-based data cleaning orchestration template method, based on the aforementioned system, includes a highly abstracted universal data cleaning template design, dynamic parameterized configuration and intelligent adaptation mechanisms, end-to-end intelligent automatic population and precise consistency verification, seamless Flink integration and efficient resource optimization, comprehensive task management, and real-time intelligent monitoring.

[0022] The present invention has the following beneficial effects:

[0023] (1) Improving the flexibility and reusability of data cleaning tasks: By introducing a highly modular template design and intelligent parameterized configuration, the present invention enables users to quickly generate and customize data cleaning tasks that meet diverse needs without in-depth programming. This flexible configuration approach significantly reduces development cycles and task deployment time, and significantly reduces the workload of repeated development through template reuse, thereby reducing long-term maintenance costs and improving system scalability.

[0024] (2) Significantly improve the execution efficiency and processing power of data cleaning tasks: Relying on Apache Flink's stream processing capabilities and powerful distributed computing framework, the present invention greatly improves the execution efficiency of data cleaning tasks through refined resource scheduling and dynamic task optimization strategies, ensuring low latency and high throughput performance in large-scale data environments. Through flexible streaming data processing modes, the present invention can not only efficiently respond to real-time data cleaning needs, but also ensure high accuracy and reliability of data processing in different data scenarios.

[0025] (3) Enhance task management and monitoring capabilities and improve the operation and maintenance experience: The present invention innovatively provides an integrated task management and monitoring interface, allowing users to monitor the execution status, performance indicators, and data flow of tasks in real time on a unified platform. This interface not only supports real-time scheduling and dynamic optimization of tasks, but also intelligently analyzes bottlenecks in the task execution process, helping users to adjust strategies in a timely manner to improve system stability and efficiency. In addition, the system also supports task history recording and log backtracking, providing strong data support for task fault diagnosis and optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the overall system framework;

[0027] Figure 2 This is a diagram illustrating the system operation logic. DETAILED DESCRIPTION

[0028] The specific implementation of the present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0029] Combine Figure 1 and Figure 2 A Flink-based data cleaning and orchestration template system, including a highly abstract general data cleaning template design module, a dynamic parameter configuration and intelligent adaptation mechanism module, an end-to-end intelligent automatic filling and precise consistency verification module, a Flink seamless integration and efficient resource optimization module, and a comprehensive task management and real-time intelligent monitoring module;

[0030] A domain-oriented template modeling method is introduced into the highly abstract general data cleaning template design module to construct a flexible and scalable general template for data cleaning tasks; through multi-level task abstraction, the core structure, input and output parameters, and rule strategies of the data cleaning process are highly parameterized and encapsulated; the highly abstract general data cleaning template design module loads data from the source data table, cleans it, and then writes it to the target data table.

[0031] This solution introduces a domain-specific template modeling (DSTM) approach to construct a flexible and scalable universal template for data cleaning tasks. Through multi-level task abstraction (MLTA), the core structure, input and output parameters, and rule policies of the data cleaning process are highly parameterized. This allows users to quickly build tasks that meet specific data sources and cleaning requirements based on configuration-driven development (CDD) without extensive code development. This template system not only reduces the complexity of task construction but also enhances cross-platform adaptability and portability.

[0032] The dynamic parameterized configuration and intelligent adaptation mechanism module proposes a dynamic parameterized configuration method based on knowledge graph, combined with an intelligent rule reasoning engine to achieve intelligent matching of templates with data sources and cleaning rules.

[0033] To overcome the limitations of traditional static task configuration, we propose a knowledge graph-based dynamic parameterization (KGDP) approach, combined with an intelligent rule inference engine (IRIE), to intelligently match templates with data sources and cleaning rules. Users only need to provide the necessary metadata, and the system automatically adjusts parameters and optimizes the task execution process based on multi-dimensional feature learning (MDFL), achieving high compatibility across data domains and improving task deployment efficiency and adaptability.

[0034] The end-to-end intelligent automatic filling and precise consistency verification module uses historical pattern-based intelligent filling to automatically identify the structural patterns and cleaning rules of the data source, infer and fill in key parameters through deep semantic analysis, and at the same time, adopts a dynamic consistency verification method based on constraint learning to perform high-precision detection of potential anomalies in cleaning rules and data streams.

[0035] In the end-to-end intelligent auto-fill and precise consistency verification module, when the user selects a template and fills in the parameters, the information is stored in the database. When the user selects a template again, the historical records in the database will be retrieved, and the parameters previously filled in by the user will be automatically retrieved and displayed on the interface, that is, the data sources and cleaning rules that the user may fill in are recommended to the user.

[0036] Pattern-Based Intelligent Autofill (PIA) technology automatically identifies the structural patterns and cleaning rules of data sources, inferring and filling in key parameters through Deep Semantic Analysis (DSA). Furthermore, the Constraint Learning-Based Dynamic Consistency Verification (CLDCV)** method performs high-precision detection of cleaning rules and potential anomalies in data streams, ensuring the integrity, consistency, and accuracy of tasks before execution, effectively reducing the rate of task failures due to configuration errors.

[0037] The Flink seamless integration and efficient resource optimization module deeply integrates Apache Flink's data flow programming paradigm and relies on its event-driven computing architecture to achieve real-time and efficient cleaning of large-scale data streams. Through reinforcement learning-based resource scheduling optimization, adaptive adjustment of computing resource allocation and parallelism configuration, combined with load-aware elastic scaling strategies, it improves the utilization rate and fault recovery capabilities of computing nodes.

[0038] In Flink's seamless integration and efficient resource optimization module, each time a data cleaning task is submitted, the server cluster needs to allocate certain resources. When many tasks are submitted, some server resources may be occupied more than others. In this case, if the submitted task is determined to require more computing resources, a relatively idle server will be assigned to this task. In addition, if too many tasks are submitted and the entire server cluster is already fully loaded, the server cluster will be automatically expanded, that is, new servers will be added to this server cluster.

[0039] Deeply integrating Apache Flink's Dataflow Programming Model (DPM) and leveraging its Event-Driven Computing (EDC) architecture, it enables efficient, real-time cleaning of large-scale data streams. Through Reinforcement Learning-Based Resource Scheduling Optimization (RLRSO), it adaptively adjusts computing resource allocation and parallelism configuration. Combined with Load-Aware Elastic Scaling (LAES) strategies, it improves compute node utilization and fault recovery capabilities, thereby optimizing system throughput, reducing computing latency, and significantly improving task execution stability.

[0040] The all-round task management and real-time intelligent monitoring module is based on a panoramic task management platform based on stream computing, providing end-to-end management capabilities from task definition, scheduling, monitoring to optimization. With the help of adaptive anomaly detection and multi-dimensional performance analysis, the system can track task execution status in real time, automatically identify abnormal patterns, and provide AI-based intelligent alarms and repair suggestions.

[0041] In the comprehensive task management and real-time intelligent monitoring module, adaptive anomaly detection and multi-dimensional performance analysis mean that after a task is submitted, the system will collect various types of task information. If it is found that the task is running too long and the server resources are basically occupied, it will automatically issue an exception notification and the multi-dimensional performance status of the server; this data will be sent to the big model and accompanied by prompt words, and the big model will return relevant intelligent alarms and repair suggestions.

[0042] The Streaming-Based Unified Task Management Platform (S-UTMP) provides end-to-end management capabilities from task definition, scheduling, monitoring, to optimization. Leveraging Adaptive Anomaly Detection (AAD) and Multi-Dimensional Performance Analysis (MDPA), the system can track task execution status in real time, automatically identify abnormal patterns, and provide AI-based intelligent alerts and remediation suggestions. Furthermore, Streaming Data Visualization (SDV) technology enables users to monitor task status through an intuitive interface, accurately analyze key performance indicators, and achieve efficient operations and maintenance and precise decision-making.

[0043] A Flink-based data cleaning orchestration template method, based on the aforementioned system, includes a highly abstracted universal data cleaning template design, dynamic parameterized configuration and intelligent adaptation mechanisms, end-to-end intelligent automatic population and precise consistency verification, seamless Flink integration and efficient resource optimization, comprehensive task management, and real-time intelligent monitoring.

[0044] Taking a data cleaning script for filtering empty data as an example, the specific implementation of the present invention is as follows:

[0045] Highly abstract general data cleaning template design (DSTM)

[0046] ●Task template selection: The user selects the "Empty Data Cleaning" template as the basic template for the data cleaning task in the system configuration interface.

[0047] ●Task configuration: The system provides flexible parameterized configuration options for each task template. Users only need to specify the input data source (for example, the name of the data table and its fields) and the output data source (for example, the target database table name and field mapping relationship) without writing additional code.

[0048] Template Generation: After selecting a template and configuring the relevant parameters, the system automatically generates a complete task script for data reading, filtering, and writing. This entire process is based on the abstract logic of the template, ensuring that the task is structured and easily extensible.

[0049] Dynamic parameterized configuration and intelligent adaptation mechanism (KGDP+IRIE)

[0050] Input data source configuration: Users provide data source metadata (such as field name and type), and the system automatically identifies the structure of the data source based on the knowledge graph. For example, the system can identify fields such as "price" and "date" and generate corresponding cleaning rules based on their characteristics.

[0051] Intelligent matching and rule inference: Based on the "Empty Data Cleansing" template selected by the user, IRIE infers filtering rules applicable to the data source based on its metadata and data characteristics. For example, for a "price" field, if there is a null value, the record will be filtered; for a "date" field, the system may check for null values based on rules and fill in the values with a default value.

[0052] Dynamic parameter adjustment: Based on the Multidimensional Feature Learning (MDFL) method, the system automatically adjusts the parameter configuration used for data stream processing in the template. For example, based on the proportion of null values in a field and the data type, the system may adjust the handling strategy for null fields or adaptively select filtering conditions.

[0053] End-to-end intelligent auto-fill and precise consistency verification (PIA+CLDCV)

[0054] Intelligent Autofill (PIA): The system uses Deep Semantic Analysis (DSA) to automatically identify the structure and patterns of data sources and infer reasonable filling of missing values based on historical data. For example, if a "Date" field is empty, the system can automatically infer a reasonable default date based on the date field format or prompt the user for confirmation.

[0055] Consistency Verification (CLDCV): Before the cleaning task begins, the system verifies the integrity and consistency of the data using Constraint Learning. The system checks whether each record field complies with predefined rules, ensuring that null values are handled as expected and avoiding data anomalies caused by improper configuration.

[0056] ●Verification example: If a field in a record has a negative value and the field is not allowed to be empty, the system will mark the record as abnormal, provide repair suggestions, or automatically filter out the record.

[0057] Flink seamless integration and efficient resource optimization (RLRSO+LAES)

[0058] ●Flink cluster selection: Users specify the Flink cluster for task execution in the task configuration, and the system automatically selects the optimal computing resources (such as memory and number of CPU cores) based on the task requirements.

[0059] Resource Scheduling Optimization (RLRSO): The system automatically adjusts the Flink cluster's computing resource allocation using a reinforcement learning-based resource scheduling optimization (RL-Based Resource Scheduling Optimization) algorithm. Based on the complexity of tasks and real-time changes in data traffic, the system intelligently adjusts parallelism and resource allocation to maximize task execution efficiency.

[0060] ●Lasting Elasticity (LAES): During task execution, the system monitors the load of data streams in real time and dynamically adjusts the number of computing nodes based on load changes to ensure high availability and high throughput of the system.

[0061] The system will automatically expand computing resources to ensure smooth execution of tasks.

[0062] Comprehensive task management and real-time intelligent monitoring (S-UTMP+AAD+SDV)

[0063] Task Management Platform (S-UTMP): Users can create, configure, schedule, and monitor tasks through an intuitive graphical interface. All task information (such as task status and execution logs) can be viewed in real time.

[0064] Anomaly Detection (AAD): The system uses adaptive anomaly detection technology to analyze potential issues during task execution in real time, such as data quality issues and resource bottlenecks. If the system detects an anomaly (such as a high percentage of null values in a field), it issues an alert and automatically adjusts task parameters or provides remediation suggestions.

[0065] ●Data visualization (SDV): The system displays the execution status of tasks through a visual interface, including task progress, computing resource usage, key performance indicators, etc., to help users quickly identify bottlenecks and optimize task processes.

[0066] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. A data cleaning and orchestration template system based on Flink, characterized by: It includes a highly abstract universal data cleaning template design module, a dynamic parameter configuration and intelligent adaptation mechanism module, an end-to-end intelligent automatic filling and precise consistency verification module, a Flink seamless integration and efficient resource optimization module, and a comprehensive task management and real-time intelligent monitoring module. A domain-oriented template modeling method is introduced into the highly abstract general data cleaning template design module to build a flexible and scalable general template for data cleaning tasks; Through multi-level task abstraction, the core structure, input and output parameters, and rule strategies of the data cleaning process are highly parameterized and encapsulated; The dynamic parameterized configuration and intelligent adaptation mechanism module proposes a dynamic parameterized configuration method based on knowledge graph, combined with an intelligent rule reasoning engine to achieve intelligent matching of templates, data sources, and cleaning rules; The end-to-end intelligent auto-population and precise consistency verification module uses historical pattern-based intelligent population to automatically identify the structural patterns and cleaning rules of the data source, infer and populate key parameters through deep semantic analysis, and simultaneously uses a dynamic consistency verification method based on constraint learning to perform high-precision detection of potential anomalies in cleaning rules and data streams. The Flink seamless integration and efficient resource optimization module deeply integrates Apache Flink's data stream programming paradigm. Leveraging its event-driven computing architecture, it enables real-time and efficient cleaning of large-scale data streams. Through reinforcement learning-based resource scheduling optimization, it adaptively adjusts computing resource allocation and parallelism configuration, combined with load-aware elastic scaling strategies, to improve computing node utilization and fault recovery capabilities. The all-round task management and real-time intelligent monitoring module is based on a panoramic task management platform based on stream computing, providing end-to-end management capabilities from task definition, scheduling, monitoring to optimization. With the help of adaptive anomaly detection and multi-dimensional performance analysis, the system can track task execution status in real time, automatically identify abnormal patterns, and provide AI-based intelligent alarms and repair suggestions.

2. A Flink-based data cleaning and orchestration template system according to claim 1, characterized in that: The highly abstract general data cleaning template design module loads data from the source data table, cleans it, and then writes it to the target data table.

3. A Flink-based data cleaning and orchestration template system according to claim 1, characterized in that: In the highly abstract general data cleaning template design module, the highly abstract general data cleaning template only needs to define the input, cleaning rules and output, and expand the input. In addition to the structured data source, the input type will also have the API type. Since different interfaces return different data contents, dynamic parameterized configuration and intelligent adaptation mechanism are required.

4. A Flink-based data cleaning and orchestration template system according to claim 1, characterized in that: In the end-to-end intelligent auto-fill and precise consistency verification module, when the user selects a template and fills in the parameters, the information is stored in the database. When the user selects a template again, the historical records in the database will be retrieved, and the parameters previously filled in by the user will be automatically retrieved and displayed on the interface, that is, the data sources and cleaning rules that the user may fill in are recommended to the user.

5. The Flink-based data cleaning and orchestration template system according to claim 1, characterized in that: In Flink's seamless integration and efficient resource optimization module, each time a data cleaning task is submitted, the server cluster needs to allocate certain resources. When many tasks are submitted, some server resources may be occupied more than others. In this case, if the submitted task is determined to require more computing resources, a relatively idle server will be assigned to this task. In addition, if too many tasks are submitted and the entire server cluster is already fully loaded, the server cluster will be automatically expanded, that is, new servers will be added to this server cluster.

6. A Flink-based data cleaning and orchestration template system according to claim 1, characterized in that: In the comprehensive task management and real-time intelligent monitoring module, adaptive anomaly detection and multi-dimensional performance analysis mean that after a task is submitted, the system will collect various types of task information. If it is found that the task is running too long and the server resources are basically occupied, it will automatically issue an exception notification and the multi-dimensional performance status of the server; this data will be sent to the big model and accompanied by prompt words, and the big model will return relevant intelligent alarms and repair suggestions.

7. A data cleaning and orchestration template method based on Flink, characterized in that: The system according to any one of claims 1 to 6 includes a highly abstract universal data cleaning template design, dynamic parameterized configuration and intelligent adaptation mechanism, end-to-end intelligent automatic filling and precise consistency verification, seamless Flink integration and efficient resource optimization, comprehensive task management and real-time intelligent monitoring.

Images