Kettle-based database dynamic sub-table processing method and Kettle-based database dynamic sub-table processing system

By using a dynamic partitioning solution based on Kettle, the flexibility and compatibility issues of existing ETL tools such as Kettle in database table partitioning are resolved. This enables efficient and flexible dynamic database table partitioning, improves query response speed and update efficiency, and reduces operation and maintenance costs.

CN121365062APending Publication Date: 2026-01-20SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
0 Cites 0 Cited by

Patent Information

Application Number
CN202511379074.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-09-25
Publication Date
2026-01-20

AI Technical Summary

Technical Problem

In existing technologies, ETL tools such as Kettle are complex and inflexible when processing database table partitioning. They cannot flexibly and efficiently split and write large tables according to user-defined rules. Furthermore, existing table partitioning tools have poor compatibility with Kettle, which increases the complexity of the data processing flow and the cost of operation and maintenance.

Method used

This paper provides a method and system for dynamic table partitioning of databases based on Kettle. It receives table partitioning configuration parameters through the Kettle graphical interface, builds a dynamic table partitioning strategy engine, supports hash partitioning, range partitioning and time partitioning algorithms, embeds a routing decision module, and adopts a multi-threaded parallel execution engine and a distributed transaction management mechanism to ensure the atomicity and efficiency of cross-table operations.

Benefits of technology

It significantly improves query response speed and update efficiency, lowers the technical threshold, makes it easy for non-professionals to use, reduces system integration difficulty and operation and maintenance costs, and supports data sharding requirements in different business scenarios.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention relates to the technical field of big data processing and data integration, in particular to a Kettle-based database dynamic sub-table processing method and system, which comprises the following steps: receiving sub-table configuration parameters through a Kettle graphical interface, the sub-table configuration parameters comprising a fragmentation key, a sub-table algorithm and a sub-table number; table division rule metadata is loaded, a dynamic table division strategy engine is constructed, and the strategy engine supports Hash fragmentation, range fragmentation and time fragmentation algorithms; the construction of the dynamic table division strategy engine comprises the following steps: automatically recommending a table division algorithm based on data sampling analysis; providing a Grouping script interface to support a user-defined fragmentation rule; maintaining table division rule version history, and supporting rule rollback; the method has the beneficial effects that the large table is split into the plurality of sub-tables according to the specified rule, so that the data volume of a single table is remarkably reduced, query and update operations can focus on data in a smaller range, and the query response speed and the update efficiency are greatly improved.
Need to check novelty before this filing date? Find Prior Art