A Data Push Method and System for Distributed Databases

Abstract

This invention discloses a data push method and system for a distributed database, relating to the field of distributed databases. The method involves selecting any node in the database cluster to initiate a waterline coordination task, establishing push tasks on all nodes, and recording push target information, push data format, filtering, and aggregation conditions through these tasks. After data is inserted into any node in the database cluster, it is determined whether the table containing the inserted data is associated with a push task. If so, the inserted data is filtered according to the filtering or aggregation conditions, formatted into a push data format, and the unique write time of the database cluster is recorded as the waterline. The waterline and the formatted data are used together to generate a logical plan, which is then converted into a physical plan and executed. Finally, it is determined whether there is push data to be sent to the push task for asynchronous push.

Description

Technical Field

[0001] This invention discloses a data push method and system for a distributed database, relating to the field of distributed databases. Background Technology

[0002] With the development of the information age, the functions of filtering, aggregating, calculating, formatting, and transmitting massive amounts of real-time data from databases to another database or other service platforms have become widely used. Existing solutions, such as using database logs for data synchronization between databases, are commonly used for replication between master and slave nodes. However, log formats are not conducive to filtering, aggregation, and calculation, nor do they support writing to different platforms. Using a separate program to apply filtering conditions through polling and then writing data to the target service via SQL statements increases the complexity of system maintenance. Similarly, providing a publish-subscribe interface for other programs to subscribe to this solution also increases the complexity of system maintenance. Summary of the Invention

[0003] This invention addresses the problems of existing technologies by providing a distributed database data push method and system. This method efficiently filters, aggregates, calculates, and converts real-time data from a database before pushing it to another database or other service platform. It ensures basic data consistency and requires no additional programming.

[0004] The specific solution proposed in this invention is as follows:

[0005] This invention provides a data push method for a distributed database. It involves selecting any node in the database cluster to initiate a water level coordination task, establishing push tasks on all nodes, and recording push target information, push data format, filtering, and aggregation conditions through these push tasks.

[0006] After inserting data into any node in the database cluster, it is determined whether the table containing the inserted data is associated with the push task. If so, the inserted data is filtered according to filtering or aggregation conditions, and the filtered data is formatted into the push data format. The unique write time of the record in the database cluster is obtained as a watermark, and the watermark and the formatted data are used together to generate a logical plan.

[0007] When converting a logical plan into a physical plan and executing it, if it is determined that there is push data to be sent to the push task, asynchronous push will be performed.

[0008] Furthermore, the data push method for a distributed database periodically performs a flush operation based on the push task to ensure that the target node returns a message indicating successful push, and sends the highest water level among all pushed data within the period to the water level coordination task.

[0009] Furthermore, the data push method for a distributed database uses a water level coordination task to receive water level lines and selects the largest water level line as the water level line for the push task to write to storage.

[0010] Furthermore, in the data push method for a distributed database, when one or more nodes in the database cluster malfunction, the push task is stopped. When all nodes recover, the push task reads the water level of the push task through the water level coordination task, and the push task restarts to push real-time data.

[0011] This invention also provides a data push system for a distributed database, including a water level coordination task module, a push task module, a judgment module, and a filtering module.

[0012] The water level coordination task module selects any node in the database cluster to start the water level coordination task, while the push task module establishes push tasks on all nodes. The push tasks record the push target information, push data format, filtering and aggregation conditions.

[0013] After inserting data into any node in the database cluster, the judgment module checks whether the table containing the inserted data is associated with a push task. If so, the filtering module filters the inserted data based on filtering or aggregation conditions, formats the filtered data into the push data format, and the push task module obtains the unique write time of the record in the database cluster as a watermark. The watermark and the formatted data are then used together to generate a logical plan.

[0014] When the push task module converts the logical plan into a physical plan and executes it, it will perform asynchronous push if it determines that there is push data to be sent to the push task.

[0015] Furthermore, the push task module of the data push system for the distributed database periodically performs a flush operation according to the push task to ensure that the target node returns a message indicating successful push, and sends the highest water level among all pushed data within the period to the water level coordination task.

[0016] Furthermore, the water level coordination task module of the data push system of the distributed database receives water level lines through the water level coordination task and selects the largest water level line as the water level line for the push task to write to storage.

[0017] Furthermore, in the data push system of the distributed database, when one or more nodes in the database cluster are abnormal, the push task module stops pushing tasks. When all nodes recover, the water level coordination task module reads the water level of the push task through the water level coordination task, and the push task module restarts the real-time data push according to the push task.

[0018] The advantages of this invention are:

[0019] This invention can efficiently filter, aggregate, and perform calculations on real-time data from a database, convert its format, and push it to another database or other service platform. It ensures basic data consistency and requires no additional programming. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0022] Example 1

[0023] This invention provides a data push method for a distributed database: select any node in the database cluster to start a water level coordination task, establish push tasks on all nodes, and record push target information, push data format, filtering and aggregation conditions through the push tasks. In this way, a one-to-many connection relationship is established between the water level coordination task and the push tasks.

[0024] After data is inserted into any node in the database cluster, the node optimizer can determine whether the table containing the inserted data is associated with a push task. If so, the inserted data is formatted into the format required for writing to storage, and the inserted data is filtered based on filtering or aggregation conditions. The filtered data is then formatted into the push data format, and the unique write time of the record in the database cluster is obtained as a watermark. The watermark and the formatted data are then used together to generate a logical plan.

[0025] When converting a logical plan into a physical plan and executing it, if it is determined that there is push data to be sent to the push task, asynchronous push will be performed.

[0026] At the same time, a flush operation is performed periodically according to the push task to ensure that the target node of the push returns a message of successful push, and the highest water level among all push data in the period is sent to the water level coordination task.

[0027] Furthermore, the task receives water level data from multiple nodes through water level coordination and selects the highest water level data as the water level data for the push task to write to storage.

[0028] When one or more nodes in the database cluster fail, the push task stops. When all nodes recover, the push task is coordinated by reading the water level of the push task through the water level coordination task. Historical data that was added to the database and whose time was greater than the water level, i.e., the data during the period of cluster failure, is read and pushed. Real-time data push is restarted according to the push task.

[0029] Example 2

[0030] This invention also provides a data push system for a distributed database, including a water level coordination task module, a push task module, a judgment module, and a filtering module.

[0031] The water level coordination task module selects any node in the database cluster to start the water level coordination task, while the push task module establishes push tasks on all nodes. The push tasks record the push target information, push data format, filtering and aggregation conditions.

[0032] After inserting data into any node in the database cluster, the judgment module checks whether the table containing the inserted data is associated with a push task. If so, the filtering module filters the inserted data based on filtering or aggregation conditions, formats the filtered data into the push data format, and the push task module obtains the unique write time of the record in the database cluster as a watermark. The watermark and the formatted data are then used together to generate a logical plan.

[0033] When the push task module converts the logical plan into a physical plan and executes it, it will perform asynchronous push if it determines that there is push data to be sent to the push task.

[0034] The information interaction and execution process between the modules in the above system are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description in the method embodiment of the present invention, and will not be repeated here.

[0035] Similarly, the system of this invention can efficiently filter, aggregate, calculate, and convert real-time data from a database before pushing it to another database or other service platform. It ensures basic data consistency and requires no additional programming.

[0036] It should be noted that not all steps and modules in the above processes and system structures are mandatory; some steps or modules can be omitted as needed. The execution order of the steps is not fixed and can be adjusted as required. The system structures described in the above embodiments can be physical or logical structures. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be implemented by certain components in multiple independent devices.

[0037] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A data push method for a distributed database, characterized by: Start the water level coordination task on any node in the database cluster, and establish push tasks on all nodes. The push task periodically performs a flush operation to ensure that the target node returns a push success message. The largest water level among all pushed data in the period is sent to the water level coordination task. The water level coordination task receives the water level and selects the largest water level as the water level for the push task to write to storage. The push task records the target information, push data format, filtering and aggregation conditions. After inserting data into any node in the database cluster, it is determined whether the table containing the inserted data is associated with the push task. If so, the inserted data is filtered according to filtering or aggregation conditions, and the filtered data is formatted into the push data format. The unique write time of the record in the database cluster is obtained as a watermark, and the watermark and the formatted data are used together to generate a logical plan. When converting a logical plan into a physical plan and executing it, if it is determined that there is push data to be sent to the push task, asynchronous push will be performed.

2. The data push method for a distributed database according to claim 1, characterized in that: When one or more nodes in the database cluster malfunction, the push task stops. When all nodes recover, the push task reads the water level of the push task through the water level coordination task, and the push task restarts to push real-time data.

3. A data push system for a distributed database, characterized by: It includes a water level coordination task module, a task push module, a judgment module, and a filtering module. The water level coordination task module selects any node in the database cluster to start the water level coordination task, and the push task module establishes a push task on each of all nodes. The push task module periodically performs a flush operation according to the push task to ensure that the target node returns a push success message. It sends the largest water level among all pushed data within the period to the water level coordination task. The water level coordination task module receives the water level through the water level coordination task and selects the largest water level as the water level for the push task to write to storage. The push task records the target information, push data format, filtering and aggregation conditions. After inserting data into any node in the database cluster, the judgment module checks whether the table containing the inserted data is associated with a push task. If so, the filtering module filters the inserted data based on filtering or aggregation conditions, formats the filtered data into the push data format, and the push task module obtains the unique write time of the record in the database cluster as a watermark. The watermark and the formatted data are then used together to generate a logical plan. When the push task module converts the logical plan into a physical plan and executes it, it will perform asynchronous push if it determines that there is push data to be sent to the push task.

4. A data push system for a distributed database according to claim 3, characterized in that: When one or more nodes in the database cluster fail, the push task module stops pushing tasks. When all nodes recover, the water level coordination task module reads the water level of the push task through the water level coordination task, and the push task module restarts pushing real-time data according to the push task.

Images