Data pushing method and system for distributed database

Abstract

The invention discloses a data pushing method and system for a distributed database, and relates to the field of distributed databases. The method comprises the following steps: selecting any node in a database cluster to start a water level line coordination task, establishing pushing tasks at all nodes, recording pushing target information, a pushing data format and filtering and aggregation conditions through the pushing tasks, inserting data into any node in the database cluster, judging whether a table into which the data is inserted is associated with the pushing tasks or not, and if yes, executing the pushing task. If yes, screening the inserted data according to a filtering or aggregation condition, formatting the screened data into a push data format, acquiring and recording unique write-in time of a database cluster as a water level line, generating a logic plan by the water level line and the formatted data, converting the logic plan into a physical plan, and executing the physical plan; and judging that the push data is sent to the push task for asynchronous push.

Description

Technical Field

[0001] The present invention discloses a data push method and system for a distributed database, relating to the field of distributed databases. Background Art

[0002] With the development of the information age, functions such as selecting, filtering, aggregating, calculating, converting formats, and transmitting massive real-time data in a database to another database or other service platforms have been widely used. In existing solutions, for example, data synchronization between databases is performed through database logs, which is commonly used for replication between primary and standby nodes. However, the log form is not conducive to filtering and aggregating calculations and does not support writing to other different platforms; while an independent program uses polling to apply filtering conditions, pulls data from the database through SQL statements, and then writes it to the target service. This method increases the complexity of the entire system maintenance; and when the database provides a publish-subscribe interface and other programs subscribe to this solution, it also increases the complexity of the entire system maintenance. Summary of the Invention

[0003] Aiming at the problems of the prior art, the present invention provides a data push method and system for a distributed database, which can efficiently implement filtering, aggregating, calculating, converting formats, and pushing real-time data in the database to another database or other service platforms. And it can ensure the basic consistency of data and does not require maintaining additional programs.

[0004] The specific solution proposed by the present invention is as follows:

[0005] The present invention provides a data push method for a distributed database, selects any node in the database cluster to start a waterline coordination task, establishes push tasks on all nodes, and records push target information, push data format, filtering and aggregation conditions through the push tasks.

[0006] After inserting data into any node in the database cluster, it is judged whether the table where the inserted data is located is associated with a push task. If so, the inserted data is screened according to the filtering or aggregation conditions, the screened data is formatted into the push data format, and the unique write time of the record database cluster is obtained as the waterline. The waterline and the formatted data are together generated into a logical plan.

[0007] When converting the logical plan into a physical plan and executing, it is judged that if there is push data sent to the push task, asynchronous push is performed.

[0008] Furthermore, the data push method for a distributed database regularly executes a flush operation according to the push task, ensures that the push target node returns a message indicating successful push, and sends the largest waterline among all push data within the period to the waterline coordination task.

[0009] Furthermore, the data pushing method of the distributed database receives watermarks through the watermark coordination task, and selects the maximum watermark as the watermark for the pushing task to be written into storage.

[0010] Furthermore, when one or more nodes in the database cluster are abnormal, the data pushing method of the distributed database stops the pushing task. When all nodes are restored, the watermark coordination task reads the watermark of the pushing task through the watermark coordination task, and resumes the real-time data pushing according to the pushing task.

[0011] The present invention also provides a data pushing system for a distributed database, including a watermark coordination task module, a pushing task module, a judgment module, and a screening module.

[0012] The watermark coordination task module selects any node in the database cluster to start the watermark coordination task. The pushing task module establishes pushing tasks on all nodes, and records the pushing target information, pushing data format, filtering and aggregation conditions through the pushing tasks.

[0013] After the judgment module inserts data into any node in the database cluster, it judges whether the table where the inserted data is located is associated with a pushing task. If so, the screening module screens the inserted data according to the filtering or aggregation conditions, formats the screened data into the pushing data format, and the pushing task module obtains the unique write time of the record database cluster as the watermark, and generates a logical plan together with the formatted data.

[0014] When the pushing task module converts the logical plan into a physical plan and executes it, if it judges that there is pushing data to be sent to the pushing task, it performs asynchronous pushing.

[0015] Furthermore, the pushing task module of the data pushing system for the distributed database regularly performs a flush operation according to the pushing task, ensures that the target node of the push returns a successful push message, and sends the maximum watermark among all the pushed data within the period to the watermark coordination task.

[0016] Furthermore, the watermark coordination task module of the data pushing system for the distributed database receives watermarks through the watermark coordination task, and selects the maximum watermark as the watermark for the pushing task to be written into storage.

[0017] Furthermore, when one or more nodes in the database cluster are abnormal, the pushing task module stops the pushing task. When all nodes are restored, the watermark coordination task module reads the watermark of the pushing task through the watermark coordination task, and the pushing task module resumes the real-time data pushing according to the pushing task.

[0018] The advantages of the present invention are:

[0019] The present invention can efficiently implement filtering, aggregation calculation, format conversion of real-time data in a database, and push the data to another database or other service platforms. It can ensure the basic consistency of data and does not require maintaining additional programs. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] The present invention will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the embodiments cited are not intended to limit the present invention.

[0022] Embodiment 1

[0023] The present invention provides a method for pushing data of a distributed database: select any node in the database cluster to start the waterline coordination task, establish a push task on each node, record the push target information, push data format, filtering and aggregation conditions through the push task, so that a one-to-many connection relationship is established between the waterline coordination task and the push task.

[0024] After inserting data into any node in the database cluster, it can be judged in the node optimizer whether the table of the inserted data is associated with a push task. If so, while formatting the inserted data into the format required for writing and storage, the inserted data is screened according to the filtering or aggregation conditions, the screened data is formatted into the push data format, and the unique write time of the record database cluster is obtained as the waterline. The waterline and the formatted data are together generated into a logical plan.

[0025] When converting the logical plan into a physical plan and executing, if it is judged that there is push data to be sent to the push task, asynchronous push is performed.

[0026] At the same time, perform a flush operation regularly according to the push task to ensure that the push target node returns a message indicating successful push, and send the largest waterline among all the push data within the period to the waterline coordination task.

[0027] And receive the waterlines sent by multiple nodes through the waterline coordination task, and select the largest waterline as the waterline of the push task to be written into storage.

[0028] When one or more nodes in the database cluster are abnormal, stop the push task. When all nodes recover, read the waterline of the push task through the waterline coordination task, read the historical data whose storage time is greater than the waterline, that is, the historical data during the cluster abnormality period for pushing, and start pushing real-time data again according to the push task.

[0029] Embodiment 2

[0030] The present invention also provides a data push system for a distributed database, which includes a waterline coordination task module, a push task module, a judgment module, and a screening module.

[0031] The waterline coordination task module selects any node in the database cluster to start the waterline coordination task. The push task module establishes push tasks on all nodes, and records push target information, push data format, filtering and aggregation conditions through the push tasks.

[0032] After data is inserted into any node in the database cluster, the judgment module determines whether the table where the inserted data is located is associated with a push task. If so, the screening module screens the inserted data according to the filtering or aggregation conditions, formats the screened data into the push data format, and the push task module obtains the unique write time of the record database cluster as the waterline, and generates a logical plan together with the formatted data.

[0033] When the push task module converts the logical plan into a physical plan and executes it, if it judges that there is push data to be sent to the push task, asynchronous push is performed.

[0034] Regarding the information interaction and execution process among the above-mentioned modules in the system, since they are based on the same concept as the method embodiment of the present invention, the specific content can be referred to the description in the method embodiment of the present invention, and will not be elaborated here.

[0035] Similarly, the system of the present invention can efficiently implement filtering, aggregating, calculating, formatting and pushing real-time data in the database to another database or other service platforms. And it can ensure the basic consistency of the data, and there is no need to maintain additional programs.

[0036] It should be noted that not all steps and modules in the above-mentioned processes and system structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted according to needs. The system structures described in the above embodiments can be physical structures or logical structures, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities respectively, or some components in multiple independent devices may be jointly implemented.

[0037] The above-mentioned embodiments are only preferred embodiments given to fully illustrate the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present invention are all within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A data push method for a distributed database, characterized in that Select any node in the database cluster to start the waterline coordination task. Establish a push task on each node. Record the push target information, push data format, filtering, and aggregation conditions through the push task. After inserting data into any node in the database cluster, determine whether the table where the inserted data is located is associated with a push task. If so, filter the inserted data according to the filtering or aggregation conditions, format the filtered data into the push data format, and obtain the record of the unique write time of the database cluster as the waterline. Generate a logical plan with the waterline and the formatted data. When converting the logical plan to a physical plan and executing it, determine whether there is push data to be sent to the push task. If so, perform asynchronous push.

2. The data push method of a distributed database according to claim 1, characterized in that Regularly execute the flush operation according to the push task to ensure that the target node of the push returns a message indicating successful push. Send the maximum waterline among all the push data within the period to the waterline coordination task.

3. The data push method of a distributed database according to claim 1, characterized in that Receive the waterline through the waterline coordination task, and select the maximum waterline as the waterline of the push task and write it to storage.

4. A data push method for a distributed database according to claim 1, characterized in that When one or more nodes in the database cluster are abnormal, stop the push task. When all nodes recover, read the waterline of the push task through the waterline coordination task, and restart the real-time data push according to the push task.

5. A data push system for a distributed database, characterized in that It includes a waterline coordination task module, a push task module, a judgment module, and a filtering module. The waterline coordination task module selects any node in the database cluster to start the waterline coordination task. The push task module establishes a push task on each node. Record the push target information, push data format, filtering, and aggregation conditions through the push task. After inserting data into any node in the database cluster, the judgment module determines whether the table where the inserted data is located is associated with a push task. If so, the filtering module filters the inserted data according to the filtering or aggregation conditions, formats the filtered data into the push data format, and the push task module obtains the record of the unique write time of the database cluster as the waterline. Generate a logical plan with the waterline and the formatted data. When the push task module converts the logical plan to a physical plan and executes it, determine whether there is push data to be sent to the push task. If so, perform asynchronous push.

6. The data push system of a distributed database according to claim 5, characterized in that The push task module regularly executes the flush operation according to the push task to ensure that the target node of the push returns a message indicating successful push. Send the maximum waterline among all the push data within the period to the waterline coordination task.

7. A data push system for a distributed database according to claim 5, characterized in that Waterline coordination The task module receives the waterline through the waterline coordination task, and selects the maximum waterline as the waterline of the push task and writes it to storage.

8. A data push system for a distributed database according to claim 5, characterized in that When one or more nodes in the database cluster are abnormal, the push task module stops the push task. When all nodes recover, the waterline coordination task module reads the waterline of the push task through the waterline coordination task, and the push task module restarts the real-time data push according to the push task.

Images