Data pushing method and system

Abstract

The invention relates to a data pushing method and system. The data push method includes: extracting task creation processing, dumping data from the source database to the database incremental log parsing server with the database instance as the boundary, obtaining incremental data through database incremental log parsing, and converting the obtained incremental The data is temporarily stored in the memory of the incremental log parsing server of the database; the message distribution process distributes the obtained incremental data as messages to one or more distributed message queue servers with corresponding topics and tags according to the required rules In a message queue, the topic represents the logical library name, and the tag represents the logical table name. Each piece of incremental data distributed as a message includes the corresponding topic and tag; and subscription push processing, the distributed message queue server is based on the subscriber's Each subscribing task subscribes to a topic and pushes the one or more message queues to the subscribing end.

Description

technical field [0001] The present invention relates to a large-scale data stream real-time consumption technology based on database incremental log analysis, and more specifically, relates to a data push method and system. Background technique [0002] With the development of the company's business, the demand for large-scale data extraction has become stronger. Data extraction refers to extracting the required data from the source database. In practical applications, relational databases are mostly used as source databases. Common data extraction methods include: (1) full data extraction, which is similar to data migration or data replication, which extracts the data of tables or views in the source from the database intact; (2) incremental data extraction Incremental data extraction, incremental data extraction only extracts new or modified data in the tables to be extracted in the database since the last extraction. Incremental data extraction is more widely used than...

AI Technical Summary

Problems solved by technology

[0004] The traditional incremental data extraction method is to regularly dump incremental data to generate corresponding data files, but this method has the following problems: it is necessary to add a timestamp field to the source-end business table, which pollutes the source-end business table to a certain extent; There is a certain degree of delay in the data; the dumped data files are too closely related to the business and are not easy to be reused; when the scheduled task is started, it often causes a large instantaneous pressure on the source database and network

[0011] This problem is also related to the above design, because the incremental data does not land, and the log storage time of the source database cannot be controlled by the data push platform, so the data push platform cannot promise the incremental data storage period to the downstream subscribers

This also reduces the ability of the data push platform to provide external services

[0012] (3). Subscribing end and extracting end depend too much

[0015] Business level: The extraction task is bounded by the database instance, and it is impossible to shield the subscriber from information that the subscriber should not care about, such as source sub-database sub-table, which leads to a stronger coupling between the extraction task and the subscription task

[0016] (4). The subscription end lacks support for common functions

Another open source project otter (the project's open source link address: https: / / github.com / alibaba / otter) supports some of the above functions, but still cannot meet the complex and flexible customization requirements

Images