Method and apparatus for trickle-down transmission of data, storage medium, and terminal

Abstract

A method and apparatus for trickle-down transmission of data, storage medium and a terminal, the method comprising: full and incremental streams are imported in parallel, The total data stream and theincremental data stream respectively comprise at least one record row, for each record row, the record row having a timestamp for uniquely identifying the data version, a status code for identifyingthe validity of the record row, and a row key, the row key being determined according to a primary key of the record row; For a plurality of record rows with the same row key, respective timestamps and status codes of the plurality of record rows are merged according to an operation type of a record row imported from an incremental data stream among the plurality of record rows, the operation typebeing used to indicate an import mode of the record row. The scheme provided by the invention can effectively realize the data real-time synchronization between the database and the big data platformand between the big data platform and the big data platform, and ensure the continuous availability of the big data platform during the synchronization period.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a data trickle transmission method and device, a storage medium, and a terminal. Background technique [0002] With the development of information technology, big data applications have penetrated into every industry and business function field today, becoming an important factor of production. Therefore, how to effectively manage and further synchronize, replicate and apply big data has become the key to providing core competitiveness and seizing market opportunities. [0003] In the application field of traditional relational databases, real-time data synchronization between databases relies more on the mechanism of first full and then incremental. Specifically, you need to export the full amount of data from the source table and then import it to the target table. After the full amount is imported, use the incremental synchronization tool to synchronize the increment...

AI Technical Summary

Problems solved by technology

[0004] However, based on such a scheme, it is often impossible to effectively synchronize data in real time between the database and the big data platform, and between the big data platform and the big data platform, and it still needs to be done through full derivatives.

However, frequent full data export and import not only increases the pressure on the source database, but also reduces the performance of the source database and reduces the continuous availability of the big data platform, because the relevant functions of the big data platform are unavailable during the derivative period.

[0005] To sum up, the existing technology cannot provide a real-time data synchronization solution that can be applied to the big data platform, so as to realize the real-time data synchronization between the database and the big data platform, and between the big data platform and the big data platform, and ensure that Continuous Availability of Big Data Platforms During Synchronization

Images