Method, device and system for data synchronizing, medium and electronic equipment

Abstract

The embodiment of the invention provides a method, a device and a system for data synchronizing, a medium and electronic equipment. The data synchronizing method comprises the following steps of detecting the change of records and table structures of logs in a database, and obtaining the change data of the records and the change data of the table structures of the logs in the database; sending thechange data of the records to a HBase data control write module; sending the change data of the table structures to a Hive table structure synchronizing module. By adopting the technical scheme, thedata synchronizing method provided by the embodiment has the advantage that the problems of inaccuracy in data and inconsistency of data structures are solved by a log analysis real-time duplicating technique, data version control and delay control, so as to reach the effect of complete consistency between the same-day data of a Hive database and the change data of a source database, and meet theT+1 high-precision data analysis and report requirements of the Hive database.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a data synchronization method, device, system, medium and electronic equipment. Background technique [0002] With the promotion and popularization of the Internet and big data technologies, data warehouses based on Hadoop and Hive distributed clusters have gradually become mainstream. At present, the data sources of the data warehouse mainly include structured data such as members and orders stored in relational databases, as well as unstructured data stored in the form of files such as user behavior, application logs, images and videos, among which members stored in relational databases Structured data such as orders and orders is the most important part of data warehouse data sources, and it is also the most valuable part of data analysis and mining. [0003] The current method of synchronizing data T+1 from the relational database to the Hive data warehouse is main...

AI Technical Summary

Problems solved by technology

[0005] (1) Data consistency is difficult to guarantee

Since the offline extraction method uses the update time field of the record as the query condition to incrementally extract data, the source-side relational database may appear that the update time field is not modified synchronously when the record is modified, or the record is deleted, resulting in this part Changed data cannot be extracted

In addition, scheduled tasks for offline extraction do not necessarily start on time at zero o'clock every day, and are often started in batches after zero o'clock according to priority. Therefore, during the time interval between zero o'clock and the start of offline extraction tasks, records that have changed the previous day may be deleted. Update again, the update time field is synchronously modified to the time of the day, so that these data cannot be extracted

[0006] (2) The consistency of the data table structure is difficult to be guaranteed

[0007] Due to the above shortcomings, the offline extraction method cannot well meet the data analysis scenarios such as financial statements that require high data consistency but relatively low data timeliness requirements, which will lead to incorrect data in financial statements and affect the enterprise normal production and operation activities

Images