Change data capture method of mysql database based on mapreduce

Abstract

The invention discloses a MapReduce-based CDC (Change Data Capture) method of an MYSQL database. The MapReduce-based CDC method comprises the steps of (1) generating a query statement 'select into outfile' of an abstract, and setting a zone bit according to a FIELDS clause; inserting an 'attribute value separator' into a line of tuples obtained by searching the database by the 'select into outfile'; generating abstract md5value and generating an output format for a searching result of 'select into outfile' according to a zone bit value; writing the searching result into a disk file outfile; (2) calculating difference by adopting a Hadoop MapReduce parallel framework; reading in two snapshoot files of old.txt and new.txt from a map end, storing a value of same keys in a Key / value structure in an iterator by a shuffle function of MapReduce, and synthesizing an output file of reduce into an insert file and a delete file, i.e obtaining a CDC result. According to the MapReduce-based CDC method disclosed by the invention, both grammar and implementation of the query statement in MYSQL is improved, a snapshoot file with the abstract can be generated by searching a data file of the database in one step, one I / O (Input / Output) is reduced by the generation of one snapshoot file, and a large amount of I / O can be reduced by multiple continuous snapshoot difference processes.

Description

technical field [0001] The invention relates to the technical field of data capture, in particular to a method for capturing changed data of a MapReduce-based MYSQL database. Background technique [0002] Change data capture (change data capture, CDC) is one of the main problems to be solved in the ETL (Extract Transform Load) process. CDC is used to capture the data of data update operations (such as insert, delete, and update) in the production database, and provide incremental data extraction for data synchronization update of enterprise application databases such as OLAP databases, report databases, data warehouses, and business intelligence databases. Serve. [0003] Existing change data capture CDC methods can be summarized into five categories: [0004] (1) CDC method based on time attribute [0005] If the database table is an append-only table (a table that only allows insertion and does not allow deletion and update operations is called an append-only table), th...

AI Technical Summary

Problems solved by technology

The CDC method based on snapshot difference can store snapshot files on a system other than the production system, so it does not have any impact on the execution efficiency of the production system and business system, the algorithm implementation is complicated, and there is a delay in the synchronization operation

Images