System and method for data migration

Abstract

The invention provides a system and a method for data migration. The system specifically comprises a configuration reading module, a reading module, a data exchange buffer module and a writing module, wherein the configuration reading module is used for reading a configuration file of a data migration task, and the configuration file comprises data reading information and data writing information; the reading module is used for reading the data according to the data reading information in the configuration file and converting the read data into an intermediate format to be written into the data exchange buffer module; the data exchange buffer module is used for temporarily storing the written intermediate format data; and the writing module is used for reading the intermediate format data from the data exchange buffer module, writing the information according to the data in the configuration file and converting the intermediate format data into a target format to be written into a target address. By the system, the data migration performance can be improved, and the maintenance cost and the development cost which are spent on data migration are lowered.

Description

technical field [0001] The present application relates to the technical field of computer system data processing, in particular to a data migration system and method. Background technique [0002] At present, with the development of network technology and the increase of information volume, a huge number of distributed heterogeneous information sources have been formed on the network. Due to the different geographical locations or the increase of new needs, various enterprises and institutions have also formed a distributed, heterogeneous and huge data collection. A very important reason for the emergence of heterogeneous data sources is the simultaneous existence of multiple commercial databases. [0003] There are various forms of heterogeneous data sources, which can be distinguished in terms of structure. They can include not only structured data in databases (such as relational databases), but also unstructured data in file systems, such as general text files, emails, ...

AI Technical Summary

Problems solved by technology

[0005] 1. Both data export and transcoding process require disk dump, because disk dump requires a lot of disk I / O (input / output, input / output), and disk is a slow device, which leads to the process of data migration It is relatively long, and the performance of data migration is low; in addition, in the long data migration process, users not only need to connect the upstream and downstream processes, monitor each process, but also monitor the disk space, which greatly increases the operation and maintenance cost;

[0006] 2. Although there are many mature data import and export tools, they generally only support one or a few specific types of databases; once you have many heterogeneous data sources, such as Mysql, Oracle, Rac, Hive and other database / file systems , and often need to import and export data between these heterogeneous data sources, then it is necessary to develop / maintain / learn to use a batch of such tools, and as the type of data source increases, the number of required import and export tools will increase linearly , and the development of import and export tools will cost a lot of development costs

Images