Data Migration Correction Method and System

Abstract

The invention relates to data migration, in particular to a big-data and extra-big-data migration and correction method. The invention provides the data migration and correction method and system. The method is applied to a system with a controller, and the controller is connected with a source data server and a target data server. The method comprises the following steps that: the source data server stores a migration file, the target data server is provided with an unused storage space used for storing the migration file, the migration file is divided into a plurality of data blocks according to the integral multiple of the number of the processor cores of the data server, and the source data server carries out calculation to obtain a data digest A independently corresponding to each data block; the data digest A is stored in the controller; the migration file is sent to the target data server; in the target data server, in a migration file transferring process, parallel computation is carried out on the data digest B of the data block in the migration file subjected to data migration; and whether the data digest A and the data digest B are the same or not are compared. The method is used for solving the problem of integrity checking during big data migration, and has the advantages of being high in reliability and high in speed.

Description

technical field [0001] This application relates to data migration, and in particular to large and very large data migration and correction methods. Background technique [0002] With the advent of the era of big data, various information systems of enterprises continue to generate massive amounts of data. While putting forward requirements for storage infrastructure, they also put forward high requirements for data migration technology and its implementation. On the other hand, according to the different forms and characteristics of the data, the data in the enterprise can usually be divided into structured data and unstructured data. Structured data refers to the data type stored in a relational database and can be expressed in a two-dimensional table structure. It is usually generated by various information systems and stored in a relational database in text format; unstructured data refers to Compared with structured data, data types that are not convenient to express in...

AI Technical Summary

Problems solved by technology

[0006] However, the method of comparing the size of the original file with the migrated file cannot fully guarantee the occurrence of various data incomplete situations such as data loss, data corruption, data semantic change, and human tampering.

Taking the file length as an example, even if the file size is the same, the file may not be opened normally due to the misalignment of some bytes.

[0007] Also compare the CRC or MD5 check value of the original file with the CRC or MD5 check value of the migrated file ((that is, the data migration process includes 3 processes: the process of data transmission, the calculation of the migration file after data transmission The process of MD5 value and the process of whether the MD5 value before and after the comparison are the same), resulting in a long time-consuming data migration process and low migration efficiency, which is not suitable for large-scale unstructured data migration projects

Images