A data migration method and device

Abstract

The invention discloses a data migration method and device, and relates to the technical field of computers. A specific implementation of the method includes: Step 1, sending the data of the corresponding business from the original database to the message server cluster through the data scheduling task corresponding to each business; Step 2, sending the data of each business through the message server cluster The data is migrated to the corresponding database sub-database. This implementation method avoids the situation that a single data abnormality prevents the migration from continuing, and the migration task is interrupted due to a single machine downtime, realizes high-availability operation of the migration service, improves data migration efficiency, has strong scalability, and makes full use of sub-databases The high-throughput performance overcomes the long-term migration process, realizes automatic verification and cleaning of migrated data, and automatically retries unsuccessfully migrated data.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a data migration method and device. Background technique [0002] With the rapid development of the Internet today, the amount of data has grown beyond imagination and has been expanding geometrically. However, with the development of time and business in any system, the data involved in the system is also gradually increasing, and the CPU of each machine , disk, and memory are all limited, and when the amount of data is very large, the performance of the database will be very poor. Therefore, more and more enterprises have adopted the Mysql storage method of sub-database and sub-table to solve the following three types of problems: 1. The cost of database storage, Mysql is free and open source; 2. The storage space of the system can be expanded by adding machines ;3. Solve the bottlenecks of CPU, disk and memory in a single machine by adding machines. But then there are new ...

AI Technical Summary

Problems solved by technology

But then there are new problems: because the data is stored in multiple database machines in the form of sub-database and sub-table, there will be two difficult problems. One is how to get data from the traditional database server Migrate all the data to multiple new sub-databases and sub-tables; second, the subsequent data will continue to grow, and the existing sub-databases will be expanded again, which will also involve the problem of data migration

[0007] It is necessary to manually verify the accuracy of the data. When the data verification fails, it is necessary to manually supplement the data. After the data verification is passed, it is also necessary to manually start the process of cleaning and deleting historical data, which is time-consuming and labor-intensive;

Images