A data migration method and system

Abstract

The embodiment of the application discloses a data migration method and system, the method comprises the following steps: a log replication server SRS records data operation logs of a source end database in real time from a target time, the data operation logs reflect data changes of the source end database, the log replication server SRS establishes a connection with a target end database, so that the target end database synchronizes incremental data of the source end database changed from the target time to the target end database according to the data operation logs. As can be seen, the embodiment of the application uses an additional log replication server SRS, and records data operation logs of the source end database from the target time by using the log replication server SRS, and the target end database synchronizes incremental data of the source end database changed from the target time to the target end database according to the data operation logs. That is, by using the log replication server SRS, the smooth and efficient migration of data is realized without affecting system business.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a data migration method and system. Background Technology

[0002] With the rapid development of computer technology, the amount and requirements for data storage are increasing. Given the large-scale retirement of outdated storage devices with limited capacity and no new ones being added, how to efficiently and smoothly migrate production databases running on older storage devices to a new platform while minimizing the impact on system operations is a pressing issue. Summary of the Invention

[0003] This application provides a data migration method that can smoothly and efficiently migrate data between old and new platforms with minimal impact on system operations.

[0004] This application provides a data migration method, the method comprising:

[0005] The Log Replication Server (SRS) records data operation logs of the source database in real time from the target time, and the data operation logs reflect the data changes of the source database.

[0006] The Log Replication Server (SRS) establishes a connection with the target database so that the target database can synchronize incremental data that has changed from the source database since the target time to the target database based on the data operation log.

[0007] Optionally, the method further includes:

[0008] The Log Replication Server (SRS) monitors in real time whether the target database has completed the synchronization of incremental data. If it has, the target database stops synchronizing.

[0009] Optionally, the method further includes:

[0010] The source database responds to the backup DUMP command by copying historical data stored before the target time to obtain backup data;

[0011] In response to the LOAD command, the target database uses the backup data to restore historical data stored before the target time.

[0012] Optionally, before the target database responds to the download LOAD command, the method further includes:

[0013] The log replication server SRS disconnected from the target database.

[0014] After the target database has finished restoring the historical data stored before the target time using the backup data, the method further includes:

[0015] The log replication server (SRS) restores its connection to the target database.

[0016] Optionally, before the log replication server (SRS) records the data operation logs of the source database in real time starting from the target time, the method further includes:

[0017] A connection is pre-established between the log replication server (SRS) and the source database.

[0018] Optionally, the source database is a Sybase ASE database, and the target database is Linux.

[0019] This application provides a data migration system, including a log replication server (SRS) and a target database;

[0020] The Log Replication Server (SRS) is used to record data operation logs of the source database in real time from the target time and establish a connection with the target database. The data operation logs reflect the data changes of the source database.

[0021] The target database is used to synchronize incremental data that has changed from the source database since the target time to the target database according to the data operation log.

[0022] Optionally, the log replication server (SRS) is used to monitor in real time whether the target database has completed synchronizing the incremental data. If it has, the target database stops synchronizing.

[0023] Optionally, the system further includes: a source database;

[0024] The source database is used to copy historical data stored before the target time in response to the backup DUMP command, so as to obtain backup data;

[0025] The target database is used to respond to the LOAD command and restore historical data stored before the target time using the backup data.

[0026] Optionally, before the target database responds to the download LOAD command, the log replication server SRS is also used to disconnect from the target database;

[0027] After the target database has finished restoring the historical data stored before the target time using the backup data, the log replication server SRS is also used to restore the connection with the target database.

[0028] This application provides a data migration method, comprising: a Log Replication Server (SRS) recording data operation logs of a source database in real time from a target time, the data operation logs reflecting data changes in the source database; and the SRS establishing a connection with a target database so that the target database can synchronize incremental data changes from the source database since the target time to the target database based on the data operation logs. Therefore, this application utilizes an additional Log Replication Server (SRS) to record data operation logs of the source database from the target time, and subsequently, the target database synchronizes incremental data changes from the source database since the target time to the target database based on these logs. In other words, by using the Log Replication Server (SRS), a smooth and efficient data migration is achieved without affecting system operations. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A flowchart illustrating a data migration method provided in this application embodiment;

[0031] Figure 2 A flowchart illustrating database migration using the DUMP / LOAD method is provided as an embodiment of this application;

[0032] Figure 3 A flowchart illustrating database migration using a log replication server is provided as an embodiment of this application.

[0033] Figure 4 This is a structural block diagram of a data migration system provided in an embodiment of this application. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0035] With the rapid development of computer technology, the amount and requirements for data storage are increasing. Given the large-scale retirement of outdated storage devices with limited capacity and no new ones being added, how to efficiently and smoothly migrate production databases running on older storage devices to a new platform while minimizing the impact on system operations is a pressing issue.

[0036] Specifically, this involves how to automatically migrate applications using the Sybase ASE database on minicomputers to x86 server platforms, thereby improving the automation level of the Sybase ASE database migration from the AIX platform to the Linux platform, simplifying the migration process, and reducing the impact on production operations in scenarios such as the obsolescence of old equipment and data center relocation.

[0037] Based on this, this application provides a data migration method, which includes: a Log Replication Server (SRS) recording data operation logs of the source database in real time from a target time, the data operation logs reflecting data changes in the source database; and the SRS establishing a connection with the target database so that the target database can synchronize incremental data changes from the source database since the target time to the target database according to the data operation logs. Thus, this application utilizes an additional Log Replication Server (SRS) to record data operation logs of the source database from the target time, and subsequently, the target database synchronizes incremental data changes from the source database since the target time to the target database according to the data operation logs. In other words, by using the Log Replication Server (SRS), a smooth and efficient data migration is achieved without affecting system operations.

[0038] See Figure 1 The figure is a flowchart of a data migration method provided in an embodiment of this application.

[0039] The data migration method provided in this embodiment includes the following steps:

[0040] S101, the Log Replication Server (SRS) records the data operation logs of the source database in real time starting from the target time.

[0041] In the embodiments of this application, the SAP Replication Server (SRS) is a real-time replication tool based on database logs, providing replication of transaction data in heterogeneous environments. The remote database can be SybaseASE, and the target database can be a Linux platform.

[0042] In the embodiments of this application, when migrating data from the source database to the target database, different migration methods can be selected depending on the amount of data to be migrated:

[0043] The first method involves migrating data if the data volume is less than or equal to 800GB, or if the downtime window is less than 12 hours. This involves using a dump backup / load physical backup and restore method, and then remounting the data files using NAS storage. The downtime window refers to the time during which the source database is shut down for data backup.

[0044] Before proceeding with the migration, you can check the migration conditions and collect basic database information, such as whether the target database version is greater than the source database version, the operating system version, and whether the tables in the source database have unique indexes. You can also configure the target database, create a database service, use the same character set and page size as the source database, copy the source database configuration parameters to the target database, create the database device and database, configure the database dboption options, add a login user, and import the sa_pass database from the source database.

[0045] The downtime window can be used to simulate DUMP backup / LOAD, that is, to simulate database migration using DUMP / LOAD and to calculate the required downtime: Execute a DUMP operation to back up the source database without compression options, and record the operation time T1. Then, copy the backup file to the target database server via the network, and record the operation time T2. On the target server, execute a LOAD operation to restore the database, perform byte order conversion, and upgrade the database version, and record the operation time T3. Execute the SP_POST_XPLOAD command to rebuild failed indexes and partitions, and record the operation time T4. Calculate the downtime required for database migration using DUMP / LOAD method: T = T1 + T2 + T3 + T4, and also calculate the size of the source database space.

[0046] refer to Figure 2 The diagram shows a flowchart of a database migration process using the DUMP / LOAD method provided in this application embodiment. As shown, database migration-related monitoring metrics can be added, all applications and background tasks can be stopped (i.e., business services can be stopped), putting the database in a silent state. A DUMP operation is performed in single-user mode to back up the source database without compression. The DUMP backup file is copied to the target database server over the network. The target server performs a LOAD operation to restore the database, perform byte order conversion, and upgrade the database version to be compatible with version 15.7. The target server executes the SP_POST_XPLOAD command to rebuild failed indexes and partitions, stops the source database, and subsequently switches the application connection to the target database, thus achieving data migration.

[0047] The second approach is to use the DUMP / LOAD method to back up the existing data of the target database if the data volume is greater than 800G or the downtime window is greater than 12 hours. In such cases, a long-term downtime of the source database will cause business services to be unable to be provided to users. Therefore, the DUMP / LOAD method can be used to back up the existing data of the target database and the incremental data can be copied in real time through SRS, thereby reducing the impact on business, shortening downtime, and effectively achieving the goal of cross-platform database upgrade and migration.

[0048] In other words, the Log Replication Server (SRS) can record data operation logs of the source database in real time, starting from a target time, which is the time after the source database has completed backing up historical data using the DUMP / LOAD method. The data operation logs reflect the data changes in the source database from the target time, i.e., they are logs of incremental data.

[0049] In the embodiments of this application, before the log replication server SRS records the data operation logs of the source database in real time from the target time, the connection between the log replication server SRS and the source database can be established in advance.

[0050] S102, the log replication server SRS establishes a connection with the target database so that the target database can synchronize incremental data that has changed from the source database since the target time to the target database according to the data operation log.

[0051] In the embodiments of this application, the Log Server (SRS) establishes a connection with the target database, so that the target database can synchronize incremental data that has changed from the source database since the target time to the target database according to the data operation log. During the process of the target database synchronizing incremental data according to the data operation log, the source database does not need to be shut down and can still provide business services, with minimal impact on business and significantly shortening the downtime of database migration.

[0052] Log Replication Server (SRS) can monitor in real time whether the target database has completed the synchronization of incremental data. If it has, the target database stops synchronizing; if it has not, the target database continues to synchronize incremental data.

[0053] In the embodiments of this application, historical data before the target time can be backed up to the target database using the DUMP / LOAD method. The source database responds to the backup DUMP command by copying the historical data stored before the target time, i.e., the existing data, to obtain the backup data. The target database responds to the download LOAD command by using the backup data to restore the historical data stored before the target time.

[0054] Specifically, when backing up existing data in the source database using the DUMP / LOAD method, the Log Replication Server (SRS) can disconnect from the target database. After the existing data backup is complete, the SRS can restore the connection to the target database to facilitate the migration of incremental data.

[0055] refer to Figure 3 The diagram illustrates a flowchart of a database migration process using a log replication server, as provided in an embodiment of this application. As shown, an SRS server can be installed, database transaction logs can be deployed, replication queues can be deployed, replication status can be deployed, and a network connectivity monitoring script between the source database, target database, and SRS can be deployed. All application services and background tasks are stopped to put the database in a silent state. A DUMP operation is performed in single-user mode to back up the source database without compression options. The real-time log replication service from the source database to the target database is configured and started. The connection from SRS to the target database is suspended, application services are started, and the DUMP backup file is copied to the target database server over the network. The target database then performs a LOAD operation. The process involves restoring and upgrading the database version to be compatible with version 15.7. On the target end, the SP_POST_XPLOAD command is executed to rebuild failed indexes and partitions. At this point, the existing data migration is complete. The SRS replication server is reinstalled on the target database, the connection between SRS and the target database is restored, and incremental data after the target time is synchronized. The synchronization monitoring script determines whether the source and target databases are synchronized. If not, synchronization continues. If so, the application service is stopped, and data consistency verification is performed. If the verification passes, the replication service is stopped, the data migration is complete, the source database is stopped, and the application service connection is switched to the target database.

[0056] The data migration method provided in this application embodiment can be implemented using an automated migration script, supporting the migration of SybaseASE databases from IBM minicomputers to x86 virtualization resource pools, and from Aix operating systems to SuSE Linux operating systems. During the migration, the Sybase ASE version is upgraded from version 15.0 and 15.5 to version 15.7.

[0057] The data migration method provided in this application also fully considers possible fault scenarios during the migration process, automatically generates monitoring alarms, and ensures that emergency plans are activated in a timely manner without affecting business continuity.

[0058] This application provides a data migration method, comprising: a Log Replication Server (SRS) recording data operation logs of a source database in real time from a target time, the data operation logs reflecting data changes in the source database; and the SRS establishing a connection with a target database so that the target database can synchronize incremental data changes from the source database since the target time to the target database based on the data operation logs. Therefore, this application utilizes an additional Log Replication Server (SRS) to record data operation logs of the source database from the target time, and subsequently, the target database synchronizes incremental data changes from the source database since the target time to the target database based on these logs. In other words, by using the Log Replication Server (SRS), a smooth and efficient data migration is achieved without affecting system operations.

[0059] Based on the data migration method provided in the above embodiments, this application also provides a data migration system, the working principle of which will be described in detail below with reference to the accompanying drawings.

[0060] See Figure 4 The figure is a structural block diagram of a data migration system provided in an embodiment of this application.

[0061] The data migration system 400 provided in this embodiment includes:

[0062] Log replication server SRS410 and target database 420;

[0063] The Log Replication Server (SRS) is used to record data operation logs of the source database in real time from the target time and establish a connection with the target database. The data operation logs reflect the data changes of the source database.

[0064] The target database is used to synchronize incremental data that has changed from the source database since the target time to the target database according to the data operation log.

[0065] Optionally, the log replication server (SRS) is used to monitor in real time whether the target database has completed synchronizing the incremental data. If it has, the target database stops synchronizing.

[0066] Optionally, the system further includes: a source database;

[0067] The source database is used to copy historical data stored before the target time in response to the backup DUMP command, so as to obtain backup data;

[0068] The target database is used to respond to the LOAD command and restore historical data stored before the target time using the backup data.

[0069] Optionally, before the target database responds to the download LOAD command, the log replication server SRS is also used to disconnect from the target database;

[0070] After the target database has finished restoring the historical data stored before the target time using the backup data, the log replication server SRS is also used to restore the connection with the target database.

[0071] Optionally, the log replication server SRS is used to pre-establish a connection between the log replication server SRS and the source database.

[0072] When describing elements of various embodiments of this application, the articles “a,” “an,” “this,” and “described” are all intended to indicate that there are one or more elements. The words “comprising,” “including,” and “having” are inclusive and mean that there may be other elements in addition to those listed.

[0073] It should be noted that those skilled in the art will understand that all or part of the processes in the above method embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0074] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the system embodiments. The system embodiments described above are merely illustrative. The units and modules described as separate components may or may not be physically separate. Furthermore, some or all of the units and modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0075] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data migration method, characterized by, The method comprises: The log replication server SRS records the data operation log of the source database in real time from the target time, and the data operation log reflects the data change of the source database; The log replication server SRS establishes a connection with the target database, so that the target database synchronizes the incremental data of the source database changed from the target time to the target database according to the data operation log; When migrating the data in the source database to the target database, different migration modes are selected according to different amounts of migrated data: If the data amount is less than or equal to 800G, or the downtime window is less than 12 hours, the DUMP backup or download LOAD physical backup and recovery mode is used to perform data migration, and the NAS storage remounting mode is used to move data files; Before the specific migration, the source and target database version compatibility is detected; The operating system version and the unique index of the source database table are verified; The target database service is configured to keep the same character set and page size as the source; The source configuration parameters are copied to the target to create corresponding database devices and database structures; Specifically, the migration steps include: Adding database migration related monitoring indicators; Stopping all applications and background tasks to stop business services, so that the source database is in a silent state; Performing DUMP operation without compression option in single-user mode to backup the historical data of the source database, and obtaining backup data; Using the NAS storage remounting mode to move data files, and copying the backup data to the target database server through the network; The target database performs LOAD operation to restore the historical data using the backup data, and performs byte sequence conversion and database version upgrade to be compatible with 15.7 version, the source database is Sybase ASE database, and the target database is Linux; The target database executes SP_POST_XPLOAD command to rebuild invalid indexes and partitions; Stopping the source database and switching the application connection to the target database; If the data amount is greater than 800G or the downtime window is greater than 12 hours, the DUMP or LOAD mode is used to backup the inventory data of the target database, and the incremental data is replicated in real time through SRS.

2. The method of claim 1, wherein, The method further comprises: The log replication server SRS monitors whether the target database synchronizes the incremental data in real time, and if the synchronization is completed, the target database stops synchronizing.

3. The method of claim 1, wherein, The method further comprises: The source database copies the historical data stored before the target time in response to the backup DUMP command to obtain backup data; The target database recovers the historical data stored before the target time using the backup data in response to the download LOAD command.

4. The method of claim 3, wherein, Before the target database responds to the download LOAD command, the method further comprises: The log replication server SRS disconnects the connection with the target database; After the target database recovers the historical data stored before the target time using the backup data, the method further comprises: The log replication server SRS resumes the connection with the target database.

5. The method according to any one of claims 1 to 4, characterized in that, Before the log replication server SRS records the data operation log of the source database in real time from the target time, the method further comprises: The connection between the log replication server SRS and the source database is established in advance.

6. A data migration system, characterized by, The system comprises a log replication server SRS and a target database; The log replication server SRS is configured to record the data operation log of the source database in real time from the target time and establish a connection with the target database, wherein the data operation log reflects the data changes of the source database; The target database is configured to synchronize the incremental data of the source database changed from the target time to the target database according to the data operation log; Different migration methods are selected according to the amount of migrated data when migrating the data in the source database to the target database: If the data amount is less than or equal to 800G or the downtime window is less than 12 hours, the DUMP backup or the download LOAD physical backup and recovery method is used to perform data migration, and the NAS storage remounting method is used to move data files; Before the specific migration, the source and target database version compatibility is detected; The operating system version and the unique index of the source database table are verified; The target database service is configured to keep the same character set and page size as the source; The source configuration parameters are copied to the target to create corresponding database devices and database structures; Specifically, the migration steps comprise: Adding database migration related monitoring indicators; Stopping all applications and background tasks to stop business services, so that the source database is in a silent state; Performing a DUMP operation without compression options in single-user mode to backup the historical data of the source database and obtain backup data; Using the NAS storage remounting method to move data files, and copying the backup data to the target database server through the network; The target database performs a LOAD operation to restore the historical data using the backup data, and simultaneously performs byte sequence conversion and database version upgrade to be compatible with the 15.7 version, wherein the source database is a Sybase ASE database, and the target database is Linux; The target database executes an SP_POST_XPLOAD command to rebuild invalid indexes and partitions; Stopping the source database and switching the application connection to the target database; If the data amount is greater than 800G or the downtime window is greater than 12 hours, the DUMP or LOAD method is used to backup the inventory data of the target database, and the incremental data is replicated in real time through the SRS.

7. The system of claim 6, wherein, The log replication server SRS is configured to monitor in real time whether the target database synchronizes the incremental data, and if so, the target database stops synchronization.

8. The system of claim 6, wherein, The system further comprises a source database; The source database is configured to copy the historical data stored before the target time in response to a backup DUMP command to obtain backup data; The target end database is configured to restore the historical data stored before the target time by using the backup data in response to a download LOAD command.

9. The system of claim 8, wherein, Before the target end database responds to the download LOAD command, the log replication server SRS is further configured to disconnect the connection with the target end database. After the target end database restores the historical data stored before the target time by using the backup data, the log replication server SRS is further configured to restore the connection with the target end database.

Images