Data migration method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data migration method and device, equipment, a storage medium and a program product, and relates to the technical field of big data processing. The method comprises the steps of determining configuration information of to-be-migrated data, then generating backup data of the to-be-migrated data based on a slave database node address, transmitting the backup data to a migration target address, and then performing data recovery on the backup data in the migration target address. And finally, synchronously copying the to-be-migrated data and the backup data after data recovery, and determining a result after synchronous copying as migrated data. According to the method, the problems that the data migration efficiency of the multi-cluster big data is low and errors are prone to occurring are effectively solved.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a data migration method, device, equipment, storage medium and program product. Background Art

[0002] There are generally two ways to migrate data in MySQL: one is to export the source data into a file in the form of SQL statements, and then execute the exported file on the target side to achieve data migration; the other is to use physical copy to back up data, and also provide data recovery functions to achieve data migration through data backup and recovery.

[0003] In the related technologies, no matter which method is used, when processing data migration with large data volumes or migration of multi-cluster data, there is a problem that a large number of manual operations are required to complete the migration. Due to the large number of manual operations, errors are prone to occur during the migration process and the migration efficiency is low. Summary of the invention

[0004] The present application provides a data migration method, device, equipment, storage medium and program product to solve the problem of low efficiency and easy error in data migration of multi-cluster big data in related technologies.

[0005] In a first aspect, the present application provides a data migration method, comprising:

[0006] Determine configuration information of the data to be migrated, the configuration information includes a node address of the data to be migrated and a migration target address, the node address includes a slave database node address;

[0007] Generate backup data of the data to be migrated based on the slave database node address, and transmit the backup data to the migration target address;

[0008] Restore the backup data in the migration target address;

[0009] The data to be migrated is synchronously copied with the backup data after data recovery, and the result after the synchronous copy is determined as the migrated data.

[0010] In a second aspect, the present application provides a data migration device, including:

[0011] A configuration module, used to determine configuration information of the data to be migrated, the configuration information includes a node address of the data to be migrated and a migration target address, the node address includes a slave database node address;

[0012] A backup module, used to generate backup data of the data to be migrated based on the slave database node address, and transmit the backup data to the migration target address;

[0013] A recovery module is used to restore the backup data in the migration target address;

[0014] The synchronization module is used to synchronously copy the data to be migrated and the backup data after data recovery, and determine the result after the synchronous copy as the migrated data.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0016] The memory stores computer-executable instructions;

[0017] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0020] The data migration method, apparatus, device, storage medium and program product provided by the embodiments of the present disclosure determine the configuration information of the data to be migrated, then generate backup data of the data to be migrated based on the address of the slave database node, and transmit the backup data to the migration target address, then perform data recovery on the backup data in the migration target address, and finally synchronously copy the data to be migrated and the backup data after data recovery, and determine the result after synchronous copying as the data after migration. Thus, by determining the configuration information of the data to be migrated, including the node and the target address, the complexity of manual configuration is reduced. Secondly, the backup data is generated by the slave database node and automatically transmitted to the target address, reducing the risk of manual intervention. Finally, through data recovery and synchronous replication, the consistency and integrity of the data are ensured, thereby effectively solving the problems of error-proneness and inefficiency in traditional methods and achieving faster and more reliable data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0022] Figure 1 An application scenario diagram of the data migration method provided in the embodiment of the present disclosure;

[0023] Figure 2 A flowchart of a data migration method provided by an embodiment of the present disclosure;

[0024] Figure 3 A flowchart of a data migration method provided by another embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of the structure of a data migration device provided by another embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of the structure of an electronic device provided by one embodiment of the present disclosure.

[0027] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] In addition, this application involves conducting big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and using artificial intelligence technology to make automated decisions, and making technical solutions that have a significant impact on personal rights and interests based on the results of automated decisions. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decisions; if the user chooses to reject, the expert decision-making process will be entered.

[0031] It should be noted that the data migration methods, devices, equipment, storage media and program products provided in the present application can be used in the field of financial technology, and can also be used in any field other than the field of financial technology. The application fields of the data migration methods, devices, equipment, storage media and program products in the present application are not limited.

[0032] Figure 1 A schematic diagram of an application scenario of the data migration method provided in this application, such as Figure 1 As shown, during the data migration process, a slave database 111 corresponding to each cluster is selected from the original cluster 110 as backup data, so as to migrate the backup data to the new cluster 120 to complete the data migration process.

[0033] It should be noted that Figure 1 The scenario shown includes only one or a specific number of original clusters, subordinate databases, and new clusters for illustration, but the present disclosure is not limited to this, that is, the number of original clusters, clusters, subordinate databases, and new clusters can be arbitrary.

[0034] MySQL data migration is usually achieved through logical export and import. This method is suitable for small-scale data, but it is slow and prone to errors in large data volumes. Another method is to back up and restore through physical copies. Although it has an advantage in speed, it still requires a lot of manual operations and configuration in a multi-cluster environment. These methods increase operational complexity and error risks when dealing with large-scale data and multi-cluster architectures, resulting in low overall migration efficiency.

[0035] Since there are many factors involved in manual operations and multi-cluster environments, it is difficult to simply improve data migration tools. It is difficult to solve the processing problems of automation and multi-cluster environments by simply optimizing data migration tools. As a result, there is a lack of effective strategies in the existing technology to optimize data migration of multi-cluster big data.

[0036] The data migration method provided by the present application reduces the complexity of manual configuration by determining the configuration information of the data to be migrated, including nodes and target addresses. Then, backup data is generated based on the slave database node and automatically transmitted to the target address, reducing the risk of manual intervention. Finally, data consistency and integrity are ensured by performing data recovery and synchronous replication at the target address, thereby achieving fast and reliable data migration.

[0037] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0038] Figure 2 Schematic diagram of the data migration method provided for this application Figure 1 ,like Figure 2 As shown, the method includes:

[0039] S201: Determine configuration information of data to be migrated.

[0040] The configuration information includes the node address of the data to be migrated and the migration target address, and the node address includes the node address of the slave database.

[0041] Specifically, this embodiment is used to provide an overall description of the main steps of the data migration method.

[0042] The executing entity in the embodiment of the present disclosure is a controller that controls the data migration process, which may specifically be a server or processor related to the database corresponding to the data migration process, or a higher-level server or higher-level controller. For the sake of convenience, they are collectively referred to as a control system here.

[0043] In the initial stage of data migration, since it involves multi-cluster data migration, the control system needs to accurately determine the configuration information of the data to be migrated. This process involves identifying and recording the node addresses of the source data and the target migration address, especially the slave database node address.

[0044] Configuration information can be entered into the configuration file by the operation and maintenance personnel or management personnel who perform data migration through the database management system, or the configuration information involving multiple clusters can be automatically obtained and edited through the network management tool. For example, the operation and maintenance personnel can select multiple database clusters to be migrated and the corresponding migration target addresses through the network tool. At this time, the network management tool can automatically obtain the login information of each database cluster and the migration target address, and automatically generate the corresponding configuration information, thereby minimizing the manual workload and reducing the possibility of human error.

[0045] To ensure the accuracy and consistency of information, you can also use configuration management tools to automate the collection and management of node information, thereby synchronizing configuration information between multiple servers and database instances to ensure that the addresses and parameters used during the migration process are the latest and correct.

[0046] In some embodiments, the operation and maintenance personnel may also configure automated scripts and upload them to the control system to periodically verify and update configuration information to adapt to dynamically changing network environments and database architectures.

[0047] In this way, the control system can be fully prepared before the migration begins, reducing the risk of migration failure due to configuration errors.

[0048] S202: Generate backup data of the data to be migrated based on the slave database node address, and transmit the backup data to the migration target address.

[0049] Specifically, after obtaining the node information, the control system generates a data backup for migration. At this time, it will select a database from the slave database to generate the data backup to ensure that the master database can work normally during the data migration process (the data in the slave database is synchronized with the master database. Therefore, backing up the data in the slave database can have the effect of backing up the data in the cluster where the slave database is located).

[0050] The process of generating backup data usually uses physical backup tools to quickly and completely copy database files to ensure data consistency and integrity.

[0051] Once the backup is generated, you can transfer the backup data to the migration target address.

[0052] In some embodiments, to ensure the security of data transmission, a secure transmission protocol may be used to ensure the security of data during transmission and optimize transmission efficiency.

[0053] At the same time, in order to further improve the transmission speed, data compression technology can be used to compress the backup data during transmission to reduce the amount of data transmission, or parallel transmission technology can be used to transmit data simultaneously using multiple threads or multiple channels.

[0054] In some embodiments, the transmission process may also include an automated error detection and retry mechanism to cope with network interruptions or other transmission failures, thereby ensuring that the backup data can reach the target address intactly and without error.

[0055] S203: Restore the backup data in the migration target address.

[0056] Specifically, after the backup data is successfully transferred to the target address, data recovery is required to ensure that the data runs normally in the new environment.

[0057] Data recovery is usually performed using database recovery tools or automated scripts that are able to reconstruct the structure and content of the database from the backup files.

[0058] During the recovery process, special attention should be paid to the version compatibility and configuration requirements of the target database to avoid data inconsistency or recovery failure. To this end, data verification can be performed before recovery to ensure the integrity and correctness of the backup data.

[0059] In some embodiments, the recovery process should record detailed log information to facilitate troubleshooting and recovery verification.

[0060] In some embodiments, in order to improve recovery efficiency, incremental recovery technology can be used during the recovery process to only recover data that has changed since the last backup.

[0061] This not only speeds up the recovery process, but also reduces the usage of target system resources, ensuring that the system can continue to operate normally during the recovery process.

[0062] S204: Synchronously copy the data to be migrated and the backup data after data recovery, and determine the result after the synchronous copy as the migrated data.

[0063] Specifically, after the data recovery is completed, synchronous replication is required to ensure the consistency of the source data (data to be migrated) and the target data (data at the migration target address after migration).

[0064] Synchronous replication can be achieved through real-time data synchronization tools to apply changes in the source database to the target database in real time, ensuring that no data updates are lost during the migration process.

[0065] After the synchronous replication is completed, the control system also needs to compare the source data and the target data to confirm the completeness and accuracy of the migration. To this end, data verification tools or scripts can be used to automatically check the consistency and integrity of the data.

[0066] In some embodiments, in order to ensure the reliability of the synchronization process, an automated monitoring and alarm mechanism may be set up to promptly discover and resolve problems that may arise during the synchronization process.

[0067] As a result, the control system can provide consistent and reliable data services after the migration is completed, meeting business continuity requirements.

[0068] The data migration method provided by the embodiment of the present application determines the configuration information of the data to be migrated, then generates the backup data of the data to be migrated based on the node address of the slave database, and transmits the backup data to the migration target address, then performs data recovery on the backup data in the migration target address, and finally synchronously replicates the data to be migrated and the backup data after data recovery, and determines the result after synchronous replication as the data after migration. Thus, by determining the configuration information of the data to be migrated, including the node and the target address, the complexity of manual configuration is reduced. Secondly, the backup data is generated by the slave database node and automatically transmitted to the target address, reducing the risk of manual intervention. Finally, through data recovery and synchronous replication, the consistency and integrity of the data are ensured, thereby effectively solving the problems of error-proneness and inefficiency in traditional methods, and achieving faster and more reliable data migration.

[0069] Figure 3 Schematic diagram of the data migration process provided for this application Figure 2,like Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, the specific process in the data migration method is described in detail. The method includes:

[0070] S301 : Determine a corresponding group number for the data to be migrated in each cluster based on the cluster information of all the data to be migrated.

[0071] Among them, the cluster information includes the addresses of each node in the cluster and the corresponding migration target address. The node address also includes the master database node address. Each master database corresponds to at least one slave database. The data in the slave database is synchronized with the data in the master database. The migration target address includes the addresses of the master database node after migration and all slave database nodes thereafter.

[0072] Specifically, this embodiment is used to illustrate the detailed process of data migration.

[0073] Before the actual data migration begins, you need to determine the configuration information of each cluster and assign a group number to each cluster whose data is to be migrated so that it can be effectively managed and scheduled during the subsequent migration process.

[0074] In the disclosed embodiment, the database of each node is configured according to the master-slave database structure to ensure that the data of the slave database is synchronized with the master database, providing consistency guarantee for migration.

[0075] The specific process of determining cluster information requires identifying the role (such as whether it is a master database or a slave database) and status (whether there is an abnormality) of each node, and assigning a group number based on this information.

[0076] In some embodiments, the group number allocation strategy can be adjusted according to the size and complexity of the cluster. For example, for a large cluster, the group numbers can be further subdivided to more finely control the migration process.

[0077] In some embodiments, the system can automatically detect the topology of the cluster and dynamically adjust the group number allocation to adapt to different network environments and data loads.

[0078] The goal of this step is to ensure that the system can quickly locate and process the data of each cluster during the subsequent migration process through reasonable group number allocation, reduce human intervention, and improve data migration efficiency.

[0079] S302: Generate configuration information of each data to be migrated based on the group number.

[0080] The configuration information includes: group number, cluster number, login information of the server where the cluster is located, database login information of each node in the cluster, backup data compression and decompression methods, and storage address of the backup data.

[0081] Specifically, to ensure effective data migration, it is necessary to generate configuration information for each data to be migrated to ensure clear operation instructions during the migration process.

[0082] The control system can automatically extract necessary data from the cluster information and organize it into a structured configuration information file to guide subsequent backup, transmission and recovery operations.

[0083] For example, the configuration information file corresponding to all clusters can be split into multiple sub-configuration files according to the group number to correspond to each cluster. Each sub-configuration file can obtain the value of the nth column, and then save each configuration information in the format of key:value, separated by commas, and one line corresponds to the configuration information of one database. For example:

[0084] Table 1 Database configuration information storage format

[0085]

[0086] In the configuration information, you can also save it as:

[0087] group_num:1,set_num:1,mysql_ip:xx.xx.xx.xx......

[0088] group_num:1,set_num:2,mysql_ip:xx.xx.xx.xx......

[0089] Finally, the control system can obtain the configuration information of the corresponding database based on the number of rows and keys.

[0090] In some embodiments, before performing subsequent data backup, it is also necessary to determine, based on the configuration information, whether the node address of the data to be migrated and the migration target address meet the environmental check requirements, wherein the environmental check requirements include determining whether the status of the database at the corresponding address meets the corresponding database status requirements, and determining whether the remote connection status between different addresses meets the corresponding connection status requirements.

[0091] Specifically, by adding an environment check step, ensure that the database status and remote connection status of the node and the target address meet the requirements.

[0092] The control system can perform environmental checks through automated scripts or tools to detect potential problems in advance and avoid failures during the migration process.

[0093] In some embodiments, the configuration information can be customized according to different migration requirements, such as selecting different compression algorithms to optimize transmission speed or storage space. Through detailed configuration information, the system can automatically perform complex operations during the migration process, reduce human errors, and improve overall migration efficiency.

[0094] S303: Based on the configuration information, use a database data in the slave database node as a backup object for the data to be migrated.

[0095] Specifically, a database in the slave database node is used as a backup object so that data backup can be performed without affecting the normal operation of the master database and without interfering with the production environment.

[0096] Based on the configuration information, the control system identifies the slave database nodes, evaluates their status and load, and then selects the most suitable node for backup.

[0097] The specific selection method can be based on factors such as network bandwidth, node load, and data consistency to optimize backup efficiency and reliability.

[0098] In some embodiments, the system can dynamically configure and verify the selection policy of the backup object, such as selecting nodes with lower load for backup during peak hours.

[0099] By reasonably selecting backup objects, the system can improve the efficiency and security of backup operations while ensuring data consistency.

[0100] S304: Stop the synchronous update process of the backup object.

[0101] Specifically, by stopping the synchronous update process of the backup object, the consistency and integrity of the data during the backup process are ensured, and the data is prevented from being changed during the backup period, thereby causing inconsistency in the backup data.

[0102] When pausing the synchronization process, the current synchronization status can also be recorded so that synchronization can be accurately resumed after the backup is completed. The control system can implement this process through automated scripts to ensure the accuracy and timeliness of the operation.

[0103] In some embodiments, the system may set a timeout mechanism to prevent the synchronization process from being suspended for a long time and affecting the normal operation of the database.

[0104] Through this step, the system can obtain a stable database snapshot, providing a reliable data source for subsequent backup operations.

[0105] S305: Generate a backup file of the backup object, and determine the backup file as backup data of the data to be migrated.

[0106] Specifically, the control system extracts data from the backup object, performs necessary compression and encryption processing, and then stores it as a backup file.

[0107] In some embodiments, after the backup data is generated, a verification code of the backup file is also generated.

[0108] Specifically, a verification code is generated for the backup file so as to verify the integrity of the data in subsequent steps.

[0109] In some embodiments, different compression algorithms and encryption techniques may be selected in the configuration information to optimize the size and security of the backup file. For example, for sensitive data, a stronger encryption algorithm may be used to improve data security.

[0110] S306: Restart the synchronous update process of the backup object.

[0111] Specifically, after the backup process is completed, it is necessary to ensure that the slave database can continue to maintain data synchronization with the master database, so it is necessary to restart the synchronization update process.

[0112] S307: Based on the configuration information, the backup data is transferred to a migration target address corresponding to the data to be migrated.

[0113] Specifically, the control system will select an appropriate transmission path and method based on the target address and transmission protocol in the configuration information to safely and quickly transmit the backup data to the migration target address.

[0114] The control system can use different transmission protocols (such as FTP, SFTP, HTTP, etc.) and encryption technologies to ensure the security and integrity of data during transmission (these transmission methods and parameters can also be configured in the configuration information).

[0115] In some embodiments, the system can dynamically adjust the transmission strategy according to the network conditions, such as selecting a better transmission path or delaying transmission when the network is congested.

[0116] S308: Decompress the backup data to obtain a decompressed file corresponding to the backup data.

[0117] Specifically, after completing the transmission of the backup data, the control system will decompress the backup data transmitted to the migration target address to obtain a decompressed file corresponding to the backup data.

[0118] In some embodiments, before decompression, it is also necessary to ensure that the verification code of the backup file passes the verification.

[0119] Specifically, by verifying the checksum of the backup file to ensure that the data is not damaged during the transmission process, and then performing the decompression operation, it can be ensured that the compressed data can be restored to a usable database file.

[0120] In some embodiments, the control system may perform a data integrity check before decompression to ensure that the decompressed data is consistent with the original data.

[0121] S309: Stop all control services of the database corresponding to the migration target address.

[0122] Specifically, by stopping all control services of the database corresponding to the migration target address, it is ensured that no other operations are performed on the database during the data recovery process, thereby avoiding data conflicts and inconsistencies.

[0123] The control system can identify the control services of the target database and stop them to ensure that all related processes have terminated. The control system can also perform a status check to ensure that the database is in a state that can be safely stopped.

[0124] In some embodiments, the system can set a recovery plan so that the control service can be quickly restarted after the data recovery is completed. In addition, the system can track the status of the control service in real time through logging and monitoring tools to promptly discover and resolve potential problems. Through this step, the system can provide a stable environment for data recovery and ensure the smooth progress of the recovery process.

[0125] S310: Delete the data directory and log directory of the database corresponding to the migration target address.

[0126] Specifically, the control system needs to clean up the existing data in the target database to provide a clean environment for the recovery of new data.

[0127] In some embodiments, the control system can set an automatic cleaning mechanism to regularly clean up old data according to preset conditions. In addition, the system can track the status of the data directory and log directory in real time through logging and monitoring tools to promptly discover and resolve potential problems. Through this step, the system can ensure that the environment of the target database is clean and provide reliable protection for the recovery of new data.

[0128] S311, restoring the decompressed file to the database corresponding to the migration target address.

[0129] Specifically, the decompressed file is restored to the database corresponding to the migration target address to ensure that the backup data can be correctly written into the target database. The data directory to which the decompressed data file is copied can be pre-configured in the configuration information or automatically generated and adjusted by the control system to ensure that the restored data can be correctly identified and used.

[0130] S312: Restart all stopped control services.

[0131] Specifically, restart all previously stopped control services to ensure that the target database can operate normally after data recovery and continue to provide data services.

[0132] S313: Synchronously copy the data to be migrated and the backup data after data recovery, and determine the result after the synchronous copy as the migrated data.

[0133] Specifically, after the data is restored, the control system will identify the differences between the data to be migrated and the restored data, and perform necessary synchronization operations to ensure that the data of the two are consistent.

[0134] In some embodiments, after completing the data migration, it is also necessary to determine whether the node address of the data to be migrated and the migration target address meet the environmental check requirements, and delete the backup data in the node address and the migration target address.

[0135] Specifically, after the synchronization is completed, the control system may perform a data consistency check again to ensure that the synchronized data is consistent with the original data and that the database environment before and after the migration is normal.

[0136] At the same time, the control system can also track the status of data synchronization in real time through logging and monitoring tools, so as to promptly discover and resolve potential problems, thereby ensuring the integrity and consistency of the data to the greatest extent possible and providing reliable guarantees for the success of data migration.

[0137] In some embodiments, during the execution of the above steps, the following steps are also included:

[0138] Based on the actions completed in the current server, the process record information of the cluster corresponding to the current server during the data migration process is updated. The process record information is used to record the actions completed during the data migration process and the node address of the corresponding cluster.

[0139] Specifically, the breakpoint resume function is realized by configuring the process record information.

[0140] The specific implementation methods include:

[0141] Step A1: In response to the data migration continuation instruction issued by the current server, determine the action to be executed based on the completed actions in the process record information.

[0142] The data migration continuation instruction is used to instruct the current server to continue to complete the data migration process when the unfinished data migration process is in a suspended state.

[0143] Specifically, during the migration process, the control system continuously records the status and results of each step and stores this information in a traceable log or file (i.e., process record information).

[0144] When a continuation instruction is issued, the system directly reads the process record information to determine the completed steps and the current migration status, thereby determining the actions to be performed.

[0145] During the recording process, timestamps and operation identifiers can also be added so that the interruption point can be more accurately located during recovery. In addition, the control system can set different recovery strategies to select the optimal recovery path based on the cause of the interruption and environmental conditions. For example, in an interruption caused by a network failure, the system can prioritize checking the network status and reestablishing the connection. Through these measures, it can be ensured that after the data migration process is interrupted, the system can quickly and accurately identify the tasks to be performed, providing a reliable basis for subsequent recovery operations.

[0146] Step A2: Based on the action to be performed, continue to complete the data migration process.

[0147] Specifically, according to the type and priority of the action to be executed, the relevant migration operation is restarted, and its execution status is monitored in real time to ensure the success of the operation.

[0148] During the sequel process, automated scripts or tools can also be added to improve the efficiency and accuracy of operations.

[0149] In some embodiments, the system may be configured with a data migration control option to automatically adjust the execution strategy or notify the administrator to perform manual intervention when a certain step fails multiple times, and automatically record the information in the process log.

[0150] In some embodiments, the process record information includes the status of all database execution actions to specifically determine the nodes where different databases are located. The format is as follows:

[0151] Total:A

[0152] xx.xx.xx.xx:A

[0153] xx.xx.xx.xx:A ...

[0155] Among them, Total:A means that the current cluster has successfully executed step A, and xx.xx.xx.xx:A means that the ip has successfully executed step A. This design is because during the migration process, there is only one target library for the operation of the old cluster, and Total is used to record the successfully executed steps. However, there are generally multiple target libraries for the operation of the new cluster, and operations will be performed concurrently. IP:A is used to distinguish the successfully executed steps of each new library in the new cluster.

[0156] For operations on the old cluster, the logic for breakpoint resumption can be as follows: before executing the first step A, determine whether the value of Total in the file is empty. If it is empty, execute step A; otherwise, skip it. After step A is successfully executed, the information in the file is changed to Total: A. Before executing the second step B, determine whether the value of Total in the file is A. If it is, execute step B; otherwise, skip it. After step B is successfully executed, the information in the file is changed to Total: A.

[0157] The breakpoint continuation logic for the operation steps of the new cluster can be as follows: before executing concurrent operations, determine whether the previous step B has been successfully executed, and if so, execute step C. Before each new database operation step C, determine whether step C has been successfully run, and if so, skip it, and if not, execute step C. After successful execution, modify the file information to ip: C. After waiting for all new databases to execute step C, determine whether all new databases have been successfully executed. If so, continue with the following operations, otherwise an error will be reported.

[0158] The data migration method provided by the embodiment of the present disclosure ensures the orderliness and automation of data migration and reduces human intervention through group number allocation and configuration information generation. Next, the backup and synchronization strategy ensures the consistency and integrity of the data while reducing the impact on the production environment. During the data transmission and recovery process, the solution ensures the security and accuracy of the data and optimizes the migration efficiency. By restarting the service and synchronizing the data, the solution ensures the normal operation of the database and the consistency of the data. In addition, the breakpoint resumption mechanism provided enhances the robustness and flexibility of the migration process, so that the migration process can be quickly resumed after an interruption. The overall solution significantly improves the efficiency of data migration, reduces the error rate, and provides reliable technical support for migration in large-scale data environments.

[0159] Figure 4 A schematic diagram of the structure of the data migration device provided in this application, such as Figure 4 As shown, the data migration device 400 provided in this embodiment includes:

[0160] Configuration module 410, used to determine configuration information of the data to be migrated, the configuration information includes the node address of the data to be migrated and the migration target address, the node address includes the slave database node address;

[0161] A backup module 420, for generating backup data of the data to be migrated based on the slave database node address, and transmitting the backup data to the migration target address;

[0162] The recovery module 430 is used to recover the backup data in the migration target address;

[0163] The synchronization module 440 is used to synchronously copy the data to be migrated and the backup data after data recovery, and determine the result after the synchronous copy as the migrated data.

[0164] In one possible implementation, the configuration module 410 is specifically used to determine the corresponding group number of the data to be migrated for each cluster based on the cluster information of all the data to be migrated, wherein the cluster information includes the addresses of each node in the cluster and the corresponding migration target address, the node address also includes the master database node address, each master database corresponds to at least one slave database, the data in the slave database is synchronized with the data in the master database, and the migration target address includes the address of the master database node after migration and all the slave database nodes thereafter; based on the group number, the configuration information of each data to be migrated is generated.

[0165] In a possible implementation, the configuration module 410 specifically includes configuration information including: group number, cluster number, login information of the server where the cluster is located, login information of the database of each node in the cluster, compression method and decompression method of backup data, and storage address of backup data.

[0166] In one possible implementation, the backup module 420 is specifically used to, based on the configuration information, use a database data from a slave database node as a backup object for the data to be migrated; stop the synchronization update process of the backup object; generate a backup file for the backup object, and determine the backup file as the backup data for the data to be migrated; restart the synchronization update process for the backup object; and based on the configuration information, transfer the backup data to the migration target address corresponding to the data to be migrated.

[0167] In one possible implementation, the recovery module 430 is specifically used to decompress the backup data to obtain a decompressed file corresponding to the backup data; stop all control services of the database corresponding to the migration target address; delete the data directory and log directory of the database corresponding to the migration target address; restore the decompressed file to the database corresponding to the migration target address; and restart all stopped control services.

[0168] In one possible implementation, the backup module 420 is also used to generate a backup file of the backup object, and after determining the backup file as the backup data of the data to be migrated, generate a verification code for the backup file; the recovery module 430 is also used to decompress the backup data, and before obtaining the decompressed file corresponding to the backup data, determine that the verification code of the backup file passes the verification.

[0169] In one possible implementation, the backup module 420 is also used to generate backup data of the data to be migrated based on the node address of the slave database, and before transmitting the backup data to the migration target address, determine, based on the configuration information, whether the node address of the data to be migrated and the migration target address meet the environmental check requirements, wherein the environmental check requirements include determining whether the status of the database at the corresponding address meets the corresponding database status requirements, and determining whether the remote connection status between different addresses meets the corresponding connection status requirements; the synchronization module 440 is also used to synchronously replicate the data to be migrated and the backup data after data recovery, and after determining the result of the synchronous replication as the data after the migration is completed, determine that the node address of the data to be migrated and the migration target address meet the environmental check requirements; and delete the backup data in the node address and the migration target address.

[0170] In one possible implementation, the synchronization module 440 is also used to update the process record information of the cluster corresponding to the current server during the data migration process based on the actions completed in the current server. The process record information is used to record the actions completed during the data migration process and the node address of the corresponding cluster.

[0171] In one possible implementation, the synchronization module 440 is specifically used to determine the actions to be executed based on the completed actions in the process record information in response to the data migration continuation instruction issued in the current server, wherein the data migration continuation instruction is used to instruct the current server to continue to complete the data migration process when the incomplete data migration process is in a paused state; and continue to complete the data migration process based on the actions to be executed.

[0172] The data migration device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0173] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0174] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0175] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0176] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0177] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0178] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0179] The present application also provides a computer program product, including a computer program, which implements the method in any of the above embodiments when executed by a processor.

[0180] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method in any of the above embodiments is implemented.

[0181] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0182] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0183] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0184] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0186] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0187] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0188] It should be further noted that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0189] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0190] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0191] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0192] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0193] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0194] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0195] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data migration method, characterized in that: The steps include: Determine configuration information of the data to be migrated, the configuration information includes a node address of the data to be migrated and a migration target address, the node address includes a slave database node address; Based on the slave database node address, generating backup data of the data to be migrated, and transmitting the backup data to the migration target address; Performing data recovery on the backup data in the migration target address; The data to be migrated and the backup data after the data recovery are synchronously copied, and the result after the synchronous copying is determined as the migrated data.

2. The method according to claim 1, characterized in that The step of determining the configuration information of the data to be migrated includes: Based on the cluster information of all the data to be migrated, a corresponding group number is determined for the data to be migrated of each cluster, wherein the cluster information includes the addresses of each node in the cluster and the corresponding migration target address, the node address also includes the master database node address, each master database corresponds to at least one slave database, the data in the slave database is kept in synchronization with the data in the master database, and the migration target address includes the addresses of the master database node after migration and all the slave database nodes thereafter; Based on the group number, configuration information of each data to be migrated is generated.

3. The method according to claim 2, characterized in that The configuration information includes: Group number, cluster number, login information of the server where the cluster is located, login information of the database of each node in the cluster, compression and decompression methods of backup data, and storage address of backup data.

4. The method according to claim 2, characterized in that: The step of generating backup data of the data to be migrated based on the slave database node address, and transmitting the backup data to the migration target address includes: Based on the configuration information, one database data from the slave database node is used as a backup object of the data to be migrated; Stop the synchronous update process of the backup object; Generate a backup file of the backup object, and determine the backup file as backup data of the data to be migrated; Restarting the synchronous update process of the backup object; Based on the configuration information, the backup data is transmitted to a migration target address corresponding to the data to be migrated.

5. The method according to claim 4, characterized in that The restoring the backup data in the migration target address includes: Decompressing the backup data to obtain a decompressed file corresponding to the backup data; Stop all control services of the database corresponding to the migration target address; Delete the data directory and log directory of the database corresponding to the migration target address; Restoring the decompressed file to the database corresponding to the migration target address; Restart all stopped control services.

6. The method according to claim 5, characterized in that After generating the backup file of the backup object and determining the backup file as the backup data of the data to be migrated, the method further includes: Generating a verification code for the backup file; Before decompressing the backup data to obtain the decompressed file corresponding to the backup data, the method further includes: Determine whether the verification code of the backup file passes the verification.

7. The method according to claim 6, characterized in that Before generating backup data of the data to be migrated based on the slave database node address and transmitting the backup data to the migration target address, the method further includes: Based on the configuration information, determining that the node address of the data to be migrated and the migration target address meet the environment check requirements, wherein the environment check requirements include determining that the state of the database of the corresponding address meets the corresponding database state requirements, and determining that the remote connection state between different addresses meets the corresponding connection state requirements; After synchronously replicating the data to be migrated and the backup data after data recovery, and determining the result of the synchronous replication as the migrated data, the method further includes: Determine that the node address of the data to be migrated and the migration target address meet the environmental inspection requirements; The backup data in the node address and the migration target address are deleted.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Based on the actions completed in the current server, the process record information of the cluster corresponding to the current server during the data migration process is updated, where the process record information is used to record the actions completed during the data migration process and the node address of the corresponding cluster.

9. The method according to claim 8, characterized in that The updating of the process record information of the cluster corresponding to the current server in the data migration process based on the actions completed in the current server includes: In response to a data migration continuation instruction issued by the current server, determining an action to be performed based on the completed actions in the process record information, wherein the data migration continuation instruction is used to instruct the current server to continue to complete the data migration process when the unfinished data migration process is in a suspended state; Based on the actions to be performed, continue to complete the data migration process.

10. A data migration device, characterized in that: include: A configuration module, used to determine configuration information of the data to be migrated, wherein the configuration information includes a node address of the data to be migrated and a migration target address, wherein the node address includes a slave database node address; A backup module, used to generate backup data of the data to be migrated based on the slave database node address, and transmit the backup data to the migration target address; A recovery module, used for recovering the backup data in the migration target address; The synchronization module is used to synchronously copy the data to be migrated and the backup data after the data is restored, and determine the result after the synchronous copy as the migrated data.

11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when being executed by a processor.

Images