Data migration method and device, computer device and readable storage medium

Abstract

The application relates to a data migration method and device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring the data volume of a current to-be-migrated data table in a source database; in the case that the data volume is greater than a preset threshold, determining a parallel migration strategy based on a first mapping relationship between data in the current to-be-migrated data table and a plurality of data nodes and a second mapping relationship between data in a target data table in a target database and the plurality of data nodes; sending a parallel migration instruction to a master node in the source database according to the parallel migration strategy; the parallel migration instruction is used to instruct the master node to control the plurality of data nodes to migrate corresponding data to the target data table in the target database in parallel; and performing data migration on a next to-be-migrated data table in the plurality of to-be-migrated data tables as a new current to-be-migrated data table until all the to-be-migrated data tables in the source database are migrated to the target database. The method can be applied to data migration in data tables in an MPP database.

Description

Technical Field

[0001] This application relates to the field of data migration technology, and in particular to a data migration method, apparatus, computer equipment, and readable storage medium. Background Technology

[0002] With the rapid development of information technology, the amount of data is growing explosively, and data migration has become a common operation in scenarios such as digital transformation, system upgrades, and architecture adjustments.

[0003] In the field of data migration, the data migration tools in related technologies are designed for single-machine databases. These tools are based on single-node data processing logic and achieve data transfer through simple file transfer or database import and export functions.

[0004] However, the methods of the related technologies are not applicable to data migration in data tables of Massively Parallel Processing (MPP) databases. Summary of the Invention

[0005] Therefore, it is necessary to provide a data migration method, apparatus, computer equipment, and readable storage medium that can be applied to data migration in data tables of an MPP database, addressing the aforementioned technical problems.

[0006] Firstly, this application provides a data migration method, including:

[0007] Get the amount of data in the current table to be migrated in the source database; the source database includes multiple tables to be migrated; the data in each table to be migrated is distributed across multiple data nodes;

[0008] When the amount of data in the current data table to be migrated exceeds a preset threshold, the parallel migration strategy of the current data table to be migrated is determined based on the first mapping relationship between the data in the current data table to be migrated and multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated and multiple data nodes in the target database.

[0009] According to the parallel migration strategy, a parallel migration instruction is sent to the master node in the source database. The parallel migration instruction is used to instruct the master node to control multiple data nodes to migrate the corresponding data to the target data table in the target database in parallel.

[0010] The next table to be migrated from among multiple tables to be migrated is treated as the new current table to be migrated, and the data migration is performed until all tables to be migrated from the source database are migrated to the target database.

[0011] In one embodiment, based on a first mapping relationship between data in the current data table to be migrated and multiple data nodes, and a second mapping relationship between data in the target data table corresponding to the current data table to be migrated and multiple data nodes in the target database, a parallel migration strategy for the current data table to be migrated is determined, including:

[0012] Based on the first and second mapping relationships, obtain the number of source data nodes corresponding to the current data table to be migrated and the number of target data nodes corresponding to the target data table.

[0013] Based on the number of source data nodes, the number of target data nodes, the first mapping relationship, and the second mapping relationship, determine the parallel migration strategy for the current data table to be migrated.

[0014] In one embodiment, based on the number of source data nodes, the number of target data nodes, a first mapping relationship, and a second mapping relationship, a parallel migration strategy for the current data table to be migrated is determined, including:

[0015] Compare the number of source data nodes with the number of target data nodes, and compare the first mapping relationship with the second mapping relationship;

[0016] Based on the comparison results of the number of data nodes and the comparison results of the mapping relationship, the parallel migration strategy for the current data table to be migrated is determined.

[0017] In one embodiment, based on the comparison results of the number of data nodes and the comparison results of the mapping relationship, a parallel migration strategy for the current data table to be migrated is determined, including:

[0018] If the comparison results of the number of data nodes and the comparison results of the mapping relationship are the same, the parallel migration strategy for the current data table to be migrated is determined to be the migration strategy between data nodes of the source database and data nodes of the target database.

[0019] If the comparison results of the number of data nodes and / or the comparison results of the mapping relationship are different, the parallel migration strategy for the current data table to be migrated is determined to be the migration strategy between the data nodes of the source database and the master node of the target database.

[0020] In one embodiment, before sending the parallel migration command to the master node in the source database according to the parallel migration strategy, the method further includes:

[0021] When the parallel migration strategy is a migration strategy between data nodes, a first service start instruction is sent to the master node in the target database. The first service start instruction is used to instruct the master node to start a data assistance service on each data node. Each data assistance service has a different port number. The port number is used to receive data sent by the corresponding data node in the source database.

[0022] In one embodiment, before sending the parallel migration command to the master node in the source database according to the parallel migration strategy, the method further includes:

[0023] In the case of a parallel migration strategy between data nodes and master nodes, a second service start command is sent to the master node in the target database. The second service start command is used to instruct the master node to start multiple data assistance services on the master node. Each data assistance service has a different port number. The port number is used to receive data sent by the corresponding data node in the source database.

[0024] In one embodiment, the method further includes:

[0025] If the amount of data in the current table to be migrated is less than or equal to a preset threshold, the migration strategy for the current data to be migrated is determined as the migration strategy between master nodes; the migration strategy between master nodes represents the strategy from the master node of the source database to the master node of the target database.

[0026] Send a third service start command to the master node in the target database; the third service start command is used to instruct the master node to start a data assistance service on the master node; the fixed port of the data assistance service is used to receive data sent by the master node in the source database;

[0027] Send a data migration command to the master node of the source data; the data migration command is used to instruct the master node to migrate the data in the current data table to be migrated to the fixed port of the data assistance service.

[0028] Secondly, this application also provides a data migration apparatus, comprising:

[0029] The acquisition module is used to acquire the amount of data in the current data table to be migrated in the source database; the source database includes multiple data tables to be migrated; the data in each data table to be migrated is distributed across multiple data nodes;

[0030] The strategy determination module is used to determine the parallel migration strategy of the current data table to be migrated when the amount of data in the current data table to be migrated is greater than a preset threshold, based on the first mapping relationship between the data in the current data table to be migrated and multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated and multiple data nodes in the target database.

[0031] The instruction sending module is used to send parallel migration instructions to the master node in the source database according to the parallel migration strategy; the parallel migration instructions are used to instruct the master node to control multiple data nodes to migrate the corresponding data to the target data table in the target database in parallel.

[0032] The loop execution module is used to perform data migration by taking the next data table to be migrated from multiple data tables to be migrated as the new current data table to be migrated, until all data tables to be migrated in the source database have been migrated to the target database.

[0033] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the content of any embodiment of the data migration method in the first aspect described above.

[0034] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the content of any embodiment of the data migration method in the first aspect described above.

[0035] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the content of any embodiment of the data migration method in the first aspect described above.

[0036] The aforementioned data migration method, apparatus, computer equipment, and readable storage medium acquire the data volume of the current data table to be migrated in the source database; the source database includes multiple data tables to be migrated; the data in each data table to be migrated is distributed across multiple data nodes; when the data volume of the current data table to be migrated exceeds a preset threshold, a parallel migration strategy for the current data table to be migrated is determined based on a first mapping relationship between the data in the current data table to be migrated and the multiple data nodes, and a second mapping relationship between the data in the target data table corresponding to the current data table to be migrated and the multiple data nodes in the target database; according to the parallel migration strategy, a parallel migration instruction is sent to the master node in the source database; the parallel migration instruction is used to instruct the master node to control multiple data nodes to migrate the corresponding data to the target data table in the target database in parallel; the next data table to be migrated among the multiple data tables to be migrated is used as the new current data table to be migrated to perform data migration, until all data tables to be migrated in the source database are migrated to the target database. This method obtains the data volume of the current table to be migrated in the source database. When the data volume exceeds a preset threshold, it determines a parallel migration strategy based on the first mapping relationship between data and data nodes in the source database and the second mapping relationship between corresponding data and data nodes in the target database. This allows for more rational resource allocation during data migration and enables the acquisition of a parallel migration strategy more suitable for the current table to be migrated. Subsequently, instructions are sent to the master node of the source database according to the parallel migration strategy. The master node then controls multiple data nodes to migrate data to the target database in parallel, fully leveraging the parallel processing capabilities of the MPP distributed database. This enables efficient migration of large-volume data tables, significantly shortening migration time and improving migration efficiency. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a diagram illustrating the application environment of a data migration method in one embodiment.

[0039] Figure 2 This is a flowchart illustrating a data migration method in one embodiment;

[0040] Figure 3 This is a flowchart illustrating a data migration method in one embodiment;

[0041] Figure 4This is a flowchart illustrating a data migration method in one embodiment;

[0042] Figure 5 This is a schematic diagram illustrating the data flow between data nodes in one embodiment;

[0043] Figure 6 This is a schematic diagram illustrating the data flow between a data node and a master node in one embodiment;

[0044] Figure 7 This is a flowchart illustrating a data migration method in one embodiment;

[0045] Figure 8 This is a schematic diagram illustrating the data flow between master nodes in one embodiment;

[0046] Figure 9 This is a flowchart illustrating a data migration method in one embodiment;

[0047] Figure 10 This is a flowchart illustrating a data migration method in one embodiment;

[0048] Figure 11 This is a structural block diagram of a data migration device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] Before providing a detailed description of the technical solution of this application, the background technology of this application will be described in detail.

[0051] Data migration refers to the process of transferring data from one storage system or database to another, with the aim of achieving seamless access and use of the data in the new environment. Data migration is typically performed during system upgrades, database replacements, storage device replacements, or cloud migrations to improve system performance, scalability, and reliability. The key is ensuring data integrity, accuracy, and consistency, and preventing data loss or corruption.

[0052] Database migration typically involves two parts: metadata migration and table data migration. Metadata migration refers to the process of migrating the database structure, definitions, and other related database objects from the source database to the target database. This includes table structures, indexes, views, stored procedures, and triggers. Effective metadata migration ensures that the target database can correctly identify and use the migrated data structures and logic. Table data migration refers to the process of migrating the actual data stored in the database from the source database to the target database. Due to the large amount of data involved, this process requires careful planning and execution to ensure data integrity and accuracy. This typically includes three parts: data export, data transfer, and data import. Data export refers to exporting database table data to data files using export tools. Data transfer refers to transferring the exported data files from the source database environment to the target database environment. This can be accomplished through network transmission, cloud storage, or physical media (such as external hard drives). Data import refers to importing the data files into the target database using the target database's import tools.

[0053] In the field of data migration, the data migration tools in related technologies are designed for single-machine databases. These tools are based on single-node data processing logic and achieve data transfer through simple file transfer or database import and export functions.

[0054] MPP databases typically employ a shared-nothing architecture (SNA), where each node independently owns memory and storage resources. This reduces resource contention and lock contention between nodes, thereby improving system scalability and fault tolerance. A typical MPP database consists of a master node and multiple data nodes (DNs). The master node is primarily responsible for coordinating the execution of queries and operations, while the data nodes are responsible for storing and processing data. In an MPP database, the master node, under an existing data partitioning strategy, distributes data evenly across the data nodes based on certain data attributes (such as primary keys, hashes, etc.), ensuring load balancing. This process is called data redistribution.

[0055] In other words, MPP databases utilize multiple independent computing nodes to process data and queries in parallel, enabling MPP databases to divide data and computing tasks into multiple subtasks for parallel execution, thereby significantly reducing processing time. During the migration of data tables, the size of the data in the data table greatly affects the data migration speed. Due to the characteristics of multiple data nodes in MPP databases, parallel migration requires more system resources, which in turn consumes more time. Therefore, under this distributed architecture, the data migration solutions in related technologies are obviously insufficient: (1) MPP databases typically store massive amounts of data at the trillion-byte (TB) or petabyte (PB) level. The methods in related technologies cannot fully utilize the advantages of multi-node storage in the MPP architecture, thus making it difficult to meet the needs of rapid migration of large amounts of data. (2) Data migration methods in related technologies usually require exporting data to local disk files. For large amounts of data at the TB or PB level, this method requires high disk capacity. (3) Table data in MPP databases are usually stored on different data nodes according to different distribution strategies. Common distribution strategies include hash distribution, random distribution, shard distribution, range distribution, and replication distribution. Therefore, when the mapping relationships between data in the source database tables and multiple data nodes differ, and the second mapping relationship between data in the target database table corresponding to the currently migrated data table and multiple data nodes in the target database also differs, data must be rebalanced according to these two mapping relationships during migration to the target database to avoid data loss or inconsistency. This presents a significant challenge for MPP database data migration. It should be noted that for MPP databases storing TB or PB level data, the time required for metadata migration is negligible compared to table data migration during the entire data migration process.

[0056] To address the aforementioned issues, this application provides a data migration method, apparatus, computer equipment, and readable storage medium. It fully considers the parallel computing and distributed storage characteristics of the MPP architecture, utilizing the multi-node architecture of the MPP database to enable each data storage node to directly participate in data transmission and execute migration tasks in parallel. This significantly improves the speed and efficiency of data migration, making it suitable for handling large data volume migrations at the TB or PB level. A streaming data transmission mechanism is used during the data migration process, migrating data directly over the network, avoiding the complete export of data to the local disk, thereby reducing disk space usage. Furthermore, the migration strategy can be dynamically adjusted based on the size of the data table to be migrated and the mapping relationship between the data in the table and the data center, making it applicable to data migration under different circumstances. The technical solution of this application will be described in detail below.

[0057] The data migration method provided in this application embodiment can be applied to, for example, Figure 1 The application environment shown.

[0058] For example, the computer device can be a server, personal computer, laptop computer, smartphone, tablet computer, mobile phone, etc. The computer device may include a processor, memory, and network interface connected via a system bus or wirelessly. The processor provides computing and control capabilities. The memory may include non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data during data migration. The network interface communicates with external terminals via a network connection, and the computer program, when executed by the processor, implements a data migration method. The computer device can be implemented using a standalone computer device or a cluster of multiple computer devices. It should be noted that the memory of the computer device is not limited to the above-mentioned memory and may also include high-speed random access memory, volatile solid-state memory, etc. Furthermore, the architecture of the computer device is not limited to the above-described cases; some components may be added or omitted.

[0059] In one exemplary embodiment, such as Figure 2 As shown, a data migration method is provided, which is applied to... Figure 1 Taking a computer device as an example, the explanation includes the following steps S101 to S104. Wherein:

[0060] S101, obtain the amount of data in the current data table to be migrated in the source database; the source database includes multiple data tables to be migrated; the data in each data table to be migrated is distributed across multiple data nodes.

[0061] In this context, the source database specifically refers to an MPP database with multiple independent data nodes. An MPP database typically includes one or more master nodes, multiple data nodes, and other auxiliary nodes. Taking a master node as an example, the master node is primarily responsible for receiving client query requests, parsing SQL, generating execution plans, coordinating the execution of queries across various data nodes, summarizing results, and returning them to the client. Data nodes are the nodes that actually store data and execute query operations. In an MPP architecture, data is sharded and distributed across multiple data nodes, with each data node responsible for processing its locally stored data portion. The number of data nodes is the core scalability of an MPP system; to achieve large-scale parallel processing, multiple data nodes are typically used.

[0062] The source database U includes a master node Mu, and the set of multiple data nodes of U can be represented as: ,in, Let represent the i-th data node of U, and M represent the number of data nodes in U. U can include K data tables, and the set of data tables can be represented as... ,in, This represents the i-th data table in U.

[0063] In this embodiment, the data table to be migrated can be any table in the source database. The computer device can obtain the amount of data in the data table to be migrated by executing a specific query statement. Alternatively, the computer device can use database system functions to query information such as the number of rows or storage space of the data table to be migrated, and use the number of rows or storage space as the amount of data in the data table to be migrated.

[0064] S102, when the amount of data in the current data table to be migrated is greater than a preset threshold, the parallel migration strategy of the current data table to be migrated is determined based on the first mapping relationship between the data in the current data table to be migrated and multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated and multiple data nodes in the target database.

[0065] The target database V includes a master node M. v The set of multiple data nodes of V can be represented as: ,in, Let N represent the i-th data node of V, and N represent the number of data nodes in V. V can include K data tables, and the set of data tables can be represented as follows: ,in, This represents the i-th data table in V.

[0066] The data migration process can be represented as: transferring a set of data tables T from the source database... s Migrate table data in the target database to a set of tables T d In the data table, there is a one-to-one correspondence between the two. The data was migrated to In, it is denoted as < >

[0067] In this embodiment, after obtaining the data volume of the current data table to be migrated, the computer device can compare the data volume of the current data table to be migrated with a preset threshold. If the comparison result shows that the data volume of the current data table to be migrated is less than or equal to the preset threshold, it indicates that the data volume of the current data table to be migrated is small, and migration through parallel migration is not required. If the comparison result shows that the data volume of the current data table to be migrated is greater than the preset threshold, it indicates that the data volume of the current data table to be migrated is large, and migration through parallel migration is required. Taking the preset threshold as T and the data volume as the number of rows in the data table R as an example, if R > T, it is determined that the data volume is large; if R <= T, it is determined that the data volume is small.

[0068] When the amount of data in the current data table to be migrated exceeds a preset threshold, the computer device can retrieve a first mapping relationship between the data in the current data table to be migrated and multiple data nodes from the source database, and a second mapping relationship between the data in the target data table and multiple data nodes from the target database. Then, the first and second mapping relationships are analyzed according to the parallel migration strategy rules to find a parallel migration strategy suitable for the current data table to be migrated. Alternatively, the computer device can input the first and second mapping relationships into a preset migration strategy determination model, and use the model for analysis to obtain a parallel migration strategy suitable for the current data table to be migrated.

[0069] S103, according to the parallel migration strategy, send a parallel migration instruction to the master node in the source database; the parallel migration instruction is used to instruct the master node to control multiple data nodes to migrate the corresponding data to the target data table in the target database in parallel.

[0070] In this embodiment, after obtaining a parallel migration strategy applicable to the current data table to be migrated, the computer device can generate a parallel migration instruction corresponding to the parallel migration strategy and send the parallel migration instruction to the master node in the source database. After receiving the parallel migration instruction, the master node controls multiple data nodes in the source database to simultaneously migrate the corresponding data to the target data table in the target database according to the migration information carried in the parallel migration instruction. In this way, the data migration process of the current data table to be migrated is completed.

[0071] S104, take the next data table to be migrated from among multiple data tables to be migrated as the new current data table to be migrated and perform data migration until all data tables to be migrated in the source database are migrated to the target database.

[0072] In this embodiment, steps S101-S103 represent the migration process of a data table in the source database. After the migration of this data table is completed, the migration of the next data table to be migrated continues. The next data table to be migrated is used as the new current data table to be migrated, and steps S101-S103 are executed to complete the data migration of the next data table to be migrated. The data migration process of the above data tables is iteratively executed until all data tables to be migrated in the source database are migrated to the target database. That is, the number of parallel migrations is less than or equal to the number of data tables in the source database. If the data volume of each data table is greater than a preset threshold, the number of parallel migrations is equal to the number of data tables in the source database; if the data volume of at least one data table is less than or equal to the preset threshold, the number of parallel migrations is less than the number of data tables in the source database.

[0073] In the above data migration method, the data volume of the current data table to be migrated in the source database is obtained; the source database includes multiple data tables to be migrated; the data in each data table to be migrated is distributed across multiple data nodes; when the data volume of the current data table to be migrated exceeds a preset threshold, a parallel migration strategy for the current data table to be migrated is determined based on a first mapping relationship between the data in the current data table to be migrated and the multiple data nodes, and a second mapping relationship between the data in the target data table corresponding to the current data table to be migrated and the multiple data nodes in the target database; according to the parallel migration strategy, a parallel migration instruction is sent to the master node in the source database; the parallel migration instruction is used to instruct the master node to control multiple data nodes to migrate the corresponding data to the target data table in the target database in parallel; the next data table to be migrated among the multiple data tables to be migrated is used as the new current data table to be migrated to perform data migration, until all data tables to be migrated in the source database are migrated to the target database. This method obtains the data volume of the current table to be migrated in the source database. When the data volume exceeds a preset threshold, it determines a parallel migration strategy based on the first mapping relationship between data and data nodes in the source database and the second mapping relationship between corresponding data and data nodes in the target database. This allows for more rational resource allocation during data migration and enables the acquisition of a parallel migration strategy more suitable for the current table to be migrated. Subsequently, instructions are sent to the master node of the source database according to the parallel migration strategy. The master node then controls multiple data nodes to migrate data to the target database in parallel, fully leveraging the parallel processing capabilities of the MPP distributed database. This enables efficient migration of large-volume data tables, significantly shortening migration time and improving migration efficiency.

[0074] Obtaining a parallel migration strategy is a prerequisite for performing parallel data migration. Therefore, in one embodiment, such as... Figure 3As shown, the specific details of the parallel migration strategy for determining the current data table to be migrated, based on the first mapping relationship between the data in the current data table to be migrated and multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated and multiple data nodes in the target database, are described below. These details include:

[0075] S201, based on the first mapping relationship and the second mapping relationship, obtain the number of source data nodes corresponding to the current data table to be migrated and the number of target data nodes corresponding to the target data table.

[0076] In this embodiment, the computer device can determine the number of data nodes associated with the data in the current data table to be migrated based on a first mapping relationship between the data in the source database and multiple data nodes, and use this number of associated data nodes as the source data node count. Simultaneously, it can also determine the number of data nodes associated with the data in the target database based on a second mapping relationship between the data in the target database corresponding to the current data table to be migrated and multiple data nodes in the target database, and use this number of associated data nodes as the target data node count.

[0077] S202, based on the number of source data nodes, the number of target data nodes, the first mapping relationship, and the second mapping relationship, determine the parallel migration strategy for the current data table to be migrated.

[0078] In this embodiment, the computer device can compare the number of source data nodes with the number of target data nodes to obtain a data node comparison result. It can also compare a first mapping relationship and a second mapping relationship to obtain a mapping relationship comparison result. Based on the data node comparison result and the mapping relationship comparison result, a parallel migration strategy suitable for the current data table to be migrated is selected from the parallel migration strategies. Alternatively, the computer device can input the number of source data nodes, the number of target data nodes, the first mapping relationship, and the second mapping relationship into a preset strategy determination model, and use the strategy determination model to analyze the two data node numbers and the two mapping relationships to determine the parallel migration strategy for the current data table to be migrated.

[0079] In the aforementioned data migration method, the number of source data nodes corresponding to the current data table to be migrated and the number of target data nodes corresponding to the target data table are obtained. Based on the number of source data nodes, the number of target data nodes, and the first and second mapping relationships, a parallel migration strategy for the current data table to be migrated is determined. This method, by obtaining the number of source and target data nodes and combining the first and second mapping relationships, can accurately obtain a parallel migration strategy suitable for the current data table to be migrated through dual analysis of node count and mapping relationships. This ensures efficient utilization of network bandwidth and computing resources, enabling data migration with optimal migration efficiency.

[0080] In one embodiment, such as Figure 4 As shown, the specific content of the parallel migration strategy for determining the current data table to be migrated based on the number of source data nodes, the number of target data nodes, the first mapping relationship, and the second mapping relationship includes:

[0081] S301, compare the number of source data nodes with the number of target data nodes, and compare the first mapping relationship with the second mapping relationship.

[0082] In this embodiment, the computer device can compare the number of source data nodes with the number of target data nodes to determine whether the number of the two data nodes is consistent. Simultaneously, it can also compare the first mapping relationship with the second mapping relationship to determine whether the distribution of the two data tables across the data nodes is consistent.

[0083] S302, Based on the comparison results of the number of data nodes and the comparison results of the mapping relationship, determine the parallel migration strategy for the current data table to be migrated.

[0084] In this embodiment, if it is determined that the number of two data nodes is the same and the distribution of two data tables on the data nodes is the same, parallel migration can be performed directly at the data node level. That is, the parallel migration strategy for the current data table to be migrated is determined to be a parallel migration strategy between data nodes. If the number of two data nodes is different, or the distribution of two data tables on the data nodes is different, parallel migration needs to be performed at the level of data nodes and master node. That is, the parallel migration strategy for the current data table to be migrated is determined to be a parallel migration strategy between data nodes and master node.

[0085] The following example illustrates the specific details of the parallel migration strategy for determining the current data table to be migrated based on the comparison results of the number of data nodes and the mapping relationship. Figure 5 As shown, the specific content includes:

[0086] If the comparison results of the number of data nodes and the comparison results of the mapping relationship are the same, the parallel migration strategy for the current data table to be migrated is determined to be the migration strategy between data nodes of the source database and data nodes of the target database.

[0087] If the comparison results of the number of data nodes and / or the comparison results of the mapping relationship are different, the parallel migration strategy for the current data table to be migrated is determined to be the migration strategy between the data nodes of the source database and the master node of the target database.

[0088] For example, consider the current data table to be migrated. The corresponding number of source data nodes is M, and the target data table is... The corresponding target data node number is N, and the first mapping relationship is D. s The second mapping relationship is D. d For example, if M=N, and D s =D d The parallel migration strategy is the migration strategy between data nodes (DN_To_DN); if M≠N, or, D s ≠D d The parallel migration strategy is the migration strategy between data nodes and master nodes (DN_To_M).

[0089] In the aforementioned data migration method, the number of source data nodes is compared with the number of target data nodes, and the first mapping relationship is compared with the second mapping relationship. Based on the comparison results of the number of data nodes and the mapping relationships, the parallel migration strategy for the current data table to be migrated is determined. By comparing the two data node counts and the two mapping relationships, this method can identify the data distribution differences between the current data table to be migrated in the source database and the target data table in the target database in advance, thereby accurately selecting a parallel migration strategy suitable for the current data table to be migrated.

[0090] The above examples illustrate the process of determining a parallel migration strategy. The following section describes the preparatory work required before performing the data migration operation. The preparatory work differs depending on the migration strategy.

[0091] First, taking the parallel migration strategy as an example of the migration strategy between data nodes, a detailed description of the preparatory work before the data migration operation is provided through an embodiment. Before sending the parallel migration instruction to the master node in the source database according to the parallel migration strategy, the method also includes:

[0092] In the case of a parallel migration strategy that is a migration strategy between data nodes, a first service start command is sent to the master node in the target database. The first service start command is used to instruct the master node to start a data assistance service on each data node. Each data assistance service has a different port number. The port number is used to receive data sent by the corresponding data node in the source database.

[0093] One of the services, data_helper, is a Transmission Control Protocol (TCP) service deployed on the target database. Its primary responsibility is to listen for network requests and receive and process data from the source database. The data_helper service listens on a specified port, waiting for connection requests from the source database. Once a connection is established, the data_helper service prepares to receive and process the transmitted data, and securely stores it in a designated location within the target database. As a core component for data reception, the data_helper service incorporates a streaming data transmission mechanism, enabling efficient data migration directly over the network.

[0094] In this embodiment, when the parallel migration strategy is a migration strategy between data nodes, the computer device can generate a first service startup instruction corresponding to the migration strategy and send the first service startup instruction to the master node in the target database. Upon receiving the first service startup instruction, the master node starts a data_helper service on each data node, with each data_helper service having a different port number. This utilizes the parallel processing capabilities and communication bandwidth of all data nodes to establish a one-to-one correspondence between data nodes in the source database and data nodes in the target database, directly completing the data migration.

[0095] Figure 5 This diagram illustrates the data flow between data nodes. As shown, during the migration process, data in each data node of the source database is migrated to the corresponding data node in the target database via the corresponding data_helper service. Furthermore, the data in multiple data nodes of the source database is migrated in parallel. Each data helper service corresponds to a port number, such as 1024, 1025, and 1026.

[0096] In the aforementioned data migration method, when the parallel migration strategy is a migration strategy between data nodes, a first service start command is sent to the master node in the target database. This first service start command instructs the master node to start a data assistance service on each data node. Each data assistance service has a different port number, used to receive data sent by the corresponding data node in the source database. Under the data node migration strategy, this method establishes a data assistance service on each data node through the master node. This data assistance service allows data to be transmitted between data nodes via the network without the need for disks, saving data transmission costs and improving resource utilization.

[0097] First, taking the parallel migration strategy as an example of the migration strategy between data nodes and master nodes, a detailed description of the preparatory work before the data migration operation is provided through an embodiment. Before sending the parallel migration instruction to the master node in the source database according to the parallel migration strategy, the method also includes:

[0098] In the case of a parallel migration strategy between data nodes and master nodes, a second service start command is sent to the master node in the target database. The second service start command is used to instruct the master node to start multiple data assistance services on the master node. Each data assistance service has a different port number. The port number is used to receive data sent by the corresponding data node in the source database.

[0099] In this embodiment, when the parallel migration strategy is a migration strategy between data nodes and master nodes, the computer device can generate a second service startup instruction corresponding to the migration strategy based on the migration strategy between data nodes and master nodes, and send the second service startup instruction to the master node in the target database. After receiving the second service startup instruction, the master node starts multiple data_helper services on the master node, and the port number of each data_helper service is different.

[0100] It should be noted that, compared to the migration strategy between data nodes, the migration strategy between data nodes and master nodes is also a parallel migration, but its performance will be affected by the processing capacity of the master node in the target database.

[0101] Figure 6This diagram illustrates the data flow between data nodes and the master node. As shown, during the migration process, data from each data node in the source database is migrated in parallel to the master node in the target database via the corresponding data_helper service. The master node in the target database then stores the data received from each data node. The diagram includes multiple data assistance services, each corresponding to a port number, such as 1024, 1025, and 1026.

[0102] In the aforementioned data migration method, when the parallel migration strategy is a migration strategy between data nodes and the master node, a second service start command is sent to the master node in the target database. This second service start command instructs the master node to start multiple data assistance services. Each data assistance service has a different port number, used to receive data sent by the corresponding data node in the source database. Under the data node-master node migration strategy, this method establishes multiple data assistance services on the master node. Through these services, data can be transmitted between the data nodes and the master node via the network, eliminating the need for disks, saving data transmission costs, and improving resource utilization.

[0103] The above examples illustrate the data migration process when the amount of data in the table to be migrated is large. The following example illustrates the data migration process when the amount of data is small. Figure 7 As shown, the data migration process includes:

[0104] S401, if the amount of data in the current data table to be migrated is less than or equal to a preset threshold, determine the migration strategy for the current data to be migrated as the migration strategy between master nodes; the migration strategy between master nodes represents the strategy from the master node of the source database to the master node of the target database.

[0105] In this embodiment, since the parallel migration strategy requires creating a data_helper service process on each data node, or establishing multiple data_helper service processes on the master node, the startup cost of the data_helper service is high. Therefore, when it is determined that the amount of data in the current data table to be migrated is less than or equal to a preset threshold, i.e., R <= T, it indicates that the amount of data in the data table is small, and parallel migration is not necessary. Based on this, the computer device can determine the migration strategy for the current data to be migrated as a migration strategy between master nodes, that is, directly transferring the data from the master node in the source database to the master node in the target database.

[0106] S402, send a third service start command to the master node in the target database; the third service start command is used to instruct the master node to start a data assistance service on the master node; the fixed port of the data assistance service is used to receive data sent by the master node in the source database.

[0107] In this embodiment, when the migration strategy is determined to be a migration strategy between master nodes, the computer device can generate a third service startup instruction corresponding to the migration strategy. After receiving the third service startup instruction, the master node in the target database starts a data_helper service on the master node, which has a fixed port.

[0108] S403, send a data migration instruction to the master node of the source data; the data migration instruction is used to instruct the master node to migrate the data in the current data table to be migrated to the fixed port of the data assistance service.

[0109] In this embodiment, the computer device can generate a data migration instruction corresponding to the migration strategy between master nodes. After receiving the data migration instruction, the master node migrates the data in the data table to be migrated to a fixed port of the data assistance service, thereby realizing the migration of data between the master nodes of the source database and the master nodes of the target database.

[0110] Figure 8 This diagram illustrates the data flow between master nodes. As shown, during the migration process, all data in the table to be migrated is transferred from the source database's master node to the target database's master node via a `data_helper` service. The port number for this `data_helper` service is 1024.

[0111] In the above data migration method, when the amount of data in the current data table to be migrated is less than or equal to a preset threshold, the migration strategy for the current data to be migrated is determined as a migration strategy between master nodes. The migration strategy between master nodes represents the strategy from the master node of the source database to the master node of the target database. A third service start command is sent to the master node in the target database. The third service start command instructs the master node to start a data assistance service on the master node. The port number of the data assistance service is used to receive data sent by the master node in the source database. A data migration command is sent to the master node of the source database. The data migration command instructs the master node to migrate the data in the current data table to be migrated to the fixed port of the data assistance service. This method triggers a direct migration strategy between master nodes when the amount of data in the data table to be migrated is less than or equal to the preset threshold. In scenarios with small data volumes, it completes the migration with a simpler link, reduces data transmission loss between multiple nodes, and improves the efficiency of data migration.

[0112] In a detailed embodiment, such as Figure 9 As shown, the data migration method includes:

[0113] S501, Get the amount of data in the current table to be migrated in the source database;

[0114] S502, if the amount of data in the current data table to be migrated is greater than a preset threshold, obtain the number of source data nodes corresponding to the current data table to be migrated and the number of target data nodes corresponding to the target data table according to the first mapping relationship and the second mapping relationship.

[0115] S503, compare the number of source data nodes with the number of target data nodes, and compare the first mapping relationship with the second mapping relationship;

[0116] S504, if the comparison results of the number of data nodes and the comparison results of the mapping relationship are the same, determine the parallel migration strategy of the current data table to be migrated as the migration strategy between data nodes of the source database and data nodes of the target database.

[0117] S505, send a first service start command to the master node in the target database; the first service start command is used to instruct the master node to start a data assistance service on each data node; each data assistance service has a different port number; the port number is used to receive data sent by the corresponding data node in the source database;

[0118] S506, if the comparison results of the number of data nodes and / or the comparison results of the mapping relationship are different, determine the parallel migration strategy of the current data table to be migrated as the migration strategy between the data nodes of the source database and the master node of the target database.

[0119] S507, send a second service start command to the master node in the target database; the second service start command is used to instruct the master node to start multiple data assistance services on the master node; each data assistance service has a different port number; the port number is used to receive data sent by the corresponding data node in the source database;

[0120] S508 sends parallel migration instructions to the master node in the source database according to the parallel migration strategy;

[0121] S509, if the amount of data in the current data table to be migrated is less than or equal to a preset threshold, determine the migration strategy for the current data to be migrated as the migration strategy between master nodes.

[0122] S510, send a third service start command to the master node in the target database; the third service start command is used to instruct the master node to start a data assistance service on the master node; the fixed port of the data assistance service is used to receive data sent by the master node in the source database;

[0123] S511, send a data migration instruction to the master node of the source data; the data migration instruction is used to instruct the master node to migrate the data in the current data table to be migrated to the fixed port of the data assistance service;

[0124] S512, determine whether the data tables to be migrated in the source database have been migrated. If yes, end the data migration; otherwise, execute step S501 until all data tables to be migrated in the source database have been migrated to the target database.

[0125] Figure 10 This is a flowchart illustrating a data migration method, which includes: S11: Determine if the data table in the source database has been migrated. If not, proceed to step S12; if yes, end. S12: Query the number of rows in the current data table to be migrated. S13: Determine if the number of rows is less than a preset threshold. If yes, proceed to step S14; if no, proceed to step S15. S14: After executing the migration strategy between master nodes, proceed to step S19. S15: Obtain the first mapping relationship between the data of the current data table to be migrated in the source database and multiple data nodes, obtain the second mapping relationship between the data of the target data table in the target database and multiple data nodes, and obtain the number of data nodes in the two databases. S16: Determine if the two mapping relationships are consistent and if the number of data nodes is consistent. If yes, proceed to step S17; if no, proceed to step S18. S17: Execute the parallel migration strategy between data nodes and proceed to step S19. S18: Execute the parallel migration strategy between data nodes and master nodes. S19: After taking the next data table to be migrated in the source database as the new current data table to be migrated, proceed to step S11.

[0126] Taking TeleDB, a distributed database derived from the open-source PostgreSQL-XL, as an example, we assume that the source TeleDB database includes one master node and two data nodes, and the target TeleDB database also includes one master node and two data nodes. The data volume is represented by the number of rows in the data table, and the threshold corresponding to the data volume is set to 300,000 for T.

[0127] For any table in the source TeleDB database, perform the following operations: Execute an SQL statement in the source database to retrieve the number of rows in the table. Execute an SQL statement in the source TeleDB database to query the mapping relationships, obtaining the first mapping relationship between the data in the table and multiple data nodes. Execute an SQL statement in the target TeleDB database to query the mapping relationships, obtaining the second mapping relationship between the data in the target table and multiple data nodes.

[0128] If the number of rows in the data table R <= 300,000, execute the migration strategy (M_To_M) between master nodes; if the number of rows in the data table R > 300,000, and M = N, D s With D d Consistent, execute the parallel migration strategy between data nodes (DN_To_DN); if the number of rows in the data table R > 300,000, but M ≠ N, or, D s With D d If inconsistent, execute the parallel migration strategy (DN_To_M) between the data node and the master node.

[0129] Before data migration, for the M_To_M strategy, the SQL statement (cmd-id) corresponding to the service startup command is executed on the master node of the target TeleDB database. The service startup command starts a data_helper service on the master node, listens on a port, and writes the cmd-id and the listening port to a file on disk. The master node of the source TeleDB database reads the file using this cmd-id, determines the port that the data_helper of the target TeleDB database is listening on, and executes the SQL statement corresponding to the data migration.

[0130] For the DN_To_DN strategy, the master node of the source TeleDB database is connected to the master node of the target TeleDB database. On each data node of the target TeleDB database, the SQL statement (cmd-id) corresponding to the service startup command is executed. The service startup command starts a data_helper service on each data node, and each data_helper listens on a port. This controls each data node of the source TeleDB database to perform data migration, completing the one-to-one data transfer between the data nodes of the two TeleDB databases.

[0131] For the DN_To_M strategy, the master node of the source TeleDB database is connected to the master node of the target TeleDB database. On the master node of the target TeleDB database, the SQL statement corresponding to the service startup command (cmd-id) is executed. The service startup command starts two data_helper services on the master node, each listening on a port, and writes the cmd-id, datanode-id, and listening port to a file and disk. This controls each data node of the source TeleDB database to perform data migration, completing the data transfer between each data node of the source TeleDB database and the master node of the target TeleDB database.

[0132] To further illustrate this solution, we will use a database backup tool (pg_dump) and a database recovery tool (pg_restore) for comparison. In practice, pg_dump is used to export the data tables from the source database, and pg_restore is used to restore these files in the target database, completing the data migration. The source and target databases can be deployed on two separate physical machines, each with a 16-core, 1-thread Intel Xeon Processor (Cascadelake) 2099.998 GHz processor, 31GB of memory, and 1TB of NVMe SSD storage, running CentOS 7.

[0133] During the data preparation phase, data from 10 tables is injected into the source database. The data migration task is to migrate these 10 tables to the target database. These 10 tables can be represented as: bmsql_config, bmsql_new_order, bmsql_order_line, bmsql_oorder, bmsql_history, bmsql_customer, bmsql_stock, bmsql_item, bmsql_district, and bmsql_warehouse, with a sequence name bmsql_hist_id_seq. The data sizes are set to 149MB (warehouse 1), 1166MB (warehouse 10), 11GB (warehouse 100), and 110GB (warehouse 1000) respectively. Simultaneously, these 10 tables are created in the target database, awaiting data migration.

[0134] The data migration table is shown in Table 1. It can be seen that under different data volumes, especially when the data volume is large, the migration time of this solution is shorter and the migration efficiency is higher.

[0135] Table 1

[0136]

[0137] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0138] Based on the same inventive concept, this application also provides a data migration apparatus for implementing the data migration method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data migration apparatus embodiments provided below can be found in the limitations of the data migration method described above, and will not be repeated here.

[0139] In one exemplary embodiment, such as Figure 11 As shown, a data migration apparatus is provided, comprising: an acquisition module 11, a strategy determination module 12, an instruction sending module 13, and a loop execution module 14, wherein:

[0140] Module 11 is used to obtain the amount of data in the current data table to be migrated in the source database; the source database includes multiple data tables to be migrated; the data in each data table to be migrated is distributed across multiple data nodes;

[0141] The strategy determination module 12 is used to determine the parallel migration strategy of the current data table to be migrated based on the first mapping relationship between the data in the current data table to be migrated and multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated and multiple data nodes in the target database when the amount of data in the current data table to be migrated is greater than a preset threshold.

[0142] The instruction sending module 13 is used to send parallel migration instructions to the master node in the source database according to the parallel migration strategy; the parallel migration instructions are used to instruct the master node to control multiple data nodes to migrate the corresponding data to the target data table in the target database in parallel.

[0143] The loop execution module 14 is used to perform data migration by taking the next data table to be migrated from multiple data tables to be migrated as the new current data table to be migrated, until all data tables to be migrated in the source database are migrated to the target database.

[0144] In an exemplary embodiment, the strategy determination module includes: a quantity acquisition unit and a strategy determination unit, wherein:

[0145] The quantity acquisition unit is used to acquire the number of source data nodes corresponding to the current data table to be migrated and the number of target data nodes corresponding to the target data table according to the first mapping relationship and the second mapping relationship.

[0146] The strategy determination unit is used to determine the parallel migration strategy of the current data table to be migrated based on the number of source data nodes, the number of target data nodes, the first mapping relationship, and the second mapping relationship.

[0147] In an exemplary embodiment, the strategy determination unit is further configured to compare the number of source data nodes with the number of target data nodes, and to compare the first mapping relationship with the second mapping relationship; and to determine the parallel migration strategy of the current data table to be migrated based on the comparison results of the number of data nodes and the comparison results of the mapping relationships.

[0148] In an exemplary embodiment, the strategy determination unit is further configured to determine, when the comparison results of the number of data nodes and the comparison results of the mapping relationship are the same, the parallel migration strategy of the current data table to be migrated is a migration strategy between data nodes of the source database and data nodes of the target database; and when the comparison results of the number of data nodes and / or the comparison results of the mapping relationship are different, the parallel migration strategy of the current data table to be migrated is a migration strategy between data nodes of the source database and data nodes of the master node of the target database.

[0149] In an exemplary embodiment, the strategy determination module further includes: a first sending unit, wherein:

[0150] The first sending unit is used to send a first service start instruction to the master node in the target database when the parallel migration strategy is a migration strategy between data nodes; the first service start instruction is used to instruct the master node to start a data assistance service on each data node; each data assistance service has a different port number; the port number is used to receive data sent by the corresponding data node in the source database.

[0151] In an exemplary embodiment, the strategy determination module further includes a second sending unit, wherein:

[0152] The second sending unit is used to send a second service start command to the master node in the target database when the parallel migration strategy is a migration strategy between data nodes and master nodes. The second service start command is used to instruct the master node to start multiple data assistance services on the master node. Each data assistance service has a different port number. The port number is used to receive data sent by the corresponding data node in the source database.

[0153] In an exemplary embodiment, the data migration apparatus further includes: a master node policy determination module, a first sending module, and a second sending module, wherein:

[0154] The master node strategy determination module is used to determine the migration strategy between master nodes when the amount of data in the current data table to be migrated is less than or equal to a preset threshold. The migration strategy between master nodes represents the strategy from the master node of the source database to the master node of the target database.

[0155] The first sending module is used to send a third service start command to the master node in the target database; the third service start command is used to instruct the master node to start a data assistance service on the master node; the fixed port of the data assistance service is used to receive data sent by the master node in the source database;

[0156] The second sending module is used to send data migration instructions to the master node of the source data; the data migration instructions are used to instruct the master node to migrate the data in the current data table to be migrated to the fixed port of the data assistance service.

[0157] Each module in the aforementioned data migration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0158] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the content of any of the embodiments of the data migration methods described above.

[0159] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the content of any one of the embodiments of the data migration method described above.

[0160] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the content of any one of the embodiments of the data migration method described above.

[0161] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0162] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0163] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0164] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data migration method, characterized in that, The method includes: Obtain the amount of data in the current data table to be migrated in the source database; the source database includes multiple data tables to be migrated; the data in each data table to be migrated is distributed across multiple data nodes; If the amount of data in the current data table to be migrated is greater than a preset threshold, a parallel migration strategy for the current data table to be migrated is determined based on the first mapping relationship between the data in the current data table to be migrated and the multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated and the multiple data nodes in the target database. According to the parallel migration strategy, a parallel migration instruction is sent to the master node in the source database; the parallel migration instruction is used to instruct the master node to control the multiple data nodes to migrate the corresponding data to the target data table in the target database in parallel. The next data table to be migrated from the plurality of data tables to be migrated is used as the new current data table to be migrated, and the data migration is performed until all data tables to be migrated in the source database are migrated to the target database. If the amount of data in the current data table to be migrated is less than or equal to a preset threshold, the migration strategy for the current data to be migrated is determined to be the migration strategy between master nodes. A third service start command is sent to the master node in the target database; the third service start command is used to instruct the master node to start a data assistance service on the master node; the fixed port of the data assistance service is used to receive data sent by the master node in the source database. Send a data migration instruction to the master node of the source data; the data migration instruction is used to instruct the master node to migrate the data in the current data table to be migrated to the fixed port of the data assistance service.

2. The method according to claim 1, characterized in that, The parallel migration strategy for the current data table to be migrated is determined based on a first mapping relationship between the data in the current data table to be migrated and the multiple data nodes, and a second mapping relationship between the data in the target data table corresponding to the current data table to be migrated and the multiple data nodes in the target database. This includes: Based on the first mapping relationship and the second mapping relationship, obtain the number of source data nodes corresponding to the current data table to be migrated and the number of target data nodes corresponding to the target data table; Based on the number of source data nodes, the number of target data nodes, the first mapping relationship, and the second mapping relationship, a parallel migration strategy for the current data table to be migrated is determined.

3. The method according to claim 2, characterized in that, The step of determining the parallel migration strategy for the current data table to be migrated based on the number of source data nodes, the number of target data nodes, the first mapping relationship, and the second mapping relationship includes: The number of source data nodes is compared with the number of target data nodes, and the first mapping relationship is compared with the second mapping relationship; Based on the comparison results of the number of data nodes and the comparison results of the mapping relationship, the parallel migration strategy of the current data table to be migrated is determined.

4. The method according to claim 3, characterized in that, The step of determining the parallel migration strategy for the current data table to be migrated based on the comparison results of the number of data nodes and the comparison results of the mapping relationship includes: If the comparison results of the number of data nodes and the comparison results of the mapping relationship are the same, the parallel migration strategy of the current data table to be migrated is determined to be the migration strategy between data nodes of the source database and data nodes of the target database. If the comparison results of the number of data nodes and / or the comparison results of the mapping relationship are different, the parallel migration strategy of the current data table to be migrated is determined to be the migration strategy between the data nodes of the source database and the master node of the target database.

5. The method according to claim 4, characterized in that, Before sending the parallel migration instruction to the master node in the source database according to the parallel migration strategy, the method further includes: When the parallel migration strategy is a migration strategy between the data nodes, a first service start instruction is sent to the master node in the target database; the first service start instruction is used to instruct the master node to start a data assistance service on each data node; each data assistance service has a different port number; the port number is used to receive data sent by the corresponding data node in the source database.

6. The method according to claim 4, characterized in that, Before sending the parallel migration instruction to the master node in the source database according to the parallel migration strategy, the method further includes: When the parallel migration strategy is a migration strategy between the data node and the master node, a second service start instruction is sent to the master node in the target database; the second service start instruction is used to instruct the master node to start multiple data assistance services on the master node; each data assistance service has a different port number; the port number is used to receive data sent by the corresponding data node in the source database.

7. The method according to any one of claims 1-6, characterized in that, The migration strategy between master nodes represents the strategy for migrating from the master node of the source database to the master node of the target database.

8. A data migration device, characterized in that, The device includes: The acquisition module is used to acquire the amount of data in the current data table to be migrated in the source database; the source database includes multiple data tables to be migrated; the data in each data table to be migrated is distributed across multiple data nodes; The strategy determination module is used to determine the parallel migration strategy of the current data table to be migrated based on the first mapping relationship between the data in the current data table to be migrated and the multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated and the multiple data nodes in the target database when the amount of data in the current data table to be migrated is greater than a preset threshold. The instruction sending module is used to send parallel migration instructions to the master node in the source database according to the parallel migration strategy; the parallel migration instructions are used to instruct the master node to control the multiple data nodes to migrate the corresponding data to the target data table in the target database in parallel. The loop execution module is used to perform data migration by taking the next data table to be migrated from the plurality of data tables to be migrated as the new current data table to be migrated, until all data tables to be migrated in the source database are migrated to the target database. The master node strategy determination module is used to determine the migration strategy between master nodes when the amount of data in the current data table to be migrated is less than or equal to a preset threshold. The first sending module is used to send a third service start command to the master node in the target database; the third service start command is used to instruct the master node to start a data assistance service on the master node; the fixed port of the data assistance service is used to receive data sent by the master node in the source database. The second sending module is used to send a data migration instruction to the master node of the source data; the data migration instruction is used to instruct the master node to migrate the data in the current data table to be migrated to the fixed port of the data assistance service.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images