Data migration method and device, computer equipment and readable storage medium

Abstract

The invention relates to a data migration method and device, computer equipment and a readable storage medium. Comprising the steps of obtaining the data volume of a current to-be-migrated data table in a source database; determining a parallel migration strategy based on a first mapping relationship between data in the current to-be-migrated data table and a plurality of data nodes and a second mapping relationship between data in a target data table in a target database and a plurality of data nodes under the condition that the data volume is greater than a preset threshold value; according to the parallel migration strategy, sending a parallel migration instruction to a main node in the source database; the parallel migration instruction is used for instructing the main node to control a plurality of data nodes to migrate corresponding data to a target data table in a target database in parallel; and taking the next to-be-migrated data table in the plurality of to-be-migrated data tables as a new current to-be-migrated data table to execute data migration until all to-be-migrated data tables in the source database are migrated to the target database. The method can be suitable for data migration in the data table in the MPP database.

Description

Technical Field

[0001] This application relates to the technical field of data migration, and particularly to a data migration method, apparatus, computer device, and readable storage medium. Background Art

[0002] With the rapid development of information technology, the amount of data has grown explosively, and data migration has become a common operation in scenarios such as digital transformation, system upgrade, and architecture adjustment.

[0003] In the field of data migration, the data migration tools in related technologies are all designed for single-machine databases. These tools are based on the data processing logic of a single node and achieve data transfer through simple file transfer or database import / export functions.

[0004] However, the methods in related technologies are not applicable to data migration in data tables in a Massively Parallel Processing (MPP) database. Summary of the Invention

[0005] Based on this, it is necessary to provide a data migration method, apparatus, computer device, and readable storage medium for the above technical problems, which can be applicable to data migration in data tables in an MPP database.

[0006] In a first aspect, this application provides a data migration method, including:

[0007] Obtain the data volume of the currently to-be-migrated data table in the source database; the source database includes multiple to-be-migrated data tables; the data in each to-be-migrated data table is correspondingly distributed on multiple data nodes;

[0008] When the data volume of the currently to-be-migrated data table is greater than a preset threshold, determine the parallel migration strategy of the currently to-be-migrated data table based on the first mapping relationship between the data in the currently to-be-migrated data table and multiple data nodes and the second mapping relationship between the data in the target data table corresponding to the currently to-be-migrated data table in the target database and multiple data nodes in the target database;

[0009] Send a parallel migration instruction to the master node in the source database according to the parallel migration strategy; the parallel migration instruction is used to instruct the master node to control multiple data nodes to parallelly migrate the corresponding data to the target data table in the target database;

[0010] Use the next to-be-migrated data table among multiple to-be-migrated data tables as the new currently to-be-migrated data table to perform data migration until all to-be-migrated data tables in the source database are migrated to the target database.

[0011] In one embodiment, based on the first mapping relationship between the data in the current data table to be migrated and multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated in the target database and multiple data nodes in the target database, determining the parallel migration strategy for the current data table to be migrated includes:

[0012] According to the first mapping relationship and the second mapping relationship, obtaining the number of source data nodes corresponding to the current data table to be migrated and the number of target data nodes corresponding to the target data table;

[0013] Based on the number of source data nodes, the number of target data nodes, the first mapping relationship and the second mapping relationship, determining the parallel migration strategy for the current data table to be migrated.

[0014] In one embodiment, based on the number of source data nodes, the number of target data nodes, the first mapping relationship and the second mapping relationship, determining the parallel migration strategy for the current data table to be migrated includes:

[0015] Comparing the number of source data nodes with the number of target data nodes, and comparing the first mapping relationship with the second mapping relationship;

[0016] According to the comparison result of the number of data nodes and the comparison result of the mapping relationship, determining the parallel migration strategy for the current data table to be migrated.

[0017] In one embodiment, according to the comparison result of the number of data nodes and the comparison result of the mapping relationship, determining the parallel migration strategy for the current data table to be migrated includes:

[0018] In the case where the comparison results of the number of data nodes and the mapping relationship are both the same, determining the parallel migration strategy for the current data table to be migrated as the migration strategy between the data nodes from the data nodes of the source database to the data nodes of the target database;

[0019] In the case where the comparison result of the number of data nodes and / or the comparison result of the mapping relationship is different, determining the parallel migration strategy for the current data table to be migrated as the migration strategy between the data nodes from the data nodes of the source database to the master nodes of the target database and the master nodes.

[0020] In one embodiment, before sending a parallel migration instruction to the master node in the source database according to the parallel migration strategy, the method further includes:

[0021] In the case where the parallel migration policy is the migration policy between data nodes, send a first service start instruction to the primary node in the target database; the first service start instruction is used to instruct the primary node to start a data assistance service on each data node; the port numbers of each data assistance service are different; the port number is used to receive data sent by the corresponding data node in the source database.

[0022] In one embodiment, before sending a parallel migration instruction to the primary node in the source database according to the parallel migration policy, the method further includes:

[0023] In the case where the parallel migration policy is the migration policy between a data node and the primary node, send a second service start instruction to the primary node in the target database; the second service start instruction is used to instruct the primary node to start multiple data assistance services on the primary node; the port numbers of each data assistance service are different; the port number is used to receive data sent by the corresponding data node in the source database.

[0024] In one embodiment, the method further includes:

[0025] In the case where the data volume of the currently to-be-migrated data table is less than or equal to a preset threshold, determine that the migration policy of the currently to-be-migrated data is the migration policy between primary nodes; the migration policy between primary nodes represents the policy from the primary node of the source database to the primary node of the target database;

[0026] Send a third service start instruction to the primary node in the target database; the third service start instruction is used to instruct the primary node to start a data assistance service on the primary node; the fixed port of the data assistance service is used to receive data sent by the primary node in the source database;

[0027] Send a data migration instruction to the primary node of the source data; the data migration instruction is used to instruct the primary node to migrate the data in the currently to-be-migrated data table to the fixed port of the data assistance service.

[0028] In a second aspect, the present application further provides a data migration device, including:

[0029] An acquisition module, configured to acquire the data volume of the currently to-be-migrated data table in the source database; the source database includes multiple to-be-migrated data tables; the data in each to-be-migrated data table is correspondingly distributed on multiple data nodes;

[0030] A policy determination module, configured to determine a parallel migration policy for the currently to-be-migrated data table based on a first mapping relationship between the data in the currently to-be-migrated data table and multiple data nodes, and a second mapping relationship between the data in the target data table corresponding to the currently to-be-migrated data table in the target database and the multiple data nodes in the target database, when the data volume of the currently to-be-migrated data table is greater than a preset threshold;

[0031] An instruction sending module, configured to send a parallel migration instruction to a master node in the source database according to the parallel migration policy; the parallel migration instruction is used to instruct the master node to control multiple data nodes to parallelly migrate the corresponding data to the target data table in the target database;

[0032] A loop execution module, configured to take the next to-be-migrated data table among multiple to-be-migrated data tables as the new currently to-be-migrated data table to perform data migration until all the to-be-migrated data tables in the source database are migrated to the target database.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the content of any one of the embodiments of the data migration method in the first aspect above is implemented.

[0034] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the content of any one of the embodiments of the data migration method in the first aspect above is implemented.

[0035] In a fifth aspect, the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the content of any one of the embodiments of the data migration method in the first aspect above is implemented.

[0036] The above data migration method, apparatus, computer device, and readable storage medium obtain the data volume of the currently to-be-migrated data table in the source database; the source database includes multiple to-be-migrated data tables; the data in each to-be-migrated data table is correspondingly distributed among multiple data nodes; when the data volume of the currently to-be-migrated data table is greater than a preset threshold, based on the first mapping relationship between the data in the currently to-be-migrated data table and the multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the currently to-be-migrated data table in the target database and the multiple data nodes in the target database, determine the parallel migration strategy for the currently to-be-migrated data table; according to the parallel migration strategy, send a parallel migration instruction to the master node in the source database; the parallel migration instruction is used to instruct the master node to control the multiple data nodes to parallelly migrate the corresponding data to the target data table in the target database; take the next to-be-migrated data table among the multiple to-be-migrated data tables as the new currently to-be-migrated data table to perform data migration until all the to-be-migrated data tables in the source database are migrated to the target database. This method obtains the data volume of the currently to-be-migrated data table in the source database, and when the data volume is greater than the preset threshold, determines the parallel migration strategy based on the first mapping relationship between the data in the source database and the data nodes, and the second mapping relationship between the corresponding data in the target database and the data nodes, which can make the resource allocation more reasonable during the data migration process and can obtain a more suitable parallel migration strategy for the currently to-be-migrated data table. Then, according to the parallel migration strategy, send an instruction to the source database master node, and the master node controls the multiple data nodes to parallelly migrate the data to the target database, giving full play to the parallel processing ability of the MPP distributed database, which can achieve efficient migration processing of large-data-volume data tables, greatly shorten the data migration time, and improve the data migration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required to be used in the description of the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0038] Figure 1 It is an application environment diagram of the data migration method in an embodiment;

[0039] Figure 2 It is a flowchart of the data migration method in an embodiment;

[0040] Figure 3 It is a flowchart of the data migration method in an embodiment;

[0041] Figure 4Schematic flow diagram of a data migration method in an embodiment;

[0042] Figure 5 Schematic diagram of data flow between data nodes in an embodiment;

[0043] Figure 6 Schematic diagram of data flow between a data node and a master node in an embodiment;

[0044] Figure 7 Schematic flow diagram of a data migration method in an embodiment;

[0045] Figure 8 Schematic diagram of data flow between master nodes in an embodiment;

[0046] Figure 9 Schematic flow diagram of a data migration method in an embodiment;

[0047] Figure 10 Schematic flow diagram of a data migration method in an embodiment;

[0048] Figure 11 Block diagram of a data migration device in an embodiment. Detailed implementation manners

[0049] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] Before introducing the technical solutions of the present application in detail, the background technology of the present application will be described in detail.

[0051] Data migration refers to the process of transferring data from one storage system or database to another storage system or database, with the aim of achieving seamless access and use of data in a new environment. Data migration is usually carried out in scenarios such as system upgrade, database replacement, storage device replacement or cloud migration, aiming to improve system performance, scalability and reliability. The key lies in ensuring data integrity, accuracy and consistency, and avoiding data loss or damage.

[0052] Data migration in a database generally consists of two parts: metadata migration and table data migration. Metadata migration refers to the process of migrating database structures, definitions, and other related database objects from the source database to the target database, specifically including table structures, indexes, views, stored procedures, and triggers, etc. Effective metadata migration can ensure that the target database can correctly identify and use the migrated data structures and logic. Table data migration refers to the process of migrating the actual stored data in the database from the source database to the target database. Due to the large amount of data involved, this process requires careful planning and execution to ensure data integrity and accuracy, which generally includes three parts: data export, data transfer, and data import. Data export means exporting the database table data to a data file through an export tool. Data transfer means transferring the exported data file from the source database environment to the target database environment. This can be done through network transfer, cloud storage, or physical media (such as an external hard drive). Data import means importing the data file into the target database using the import tool of the target database.

[0053] In the field of data migration, the data migration tools in related technologies are all designed for single-machine databases. These tools are based on the data processing logic of a single node and achieve data transfer through simple file transfer or database import / export functions.

[0054] MPP databases usually adopt a shared-nothing architecture (SNA), that is, each node independently owns memory and storage resources, reducing resource competition and lock contention between nodes, thereby enhancing the scalability and fault tolerance of the system. A typical MPP database consists of a master node and multiple data nodes (DNs). The master node is mainly responsible for coordinating the execution of queries and operations, while the data nodes are responsible for storing and processing data. In an MPP database, under the existing data partitioning strategy, the master node distributes the data evenly to each data node according to certain attributes of the data (such as primary key, hash, etc.), thus ensuring load balancing. This process is called data redistribution.

[0055] That is to say, an MPP database utilizes multiple independent computing nodes to process data and queries in parallel, enabling the MPP database to split data and computing tasks into multiple subtasks for parallel execution, thereby significantly reducing processing time. During the migration process of a data table, the size of the data volume in the data table highly affects the data migration speed. Due to the characteristics of multiple data nodes in the MPP database, parallel migration consumes more system resources, and consequently, more time is consumed. Therefore, in such a distributed architecture, the data migration solutions in related technologies have obvious deficiencies: (1) MPP databases usually store massive amounts of data at the terabyte (TB) or petabyte (PB) level. The methods in related technologies cannot fully utilize the advantages of multi-node storage in the MPP architecture, so it is difficult to meet the requirements of fast migration of large data volumes. (2) The data migration methods in related technologies usually require exporting data to local disk files. For large data volumes at the TB or PB level, this method has high requirements for disk capacity. (3) Usually, the table data in an MPP database is stored on different data nodes according to different distribution strategies. Common distribution strategies include hash distribution, random distribution, Shard distribution, range distribution, and replication distribution, etc. Therefore, when the mapping relationship between the data in the data table in the source database and multiple data nodes is different from the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated in the target database and multiple data nodes in the target database, data must be rebalanced according to the two mapping relationships when migrating to the target database to avoid data loss or inconsistency, which also poses a great challenge to the data migration of MPP databases. It should be noted that for an MPP database storing data at the TB or PB level, during the entire data migration process, compared with table data migration, the time required for metadata migration is almost negligible.

[0056] In response to the above problems, the present application provides a data migration method, apparatus, computer device, and readable storage medium, which can fully consider the parallel computing and distributed storage characteristics of the MPP architecture, utilize the multi-node architecture of the MPP database, and enable each data storage node to directly participate in data transmission and execute migration tasks in parallel. This can not only greatly improve the speed and efficiency of data migration but also be applicable to the migration of large data volumes at the TB or PB level. During the data migration process, a streaming data transmission mechanism is used to directly migrate data through the network, avoiding fully exporting data to local disks, thereby reducing the occupation of disk space. Additionally, it can dynamically adjust the corresponding migration strategy based on the size of the data table to be migrated and the mapping relationship between the data in the data table and the data center, and can be applicable to data migration in different situations. Next, the technical solutions of the present application will be introduced in detail.

[0057] The data migration method provided in the embodiments of the present application can be applied to, for exampleFigure 1 in the application environment shown.

[0058] For example, the computer device can be a server, a personal computer, a laptop computer, a smart phone, a tablet computer, a smart mobile phone, etc. The computer device can include a processor, a memory, and a network interface connected by a system bus or wirelessly. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device can include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data during the data migration process. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a data migration method. Among them, the computer device can be implemented by an independent computer device or a computer device cluster composed of multiple computer devices. It should be noted that the memory of the computer device is not limited to the above-mentioned memory, and can also include high-speed random access memory, volatile solid-state memory, etc. In addition, the composition architecture of the computer device is not limited to the above situation, and some components can also be added or omitted.

[0059] In an exemplary embodiment, as Figure 2 shown, a data migration method is provided. Taking the computer device in Figure 1 as an example, the method includes the following steps S101 to S104. Among them:

[0060] S101, obtain the data volume of the currently to-be-migrated data table in the source database; the source database includes multiple to-be-migrated data tables; the data in each to-be-migrated data table is correspondingly distributed on multiple data nodes.

[0061] Among them, the source database specifically refers to an MPP database with multiple independent data nodes. An MPP database usually includes one or more master nodes, multiple data nodes, and other auxiliary nodes. Taking one master node as an example, the master node is mainly responsible for receiving query requests from clients, parsing SQL, generating an execution plan, coordinating each data node to execute the query, summarizing the results, and returning them to the client. The data node is the node that actually stores data and executes query operations. In the MPP architecture, data is sharded and distributed on multiple data nodes, and each data node is responsible for processing the data part stored locally. The number of data nodes is the core expansion ability of the MPP system. In order to achieve large-scale parallel processing, there are usually multiple data nodes.

[0062] The source database U includes a master node Mu, and the set of multiple data nodes of U can be expressed as: , where represents the i-th data node of U, and M represents the number of data nodes of U. U may include K data tables, and the data table set can be represented as , where represents the i-th data table in U.

[0063] In the embodiments of the present application, the currently to-be-migrated data table may be any data table in the source database, and the computer device may obtain the data volume of the currently to-be-migrated data table by executing a specific query statement. Alternatively, the computer device may also use database system functions to query information such as the number of rows or storage space of the currently to-be-migrated data table, and use the information such as the number of rows or storage space as the data volume of the currently to-be-migrated data table.

[0064] S102, when the data volume of the currently to-be-migrated data table is greater than a preset threshold, based on the first mapping relationship between the data in the currently to-be-migrated data table and multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the currently to-be-migrated data table in the target database and multiple data nodes in the target database, determine the parallel migration strategy of the currently to-be-migrated data table.

[0065] Among them, the target database V includes a master node M v , and the multiple data node sets of V can be represented as: , where represents the i-th data node of V, and N represents the number of data nodes of V. V may include K data tables, and the data table set can be represented as , where represents the i-th data table in V.

[0066] The process of data migration can be represented as: migrating the table data in the data table set T s in the source database to the data table set T d in the target database, and there is a one-to-one correspondence between the two, that is, the data of the data table is migrated to , denoted as < .

[0067] In the embodiment of the present application, after obtaining the data volume of the currently to-be-migrated data table, the computer device may compare the data volume of the currently to-be-migrated data table with a preset threshold. If the comparison result shows that the data volume of the currently to-be-migrated data table is less than or equal to the preset threshold, it indicates that the data volume of the currently to-be-migrated data table is small and there is no need to migrate it through the parallel migration method. If the comparison result shows that the data volume of the currently to-be-migrated data table is greater than the preset threshold, it indicates that the data volume of the currently to-be-migrated data table is large and it is necessary to migrate it through the parallel migration method. Taking the preset threshold as T and the data volume as the number of rows R of the data table as an example, if R>T, it is determined that the data volume is large; if R<=T, it is determined that the data volume is small.

[0068] When it is determined that the data volume of the currently to-be-migrated data table is greater than the preset threshold, the computer device may search for the first mapping relationship between the data in the currently to-be-migrated data table and multiple data nodes in the source database, and, search for the second mapping relationship between the data in the target data table and multiple data nodes in the target database. Then, analyze the first mapping relationship and the second mapping relationship according to the parallel migration strategy rules to find out the parallel migration strategy applicable to the currently to-be-migrated data table. Alternatively, the computer device may also input the first mapping relationship and the second mapping relationship into a preset migration strategy determination model, and use the migration strategy determination model for analysis to obtain the parallel migration strategy applicable to the currently to-be-migrated data table.

[0069] S103. Send a parallel migration instruction to the master node in the source database according to the parallel migration strategy; the parallel migration instruction is used to instruct the master node to control multiple data nodes to parallelly migrate the corresponding data to the target data table in the target database.

[0070] In the embodiment of the present application, after obtaining the parallel migration strategy applicable to the currently to-be-migrated data table, the computer device may generate a parallel migration instruction corresponding to the parallel migration strategy and send the parallel migration instruction to the master node in the source database. After receiving the parallel migration instruction, the master node controls multiple data nodes in the source database to simultaneously migrate the corresponding data to the target data table in the target database according to the migration information carried by the parallel migration instruction. In this way, the data migration process of the currently to-be-migrated data table is completed.

[0071] S104. Take the next to-be-migrated data table among the multiple to-be-migrated data tables as the new currently to-be-migrated data table to perform data migration until all the to-be-migrated data tables in the source database are migrated to the target database.

[0072] In the embodiment of the present application, the above steps S101 - S103 are the migration process of a data table in the source database. After the migration of this data table is completed, continue with the migration of the next data table to be migrated, take the next data table to be migrated as the new current data table to be migrated, and execute the steps of S101 - S103 to complete the data migration of the next data table to be migrated. Iteratively execute the data migration process of the above data tables until all the data tables to be migrated in the source database are migrated to the target database. That is to say, the number of parallel migrations is less than or equal to the number of data tables in the source data. If the data volume of each data table is greater than the preset threshold, the number of parallel migrations is equal to the number of data tables in the source data; if the data volume of at least one data table is less than or equal to the preset threshold, the number of parallel migrations is less than the number of data tables in the source data.

[0073] In the above data migration method, obtain the data volume of the current data table to be migrated in the source database; the source database includes multiple data tables to be migrated; the data in each data table to be migrated is correspondingly distributed among multiple data nodes; when the data volume of the current data table to be migrated is greater than the preset threshold, based on the first mapping relationship between the data in the current data table to be migrated and the multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated in the target database and the multiple data nodes in the target database, determine the parallel migration strategy of the current data table to be migrated; according to the parallel migration strategy, send a parallel migration instruction to the master node in the source database; the parallel migration instruction is used to instruct the master node to control the multiple data nodes to parallelly migrate the corresponding data to the target data table in the target database; take the next data table to be migrated among the multiple data tables to be migrated as the new current data table to be migrated to execute data migration until all the data tables to be migrated in the source database are migrated to the target database. This method can make the resource allocation in the data migration process more reasonable and obtain a more suitable parallel migration strategy for the current data table to be migrated by obtaining the data volume of the current data table to be migrated in the source database and determining the parallel migration strategy based on the first mapping relationship between the data in the source database and the data nodes and the second mapping relationship between the corresponding data in the target database and the data nodes when the data volume is greater than the preset threshold. Then, send an instruction to the source database master node according to the parallel migration strategy, and the master node controls the multiple data nodes to parallelly migrate the data to the target database, giving full play to the parallel processing ability of the MPP distributed database, and can achieve efficient migration processing of large - data - volume data tables, greatly shortening the data migration time and improving the data migration efficiency.

[0074] Obtaining the parallel migration strategy is a prerequisite for performing data parallel migration. Then, in one embodiment, as Figure 3As shown in the figure, the specific content of determining the parallel migration strategy of the current data table to be migrated based on the first mapping relationship between the data in the current data table to be migrated and multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated in the target database and multiple data nodes in the target database is introduced. The specific content includes:

[0075] S201. According to the first mapping relationship and the second mapping relationship, obtain the number of source data nodes corresponding to the current data table to be migrated and the number of target data nodes corresponding to the target data table.

[0076] In the embodiment of the present application, the computer device can determine the number of data nodes associated with the data in the current data table to be migrated based on the first mapping relationship between the data in the current data table to be migrated in the source database and multiple data nodes, and use the number of associated data nodes as the number of source data nodes. At the same time, it can also determine the number of data nodes associated with the data in the target database based on the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated in the target database and multiple data nodes in the target database, and use the number of associated data nodes as the number of target data nodes.

[0077] S202. Based on the number of source data nodes, the number of target data nodes, the first mapping relationship, and the second mapping relationship, determine the parallel migration strategy of the current data table to be migrated.

[0078] In the embodiment of the present application, the computer device can compare the number of source data nodes with the number of target data nodes to obtain a data node comparison result. And compare the first mapping relationship with the second mapping relationship to obtain a mapping relationship comparison result. Then, according to the data node comparison result and the mapping relationship comparison result, screen out the parallel migration strategy applicable to the current data table to be migrated from the parallel migration strategies. Alternatively, the computer device can also input the number of source data nodes, the number of target data nodes, the first mapping relationship, and the second mapping relationship into a preset policy determination model, and use the policy determination model to analyze the two data node numbers and the two mapping relationships to determine the parallel migration policy of the current data table to be migrated.

[0079] In the above data migration method, the number of source data nodes corresponding to the data table to be migrated currently is obtained, and the number of target data nodes corresponding to the target data table is obtained; based on the number of source data nodes, the number of target data nodes, the first mapping relationship, and the second mapping relationship, a parallel migration strategy for the data table to be migrated currently is determined. This method can accurately obtain a parallel migration strategy adapted to the data table to be migrated currently through obtaining the number of source data nodes and target data nodes and combining the first mapping relationship and the second mapping relationship, and can ensure the efficient utilization of network bandwidth and computing resources, and perform data migration with the best migration efficiency.

[0080] In one embodiment, as Figure 4 shown, the specific content of determining the parallel migration strategy of the data table to be migrated currently based on the number of source data nodes, the number of target data nodes, the first mapping relationship, and the second mapping relationship includes:

[0081] S301, compare the number of source data nodes with the number of target data nodes, and compare the first mapping relationship with the second mapping relationship.

[0082] In the embodiments of the present application, the computer device can compare the number of source data nodes with the number of target data nodes to determine whether the numbers of the two data nodes are the same. At the same time, the first mapping relationship can also be compared with the second mapping relationship to determine whether the distribution methods of the two data tables on the data nodes are the same.

[0083] S302, determine the parallel migration strategy of the data table to be migrated currently according to the comparison result of the number of data nodes and the comparison result of the mapping relationship.

[0084] In the embodiments of the present application, if it is determined that the numbers of the two data nodes are the same and the distribution methods of the two data tables on the data nodes are the same, parallel migration can be directly performed from the level of the data nodes, that is, it is determined that the parallel migration strategy of the data table to be migrated currently is the parallel migration strategy between the data nodes. If the numbers of the two data nodes are different, or the distribution methods of the two data tables on the data nodes are different, parallel migration needs to be performed from the level of the data nodes and the master node, that is, it is determined that the parallel migration strategy of the data table to be migrated currently is the parallel migration strategy between the data nodes and the master node.

[0085] Next, through an embodiment, the specific content of determining the parallel migration strategy of the data table to be migrated currently according to the comparison result of the number of data nodes and the comparison result of the mapping relationship is introduced, as Figure 5 shown, the specific content includes:

[0086] When the comparison results of the number of data nodes and the comparison results of the mapping relationships are both the same, determine that the parallel migration strategy for the currently to-be-migrated data table is the migration strategy between the data nodes from the data nodes of the source database to the data nodes of the target database;

[0087] When the comparison result of the number of data nodes and / or the comparison result of the mapping relationship is different, determine that the parallel migration strategy for the currently to-be-migrated data table is the migration strategy between the data nodes from the data nodes of the source database to the master node of the target database and the master node.

[0088] Exemplarily, taking the currently to-be-migrated data table with the corresponding number of source data nodes being M, and the target data table with the corresponding number of target data nodes being N, the first mapping relationship being D s and the second mapping relationship being D d as an example, if M = N and D s = D d , the parallel migration strategy is the migration strategy between data nodes (DN_To_DN); if M ≠ N, or D s ≠ D d , the parallel migration strategy is the migration strategy between data nodes and the master node (DN_To_M).

[0089] In the above data migration method, the number of source data nodes is compared with the number of target data nodes, and the first mapping relationship is compared with the second mapping relationship; according to the comparison results of the number of data nodes and the comparison results of the mapping relationships, the parallel migration strategy for the currently to-be-migrated data table is determined. By comparing the two numbers of data nodes and the two mapping relationships, this method can identify in advance the data distribution differences between the currently to-be-migrated data table in the source database and the target data table in the target database, so as to accurately screen out the parallel migration strategy suitable for the currently to-be-migrated data table.

[0090] The above embodiments are all introductions to the determination process of the parallel migration strategy. Next, the preparations before performing the data migration operation will be introduced. For different migration strategies, the preparations are different.

[0091] First, taking the parallel migration strategy as the migration strategy between data nodes as an example, through an embodiment, the preparations before the data migration operation will be introduced in detail. Before sending a parallel migration instruction to the master node in the source database according to the parallel migration strategy, the method further includes:

[0092] In the case where the parallel migration strategy is the migration strategy between data nodes, send a first service start instruction to the primary node in the target database; the first service start instruction is used to instruct the primary node to start a data assistance service on each data node; the port numbers of each data assistance service are different; the port number is used to receive data sent by the corresponding data node in the source database.

[0093] Among them, a data_helper service is a Transmission Control Protocol (TCP) service deployed on the target database, which is mainly responsible for listening for network requests and receiving and processing data from the source database. The data_helper service waits for connection requests from the source database by listening on a specified port. Once the connection is established, the data_helper service is ready to receive and process the transmitted data and store it securely at a specified location in the target database. As the core component for data reception, the data_helper service introduces a streaming data transmission mechanism, which can achieve efficient data migration directly through the network.

[0094] In the embodiment of the present application, when the parallel migration strategy is the migration strategy between data nodes, the computer device can generate a first service start instruction corresponding to the migration strategy based on the migration strategy between data nodes and send the first service start instruction to the primary node in the target database. After receiving the first service start instruction, the primary node starts the data_helper service on each data node, and the port numbers of each data_helper service are inconsistent. That is, by utilizing the parallel processing capabilities and communication bandwidth of all data nodes, a one-to-one correspondence is established between the data nodes in the source database and the data nodes in the target database, and the data migration is directly completed.

[0095] Figure 5 It is a schematic diagram of the data flow between data nodes. It can be seen from the figure that during the migration of the data in the current data table to be migrated, the data in each data node in the source database is migrated to the corresponding data node in the target database through the corresponding data_helper service. Moreover, the data in multiple data nodes in the source database is migrated in parallel. Each data assistance service corresponds to a port number, for example, 1024, 1025, and 1026.

[0096] In the above data migration method, when the parallel migration strategy is the migration strategy between data nodes, a first service start instruction is sent to the master node in the target database; the first service start instruction is used to instruct the master node to start a data assistance service on each data node; the port numbers of each data assistance service are different; the port number is used to receive data sent by the corresponding data node in the source database. In the migration strategy between data nodes, by establishing a data assistance service on each data node by the master node, through this data assistance service, data can be transmitted between data nodes through the network without introducing a disk, saving the cost of data transmission and improving resource utilization.

[0097] First, taking the parallel migration strategy as the migration strategy between a data node and a master node as an example, the preparation work before the data migration operation is introduced in detail through an embodiment. Before sending a parallel migration instruction to the master node in the source database according to the parallel migration strategy, the method further includes:

[0098] When the parallel migration strategy is the migration strategy between a data node and a master node, a second service start instruction is sent to the master node in the target database; the second service start instruction is used to instruct the master node to start multiple data assistance services on the master node; the port numbers of each data assistance service are different; the port number is used to receive data sent by the corresponding data node in the source database.

[0099] In the embodiment of the present application, when the parallel migration strategy is the migration strategy between a data node and a master node, the computer device can generate a second service start instruction corresponding to the migration strategy based on the migration strategy between the data node and the master node, and send the second service start instruction to the master node in the target database. After receiving the second service start instruction, the master node starts multiple data_helper services on the master node, and the port numbers of each data_helper service are inconsistent.

[0100] It should be noted that compared with the migration strategy between data nodes, the migration strategy between a data node and a master node is also parallel migration, but its performance is affected by the processing capacity of the master node in the target database.

[0101] Figure 6It is a schematic diagram of the data flow between the data nodes and the master node. It can be seen from the figure that during the migration of the data in the currently to-be-migrated data table, the data in each data node in the source database is migrated in parallel to the master node in the target database through the corresponding data_helper service, and the master node in the target database then stores the data received from each data node. The figure includes multiple data assistance services, and each data assistance service corresponds to a port number, for example, 1024, 1025, and 1026.

[0102] In the above data migration method, when the parallel migration strategy is the migration strategy between the data node and the master node, a second service start instruction is sent to the master node in the target database; the second service start instruction is used to instruct the master node to start multiple data assistance services on the master node; the port numbers of each data assistance service are different; the port number is used to receive the data sent by the corresponding data node in the source database. In the migration strategy between the data node and the master node, this method establishes multiple data assistance services on the master node through the master node. Through the multiple data assistance services, data can be transmitted between the data node and the master node through the network without introducing a disk, saving the cost of data transmission and improving resource utilization.

[0103] The above embodiments are all introductions to the data migration process when the data volume of the currently to-be-migrated data table is large. Next, an embodiment is used to introduce the data migration process when the data volume is small, as Figure 7 shown, the data migration process includes:

[0104] S401, when the data volume of the currently to-be-migrated data table is less than or equal to a preset threshold, determine that the migration strategy of the currently to-be-migrated data is the migration strategy between the master nodes; the migration strategy between the master nodes represents the strategy from the master node of the source database to the master node of the target database.

[0105] In the embodiments of the present application, since the parallel migration strategy requires creating a data_helper service process on each data node or establishing multiple data_helper service processes on the master node, the startup cost of the data_helper service is relatively high. Therefore, when it is determined that the data volume of the currently to-be-migrated data table is less than or equal to the preset threshold, that is, R <= T, it means that the data volume of this data table is small and parallel migration is not required. Based on this, the computer device can determine that the migration strategy of the currently to-be-migrated data is the migration strategy between the master nodes, that is, directly transmit the data from the master node in the source database to the master node in the target database.

[0106] S402. Send a third service start instruction to the master node in the target database. The third service start instruction is used to instruct the master node to start a data assistance service on the master node. The fixed port of the data assistance service is used to receive the data sent by the master node in the source database.

[0107] In an embodiment of the present application, when it is determined that the migration strategy is a migration strategy between master nodes, the computer device may generate a third service start instruction corresponding to the migration strategy according to the migration strategy between master nodes. After receiving the third service start instruction, the master node in the target database starts a data_helper service on the master node, and the data_helper service has a fixed port.

[0108] S403. Send a data migration instruction to the master node of the source data. The data migration instruction is used to instruct the master node to migrate the data in the currently to-be-migrated data table to the fixed port of the data assistance service.

[0109] In an embodiment of the present application, the computer device may generate a data migration instruction corresponding to the migration strategy between master nodes based on the migration strategy between master nodes. After receiving the data migration instruction, the master node migrates the data in the currently to-be-migrated data table to the fixed port of the data assistance service, so as to implement the migration of data between the master node of the source database and the master node of the target database.

[0110] Figure 8 It is a schematic diagram of the data flow between master nodes. It can be seen from the figure that during the migration of the data in the currently to-be-migrated data table, all the data is migrated from the master node of the source database to the master node of the target database through a data_helper service. The port number corresponding to the data_helper service is 1024.

[0111] In the above data migration method, when the data volume of the currently to-be-migrated data table is less than or equal to a preset threshold, it is determined that the migration strategy of the currently to-be-migrated data is a migration strategy between master nodes. The migration strategy between master nodes represents the strategy from the master node of the source database to the master node of the target database. Send a third service start instruction to the master node in the target database. The third service start instruction is used to instruct the master node to start a data assistance service on the master node. The port number of the data assistance service is used to receive the data sent by the master node in the source database. Send a data migration instruction to the master node of the source data. The data migration instruction is used to instruct the master node to migrate the data in the currently to-be-migrated data table to the fixed port of the data assistance service. This method triggers the direct migration strategy between master nodes when the data volume of the to-be-migrated data table is less than or equal to the preset threshold, completes the migration with a more concise link in the small data volume scenario, reduces the transmission loss of data between multiple nodes, and improves the efficiency of data migration.

[0112] In a detailed embodiment, as Figure 9 shown, the data migration method includes:

[0113] S501, obtaining the data volume of the currently to-be-migrated data table in the source database;

[0114] S502, when the data volume of the currently to-be-migrated data table is greater than a preset threshold, obtaining the number of source data nodes corresponding to the currently to-be-migrated data table and the number of target data nodes corresponding to the target data table according to the first mapping relationship and the second mapping relationship;

[0115] S503, comparing the number of source data nodes with the number of target data nodes, and comparing the first mapping relationship with the second mapping relationship;

[0116] S504, when the comparison results of the number of data nodes and the comparison results of the mapping relationships are both the same, determining that the parallel migration strategy for the currently to-be-migrated data table is the migration strategy between the data nodes from the data nodes of the source database to the data nodes of the target database;

[0117] S505, sending a first service start instruction to the master node in the target database; the first service start instruction is used to instruct the master node to start a data assistance service on each data node; the port numbers of each data assistance service are different; the port numbers are used to receive the data sent by the corresponding data node in the source database;

[0118] S506, when the comparison result of the number of data nodes and / or the comparison result of the mapping relationships is different, determining that the parallel migration strategy for the currently to-be-migrated data table is the migration strategy between the data nodes from the data nodes of the source database to the master node of the target database;

[0119] S507, sending a second service start instruction to the master node in the target database; the second service start instruction is used to instruct the master node to start multiple data assistance services on the master node; the port numbers of each data assistance service are different; the port numbers are used to receive the data sent by the corresponding data node in the source database;

[0120] S508, sending a parallel migration instruction to the master node in the source database according to the parallel migration strategy;

[0121] S509, when the data volume of the currently to-be-migrated data table is less than or equal to the preset threshold, determining that the migration strategy for the currently to-be-migrated data is the migration strategy between the master nodes;

[0122] S510. Send a third service start instruction to the master node in the target database; the third service start instruction is used to instruct the master node to start a data assistance service on the master node; the fixed port of the data assistance service is used to receive data sent by the master node in the source database.

[0123] S511. Send a data migration instruction to the master node of the source data; the data migration instruction is used to instruct the master node to migrate the data in the currently to-be-migrated data table to the fixed port of the data assistance service.

[0124] S512. Determine whether the to-be-migrated data table in the source database has been migrated. If so, end the data migration; if not, execute step S501 until all the to-be-migrated data tables in the source database have been migrated to the target database.

[0125] Figure 10 It is a schematic flow diagram of a data migration method. The data migration method includes: S11: Determine whether the data tables in the source database have been migrated. If not, execute step S12; if so, end. S12: Query the number of rows of the currently to-be-migrated data table. S13: Determine whether the number of rows is less than a preset threshold. If so, execute step S14; if not, execute step S15. S14: After executing the migration strategy between the master nodes, execute step S19. S15: Obtain the first mapping relationship between the data in the currently to-be-migrated data table in the source database and multiple data nodes, obtain the second mapping relationship between the data in the target data table in the target database and multiple data nodes, and obtain the number of data nodes in the two databases. S16: Determine whether the two mapping relationships are consistent and whether the number of data nodes in the two databases is consistent. If so, execute step S17; if not, execute step S18. S17: Execute the parallel migration strategy between the data nodes and then execute step S19. S18: Execute the parallel migration strategy between the data nodes and the master node. S19: Take the next to-be-migrated data table in the source database as the new currently to-be-migrated data table and then execute step S11.

[0126] Taking the TeleDB distributed database derived from the open-source PostgreSQL-XL as an example, assume that the source TeleDB database includes one master node and two data nodes, and the target TeleDB database also includes one master node and two data nodes. The data volume is represented by the number of rows of the data table, and the set threshold corresponding to the data volume is T = 300,000.

[0127] For any data table in the source TeleDB database, perform the following operations: execute an SQL statement to obtain the number of rows of the table in the source database, and get the number of rows of the data table. Execute an SQL statement to query the mapping relationship in the source TeleDB database to obtain the first mapping relationship between the data in the data table and multiple data nodes, and execute an SQL statement to query the mapping relationship in the target TeleDB database to obtain the second mapping relationship between the data in the target data table and multiple data nodes.

[0128] If the number of rows of the data table R <= 300,000, execute the migration strategy between master nodes (M_To_M); if the number of rows of the data table R > 300,000, and M = N, D s and D d are consistent, execute the parallel migration strategy between data nodes (DN_To_DN); if the number of rows of the data table R > 300,000, but M ≠ N, or, D s and D d are inconsistent, execute the parallel migration strategy from data node to master node (DN_To_M).

[0129] Before data migration, for the M_To_M strategy, execute the SQL statement (cmd-id) corresponding to the service start instruction on the master node of the target TeleDB database. The service start instruction will start a data_helper service on the master node, listen on a port, and write the cmd-id and the listening port to a file for disk storage. The master node of the source TeleDB database reads the file through this cmd-id to determine the port listened on by the data_helper of the target TeleDB database, and executes the SQL statement corresponding to data migration.

[0130] For the DN_To_DN strategy, connect the master node of the source TeleDB database to the master node of the target TeleDB database, and execute the SQL statement (cmd-id) corresponding to the service start instruction on each data node of the target TeleDB database. The service start instruction will start a data_helper service on each data node, and each data_helper listens on a port. Control each data node of the source TeleDB database to perform data migration to complete the one-to-one data transfer between the data nodes in the two TeleDB databases.

[0131] For the DN_To_M strategy, connect the primary node of the source TeleDB database to the primary node of the target TeleDB database. Execute the SQL statement (cmd-id) corresponding to the service startup instruction on the primary node of the target TeleDB database. The service startup instruction will start two data_helper services on the primary node. Each data_helper listens on a port and writes the cmd-id, datanode-id, and the listening port to a file for disk storage. Control each data node of the source TeleDB database to perform data migration to complete the data transfer between each data node of the source TeleDB database and the primary node of the target TeleDB database.

[0132] To further introduce this solution, for example, use the database backup tool (pg_dump) and the database restoration tool (pg_restore) for comparison. In the specific operation, use pg_dump to export the data tables in the source database and use pg_restore to restore these files in the target database to complete the data migration. The source database and the target database can be deployed on two physical machines with 16 cores, 1 thread, Intel Xeon Processor (Cascadelake) 2099.998 GHz processors, 31G of memory, and 1TB of NVMe SSD storage respectively, and the operating system is centos 7.

[0133] In the data preparation stage, inject data for 10 tables into the source database. The task of data migration is to migrate these 10 tables to the target database. The 10 tables can be represented as: bmsql_config, bmsql_new_order, bmsql_order_line, bmsql_oorder, bmsql_history, bmsql_customer, bmsql_stock, bmsql_item, bmsql_district, bmsql_warehouse, and a sequence is bmsql_hist_id_seq. Set the data volume to 149MB (1 warehouse), 1166MB (10 warehouses), 11GB (100 warehouses), and 110GB (1000 warehouses) respectively. At the same time, create the 10 tables in the target database and wait for data migration.

[0134] The data migration table is shown in Table 1. It can be found that under different data volumes, especially when the data volume is large, the migration time of this solution is shorter and the migration efficiency is higher.

[0135] Table 1

[0136]

[0137] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0138] Based on the same inventive concept, an embodiment of the present application further provides a data migration device for implementing the data migration method involved above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more of the following data migration device embodiments can refer to the limitations on the data migration method in the above text, and will not be repeated here.

[0139] In an exemplary embodiment, as Figure 11 shown, a data migration device is provided, including: an acquisition module 11, a policy determination module 12, an instruction sending module 13, and a loop execution module 14, where:

[0140] The acquisition module 11 is used to acquire the data volume of the currently to-be-migrated data table in the source database; the source database includes multiple to-be-migrated data tables; the data in each to-be-migrated data table is correspondingly distributed on multiple data nodes;

[0141] The policy determination module 12 is used to determine the parallel migration policy of the currently to-be-migrated data table based on the first mapping relationship between the data in the currently to-be-migrated data table and multiple data nodes and the second mapping relationship between the data in the target data table corresponding to the currently to-be-migrated data table in the target database and multiple data nodes in the target database when the data volume of the currently to-be-migrated data table is greater than a preset threshold;

[0142] The instruction sending module 13 is used to send a parallel migration instruction to the master node in the source database according to the parallel migration policy; the parallel migration instruction is used to instruct the master node to control multiple data nodes to parallelly migrate the corresponding data to the target data table in the target database;

[0143] The loop execution module 14 is used to perform data migration by taking the next data table to be migrated in multiple data tables to be migrated as the new current data table to be migrated until all the data tables to be migrated in the source database are migrated to the target database.

[0144] In an exemplary embodiment, the above-mentioned policy determination module includes: a quantity acquisition unit and a policy determination unit, where:

[0145] The quantity acquisition unit is used to obtain the number of source data nodes corresponding to the current data table to be migrated and the number of target data nodes corresponding to the target data table according to the first mapping relationship and the second mapping relationship;

[0146] The policy determination unit is used to determine the parallel migration policy of the current data table to be migrated based on the number of source data nodes, the number of target data nodes, the first mapping relationship and the second mapping relationship.

[0147] In an exemplary embodiment, the above-mentioned policy determination unit is further used to compare the number of source data nodes with the number of target data nodes, and compare the first mapping relationship with the second mapping relationship; according to the comparison result of the number of data nodes and the comparison result of the mapping relationship, determine the parallel migration policy of the current data table to be migrated.

[0148] In an exemplary embodiment, the above-mentioned policy determination unit is further used to determine that the parallel migration policy of the current data table to be migrated is the migration policy between the data nodes from the data nodes of the source database to the data nodes of the target database when the comparison results of the number of data nodes and the mapping relationship are both the same; when the comparison result of the number of data nodes and / or the comparison result of the mapping relationship is different, determine that the parallel migration policy of the current data table to be migrated is the migration policy between the data nodes from the data nodes of the source database to the main node of the target database and the main node.

[0149] In an exemplary embodiment, the above-mentioned policy determination module further includes: a first sending unit, where:

[0150] The first sending unit is used to send a first service start instruction to the main node in the target database when the parallel migration policy is the migration policy between data nodes; the first service start instruction is used to instruct the main node to start a data assistance service on each data node; the port numbers of each data assistance service are different; the port numbers are used to receive data sent by the corresponding data nodes in the source database.

[0151] In an exemplary embodiment, the above-mentioned policy determination module further includes: a second sending unit, where:

[0152] A second sending unit, configured to send a second service start instruction to the master node in the target database when the parallel migration policy is a migration policy between a data node and a master node; the second service start instruction is used to instruct the master node to start multiple data assistance services on the master node; the port number of each data assistance service is different; the port number is used to receive data sent by the corresponding data node in the source database.

[0153] In an exemplary embodiment, the above data migration device further includes: a master node policy determination module, a first sending module, and a second sending module, where:

[0154] The master node policy determination module is configured to determine that the migration policy of the currently to-be-migrated data table is a migration policy between master nodes when the data volume of the currently to-be-migrated data table is less than or equal to a preset threshold; the migration policy between master nodes represents the policy from the master node of the source database to the master node of the target database;

[0155] The first sending module is configured to send a third service start instruction to the master node in the target database; the third service start instruction is used to instruct the master node to start a data assistance service on the master node; the fixed port of the data assistance service is used to receive data sent by the master node in the source database;

[0156] The second sending module is configured to send a data migration instruction to the master node of the source data; the data migration instruction is used to instruct the master node to migrate the data in the currently to-be-migrated data table to the fixed port of the data assistance service.

[0157] Each module in the above data migration device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the computer device in the form of hardware or be independent of the processor, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above respective modules.

[0158] In an exemplary embodiment, a computer device is provided, including a memory and a processor, where a computer program is stored in the memory, and when the processor executes the computer program, the content of any one of the above data migration method embodiments is implemented.

[0159] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the content of any one of the above data migration method embodiments is implemented.

[0160] In an embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the content of any one of the above data migration method embodiments is implemented.

[0161] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0162] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0163] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0164] The above embodiments only express several implementation manners of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several deformations and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application shall be subject to the appended claims.

Claims

1. A data migration method, characterized in that, The method includes: Obtaining the data volume of the current data table to be migrated in the source database; the source database includes multiple data tables to be migrated; the data in each data table to be migrated is correspondingly distributed among multiple data nodes; When the data volume of the current data table to be migrated is greater than a preset threshold, based on the first mapping relationship between the data in the current data table to be migrated and the multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated in the target database and the multiple data nodes in the target database, determining the parallel migration strategy for the current data table to be migrated; According to the parallel migration strategy, sending a parallel migration instruction to the master node in the source database; the parallel migration instruction is used to instruct the master node to control the multiple data nodes to parallelly migrate the corresponding data to the target data table in the target database; Taking the next data table to be migrated among the multiple data tables to be migrated as the new current data table to be migrated to perform data migration until all the data tables to be migrated in the source database are migrated to the target database.

2. The method according to claim 1, characterized in that The determining the parallel migration strategy for the current data table to be migrated based on the first mapping relationship between the data in the current data table to be migrated and the multiple data nodes, and the second mapping relationship between the data in the target data table corresponding to the current data table to be migrated in the target database and the multiple data nodes in the target database includes: According to the first mapping relationship and the second mapping relationship, obtaining the number of source data nodes corresponding to the current data table to be migrated and the number of target data nodes corresponding to the target data table; Based on the number of source data nodes, the number of target data nodes, the first mapping relationship and the second mapping relationship, determining the parallel migration strategy for the current data table to be migrated.

3. The method according to claim 2, wherein The determining the parallel migration strategy for the current data table to be migrated based on the number of source data nodes, the number of target data nodes, the first mapping relationship and the second mapping relationship includes: Comparing the number of source data nodes with the number of target data nodes, and comparing the first mapping relationship with the second mapping relationship; According to the comparison result of the number of data nodes and the comparison result of the mapping relationship, determining the parallel migration strategy for the current data table to be migrated.

4. The method according to claim 3, characterized in that, The determining the parallel migration strategy for the current data table to be migrated according to the comparison result of the number of data nodes and the comparison result of the mapping relationship includes: When the comparison result of the number of data nodes and the comparison result of the mapping relationship are both the same, determining the parallel migration strategy for the current data table to be migrated as the migration strategy between the data nodes from the data nodes of the source database to the data nodes of the target database; In the case where the comparison result of the number of data nodes and / or the comparison result of the mapping relationship is different, determine that the parallel migration strategy of the currently to-be-migrated data table is the migration strategy between the data nodes of the source database and the master node of the target database.

5. The method according to claim 4, wherein Before sending a parallel migration instruction to the master node in the source database according to the parallel migration strategy, the method further includes: In the case where the parallel migration strategy is the migration strategy between data nodes, send a first service start instruction to the master node in the target database; the first service start instruction is used to instruct the master node to start a data assistance service on each data node; the port numbers of each data assistance service are different; the port number is used to receive data sent by the corresponding data node in the source database.

6. The method according to claim 4, wherein Before sending a parallel migration instruction to the master node in the source database according to the parallel migration strategy, the method further includes: In the case where the parallel migration strategy is the migration strategy between a data node and a master node, send a second service start instruction to the master node in the target database; the second service start instruction is used to instruct the master node to start multiple data assistance services on the master node; the port numbers of each data assistance service are different; the port number is used to receive data sent by the corresponding data node in the source database.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: In the case where the data volume of the currently to-be-migrated data table is less than or equal to a preset threshold, determine that the migration strategy of the currently to-be-migrated data is the migration strategy between master nodes; the migration strategy between master nodes represents the strategy from the master node of the source database to the master node of the target database; Send a third service start instruction to the master node in the target database; the third service start instruction is used to instruct the master node to start a data assistance service on the master node; the fixed port of the data assistance service is used to receive data sent by the master node in the source database; Send a data migration instruction to the master node of the source data; the data migration instruction is used to instruct the master node to migrate the data in the currently to-be-migrated data table to the fixed port of the data assistance service.

8. A data migration device, characterized in that, The device includes: An acquisition module, configured to acquire the data volume of the currently to-be-migrated data table in the source database; the source database includes multiple to-be-migrated data tables; the data in each to-be-migrated data table is correspondingly distributed on multiple data nodes; A strategy determination module, configured to, in the case where the data volume of the currently to-be-migrated data table is greater than a preset threshold, determine the parallel migration strategy of the currently to-be-migrated data table based on the first mapping relationship between the data in the currently to-be-migrated data table and the multiple data nodes and the second mapping relationship between the data in the target data table corresponding to the currently to-be-migrated data table in the target database and the multiple data nodes in the target database; An instruction sending module, configured to send a parallel migration instruction to a master node in the source database according to the parallel migration strategy; the parallel migration instruction is used to instruct the master node to control the multiple data nodes to parallelly migrate corresponding data to a target data table in the target database; A loop execution module, configured to use the next to-be-migrated data table among the multiple to-be-migrated data tables as a new current to-be-migrated data table to perform data migration until all the to-be-migrated data tables in the source database are migrated to the target database.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images