Data migration method and device, storage medium and processor

Abstract

The embodiment of the invention provides a data migration method and device, a storage medium and a processor, and relates to the technical field of databases. The method comprises the steps that in the full data migration process, the following incremental data pre-merging processing is executed, incremental data are continuously obtained, incremental data rows are analyzed, the incremental data rows comprise table names, DML types, primary key values and complete data row content, and the DML types comprise insertion, updating and / or deletion; combining the incremental data rows of the same DML type to obtain disk files corresponding to different DML types; and after the full migration of the data is completed, respectively carrying out parallel loading on the disk files corresponding to different DML types based on a preset type sequence. The method is used for improving the data migration efficiency so as to better meet the data migration requirement of a system with a large data volume.

Description

Technical Field

[0001] This application relates to the field of database technology, specifically to a data migration method, apparatus, storage medium, and processor. Background Technology

[0002] Database data migration generally refers to the process of exporting data from the source database and then writing the data into the target database. Common migration methods include full migration and full-incremental migration.

[0003] A full migration involves continuously querying and reading data from the source database at a specific point in time. If a transaction occurs in the database during this migration process, it cannot guarantee that the latest data of the transaction changes will be obtained. Therefore, to ensure the accuracy of the migrated data, a full migration often requires the database to be in a static, transaction-free state, which in a production system is almost equivalent to the moment of system downtime.

[0004] Full incremental migration does not require the source database to stop transactions. Instead, during the full migration process, log analysis technology is used to continuously capture and cache all transaction records that occur in the source database. After the full migration is completed, the cached incremental data is then played back to the target database in sequence, ensuring the accuracy of the migration data without shutting down the source database.

[0005] Online systems operate 24 / 7 to provide services, especially in the financial industry where guaranteed annual system availability is crucial, making it difficult to request significant downtime. Therefore, when dealing with small data volumes, assuming the migration can be completed within a tolerable business timeframe, a full migration approach can be chosen. This is essentially determined by the downtime the source database can support. More often, however, with larger data volumes or systems that cannot tolerate prolonged downtime, the migration might take several hours or even days to complete at a normal, stable migration speed. In such cases, a full-incremental migration is used. This involves migrating both full and incremental data without shutting down the source database. The migration process brings the target and source databases to near real-time synchronization. At this point, a brief downtime of the source database is possible. Once the target database completes the final incremental data write within the expected short period, the entire data migration is finished.

[0006] Data migration is an unconventional system operation mode. During this process, the source database, target database, and migration program operate under different system loads. The source database server, which is extracting data, often experiences high bandwidth consumption and increased I / O. The migration program and target database are both under intensive CPU and I / O conditions. Therefore, optimizing data migration time is crucial for the success of the migration project. Shorter time means a higher success rate and also gives engineers controlling the migration implementation greater confidence.

[0007] In open-source products, MySQL's parallel replication functionality shares some similarities with database data migration scenarios, but this approach cannot guarantee the original order and consistency of incremental data. Building upon MySQL's parallel replication functionality, parallel incremental data loading can be performed at the table level. For a single table, this ensures data consistency and the timing of data replay; globally, after multiple tables are loaded in parallel, the final data across all tables becomes consistent.

[0008] However, while the table-level parallel incremental data loading scheme can improve the overall loading speed of incremental data to some extent, it still has the following drawbacks from both a technical and practical perspective: 1. Loading parallelism depends on data distribution: Because incremental data loading is performed at the table level, when one (or several) tables contain a large proportion of data, parallel loading will complete the tables with smaller data volumes ahead of time. The remaining large tables will become the overall bottleneck, and the data loading parallelism will continuously decrease as more tables are processed. Normally, data follows a normal distribution, and the overall parallelism is lower than the number of tables (TABLE_NUM). In extreme cases, all incremental data may belong to a single table, and the solution will degenerate into a single-threaded serial process. Data distribution is an objective reality of the application system; in most cases, there will be a long tail of processing involving a small number of large tables.

[0009] 2. Parallelism Constraint: The parallelism of incremental data loading has a maximum value of the number of tables, TABLE_NUM. During the actual loading of the target database, the database's load capacity needs to be considered. Assuming the database can stably support L parallel DML operations, exceeding the parallelism limit L will cause database processing delays, potentially leading to a decrease in load capacity. When the number of tables, TABLE_NUM, is much smaller than L, it indicates low-parallel loading that cannot fully utilize database resources. When the number of tables, TABLE_NUM, is large and exceeds L, the parallelism cannot be directly set to TABLE_NUM (resulting in a slower loading speed) and needs to be analyzed in conjunction with the database load; it will often not exceed L.

[0010] 3. Low Single-Table Data Loading Performance: When faced with a large number of incremental DML operations on a single table, loading them row by row sequentially into the database results in low overall performance. As in the example above, the normal execution time for each incremental DML operation is approximately 10 milliseconds. When migrating a system with hundreds of millions of rows in a single table, a full migration of 2TB of data would require 24 hours. The migration process caches all incremental data from a single day for the system. When the number of incremental DML operations on a single table exceeds 3 million, loading would take 8 hours. The total amount of data being migrated and the system's TPS will cause a linear increase in the incremental data volume, making the low single-table incremental DML loading performance a bottleneck for parallel loading.

[0011] Therefore, although existing technologies have improved the efficiency of data migration, they are still insufficient to meet the data migration needs of systems with large data volumes. Summary of the Invention

[0012] The purpose of this application is to provide a data migration method, apparatus, storage medium, and processor to improve the efficiency of data migration and better meet the data migration needs of systems with large data volumes.

[0013] To achieve the above objectives, the first aspect of this application provides a data migration method, comprising: during a full data migration process, performing the following incremental data pre-merging process: continuously acquiring incremental data and parsing incremental data rows, wherein the incremental data rows include table name, DML type, primary key value, and complete data row content, wherein the DML type includes insert, update, and / or delete; merging incremental data rows of the same DML type to obtain disk files corresponding to different DML types; and after the full data migration is completed, loading the disk files corresponding to different DML types in parallel based on a preset type order.

[0014] In this embodiment, merging incremental data rows of the same DML type to obtain disk files corresponding to different DML types includes performing the following processing on incremental data rows of DML type "insertion": storing the acquired incremental data rows in a first memory list object; determining whether the storage size of the first memory list object has reached a first storage threshold; if it has not reached the first storage threshold, continuing to acquire the incremental data rows; if it has reached the first storage threshold, writing all data of the first memory list object into a first disk file and determining whether the size of the first disk file has reached the first disk threshold; and if it has reached the first disk threshold, creating a new first disk file and writing to the new first disk file during the next write operation.

[0015] In this embodiment, merging incremental data rows of the same DML type to obtain disk files corresponding to different DML types includes performing the following processing on incremental data rows of the DML type "Update": obtaining the first primary key value of the incremental data row; determining whether the first primary key value is contained in a second memory list object; if it is contained, deleting the incremental data row corresponding to the first primary key value in the second memory list object; determining whether an index with the primary key value of the first primary key value is contained in an index list object; if it is contained, searching for the data row corresponding to the first primary key value in the disk file of the DML type "Update" according to the index; if found, processing the data row corresponding to the first primary key value. The process involves: deleting the data and removing the index from the index list object; storing the incremental data row in a second memory list object; determining whether the storage size of the second memory list object has reached a second storage threshold; if it has not reached the second storage threshold, continuing to acquire the incremental data row; if it has reached the second storage threshold, writing all the data from the second memory list object to a second disk file and determining whether the size of the second disk file has reached the second disk threshold; if it has reached the second disk threshold, creating a new second disk file and writing to the new second disk file during the next write operation; and writing the index of the incremental data row into the index list object.

[0016] In this embodiment, merging incremental data rows of the same DML type to obtain disk files corresponding to different DML types includes performing the following processing on incremental data rows of the DML type "delete": storing the acquired incremental data rows in a third memory list object; determining whether the storage size of the third memory list object has reached a third storage threshold; if it has not reached the third storage threshold, continuing to acquire the incremental data rows; if it has reached the third storage threshold, writing all the data of the third memory list object into a third disk file and determining whether the size of the third disk file has reached the third disk threshold; and if it has reached the third disk threshold, creating a new third disk file and writing it to the new third disk file during the next write operation.

[0017] In this embodiment of the application, after the full data migration is completed, the disk files corresponding to different DML types are loaded in parallel based on a preset type order, including: after the full data migration is completed, an incremental file composed of the disk files is obtained, wherein the filename of each disk file in the incremental file contains its corresponding DML type; and the disk files with corresponding filenames in the incremental file are read in parallel in the order of insertion, update and deletion, and the reading results are submitted to the database.

[0018] In this embodiment, parallel reading of disk files with corresponding filenames in the incremental file and submitting the reading results to the database includes performing the following processing on disk files of DML type "insert": reading the set of filenames in the incremental file whose filenames contain the inserted disk files, submitting them to a worker thread group, wherein the number of threads in the thread group is set based on database load tuning; all threads in the worker thread group perform parallel processing, wherein each thread performs the following processing on each allocated disk file: reading each incremental data row of the disk file; and performing DML merging on the reading results based on a preset first optimal batch step size, and submitting the merged results to the database.

[0019] In this embodiment, parallel reading of disk files with corresponding filenames in the incremental file and submitting the reading results to the database includes performing the following processing on disk files of DML type update: reading the set of filenames in the incremental file whose filenames contain updated disk files, and submitting them to a worker thread group, wherein the number of threads in the thread group is set based on database load tuning; all threads in the worker thread group perform parallel processing, wherein each thread performs the following processing on each allocated disk file: reading each incremental data row of the disk file; and sending multiple read results to the database at once in batch processing mode based on a preset second optimal batch step size.

[0020] In this embodiment, parallel reading of disk files with corresponding filenames in the incremental file and submitting the reading results to the database includes performing the following processing on disk files with DML type "delete": reading a set of filenames in the incremental file whose filenames contain the deleted disk files, submitting them to a worker thread group, wherein the number of threads in the thread group is set based on database load tuning; all threads in the worker thread group perform parallel processing, wherein each thread performs the following processing on each allocated disk file: reading each incremental data row of the disk file; and performing DML merging on the reading results based on a preset first optimal batch step size, and submitting the merged results to the database.

[0021] In this embodiment of the application, the method further includes: after the full data migration is completed, continuing to perform the incremental data pre-merging process until the first condition is met, and then loading the disk files corresponding to different DML types in parallel based on a preset type order, wherein the first condition includes receiving an incremental migration instruction, the source database TPS entering a trough or the source database TPS entering a preset downtime interval.

[0022] A second aspect of this application provides a data migration apparatus, comprising: a data pre-merging module, configured to perform the following incremental data pre-merging processing during a full data migration process: continuously acquiring incremental data and parsing incremental data rows, wherein the incremental data rows include table names, DML types, primary key values, and complete data row content, wherein the DML types include insert, update, and / or delete; merging incremental data rows of the same DML type to obtain disk files corresponding to different DML types; and a parallel loading module, configured to perform parallel loading of disk files corresponding to different DML types based on a preset type order after the full data migration is completed.

[0023] A third aspect of this application provides a processor configured to perform the data migration method described above.

[0024] A fourth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the data migration method described above.

[0025] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the data migration method described above.

[0026] The method proposed in this invention includes two steps: full data migration and incremental data loading and reordering. During the full data migration, incremental data is merged without boundaries, significantly reducing the number of times incremental data is written to the database. In the incremental data loading and reordering, data is loaded in parallel according to a specified DML type order. The degree of parallelism is independent of data distribution and is not constrained by the number of tables, allowing selection of the optimal parallelism for the database, which helps to fully utilize the database's full load capacity. Furthermore, during the full migration, pre-merging processing of continuously increasing incremental data can be completed in near real-time. Once the full migration is complete, parallel loading of the newly sorted incremental data is initiated directly, thereby maximizing the savings in overall migration time and improving the overall efficiency of data migration.

[0027] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0028] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 This illustration schematically shows an application environment diagram of the data migration method according to an embodiment of this application; Figure 2The illustration shows a schematic flowchart of a data migration method according to an embodiment of this application; Figure 3 This illustration schematically depicts a process diagram of full data migration according to an embodiment of this application; Figure 4 This illustration schematically depicts a process diagram of incremental data loading and reordering according to an embodiment of this application; Figure 5 This schematic diagram illustrates a structural block diagram of a data migration apparatus according to an embodiment of the present application; Figure 6 The diagram illustrates the internal structure of a computer device according to an embodiment of this application.

[0029] Explanation of reference numerals in the attached figures 400 - Data Migration Device; 402 - Data Pre-merging Module; 404 - Parallel loading of modules. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0031] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0032] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0033] The acquisition, transmission, storage, use, and processing of data in this application comply with relevant laws and regulations. Furthermore, it should be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0034] It's important to note that the MySQL parallel replication mentioned above does not refer to the schema-level parallel replication in earlier versions of MySQL 5.6. Schema-level parallel replication, in essence, seeks a coarse-grained parallel dimension, requiring data across different databases to be parallel. In data migration scenarios, multiple schemas may not exist, and even multiple schemas cannot guarantee data balance, resulting in relatively low efficiency. MySQL group-based parallel replication (LOGICAL_CLOCK parallel replication mode) marks group information in the binlog log when transactions are committed on the master database. This indicates that multiple transactions belonging to the same group do not conflict, allowing transactions within a group commit to be replayed in parallel on the standby database. This feature identifies consecutive, conflict-free DML operations from a continuous incremental data stream and loads them in parallel using multi-threading on the target database, improving performance.

[0035] However, the incremental data generated during data migration optimizes the migration of already accumulated incremental data, which differs from the near real-time synchronization scenario of database master and slave nodes. The parallel replication technology of the open-source database MySQL addresses the latency in near real-time synchronization, and group commit is used for transaction grouping within a short real-time window. The incremental data collection program cannot execute logic similar to how the MySQL master groups transaction sets (MySQL determines conflicts between transactions), nor can it group large amounts of already accumulated incremental data. Therefore, MySQL's parallel replication technology cannot be used to handle incremental data synchronization in data migration scenarios.

[0036] The method described above refers to breaking the original order of two incremental DML operations that originally had a sequential sequence, loading them simultaneously during program execution, which cannot guarantee the original order and the consistency of the incremental data. For example, if operation 1 modifies the value to A, and operation 2, following operation 1, modifies the value to B, the final value is B. If operation 2 is executed first and operation 1 is executed later in parallel execution, the final result will be the incorrect result A. Therefore, the above-described MySQL parallel replication function cannot meet the scenario of this invention.

[0037] From a technical perspective, the aforementioned MySQL parallel replication function also provides a method for accelerating the processing of incremental data. This can be achieved by selecting a dimension for the incremental data that does not affect data consistency and loading it in parallel. Choosing a table-level dimension with a lower granularity than the schema allows for parallel incremental data loading at the table level. For a single table, this ensures data consistency and the timing of data replay. Globally, after multiple tables are loaded in parallel, the final data across all tables becomes consistent.

[0038] For example, the original serial single-threaded sequence is: Table01-DML01 —> Table01-DML02 —> Table02-DML03 —> Table01-DML04 —> Table02-DML05.

[0039] Table01-DML01 represents an incremental DML operation on Table01.

[0040] The optimized structure is as follows, with parallel processing at the table level: Thread 1: Table01-DML01 —> Table01-DML02 —> Table01-DML04.

[0041] Thread 2: Table02-DML03 → Table02-DML05.

[0042] It is intuitive to see that the original 5 database operations are reduced to 3. When there are TABLE_NUM tables, and ideally each table has the same number of DML operations, under the premise that the database load is not under pressure and is not a bottleneck, it is equivalent to dividing the incremental data DATA_SIZE (REAL) into TABLE_NUM parts, which can reduce the processing time to about 1 / TABLE_NUM of the original.

[0043] However, table-level parallel incremental data loading schemes still suffer from problems such as loading parallelism depending on data distribution, parallelism constraints, and low performance when loading data from a single table. To address these issues, this invention proposes a data migration method.

[0044] The data migration method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Based on the instructions from terminal 102, server 104 performs data migration. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0045] For the full migration phase, assuming the amount of data to be migrated is DATA_SIZE(FULL), and BPS_T(FULL) is the amount of data migrated to the target database per second, then the time required for the full migration is T(FULL) = DATA_SIZE(FULL) / BPS_T(FULL). This formula omits the description of data extraction from the source database because the data extraction speed from the source database is often several times faster than the writing speed to the target database. The key full migration metric BPS(FULL) requires attention to the target database performance and related performance tuning parameters. Since the data is a single INSERT operation involving large amounts of data across multiple tables, there are various ways to optimize it, such as the method of writing data to the target database and the data batch size.

[0046] Define BPS_S(REAL) as the average amount of real-time incremental data cached per second during the full phase. Then, the incremental data DATA_SIZE(REAL) during the full phase = BPS_S(REAL) × T(FULL). The incremental data is in the form of INSERT, UPDATE, and DELETE statements (hereinafter referred to as incremental DML) for different data in different tables. The data is irregular, and batch merging operations cannot be performed when replaying incremental data. It is necessary to load each row of data into the target database.

[0047] For example, consider an application system with a TPS of 100. The original system is migrating 200GB of data at a full migration speed of 80GB / hour. During the non-stop migration, incremental transactions are continuously generated, with each transaction containing the addition (or modification, deletion) of 4 rows of data. Assuming the operation type ratio is INSERT:UPDATE:DELETE = 5:4:1, then 100 × 4 = 400 DML operations will be generated every second (200 INSERT, 160 UPDATE, and 40 DELETE). The full migration time is 200 / 80 = 2.5 hours. The incremental data generated during the full migration phase is 400 × 2.5 × 3600 = 3.6 million. Each incremental DML operation is sent to the target database, with a normal execution time of approximately 10 milliseconds. Therefore, executing 3.6 million incremental DML operations sequentially will take 3,600,000 × 10 / (1000 × 3600) = 10 hours. At this point, the incremental data migration time has exceeded the full data migration time, becoming the bottleneck of the overall migration time.

[0048] The incremental data size (DATA_SIZE(REAL)) is affected by two metrics: the full data size (DATA_SIZE(FULL)) and the incremental data production rate (BPS_S(REAL)). DATA_SIZE(FULL) is positively correlated; a sufficiently long full data migration phase results in more incremental data. Common migration practice dictates choosing a time when system TPS is low to minimize DATA_SIZE(REAL). However, these two metrics are objective to the application system. Even if the time range with the lowest BPS_S(REAL) is chosen, if the overall full data migration time is too long, there will be a period where the system enters a normal TPS range. For example, in the case of migrating tens of terabytes of data, if the full data migration phase spans several days, and the incremental phase also takes several days, the migration implementation will be extremely difficult. Since it's impossible to prevent the migration system from generating a sufficient amount of incremental data through meticulous planning, optimizing incremental data migration becomes a key technology in data migration. Therefore, the method proposed in this invention aims to improve the performance of loading a certain amount of accumulated incremental data generated during the migration phase into the target database, thereby shortening the overall migration time.

[0049] Figure 2 A schematic flowchart illustrating a data migration method according to an embodiment of this application is shown. Figure 2 As shown, in one embodiment of this application, a data migration method is provided. This embodiment mainly applies this method to the above-mentioned... Figure 1 Taking server 104 as an example, the following steps are included: Step 202: During the full data migration process, perform incremental data pre-merging processing.

[0050] Step 204: After the full data migration is completed, disk files corresponding to different DML types are loaded in parallel based on a preset type order.

[0051] The data merging process includes the following steps: Step 2021: Continuously acquire incremental data and parse out incremental data rows. The incremental data rows include table name, DML type, primary key value and complete data row content. The DML type includes INSERT, UPDATE and / or DELETE.

[0052] Step 2022: Merge incremental data rows of the same DML type to obtain disk files corresponding to different DML types.

[0053] In incremental data migration scenarios, analyzing the source database transaction logs reveals three types of DML operations: INSERT, UPDATE, and DELETE. The WHERE clauses of UPDATE and DELETE statements are based on the primary key (or unique index), and INSERT statements also include the primary key column and its value. This is the standard implementation of incremental log analysis technology. The table being incrementally migrated must have a primary key, which is typically of numeric or character data type and can be a single column or a combination of multiple columns. The acquisition of incremental data is continuous during the migration process, and the amount of data cached per unit time depends on the transaction TPS of the source database.

[0054] The method proposed in this invention includes two steps: full data migration and incremental data loading and reordering. During the full data migration, incremental data is merged without boundaries. By grouping the insert, update, and delete DML operations of each table in the incremental data, each table is divided into three DML type data groups. Simultaneously, DML operations of the same type are merged; for example, multiple INSERT operations are merged into a single batch INSERT set, i.e., a disk file. This merging significantly reduces the number of times incremental data is written to the database. The incremental data pre-merging process is performed on the entire data set, independent of custom grouping markers, and independent of whether the same operation is continuous within the incremental DML operations; it does not have the inherent boundary concept of a certain batch of data.

[0055] In incremental data loading and reordering, the merged batch DML sets of each table are loaded in parallel according to the specified DML type order. The degree of parallelism does not depend on the data distribution and is not constrained by the number of tables. The optimal parallelism of the database can be selected, which helps to fully utilize the database's full load capacity.

[0056] Through the above steps, while ensuring eventual consistency of incremental data, a unified loading order for incremental data is achieved by changing the loading order of each incremental DML operation. Compared with the original data where various DML types were arranged in an interleaved order, this ensures that each database load is a highly efficient batch dataset operation, greatly improving the performance of single-table data loading.

[0057] Furthermore, the method proposed in this invention is divided into two stages, with the end of the first stage marking the beginning of the second. The timing of merging and sorting incremental data coincides precisely with the full migration phase, ending upon completion of the full migration. For continuously generated incremental data, the original cached incremental data is now stored in the merged and sorted data format according to the improved method of this invention. Utilizing the full migration time, pre-merging processing of continuously increasing data can be completed in near real-time. Once the full migration is complete, parallel loading of the newly sorted incremental data is initiated directly, thereby maximizing the savings in overall migration time and improving the overall efficiency of data migration.

[0058] Figure 2 This is a flowchart illustrating a data migration method in one embodiment. It should be understood that, although... Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0059] For the same table, the merging process differs depending on the DML type. Figure 3 This illustration schematically depicts a full data migration process according to an embodiment of this application, such as... Figure 3 As shown, merging incremental data rows of the same DML type to obtain disk files corresponding to different DML types includes performing the following processing on incremental data rows of DML type "Insert": storing the acquired incremental data rows in a first memory list object "InsertRowList"; determining whether the storage size of the first memory list object has reached a first storage threshold; if it has not reached the first storage threshold, continuing to acquire incremental data rows; if it has reached the first storage threshold, writing all data of the first memory list object to a first disk file and determining whether the size of the first disk file has reached the first disk threshold; and if it has reached the first disk threshold, creating a new first disk file and writing to the new first disk file in the next write operation.

[0060] By processing incremental data rows of type "Insert" using the above methods, frequent single-row I / O operations are avoided, improving the efficiency of data storage to disk. For disk files of type "Insert" using DML, the filename is prefixed with "INSERT_" and indicates the DML type. Setting a first disk size threshold for file size facilitates control over the number of files per second (TPS) across different source databases. This value can be set according to actual conditions, for example, between 50MB and 100MB, to prevent subsequent multi-threaded loading of file data into memory and excessive memory consumption.

[0061] In DML operations on the same table, DML operations with the same primary key value belong to the same row of data, while DML operations with different primary key values ​​belong to different rows of data. When merging incremental data rows with the DML type "update," the merging of DML operations of the same type belonging to the same row of data within the same table is performed. It should be noted that this invention does not focus on primary key type. Specifically, merging incremental data rows of the same DML type to obtain disk files corresponding to different DML types includes performing the following processing on incremental data rows with the DML type "update": obtaining the first primary key value of the incremental data row; determining whether the first primary key value is contained in a second memory list object; if contained, deleting the incremental data row corresponding to the first primary key value in the second memory list object; determining whether an index with the primary key value of the first primary key value is contained in an index list object; if contained, searching for the data row corresponding to the first primary key value in the disk file with the DML type "update" based on the index; if found, deleting the data row corresponding to the first primary key value. The process involves: removing the index from the index list object; storing the incremental data row in a second memory list object; determining whether the storage size of the second memory list object has reached a second storage threshold; if it has not reached the second storage threshold, continuing to acquire the incremental data row; if it has reached the second storage threshold, writing all the data of the second memory list object to a second disk file and determining whether the size of the second disk file has reached the second disk threshold; if it has reached the second disk threshold, creating a new second disk file and writing to the new second disk file during the next write operation; and writing the index of the incremental data row into the index list object.

[0062] Specifically, a HashMap object `updatePkNestingMap` is created in memory for indexing; this is the index list object, and its structure is two-level. <tableName,<pkValue,updateFileName> The inner Map retrieves the primary key value pkValue and the storage file name key value updateFileNam through the table name tableName. The inner Map then uses the primary key value to determine whether the same row of data for the incremental data row being updated has already been stored in the disk file of DML type update. The two-level structure grouped by table improves the efficiency of determining whether each primary key value already exists. The execution time is in the microsecond range for data of hundreds of millions.

[0063] Please refer to Figure 3In practice, before storing the acquired incremental data rows into the second memory list object `UpdateRowList`, for each incremental data row of type `Update` (DML type `Update`), it is determined whether there is a previous identical row. First, this is checked in the second memory list object `InsertRowList`; if found, it is removed. Then, it searches in `updatePkNestingMap`. If it exists, the data row with that `pk` value is deleted from the corresponding disk file, and the key-value pair of that primary key and filename is removed from `updatePkNestingMap`. Then, the acquired incremental data rows are stored in the second memory list object `UpdateRowList`. When the number of incremental DML rows stored in the second memory list object `UpdateRowList` reaches a set second storage threshold, all data rows in the second memory list object `UpdateRowList` are written to the disk file of type `Update` (DML type `Update`), with the filename prefixed with "UPDATE_" to indicate the DML type. Simultaneously, all data rows are written to the index `updatePkNestingMap`. When the file size exceeds the set size, the logic for handling DML type `Insert` is the same as described above.

[0064] Through the above processing, for updates that occur repeatedly in the same row of data, the last update is used as the final value. This method reduces the overall number of incremental DML operations, especially in scenarios where frequently updated hot data occurs. For example: UPDATE TABLE SET COLUMNS =values1 WHERE PK=pk1; UPDATE TABLE SET COLUMNS =values2 WHERE PK=pk1; Merge into: UPDATE TABLE SET COLUMNS =values2 WHERE PK=pk1.

[0065] As shown above, to determine whether the cached incremental DML contains duplicate rows for each UPDATE operation, a collection of primary key values ​​needs to be constructed and reside in program memory for real-time determination of duplicate rows. It is estimated that a collection of ten million primary key values ​​would occupy approximately 1.4GB of memory, and multiplied by this to one hundred million, or 14GB. Therefore, this is not a significant burden for modern server configurations.

[0066] Furthermore, merging incremental data rows of the same DML type to obtain disk files corresponding to different DML types includes performing the following processing on incremental data rows of the DML type "delete": storing the acquired incremental data rows in a third memory list object; determining whether the storage size of the third memory list object has reached a third storage threshold; if it has not reached the third storage threshold, continuing to acquire the incremental data rows; if it has reached the third storage threshold, writing all the data of the third memory list object into a third disk file and determining whether the size of the third disk file has reached the third disk threshold; and if it has reached the third disk threshold, creating a new third disk file and writing to the new third disk file in the next write operation.

[0067] Unlike updates, the above merging is not necessary for incremental data rows with DML type "delete". For example: DELETE FROM TABLE WHERE PK=pk1; DELETE FROM TABLE WHERE PK=pk1, DELETE statements for the same row of data are equivalent to each other, and the DELETE operation executed later has no effect on batch deletion, so it can be directly written to the disk file.

[0068] Please refer to Figure 3 In practice, the acquired incremental data rows are stored in a third memory list object, DeleteRowList. When the number of incremental DML rows stored in DeleteRowList reaches a set third disk threshold, all data rows in DeleteRowList are written to the corresponding disk file. The filename is prefixed with "DELETE_" to indicate the DML type. When the file size exceeds the set size, the same logic applies to inserting DML rows.

[0069] Furthermore, after the full data migration is completed, based on a preset type order, the disk files corresponding to different DML types are loaded in parallel, including: after the full data migration is completed, an incremental file composed of the disk files is obtained, wherein the filename of each disk file in the incremental file contains its corresponding DML type; and the disk files with corresponding filenames in the incremental file are read in parallel in the order of insertion, update and deletion, and the reading results are submitted to the database.

[0070] The method proposed in this invention loads incremental data in parallel according to DML type order, rather than by table granularity. This parallel loading method is key to improving parallelism. During the incremental data pre-merging process, incremental data in all tables is already classified into three categories: insert, update, and delete. These are then uniformly sorted according to the order of insert, update, and delete. Insert operations on all tables are executed in parallel first, followed by updates, and finally deletes. This sorting by DML type removes the inherent sequential order of the original incremental DML operations, eliminating any serial loading requirements for all DML operations of the same type. This allows all DML operations of the same type to be parallelized, and the loading parallelism is independent of the number of tables, with no upper limit constraint when database load capacity is not considered.

[0071] DML type sorting alters the execution order of all DML operations (within the same table or across different tables), necessitating ensuring eventual data consistency after reordering. For DML operations on different tables, changing the execution order does not affect the eventual consistency of the table data; that is, DML operations on clearly different tables can be performed concurrently. Similarly, for DML operations on different rows of data within the same table, changing the execution order also does not affect the eventual consistency of the data; that is, DML operations on different rows of the same table can be performed concurrently. However, for DML operations on the same row of data within the same table, the actual transaction operations of the application database should be used as the reference benchmark. Incremental data should be interpreted as data generated after the source database has normally processed the transactions submitted by the application, with each increment representing an exact data change in the source database.

[0072] During migration, no primary key value will be reused. This adheres to database primary key usage guidelines: firstly, primary key values ​​must be unique within the table, a database requirement; secondly, in mainstream databases, primary keys can use auto-incrementing sequences or be generated by the application through custom code to ensure that each generated primary key value is unique. Therefore, it is assumed that database primary keys will not be reused under normal circumstances. Although manually correcting data may result in previously used primary key values ​​being written to the database, this special case is not considered in the data migration scenario. Furthermore, under high concurrency, applications may submit identical DML operations or DML operations that do not change the data and are executed without modification. The database guarantees that incremental DML operations on the same row of data will not affect the correctness of these operations.

[0073] Based on the above premise, in incremental data processing, for DML operations on the same row of data in the same table, the following conditions must be met: (1) Since the primary key value will not be reused, there will be no two INSERTs.

[0074] (2) If UPDATE or DELETE appears first, it means that the primary key value exists in the full data and INSERT will not appear again.

[0075] (3) If INSERT appears first, it means that the primary key value does not exist in the full data. UPDATE or DELETE may appear later.

[0076] (4) Multiple UPDATE statements can exist, indicating continuous updates, while only one DELETE statement normally exists.

[0077] (5) DELETE is the final operation for a row of data. Since the original data row has been deleted, there will be no UPDATE afterward. At this time, the UPDATE is a pseudo operation. There will also be no INSERT afterward because, based on the correctness of D above, the row with the primary key value has already been included in the full data.

[0078] The above conditions can be summarized in Table 1, where INSERT is abbreviated as I, UPDATE as U, and DELETE as D. "I->U" means that I is executed first, and then U is executed.

[0079] Table 1. DML order of identical rows in the same table

[0080] For non-existent sequences, there's no need to consider reordering or eventual consistency of the sorted data. After filtering, the remaining 5 correct DML sequence combinations are: I->U can be generalized to I->U1->…->UN.

[0081] I->D U->U1 can be generalized to U->…->UN.

[0082] U->D can be generalized to U->…->UN->D.

[0083] I->U->D can be generalized to I->U->…->UN->D.

[0084] For the five sorting methods and their generalized forms of DML operations on the same row of data in the same table, it can be considered that regardless of the sorting, it can be regarded as the order of insertion, update, and deletion. Therefore, the method proposed in this invention performs parallel loading of all incremental DML operations in the order of insertion, update, and deletion, whether in the same or different tables, or in the same or different rows of the same table, thereby ensuring the eventual consistency of data loading.

[0085] Figure 4 This illustration schematically depicts a flowchart of incremental data loading and reordering according to an embodiment of this application, such as... Figure 4As shown, parallel reading of disk files with corresponding filenames in the incremental file and submission of the reading results to the database includes performing the following processing on disk files of DML type "insert": reading the set of filenames in the incremental file whose filenames contain the inserted disk files, submitting them to a worker thread group, where the number of threads in the thread group is set based on database load tuning, and the number of threads is the overall parallelism; all threads in the worker thread group perform parallel processing, wherein each thread performs the following processing on each allocated disk file until all rows of the disk file are read, a single disk file is processed, and the next file is allocated for processing: reading each incremental data row of the disk file; and performing DML merging on the reading results based on a preset first optimal batch step size, and submitting the merged results to the database.

[0086] For incremental data rows with DML type "insert", the main approach is to use SQL standards for DML merging, for example: INSERT INTO TABLE (PK,COLUMNS) VALUES (pk1,value1); INSERT INTO TABLE (PK,COLUMNS) VALUES (pk2,value2); ... INSERT INTO TABLE (PK,COLUMNS) VALUES (pkN,valueN); Merge into: INSERT INTO TABLE (PK, COLUMNS) VALUES (pk1,value1),(pk2,value2),…(pkN,valueN).

[0087] Here, N is the first optimal batch step size, which can be set according to the actual situation. Due to the limited SQL length that different databases can execute, a safe value for N is generally between 500 and 1000. In addition, different databases have different processing capabilities for SQL of different lengths. For example, for databases that prefer small transactions and high concurrency, N can be set smaller, and vice versa.

[0088] Furthermore, parallel reading of disk files corresponding to the filenames in the incremental file and submitting the reading results to the database includes performing the following processing on disk files of DML type update: reading the set of filenames in the incremental file whose filenames contain updated disk files, and submitting them to a worker thread group, the number of threads in the thread group being set based on database load tuning; all threads in the worker thread group performing parallel processing, wherein each thread performs the following processing on each allocated disk file: reading each incremental data row of the disk file; and sending multiple read results to the database at once in batch processing mode based on a preset second optimal batch step size.

[0089] For incremental data rows of DML type "update", merging cannot be performed at the SQL level, and multiple UPDATE statements cannot be merged into a single UPDATE. Therefore, this invention uses a batch processing method to send multiple UPDATE statements to the database for execution at once. The second optimal batch step size can be set according to actual conditions.

[0090] Furthermore, parallel reading of disk files with corresponding filenames in the incremental file and submitting the reading results to the database includes performing the following processing on disk files with DML type "delete": reading the set of filenames in the incremental file whose filenames contain the deleted disk files, submitting them to a worker thread group, the number of threads in the thread group being set based on database load tuning; all threads in the worker thread group performing parallel processing, wherein each thread performs the following processing on each allocated disk file: reading each incremental data row of the disk file; and performing DML merging on the reading results based on a preset first optimal batch step size, and submitting the merged results to the database.

[0091] For incremental data rows with DML type "delete", the SQL standard is also used, where PK is the primary key column of the table: DELETE FROM TABLE WHERE PK=pk1; DELETE FROM TABLE WHERE PK=pk2; ... DELETE FROM TABLE WHERE PK=pkN; Merge into: DELETE FROM TABLE WHERE PK IN (pk1, pk2,...pkN).

[0092] In the second phase described above, data is loaded in parallel according to DML type. Throughout the process, some worker threads will be idle only if the order of DML types is too short, such as when the number of data files for a certain type of DML is less than the number of worker threads. The parallelism of incremental data loading is not constrained by databases, tables, or other artificial logical groupings. The most suitable parallelism based on the database's load capacity can be directly selected. The parallelism can remain stable and undiminished throughout the incremental migration process, which is the theoretical maximum value under performance tuning of the target database during data migration.

[0093] According to the above processing flow, completing a full migration is a prerequisite for moving from the first stage to the second stage. In some embodiments of the present invention, after the full migration is completed, the second stage may not be started immediately. Instead, pre-merging processing of the collected incremental data continues until a suitable window is reached. Then, based on a preset type order, disk files corresponding to different DML types are loaded in parallel. A suitable window is, for example, when the source database TPS is at a low point or during a manually set downtime period. The user is given the option to initiate the second stage of parallel loading, which is then activated only after receiving an incremental migration instruction, thereby improving the flexibility of data migration.

[0094] The data migration method proposed in this invention reorders incremental data loading according to DML types, breaking the temporal relationship between the original data. This ensures that regardless of whether the data distribution in the database is uniform at the table level or concentrated in large tables, no large data tables become long-tail bottlenecks throughout the incremental migration cycle. By merging DML operations, the number of incremental DML operations can be reduced to a certain extent, and each database write completes hundreds of times more incremental data migration, while also reducing the number of database writes. This significantly improves data loading performance per unit time, shortening the incremental migration time from several hours to minutes. The data pre-merging and loading reordering acceleration scheme in this invention is integrated into the incremental migration technology solution and can be universally applied to any data throughout the entire incremental migration process, without any additional time, local, or data-related requirements, thus achieving a global improvement in incremental migration.

[0095] In one embodiment, such as Figure 5 As shown, a data migration device is provided, including a data pre-merging module and a parallel loading module, wherein: The data pre-merging module 402 is used to perform the following incremental data pre-merging process during the full data migration: continuously acquire incremental data and parse out incremental data rows, wherein the incremental data rows include table name, DML type, primary key value and complete data row content, wherein the DML type includes insert, update and / or delete; merge incremental data rows of the same DML type to obtain disk files corresponding to different DML types.

[0096] The parallel loading module 404 is used to load disk files corresponding to different DML types in parallel based on a preset type order after the full data migration is completed.

[0097] The data migration device includes a processor and a memory. The aforementioned data pre-merging module and parallel loading module are stored in the memory as program units, and the processor executes the aforementioned program modules stored in the memory to implement the corresponding functions.

[0098] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and data migration methods are implemented by adjusting kernel parameters.

[0099] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0100] This application provides a computer program product, including a computer program, characterized in that the computer program implements the above-described data migration method when executed by a processor.

[0101] This application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the data migration method described above.

[0102] This application provides a processor for running a program, wherein the program executes the above-described data migration method during runtime.

[0103] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor A01, a network interface A02, memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The database stores the data to be migrated. The network interface A02 communicates with external terminals via a network connection. When the computer program B02 is executed by the processor A01, it implements a data migration method.

[0104] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0105] In one embodiment, the data migration apparatus provided in this application can be implemented as a computer program, and the computer program can be implemented as follows: Figure 6 It runs on the computer device shown. The computer device's memory can store the various program modules that make up the data migration device, for example, Figure 5 The data pre-merging module and parallel loading module are shown. The computer program, composed of these various program modules, causes the processor to execute the steps in the data migration methods of the various embodiments of this application described in this specification.

[0106] Figure 6 The computer equipment shown can be used as follows Figure 5 The data pre-merging module in the data migration apparatus shown performs step 202. The computer device can perform step 204 via the parallel loading module.

[0107] This application provides an apparatus including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the data migration methods of various embodiments.

[0108] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform steps in the data migration methods of various embodiments.

[0109] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0114] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0115] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0116] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0117] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data migration method, characterized by, The method comprises: In the data full migration process, the following incremental data pre-merging processing is performed: Continuously acquiring incremental data and parsing incremental data rows, the incremental data rows comprising table names, DML types, primary key values and complete data row contents, the DML types comprising insertion, update and / or deletion; Merging the incremental data rows of the same DML type to obtain disk files corresponding to different DML types; and After the data full migration is completed, based on a preset type order, the disk files corresponding to different DML types are loaded in parallel.

2. The method of claim 1, wherein, Merging the incremental data rows of the same DML type to obtain disk files corresponding to different DML types comprises performing the following processing on the incremental data rows of the DML type of insertion: Storing the acquired incremental data rows in a first memory list object; Judging whether the storage amount of the first memory list object reaches a first storage threshold; If the first storage threshold is not reached, continuously acquiring the incremental data rows, if the first storage threshold is reached, writing all data of the first memory list object into a first disk file, and judging whether the size of the first disk file reaches a first disk threshold; And If the first disk threshold is reached, establishing a new first disk file, and writing into the new first disk file in the next writing.

3. The method of claim 1, wherein, Merging the incremental data rows of the same DML type to obtain disk files corresponding to different DML types comprises performing the following processing on the incremental data rows of the DML type of update: Acquiring a first primary key value of the incremental data row; Judging whether the first primary key value is contained in a second memory list object, if yes, deleting the incremental data row corresponding to the first primary key value in the second memory list object; Judging whether an index with the first primary key value is contained in an index list object, if yes, according to the index, searching for the data row corresponding to the first primary key value in the disk file of the DML type of update, if found, deleting the data row corresponding to the first primary key value, and deleting the index in the index list object; Storing the incremental data row in a second memory list object; Judging whether the storage amount of the second memory list object reaches a second storage threshold; If the second storage threshold is not reached, continuously acquiring the incremental data rows, if the second storage threshold is reached, writing all data of the second memory list object into a second disk file, and judging whether the size of the second disk file reaches a second disk threshold; If the second disk threshold is reached, establishing a new second disk file, and writing into the new second disk file in the next writing; And Writing the index of the incremental data row in the index list object.

4. The method of claim 1, wherein, Merging the incremental data rows of the same DML type to obtain disk files corresponding to different DML types comprises performing the following processing on the incremental data rows of the DML type of deletion: Storing the acquired incremental data rows in a third memory list object; determining whether a storage amount of the third memory list object reaches a third storage threshold value; if the third storage threshold value is not reached, continuing to acquire the incremental data rows, and if the third storage threshold value is reached, writing all data of the third memory list object into a third disk file, and determining whether a size of the third disk file reaches a third disk threshold value; and if the third disk threshold value is reached, establishing a new third disk file, and writing into the new third disk file in the next writing.

5. The method of claim 1, wherein, After the full data migration is completed, the disk files corresponding to different DML types are loaded in parallel based on a preset type order, including: After the full data migration is completed, an incremental file is obtained, and the file name of each disk file in the incremental file contains the corresponding DML type; and According to the order of insertion, update and deletion in turn, the disk files corresponding to the file names in the incremental file are read in parallel, and the read results are submitted to the database.

6. The method of claim 5, wherein, Parallel reading of the disk files corresponding to the file names in the incremental file and submitting the read results to the database includes the following processing for the disk files of the DML type of insertion: Read the file name set of the disk file containing the insertion in the incremental file, and submit to a working thread group, the number of threads of the thread group being based on the database load tuning setting; All threads of the working thread group perform parallel processing, wherein each thread performs the following processing on each assigned disk file: Read each incremental data row of the disk file; and Based on a preset first optimal batch step, the read results are merged by DML, and the merged results are submitted to the database.

7. The method of claim 5, wherein, Parallel reading of the disk files corresponding to the file names in the incremental file and submitting the read results to the database includes the following processing for the disk files of the DML type of update: Read the file name set of the disk file containing the update in the incremental file, and submit to a working thread group, the number of threads of the thread group being based on the database load tuning setting; All threads of the working thread group perform parallel processing, wherein each thread performs the following processing on each assigned disk file: Read each incremental data row of the disk file; and Based on a preset second optimal batch step, multiple read results are sent to the database in batch processing at a time.

8. The method of claim 5, wherein, Parallel reading of the disk files corresponding to the file names in the incremental file and submitting the read results to the database includes the following processing for the disk files of the DML type of deletion: Read the file name set of the disk file containing the deletion in the incremental file, and submit to a working thread group, the number of threads of the thread group being based on the database load tuning setting; All threads of the working thread group perform parallel processing, wherein each thread performs the following processing on each assigned disk file: Read each incremental data row of the disk file; and Based on a preset first optimal batch step, the read results are merged by DML, and the merged results are submitted to the database.

9. The method of claim 1, wherein, The method further includes: After the data full migration is completed, the incremental data pre-merging processing is continuously performed until a first condition is met, and then disk files corresponding to different DML types are loaded in parallel based on a preset type order. The first condition includes obtaining an incremental migration instruction, the source database TPS entering a trough, or the source database TPS entering a preset shutdown interval.

10. A data migration apparatus, characterized by comprising: The apparatus includes: A data pre-merging module configured to perform the following incremental data pre-merging processing during data full migration: continuously obtaining incremental data and parsing incremental data rows, the incremental data rows including table names, DML types, primary key values, and complete data row contents, the DML types including insertion, update, and / or deletion; merging incremental data rows of the same DML type to obtain disk files corresponding to different DML types; and A parallel loading module configured to load the disk files corresponding to different DML types in parallel based on a preset type order after the data full migration is completed.

11. A processor, comprising: The computer program is configured to implement the data migration method according to any one of claims 1 to 9 when executed by a processor.

12. A machine-readable storage medium having stored thereon instructions, the instructions being executable by a machine to cause the machine to perform operations comprising: The computer program is configured to implement the data migration method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product comprising a computer program, characterized in that, The computer program is configured to implement the data migration method according to any one of claims 1 to 9 when executed by a processor.

Images