Data migration method and device, equipment, storage medium and program product

Abstract

The invention provides a data migration method and device, equipment, a storage medium and a program product, and is applied to the technical field of big data. The method comprises the following steps: acquiring a system revision identifier corresponding to total data in a source end database; exporting the total data to obtain a to-be-transmitted file, and performing format conversion on the to-be-transmitted file to obtain a target format file; importing the target format file into a target end database through a preset physical importing mode; the target end database is in an initial state that the data table is empty; determining incremental data according to the system revision identifier from the first moment, and storing the incremental data into a cache file; the first moment is a full data export starting moment; and at a second moment after the target format file is imported, updating the incremental data in the cache file to a target end database through a preset logic importing mode. In this way, the data migration efficiency can be improved, and meanwhile, the service continuity can be ensured.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data migration method, device, equipment, storage medium and program product. Background Art

[0002] As digital transformation continues to deepen in various fields, the scale of various types of data continues to increase. In scenarios such as system upgrades, data migration has become a necessary process to ensure the normal implementation of business functions.

[0003] In the related art, data migration usually requires importing data from the source end to the target end in a downtime state. This data migration method is inefficient and has a long downtime, and cannot ensure business continuity. Summary of the invention

[0004] The present application provides a data migration method, apparatus, device, storage medium and program product, which can improve data migration efficiency and ensure business continuity.

[0005] In a first aspect, the present application provides a data migration method, comprising:

[0006] Obtain the system revision ID corresponding to all data in the source database;

[0007] Exporting the full amount of data to obtain a file to be transmitted, and converting the format of the file to be transmitted to obtain a file in a target format;

[0008] Importing the target format file into the target-side database by a preset physical import method; the target-side database is in an initial state where the data table is empty;

[0009] Starting from a first moment, incremental data is determined according to the system revision identifier, and the incremental data is stored in a cache file; the first moment is the start moment of the full data export;

[0010] At a second moment after the target format file is imported, the incremental data in the cache file is updated to the target-side database through a preset logic import method.

[0011] In a possible implementation manner, converting the format of the file to be transmitted to obtain a file in a target format includes:

[0012] The file to be transmitted is format converted, and the file to be transmitted is segmented according to a preset capacity to obtain a plurality of target format files; the target format corresponding to the target format file includes a comma separated value CSV format.

[0013] In a possible implementation, the method further includes:

[0014] Taking the data table level of the source database as the concurrency granularity, the export process of the full data and the format conversion process of the to-be-transmitted file are cross-merged and executed.

[0015] In a possible implementation manner, importing the target format file into the target end database by using a preset physical import method includes:

[0016] Encode the target format file into a target key-value pair, and store the target key-value pair in a local temporary directory;

[0017] Upload the target key-value pairs from the local temporary directory to each target node corresponding to the target database;

[0018] The target key-value pair is inserted from the target node into the underlying storage corresponding to the target database through the target interface.

[0019] In a possible implementation manner, updating the incremental data in the cache file to the target-side database by using a preset logic import method includes:

[0020] Reading the incremental data from the cache file in the order of storage time;

[0021] Encoding the incremental data into a structured query statement SQL;

[0022] The incremental data is imported into the target database by executing the SQL in a preset import order.

[0023] In a possible implementation, the source database is an Oracle database; and the target database is a TiDB database.

[0024] In a second aspect, the present application provides a data migration device, including:

[0025] The acquisition module is used to obtain the system revision identifier corresponding to the full amount of data in the source database;

[0026] An export module, used to export the full amount of data to obtain a file to be transmitted, and perform format conversion on the file to be transmitted to obtain a file in a target format;

[0027] A first import module, used to import the target format file into a target-side database by a preset physical import method; the target-side database is in an initial state where the data table is empty;

[0028] A determination module, configured to determine incremental data according to the system revision identifier and store the incremental data in a cache file starting from a first moment; the first moment is a start moment of exporting the full amount of data;

[0029] The second import module is used to update the incremental data in the cache file to the target end database through a preset logical import method at a second moment after the target format file is imported.

[0030] In a possible implementation manner, the export module is specifically used to:

[0031] The file to be transmitted is format converted, and the file to be transmitted is segmented according to a preset capacity to obtain a plurality of target format files; the target format corresponding to the target format file includes a comma separated value CSV format.

[0032] In a possible implementation manner, the device is further used for:

[0033] Taking the data table level of the source database as the concurrency granularity, the export process of the full data and the format conversion process of the to-be-transmitted file are cross-merged and executed.

[0034] In a possible implementation manner, the first import module is specifically used to:

[0035] Encode the target format file into a target key-value pair, and store the target key-value pair in a local temporary directory;

[0036] Upload the target key-value pairs from the local temporary directory to each target node corresponding to the target database;

[0037] The target key-value pair is inserted from the target node into the underlying storage corresponding to the target database through the target interface.

[0038] In a possible implementation manner, the second import module is specifically used to:

[0039] Reading the incremental data from the cache file in the order of storage time;

[0040] Encoding the incremental data into a structured query statement SQL;

[0041] The incremental data is imported into the target database by executing the SQL in a preset import order.

[0042] In a possible implementation, the source database is an Oracle database; and the target database is a TiDB database.

[0043] In a third aspect, the present application provides a data migration device, including: a processor and a memory;

[0044] The memory stores computer-executable instructions;

[0045] The processor executes the computer-executable instructions stored in the memory to implement the data migration method as described in any one of the first aspects.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed, they are used to implement the data migration method described in any one of the first aspects.

[0047] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed, implements the data migration method described in any one of the first aspects.

[0048] The data migration method, device, equipment, storage medium and program product provided by the present application obtain the system revision identifier corresponding to the full amount of data in the source database; export the full amount of data to obtain the file to be transmitted, and convert the format of the file to be transmitted to obtain the target format file; import the target format file into the target database through a preset physical import method; the target database is in an initial state where the data table is empty; starting from the first moment, the incremental data is determined according to the system revision identifier, and the incremental data is stored in the cache file; the first moment is the start moment of the full amount of data export; at the second moment after the target format file is imported, the incremental data in the cache file is updated to the target database through a preset logical import method. In the present application, the electronic device exports the full amount of data and converts it into a target format file, and stores the incremental data at the same time, and then first imports the target format file into the target database in the initial state based on the preset physical import method, and then updates the incremental data to the target database based on the preset logical import method, which can improve the efficiency of data migration and ensure business continuity. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a data migration method provided in an embodiment of the present application;

[0050] Figure 2 A flowchart of another data migration method provided in an embodiment of the present application;

[0051] Figure 3 A technical logic diagram of a data migration method provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of the structure of a data migration device provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of the structure of a data migration device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the technical solution of the present application, the present application is further described in detail below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments and drawings described herein are only used to explain the present application, and are not intended to limit the present application.

[0055] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation portals for users to choose to authorize or refuse. In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user data information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0056] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0057] It should be noted that the data migration method, device, equipment, storage medium and program product of the present application can be used in the field of big data technology, and can also be used in any field outside the field of big data technology. The present application does not limit the specific application field of the data migration method, device, equipment, storage medium and program product.

[0058] The following is a detailed explanation of the terms involved in this application:

[0059] Open source distributed relational database (Ti Distributed Database, TiDB): is a hybrid distributed database that can support both online transaction processing and online analytical processing. It has important features such as horizontal expansion or reduction, high availability, support for real-time online transaction processing and online analytical processing (Hybrid Transactional and AnalyticalProcessing, HTAP), cloud native, and compatibility with the relational database management system MySQL protocol and MySQL ecosystem.

[0060] Distributed transaction key-value storage database (Ti Key-Value, TiKV): a distributed transactional key-value database that provides a distributed transaction interface that meets the constraints of database transaction characteristics (Atomicity, Consistency, Isolation, Durability, ACID), and ensures multi-copy data consistency and high availability through the distributed consistency (Raft) protocol. As the storage layer of TiDB, TiKV can provide persistence and read-write services for the data written by users to TiDB, and also stores TiDB's statistical information data.

[0061] With the continuous development of Internet technology, digital transformation has been continuously deepened in various fields, and the scale of various types of data has also increased. In scenarios such as system upgrades, data migration has become an indispensable process to ensure the normal implementation of business functions.

[0062] Data migration refers to the process of exporting data from a source database and then importing the data into a target database. Data import refers to the process of importing data files into a database. In related technologies, data migration usually involves importing all data from a source database into a target database while the database is down. This method of data migration is slow and inefficient when faced with large-scale data imports, and the downtime is long, making it impossible to ensure business continuity.

[0063] In order to solve the above problems, the data migration method, device, equipment, storage medium and program product in the present application, the electronic device obtains the system revision identifier corresponding to the full amount of data in the source database, exports the full amount of data as a file to be transmitted, and performs format conversion to obtain a target format file. After that, the electronic device can import the target format file into the target database in the initial state through a preset physical import method. At the beginning of the full data export, the electronic device can determine and record the incremental data according to the system revision identifier. At the second moment after the target format file is fully imported into the target database, the electronic device can update the incremental data to the target database through a preset logical import method. In this way, the electronic device can combine the advantages of physical import and logical import in the data migration process, which can not only improve the efficiency of data migration, but also ensure the continuity of business.

[0064] The scheme shown in the present application is described in detail below through specific embodiments. It should be noted that the following embodiments can exist independently or in combination with each other, and the same or similar contents will not be described repeatedly in different embodiments.

[0065] Figure 1 A flowchart of a data migration method provided in an embodiment of the present application. Figure 1, the data migration method may include:

[0066] S101. Obtain the system revision identifier corresponding to the full amount of data in the source database.

[0067] The execution subject of the embodiment of the present application may be an electronic device, such as a server, etc., and may specifically be a data migration system installed in the electronic device, or may be a data migration device provided in the electronic device. The data migration device may be implemented by software, or may be implemented by a combination of software and hardware. For ease of understanding, the following description will be made by taking the execution subject as an electronic device as an example.

[0068] In an embodiment of the present application, the source database may refer to the database at the data export end during the data migration process. The full data may refer to all the data that needs to perform data migration. The system revision identifier (System Change Number, SCN) may refer to the internal identifier corresponding to the full data in the source database, which is used to characterize the submission order of each transaction in the source database. Exemplarily, when the source database is an Oracle database, the electronic device can obtain the system revision identifier of the full data in the Oracle database, and can subsequently determine the incremental data in the data migration process based on the system revision identifier.

[0069] Specifically, when the electronic device needs to migrate data, it can first obtain the system revision identifier of the full data in the source database through the structured query language (SQL), so that the system revision identifier of the initial state can be obtained.

[0070] S102: Export the full amount of data to obtain a file to be transmitted, and convert the file to be transmitted into a format to obtain a file in a target format.

[0071] In the embodiment of the present application, the file to be transmitted may refer to a data file obtained by exporting the full amount of data in the source database, which may be represented as an XF1 file. The target format file may refer to a full amount of data file in the target format, which may refer to comma-separated values ​​(CSV), JS key-value pair data format (JavaScript Object Notation, JSON) or Extensible Markup Language (XML), etc. Of course, the target format may also be other formats, which is not limited in the embodiment of the present application.

[0072] In this step, after obtaining the system revision identifier of the full data in the source database, the electronic device can export the full data. For example, it can trigger the export operation of the full data based on the export command (expdp) to obtain the file to be transmitted; then the electronic device can perform format conversion on the file to be transmitted to obtain a target format file to facilitate subsequent data import processing.

[0073] S103, importing the target format file into the target-side database through a preset physical import method; the target-side database is in an initial state where the data table is empty.

[0074] In the embodiment of the present application, the preset physical import mode may refer to a preset full data physical import mode. The target database may refer to a destination database for receiving data during the data migration process, and the target database may specifically be TiDB, etc. The target database is in an initial state where the data table is empty, that is, the target database does not include any data.

[0075] In this step, after the electronic device converts the format of the file to be transferred to obtain the target format file, it can import the target format file corresponding to the full data of the source database into the target database based on the preset physical import method. The preset physical import method may refer to the physical import mode in the data import tool software, for example, the data import tool software may be TiDB Lightning, etc. The preset physical import mode has a higher import efficiency, but it is more stringent in terms of use conditions, for example, the data table in the target database must be empty, etc.

[0076] S104. Starting from the first moment, determine the incremental data according to the system revision identifier, and store the incremental data in a cache file; the first moment is the start moment of exporting the full data.

[0077] In the embodiment of the present application, the first moment may refer to the start moment of exporting the full data of the source database. The incremental data may refer to the data that has changed in the full data of the source database since the first moment. The buffer file may refer to a buffer file for storing incremental data, which may be represented by XF1 BUFFER.

[0078] In this step, the electronic device starts to collect incremental data according to the system revision mark of the full data at the first moment of exporting the full data, and stores the incremental data in the cache file. In this way, the electronic device can determine and record the incremental data during the execution of the preset physical import method, realize the comprehensive and accurate collection of incremental data, and ensure the integrity and reliability of data migration.

[0079] S105. At a second moment after the target format file is imported, the incremental data in the cache file is updated to the target end database through a preset logical import method.

[0080] In an embodiment of the present application, the second moment may refer to the moment corresponding to the completion of the import of the target format file. The preset logical import method may refer to a preset incremental data logical import mode. Specifically, at the second moment when the import of the target format file is completed, the electronic device may update the incremental data in the cache file XF1 BUFFER to the target end database through a preset logical import method, such as the logical import mode in the data import tool software TiDBLightning, to achieve the update of incremental data during data migration.

[0081] The data migration method provided by the embodiment of the present application is that the electronic device obtains the system revision identifier corresponding to the full amount of data in the source database; exports the full amount of data to obtain the file to be transmitted, and performs format conversion on the file to be transmitted to obtain the target format file; imports the target format file into the target database through a preset physical import method; the target database is in an initial state where the data table is empty; starting from the first moment, the incremental data is determined according to the system revision identifier, and the incremental data is stored in the cache file; the first moment is the start moment of the full amount of data export; at the second moment after the target format file is imported, the incremental data in the cache file is updated to the target database through a preset logical import method. In the present application, the electronic device exports the full amount of data and converts it into a target format file, stores the incremental data at the same time, and then first imports the target format file into the target database in the initial state based on the preset physical import method, and then updates the incremental data to the target database based on the preset logical import method, which can improve the efficiency of data migration and ensure business continuity.

[0082] Based on the above embodiments, Figure 2 This is a flowchart of another data migration method provided in an embodiment of the present application. Figure 2 , the data migration method may include:

[0083] S201. Obtain the system revision identifier corresponding to the full amount of data in the source database.

[0084] In a possible implementation, the source database is an Oracle database; the target database is a TiDB database.

[0085] In the embodiment of the present application, the source database in the data migration process can be an Oracle database, and the target database can be a TiDB database, so that large amounts of data can be migrated online between heterogeneous databases to meet the actual needs of users. Of course, the source database and the target database can also be other types of databases, which are not limited in the embodiment of the present application.

[0086] S202: Export all data to obtain a file to be transmitted, and convert the file to be transmitted to obtain a file in a target format.

[0087] In a possible implementation manner, the format conversion process of the file to be transmitted in step S202 may be implemented in the following manner:

[0088] The file to be transferred is converted into a format, and the file to be transferred is segmented according to a preset capacity to obtain a plurality of target format files; the target formats corresponding to the target format files include a comma separated value (CSV) format.

[0089] In an embodiment of the present application, the preset capacity may refer to the capacity size of a preset target format file. Specifically, after the electronic device exports the full amount of data in the source database as the file to be transmitted XF1, it may perform format conversion and segmentation processing on the file to be transmitted XF1 to obtain multiple CSV files, which are the target format files. It should be noted that when the electronic device segments the file to be transmitted, it may evenly segment it according to the preset capacity. The value of the preset capacity cannot be too large or too small. For example, the preset capacity of a CSV file may be 512 megabytes (MB). This can avoid the situation where the preset capacity is too small, resulting in too many CSV files and high management costs, and can also avoid the situation where the preset capacity is too large, resulting in a long processing time for a single CSV file and not conducive to concurrency.

[0090] In a possible implementation, the export process of the full amount of data in the source database in step S201 and the format conversion process of the to-be-transferred file in step S202 can be combined and executed, which can further improve the efficiency of data migration. The data migration method may include the following steps:

[0091] Taking the data table level of the source database as the concurrency granularity, the export process of the full data and the format conversion process of the files to be transferred are cross-merged and executed.

[0092] In the embodiment of the present application, the concurrency granularity may refer to the smallest data unit in the concurrency control. When the electronic device exports the full amount of data in the source database as the file to be transmitted XF1, it may cross-merge and perform the format conversion and segmentation process of the file to be transmitted, and the electronic device may use the data table level as the concurrency granularity, which can reduce the memory and CPU overhead, and can further improve the efficiency of data migration and shorten the time consumed by data migration.

[0093] S203, encoding the target format file into a target key-value pair, and storing the target key-value pair in a local temporary directory; uploading the target key-value pair from the local temporary directory to each target node corresponding to the target end database.

[0094] S204. Insert the target key-value pair from the target node into the underlying storage corresponding to the target database through the target interface; the target database is in an initial state where the data table is empty.

[0095] In an embodiment of the present application, the target key-value pair may refer to the key-value pair obtained from the target format file. The local temporary directory may refer to the local directory storing the key-value pair. The target node may refer to the node corresponding to the target database, such as the TiKV node in the TiDB database. The target interface may refer to the data import interface in the target database, such as the TiKV Ingest interface in the TiDB database. The underlying storage may refer to the underlying storage of the target node in the target database, such as the underlying storage (RocksDB) of the TiKV node in the TiDB database.

[0096] In this step, after the electronic device has exported all the data in the source database and converted the format to obtain the target format file, it can import the target format file into the target database in the initial state through a preset physical import method. Specifically, the electronic device can first encode the target format file to obtain the target key-value pair, and then sort and store the target key-value pair in a local temporary directory; then the electronic device can upload the target key-value pair to each target node of the target database; then the electronic device can insert the target key-value pair from the target node into the underlying storage of the target database through the target interface to realize the data import of the target format file. In this way, for the target database in the initial state, the electronic device imports the data of the target format file based on the preset physical import method, and the data import speed is faster and more efficient.

[0097] S205. Starting from the first moment, determine the incremental data according to the system revision identifier, and store the incremental data in a cache file; the first moment is the start moment of exporting the full data.

[0098] In an embodiment of the present application, at the first moment when the full data is exported, the electronic device can determine the incremental data of the full data according to the system revision identifier of the full data, and store the incremental data in a cache file. For example, the electronic device can determine the system revision identifier of the full data at the first moment, and then determine the system revision identifier after a certain incremental operation is completed. Then, the electronic device can determine the incremental data according to the system revision identifier at the first moment and the system revision identifier after the incremental operation is completed through an SQL query statement such as an export command.

[0099] S206. At the second moment after the target format file is imported, the incremental data is read from the cache file in the order of storage time; the incremental data is encoded into a structured query statement SQL; and the incremental data is imported into the target database by executing SQL according to a preset import order.

[0100] In an embodiment of the present application, at the second moment after the target format file is imported, the electronic device can read the incremental data one by one from the cache file in the order of the storage time of the incremental data in the cache file, and then update the incremental data to the target database through a preset logical import method. Specifically, the electronic device can first encode the incremental data into a structured query statement SQL through the logical import mode in the data import tool software TiDB Lightning, and then execute the structured query statement through a preset import order to update the incremental data to the target database. The preset import order can be an import order of first deletion and then insertion; after all the incremental data in the mixed storage file are imported, the incremental data is leveled and the data migration process ends. In this way, the electronic device plays back the incremental data to the target database through a preset logical import method, which can ensure the real-time and consistency of the data during the data migration process.

[0101] It should be noted that when electronic devices perform data migration based on the data import tool software TiDB Lightning, the back-end modes corresponding to the preset physical import method and the preset logical import method may be different. For example, the first back-end mode corresponding to the preset physical import method may be local (local), and the second back-end mode corresponding to the preset logical import method may be the target database mode (tidb).

[0102] For example, Figure 3A technical logic diagram of a data migration method provided for an embodiment of the present application. As shown in 3, the electronic device first obtains the system revision identifier of the full data in the source data, and then exports the full data as a file to be transmitted XF1; at the first moment of the full data export process, the electronic device starts incremental data collection according to the system revision identifier, and accumulates and stores these incremental data in the cache file XF1 BUFFER. The electronic device can convert the format of the file to be transmitted to obtain a target format file, and specifically can convert and divide the file to be transmitted XF1 into multiple CSV files according to a preset capacity.

[0103] Then, the electronic device can import the target format file into the target end database through the preset physical import method. After the target format file corresponding to the full data is imported, the electronic device can read the incremental data from the cache file XF1 BUFFER through the preset logical import method, and update the incremental data to the target database according to the preset import order of first deleting and then inserting. After all the incremental data in the cache file XF1 BUFFER is imported, the data migration process ends. In this way, based on the preset physical import method and the preset logical import method, the electronic device can not only ensure the efficiency of data migration, but also realize the update of incremental data, thereby ensuring the integrity and availability of the data.

[0104] In the related art, the data import tool software TiDB Lightning supports two import modes: traditional physical import mode and traditional logical import mode; the traditional physical import mode has high import efficiency, but the import conditions are relatively harsh, and it is impossible to obtain incremental data, and it is only applicable to the full import process under the static source end; and although the traditional logical import supports incremental data appending, the import speed of the traditional logical import is not high, the data import takes a long time, and the efficiency is low. In addition, there are some data migration tools in the related art, such as TiDB Data Migration (DM), etc. This DM supports full data migration and incremental data synchronization from a database compatible with the MySQL protocol to TiDB, but this DM is only applicable to data migration of small data volumes and cannot be applied to large-scale data migration.

[0105] It can be seen that in the relevant technology, DM is suitable for databases with small data volumes (for example, less than 1 terabyte (TB)), and has low efficiency and low applicability for large-scale data migration. The traditional physical import mode in the data import tool software TiDB Lightning cannot achieve incremental data migration; the traditional logical import mode in the data import tool software TiDB Lightning has a slow import speed and low data migration efficiency during large-scale data migration.

[0106] In an embodiment of the present application, the electronic device obtains the system revision identifier in the source database, records the incremental data at the first moment of the full data export and stores it in the cache file; after the electronic device exports the full data as a file to be transmitted and performs format conversion to obtain a target format file, it can import the target format file into the target database based on a preset physical import method to ensure the import speed of the full data; at the second moment after the preset physical import method is completed, the electronic device can read the incremental data from the cache file based on a preset logical import method, and update the incremental data to the target database. In this way, the electronic device can take advantage of both physical and logical imports during the data migration process, making full use of the high efficiency of physical import, and using the logical import method to supplement the disadvantage that physical import does not support real-time increments, ensuring the efficiency of data migration and business continuity, and ensuring the data integrity and availability of data migration.

[0107] Exemplarily, Table 1 shows the time taken for each stage of data migration provided by an exemplary embodiment of the present application.

[0108] Table 1

[0109]

[0110] As shown in Table 1, the total amount of data in the source database is 61 gigabytes (GB). After the full amount is exported, the file to be transferred XF1 is obtained, the file to be transferred XF1 is converted into the target format file (CSV file), and then imported into the TiDB database through the preset physical import method, with an average speed of 133.58GB per hour; if the export process of the full amount of data and the format conversion process of the file to be transferred are cross-merged and executed, the preset physical import method storage speed is 114.6GB per hour. Of course, due to the different data sizes of the full amount of data and the different configurations of the migration intermediate machines, whether to cross-merge can be determined based on the actual scenario, and the embodiments of the present application do not limit this.

[0111] In the embodiment of the present application, the data migration process of the electronic device combines the advantages of logical import and physical import, which can effectively supplement the usage scenario restrictions of physical import, support the migration of large amounts of data, and ensure the integrity and accuracy of data migration. In addition, the data migration method does not have technical dependence on the transaction processing and isolation level of the database, and has transaction and isolation level independence. Regardless of the transaction processing mechanism of the source database or the target database, the migration process can proceed smoothly. In addition, the data migration method can further improve the continuity of the business, ensure that business operations are not affected while data is being migrated, and ensure the continuity of business operations.

[0112] Figure 4This is a schematic diagram of the structure of a data migration device provided in an embodiment of the present application. Figure 4 , the data migration device 40 may include:

[0113] An acquisition module 41 is used to acquire a system revision identifier corresponding to all data in a source database;

[0114] The export module 42 is used to export the full amount of data to obtain a file to be transmitted, and to convert the format of the file to be transmitted to obtain a file in a target format;

[0115] The first import module 43 is used to import the target format file into the target database through a preset physical import method; the target database is in an initial state where the data table is empty;

[0116] A determination module 44 is used to determine incremental data according to the system revision identifier starting from a first moment, and store the incremental data in a cache file; the first moment is the start moment of exporting the full amount of data;

[0117] The second import module 45 is used to update the incremental data in the cache file to the target end database through a preset logical import method at a second moment after the target format file is imported.

[0118] In a possible implementation, the export module 42 is specifically configured to:

[0119] The file to be transferred is converted into a format, and the file to be transferred is segmented according to a preset capacity to obtain a plurality of target format files; the target formats corresponding to the target format files include a comma separated value (CSV) format.

[0120] In a possible implementation manner, the device 40 is further used for:

[0121] Taking the data table level of the source database as the concurrency granularity, the export process of the full data and the format conversion process of the files to be transferred are cross-merged and executed.

[0122] In a possible implementation manner, the first import module 43 is specifically configured to:

[0123] Encode the target format file into target key-value pairs and store the target key-value pairs in a local temporary directory;

[0124] Upload the target key-value pairs from the local temporary directory to each target node corresponding to the target database;

[0125] Through the target interface, the target key-value pair is inserted from the target node into the underlying storage corresponding to the target database.

[0126] In a possible implementation manner, the second import module 45 is specifically used to:

[0127] Read incremental data from the cache file in the order of storage time;

[0128] Encode incremental data into structured query statements SQL;

[0129] Import incremental data into the target database by executing SQL in the preset import order.

[0130] In a possible implementation, the source database is an Oracle database; the target database is a TiDB database.

[0131] The data migration device 40 provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0132] Figure 5 This is a schematic diagram of the structure of a data migration device provided in an embodiment of the present application. Figure 5 The data migration device 50 may include: a memory 51 and a processor 52. Exemplarily, the memory 51 and the processor 52 are interconnected via a bus 53.

[0133] The memory 51 is used to store program instructions;

[0134] The processor 52 is used to execute the program instructions stored in the memory to implement the data migration method shown in the above embodiment.

[0135] Figure 5 The data migration device 50 shown can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0136] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above-mentioned data migration method.

[0137] The embodiment of the present application may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the above-mentioned data migration method.

[0138] It should be noted that the processor mentioned in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0139] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DR RAM). It should be noted that when the processor is a general purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) is integrated in the processor. It should be noted that the memory described herein is intended to include but is not limited to these and any other suitable types of memory.

[0140] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0141] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0144] Regarding the various modules / units contained in the various devices and products described in the above embodiments, they may be software modules / units, or hardware modules / units, or they may be partially software modules / units and partially hardware modules / units. Each device and product may be applied to or integrated in a chip, a chip module or a terminal device. Exemplarily, for each device and product applied to or integrated in a chip, each module / chip contained therein may be implemented in the form of hardware such as circuits, or at least some modules / units may be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining modules / units may be implemented in the form of hardware such as circuits.

[0145] In the present application, the term "include" and its variations may refer to non-restrictive inclusion; the term "or" and its variations may refer to "and / or". The terms "first", "second", etc. in the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In the present application, "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previously associated objects are in an "or" relationship.

[0146] The above are only some embodiments of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data migration method, characterized in that: include: Obtain the system revision ID corresponding to all data in the source database; Exporting the full amount of data to obtain a file to be transmitted, and converting the format of the file to be transmitted to obtain a file in a target format; Importing the target format file into the target end database through a preset physical import method; The target-side database is in an initial state where the data table is empty; Starting from the first moment, determining incremental data according to the system revision identifier, and storing the incremental data in a cache file; The first time is the start time of exporting the full amount of data; At a second moment after the target format file is imported, the incremental data in the cache file is updated to the target-side database through a preset logic import method.

2. The method according to claim 1, characterized in that The step of converting the format of the file to be transmitted to obtain a file in a target format includes: The file to be transmitted is format converted, and the file to be transmitted is segmented according to a preset capacity to obtain a plurality of target format files; the target format corresponding to the target format file includes a comma separated value CSV format.

3. The method according to claim 1, characterized in that The method further comprises: Taking the data table level of the source database as the concurrency granularity, the export process of the full data and the format conversion process of the to-be-transmitted file are cross-merged and executed.

4. The method according to claim 1, characterized in that The step of importing the target format file into the target end database by a preset physical import method includes: Encode the target format file into a target key-value pair, and store the target key-value pair in a local temporary directory; Upload the target key-value pairs from the local temporary directory to each target node corresponding to the target database; The target key-value pair is inserted from the target node into the underlying storage corresponding to the target database through the target interface.

5. The method according to claim 1, characterized in that The updating of the incremental data in the cache file to the target end database by means of a preset logic import method includes: Reading the incremental data from the cache file in the order of storage time; Encoding the incremental data into a structured query statement SQL; The incremental data is imported into the target database by executing the SQL in a preset import order.

6. The method according to any one of claims 1 to 5, characterized in that: The source database is an Oracle database; the target database is a TiDB database.

7. A data migration device, characterized in that: include: The acquisition module is used to obtain the system revision identifier corresponding to the full amount of data in the source database; An export module, used to export the full amount of data to obtain a file to be transmitted, and perform format conversion on the file to be transmitted to obtain a file in a target format; A first import module, used for importing the target format file into the target end database through a preset physical import method; The target-side database is in an initial state where the data table is empty; A determination module, configured to determine incremental data according to the system revision identifier starting from a first moment, and store the incremental data in a cache file; The first time is the start time of exporting the full amount of data; The second import module is used to update the incremental data in the cache file to the target end database through a preset logical import method at a second moment after the target format file is imported.

8. A data migration device, characterized in that: include: Processor, memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the data migration method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the data migration method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the data migration method according to any one of claims 1 to 6.

Images