Data migration method, device, equipment and readable storage medium

Abstract

The application provides a data migration method and device, equipment and a readable storage medium, and relates to the technical field of big data. The method comprises the following steps: acquiring a plurality of log files for operating a source database; performing analysis processing on the plurality of log files to obtain a plurality of operation information, wherein the operation information comprises an operation statement, and an identifier and a global sequence number of a transaction corresponding to the operation statement; performing sorting processing on the plurality of operation information according to the global sequence number to obtain a plurality of sorted operation information, and determining at least one operation information corresponding to each transaction in the plurality of sorted operation information; and generating a transaction file according to the at least one operation information corresponding to each transaction, and sending the transaction file to a target device. The migration efficiency of the log file is improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data migration method, apparatus, device, and readable storage medium. Background Technology

[0002] As data storage demands expand, databases are becoming increasingly diverse. If the database storing the data changes, the data in the source database needs to be migrated to the target device.

[0003] During database migration, log files from the source database also need to be migrated to the target device. While structured data replication software can be used to migrate log files, this requires deploying numerous other components, resulting in low migration efficiency. Summary of the Invention

[0004] This application provides a data migration method, apparatus, device, and readable storage medium to solve the problem of low migration efficiency of log files.

[0005] Firstly, this application provides a data migration method, including:

[0006] Retrieve multiple log files that were used to operate on the source database;

[0007] The multiple log files are parsed to obtain multiple operation information entries, which include operation statements, as well as the identifier and global sequence number of the transaction corresponding to the operation statement.

[0008] The multiple operation information is sorted according to the global sequence number to obtain sorted multiple operation information, and at least one operation information corresponding to each transaction is determined in the sorted multiple operation information.

[0009] A transaction file is generated based on at least one operation information corresponding to each transaction, and the transaction file is sent to the target device.

[0010] In one possible implementation, multiple log files that operate on the source database are obtained, including:

[0011] Get the log file download progress;

[0012] Based on the log file download progress, multiple data blocks are downloaded from the source database and stored on the disk in the form of log files to obtain the multiple log files.

[0013] In one possible implementation, multiple data blocks are downloaded from the source database, and the multiple data blocks are stored on a disk in the form of log files to obtain the multiple log files, including:

[0014] The multiple data blocks are downloaded in parallel using multiple download threads, and the multiple data blocks are stored in a data block queue;

[0015] The sorting thread retrieves the multiple data blocks from the data block queue and sorts them according to their data block information to obtain multiple ordered data blocks. The data block information includes the identifier of the log file corresponding to the data block and the sequence number of the data block in the corresponding log file.

[0016] The multiple ordered data blocks are stored on the disk in the form of log files to obtain the multiple log files.

[0017] In one possible implementation, the plurality of ordered data blocks are stored on a disk in the form of log files to obtain the plurality of log files, including:

[0018] The multiple data blocks are stored in a cache in the order they are ordered. When the amount of data in the cache is greater than or equal to a first threshold, the data blocks in the cache are stored on the disk until the multiple log files are obtained.

[0019] In one possible implementation, after determining at least one operation message corresponding to each transaction from the sorted multiple operation messages, the method further includes:

[0020] For any given transaction, if the memory occupied by at least one operation information corresponding to the transaction is greater than or equal to a second threshold, then at least one operation information corresponding to the transaction is stored on the disk in the form of a file to obtain the transaction file corresponding to the transaction.

[0021] If the memory occupied by at least one operation information corresponding to the transaction is less than the second threshold, then at least one operation information corresponding to the transaction is stored in the cache until the amount of data in the cache is greater than or equal to the third threshold. Then, the operation information in the cache is stored in the form of a file on the disk to obtain the corresponding transaction file.

[0022] In one possible implementation, the method further includes:

[0023] Identify the first transaction to be rolled back;

[0024] If the first transaction is in the cache, then delete the first transaction from the cache;

[0025] If the first transaction is located on a disk, then delete the first transaction on that disk.

[0026] Secondly, this application provides a data migration apparatus, including an acquisition module, a first processing module, a second processing module, a first determination module, a generation module, and a sending module, wherein:

[0027] The acquisition module is used to acquire multiple log files that are used to operate on the source database;

[0028] The first processing module is used to parse and process the multiple log files to obtain multiple operation information, the operation information including operation statements, and the identifier and global sequence number of the transaction corresponding to the operation statement;

[0029] The second processing module is used to sort the multiple operation information entries according to the global sequence number to obtain sorted multiple operation information entries.

[0030] The first determining module is used to determine at least one operation information corresponding to each transaction from the sorted multiple operation information;

[0031] The generation module is used to generate a transaction file based on at least one operation information corresponding to each transaction.

[0032] The sending module is used to send the transaction file to the target device.

[0033] In one possible implementation, the acquisition module is specifically used for:

[0034] Get the log file download progress;

[0035] Based on the log file download progress, multiple data blocks are downloaded from the source database and stored on the disk in the form of log files to obtain the multiple log files.

[0036] In one possible implementation, the acquisition module is specifically used for:

[0037] The multiple data blocks are downloaded in parallel using multiple download threads, and the multiple data blocks are stored in a data block queue;

[0038] The sorting thread retrieves the multiple data blocks from the data block queue and sorts them according to their data block information to obtain multiple ordered data blocks. The data block information includes the identifier of the log file corresponding to the data block and the sequence number of the data block in the corresponding log file.

[0039] The multiple ordered data blocks are stored on the disk in the form of log files to obtain the multiple log files.

[0040] In one possible implementation, the acquisition module is specifically used for:

[0041] The multiple data blocks are stored in a cache in the order they are ordered. When the amount of data in the cache is greater than or equal to a first threshold, the data blocks in the cache are stored on the disk until the multiple log files are obtained.

[0042] In one possible implementation, the device further includes a first storage module and a second storage module, wherein:

[0043] The first storage module is specifically used to, for any transaction, if the memory occupied by at least one operation information corresponding to the transaction is greater than or equal to a second threshold, store at least one operation information corresponding to the transaction in the form of a file to the disk to obtain the transaction file corresponding to the transaction;

[0044] The second storage module is specifically used to store at least one operation information corresponding to the transaction in a cache if the memory occupied by at least one operation information is less than the second threshold, until the amount of data in the cache is greater than or equal to the third threshold, and then store the operation information in the cache in the form of a file to the disk to obtain the corresponding transaction file.

[0045] In one possible implementation, the device further includes a second determining module, a first deleting module, and a second deleting module, wherein:

[0046] The second determining module is specifically used to determine the first transaction to be rolled back;

[0047] The first deletion module is specifically used to delete the first transaction from the cache if the first transaction is located in the cache.

[0048] The second deletion module is specifically used to delete the first transaction from the disk if the first transaction is located on the disk.

[0049] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor.

[0050] The memory stores computer-executed instructions;

[0051] The processor executes computer execution instructions stored in the memory, causing the processor to perform the data migration method according to any one of the first aspects.

[0052] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data migration method described in any of the first aspects.

[0053] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the data migration method described in any of the first aspects.

[0054] The data migration method, apparatus, device, and readable storage medium provided in this application acquire multiple log files that operate on the source database, parse and process the multiple log files, sort the multiple operation information obtained according to the global sequence number, determine at least one operation information corresponding to each transaction from the sorted multiple operation information, generate a transaction file, and send the transaction file to the target device, thereby improving the migration efficiency of log files. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0056] Figure 1 A schematic diagram illustrating the application scenarios provided in the embodiments of this application;

[0057] Figure 2 A flowchart illustrating a data migration method provided in an embodiment of this application;

[0058] Figure 3 A flowchart illustrating the process of determining the data segment to be downloaded, provided for an embodiment of this application;

[0059] Figure 4 This is a schematic diagram illustrating the process of retrieving multiple data blocks from a disk, provided as an embodiment of this application.

[0060] Figure 5 A flowchart illustrating the parsing thread provided in an embodiment of this application;

[0061] Figure 6 This is a schematic diagram of the process for sending transaction files provided in an embodiment of this application;

[0062] Figure 7 A flowchart illustrating another data migration method provided in an embodiment of this application;

[0063] Figure 8 A schematic diagram of the data download process provided for embodiments of this application;

[0064] Figure 9A flowchart illustrating the data block sorting process provided in this application embodiment;

[0065] Figure 10 A flowchart illustrating another data migration method provided in an embodiment of this application;

[0066] Figure 11 A flowchart illustrating the sorting process provided in an embodiment of this application;

[0067] Figure 12 This is a schematic diagram of the structure of a data migration device provided in an embodiment of this application;

[0068] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0069] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0070] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0071] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application. Please refer to [link / reference]. Figure 1 It includes a database 101, a processing device 102, and a target device 103.

[0072] Database 101 can be used to store business data, and the log file can record changes in the business data in database 101. The location of data storage in the database can be determined based on the log file.

[0073] Processing device 102 can migrate data from database 101 to target device 103. It employs multiple threads for data migration, downloading corresponding data from the log files of database 101, parsing the log files, sorting the parsed operation information, determining the operation information corresponding to each transaction, and finally sending the transaction file to target device 103. Target device 103 can obtain the transaction file as needed; target device 103 may include databases, terminal devices, etc.

[0074] In related technologies, log files can be migrated using structured data replication software. However, when migrating log files using structured data replication software, it is necessary to deploy many other components, resulting in low migration efficiency.

[0075] In this embodiment, multiple log files representing operations on the source database can be obtained, parsed, and the resulting operation information is sorted according to a global sequence number. From the sorted operation information, at least one operation information corresponding to each transaction is determined, a transaction file is generated, and the transaction file is sent to the target device. This process improves the efficiency of log file migration.

[0076] The method described in this application will now be illustrated through specific embodiments. It should be noted that the following embodiments may exist independently or in combination with each other; identical or similar content will not be repeated in different embodiments.

[0077] Figure 2 This is a flowchart illustrating a data migration method provided in an embodiment of this application. Please refer to... Figure 2 The method may include:

[0078] S201. Obtain multiple log files that were used to operate on the source database.

[0079] The execution entity in this application embodiment can be a processing device or a data migration device installed in the processing device. The data migration device can be implemented by software or by a combination of software and hardware.

[0080] Multiple log files used to operate on the source database can be obtained in the following way: obtain the log file download progress; based on the log file download progress, download multiple data blocks from the source database and store the multiple data blocks to the disk in the form of log files to obtain multiple log files.

[0081] The source database can continuously record the operation process through log files. When the processing device obtains the log files, it needs to determine the download progress of the log files and identify the data blocks that have not been downloaded based on the download progress.

[0082] A log file can contain multiple data blocks.

[0083] For example, suppose that at time 1, a total of 3 log files are generated in the source database, namely log files 1-3. At this time, the log file download progress is 2. Then it can be determined that multiple data blocks corresponding to log files 1 and 2 have been downloaded, and multiple data blocks corresponding to log file 3 have not been downloaded.

[0084] To facilitate the description of the embodiments of this application, the following will be combined with... Figure 3 This application describes the process of determining the data segment to be downloaded based on the download progress of the log file.

[0085] Figure 3 This is a flowchart illustrating the process of determining a data segment to be downloaded, as provided in an embodiment of this application. The actual progress of the log file in the source database is greater than the progress that the processing device can perceive. After perceiving the log file progress, multiple data blocks corresponding to the log file to be downloaded are determined based on the current log file download progress; these multiple data blocks are the data segments to be downloaded. As the actual progress of the log file continuously increases, the perceived progress of the log file can be continuously updated, allowing new data segments to be downloaded to be determined and distributed to multiple threads to download multiple data blocks.

[0086] The data segment to be downloaded may include the identifier of the corresponding log file and the offset of the data segment corresponding to the log file identifier.

[0087] S202. Parse and process multiple log files to obtain multiple operation messages.

[0088] It can parse and process multiple data blocks in any log file to obtain the operation information corresponding to the data blocks.

[0089] Operation information may include the operation statement, as well as the identifier and global sequence number of the transaction corresponding to the operation statement.

[0090] For example, suppose there are two log files, namely log file 1 and log file 2. Log file 1 contains data block 1 and data block 2, and log file 2 contains data block 3. After parsing and processing the three data blocks, the operation information shown in Table 1 can be obtained.

[0091] Table 1

[0092]

[0093] A global sequence number can refer to the numbering of multiple data blocks in their order.

[0094] After multiple parsing threads parse the data block, the order of the multiple operation information is different from the order of the multiple data blocks because the data block parsing is completed at different times.

[0095] For example, suppose there are 3 data blocks. The order of data blocks 1-3 in the data block queue is data block 1, data block 2, and data block 3, with global sequence numbers 1, 2, and 3 respectively. After being parsed by multiple parsing threads, data block 1 corresponds to operation information 1, data block 2 corresponds to operation information 2, and data block 3 corresponds to operation information 3. If the order of data blocks 1-3 at the time of parsing completion is data block 1, data block 3, and data block 2, then the order of operation information in the operation information queue is operation information 1, operation information 3, and operation information 2. That is, the order of operation information after parsing is different from the order of data blocks.

[0096] Multiple operation information items after being parsed out of order can be sorted using a global sequence number.

[0097] A transaction can refer to a business event, and a business event can include multiple operational processes; that is, a transaction can correspond to multiple operational information. For example, suppose transaction 1 is a business transaction that can include increasing the data of account 1 by A and decreasing the data of account 2 by A. Then, increasing the data of account 1 by A is operational information 1, and decreasing the data of account 2 by A is operational information 2. That is, the operational information corresponding to transaction 1 are operational information 1 and operational information 2.

[0098] Since multiple log files are stored on the disk, multiple log files can be retrieved from the disk, and multiple data blocks can be determined based on these multiple log files.

[0099] To facilitate the description of the embodiments of this application, the following will be combined with... Figure 4 This section explains the process of retrieving multiple data blocks from the disk.

[0100] Figure 4 This is a schematic diagram illustrating the process of retrieving multiple data blocks from a disk, as provided in an embodiment of this application. The disk includes multiple log files. Data corresponding to the log file's identifier can be retrieved, and this data can be used to identify at least one data block. After retrieving the data corresponding to the log file's identifier, the corresponding data block can be determined based on the identifier of the next log file. By repeating this process, a data block queue consisting of multiple data blocks can be obtained.

[0101] Below, in conjunction with Figure 5 The process of parsing threads in this application is explained.

[0102] Figure 5This is a flowchart illustrating the parsing thread provided in an embodiment of this application. Data blocks can be retrieved from the data block queue. Parsing the data blocks yields operation information, including operation statements, the identifier of the corresponding transaction, and a global sequence number. After adding a memory lock, the operation information is placed into a key information set, and then the memory lock is released. The memory lock allows the key information set to be read but not modified. The parsing progress can be updated using the global sequence number, which facilitates multiple parsing threads performing the parsing process simultaneously.

[0103] S203. Sort multiple operation messages according to the global sequence number to obtain sorted operation messages.

[0104] After multiple parsing threads parse multiple data blocks, multiple operation messages can be sorted according to the global sort number so that the multiple operation messages are arranged in the order of operations recorded in the log file.

[0105] For example, suppose there are 3 operation messages, namely operation message 1-3. The operation statement, transaction identifier and global sequence number can be as shown in Table 2. After sorting according to the global sequence number, the arrangement order can be as shown in Table 3.

[0106] Table 2

[0107] Operation Information 1 Operation statement 1 Transaction 1 3 Operation Information 2 Operation statement 2 Transaction 2 2 Operation Information 3 Operation statement 3 Transaction 2 1

[0108] Table 3

[0109] Operation Information 3 Operation statement 3 Transaction 2 1 Operation Information 2 Operation statement 2 Transaction 2 2 Operation Information 1 Operation statement 1 Transaction 1 3

[0110] S204. Determine at least one operation message corresponding to each transaction from the sorted multiple operation messages.

[0111] Based on the transaction identifier, at least one operation information corresponding to each transaction can be determined, and the at least one operation information is arranged in the order of operation.

[0112] For example, assuming that the sorted operation information is as shown in Table 3, the operation information corresponding to transaction 2 can be determined as operation information 3 and operation information 2 according to the transaction identifier, and the operation time of operation information 3 is earlier than the operation time of operation information 2; the operation information corresponding to transaction 1 is operation information 1.

[0113] S205. Generate a transaction file based on at least one operation information corresponding to each transaction.

[0114] At least one operation information corresponding to a transaction can be persisted and stored on disk as a transaction file.

[0115] Persistent storage refers to storing temporarily stored data on a disk, thus transforming the data into data that can be stored for a long time.

[0116] Any transaction file can contain operation information corresponding to at least one transaction.

[0117] The memory usage of transaction files can be preset according to requirements, but this application embodiment does not limit the preset value.

[0118] For example, assuming the default value of the transaction file can be 300M, and the operation information and memory corresponding to multiple transactions can be as shown in Table 4, then transaction file 1 and transaction file 2 can be generated. Transaction file 1 includes transaction 1 and transaction 2, and transaction file 2 includes transaction 3.

[0119] Table 4

[0120]

[0121] S206. Send the transaction file to the target device.

[0122] To facilitate the description of the embodiments of this application, the following will be combined with... Figure 6 The process of sending transaction files to the target device is described.

[0123] Figure 6 This is a schematic diagram illustrating the process of sending transaction files according to an embodiment of this application. Multiple transaction files are stored on the disk. After obtaining any one of the transaction files, the data in the transaction file is read, and the parsing result (operation information) is determined based on the data. The parsing result is stored in a parsing result queue. This process is repeated, and the parsing result queue may include multiple operation information entries. Finally, the parsing result queue is sent to the target device.

[0124] Once any transaction file has been sent to the target device, it can be deleted from the disk.

[0125] The data migration method provided in this application embodiment can parse and process multiple log files of operations performed in the source database to obtain multiple operation information. The operation information includes operation statements, as well as the identifier and global sequence number of the transaction corresponding to the operation statement. Based on the identifier and global sequence number of the transaction, at least one operation information corresponding to the transaction can be determined, a transaction file can be generated, and the transaction file can be sent to the target device, thereby improving the migration efficiency of log files.

[0126] Figure 7 This is a flowchart illustrating another data migration method provided in an embodiment of this application. Please refer to... Figure 7 The method may include:

[0127] S701, Get the log file download progress.

[0128] The execution process of S701 can be found in the execution process of S201, and will not be repeated here.

[0129] S702: Download multiple data blocks in parallel using multiple download threads and store the data blocks in a data block queue.

[0130] Any data segment to be downloaded can obtain the corresponding data based on the log file identifier and the data segment offset, and then divide the data into multiple data blocks.

[0131] Below, in conjunction with Figure 8 This application describes the data download process for any one of the download threads.

[0132] Figure 8 This is a schematic diagram of the data download process provided in this application embodiment. The queue of data segments to be downloaded includes multiple data segments. A data segment to be downloaded can be obtained from the queue. Based on the log file identifier and the corresponding data segment offset within the data segment, the data corresponding to that data segment is retrieved from the source database. This data identifies multiple data blocks, which are then placed into the data block queue. After retrieving the data corresponding to the data segment to be downloaded from the source database, a new data segment to be downloaded can be retrieved, and the above process can be repeated. The data block queue includes multiple data blocks.

[0133] Because multiple download threads execute the download process concurrently, multiple data blocks can be obtained at the same time, resulting in an unordered arrangement of multiple data blocks in the obtained data block queue.

[0134] For example, suppose there are two threads concurrently executing the download process, namely thread 1 and thread 2, and there are two data segments to be downloaded, namely data segment 1 and data segment 2. Suppose that thread 1 downloads the data corresponding to data segment 1 from the source database, obtaining data block 1 at time 1 and data block 2 at time 2; and thread 2 downloads the data corresponding to data segment 1 from the source database, obtaining data block 3 at time 3. If time 1 is earlier than time 3, and time 3 is earlier than time 2, then the sorting order of the data blocks in the data block queue is data block 1, data block 3, and data block 2.

[0135] S703: Obtain multiple data blocks from the data block queue through a sorting thread.

[0136] S704. Sort the multiple data blocks according to their data block information to obtain multiple ordered data blocks.

[0137] Data block information may include the identifier of the log file corresponding to the data block, and the sequence number of the data block in the corresponding log file.

[0138] At least one corresponding data block can be identified by the identifier of the log file, and then the data blocks in the log file can be sorted according to their sequence number in the corresponding log file.

[0139] For example, assuming there are 3 data blocks, the data block information can be as shown in Table 5. The data block corresponding to log file 1 is data block 1, and the data blocks corresponding to log file 2 are data block 1 and data block 2. Then, the data blocks are sorted according to the sequence number in the log file, and the resulting ordered data blocks can be as shown in Table 6.

[0140] Table 5

[0141] Data block 1 Log file 1 11 Data block 2 Log file 2 22 Data Block 3 Log file 2 21

[0142] Table 6

[0143] Data Block 1 Log file 1 11 Data Block 3 Log file 2 21 Data block 2 Log file 2 22

[0144] S705. Store multiple ordered data blocks to disk in the form of log files to obtain multiple log files.

[0145] After persisting multiple ordered data blocks to the disk, the data needs to be retrieved from the disk again during the data parsing process, thus decoupling the data download process from the data parsing process.

[0146] Multiple data blocks are stored in the cache in sequence, and when the amount of data in the cache is greater than or equal to the first threshold, the data blocks in the cache are stored to the disk, until multiple log files are obtained.

[0147] To facilitate the description of the embodiments of this application, the following will be combined with... Figure 9 The sorting process of multiple data blocks in this application is described.

[0148] Figure 9 This is a schematic diagram illustrating the data block sorting process provided in this application embodiment. Data is retrieved from the data block queue, and this data includes multiple data blocks. The multiple data blocks are sorted by the identifier of the log file and the sequence number of the data block in the corresponding log file to obtain multiple ordered data blocks, and the multiple ordered data blocks are cached.

[0149] If the cache size is greater than the first threshold, then the multiple ordered data blocks in the cache are persisted to disk; if the cache size is less than the first threshold, the process of fetching data is repeated until the cache size is greater than the first threshold, at which point the multiple ordered data blocks in the cache are persisted to disk. Additionally, if the last data block corresponding to the identifier of any log file is already in the cache, then the multiple ordered data blocks in the cache are persisted to disk.

[0150] After persisting the ordered data blocks in the cache to disk, the cache can be cleared, and then you can return to the data retrieval step. This process can continuously persist the data in the data block queue to disk.

[0151] Multiple data blocks on the disk are stored in the form of log files, and each log file corresponds to at least one data block.

[0152] S706. Parse and process multiple log files to obtain multiple operation messages.

[0153] S707. Sort multiple operation messages according to the global sequence number to obtain sorted operation messages.

[0154] S708. Determine at least one operation message corresponding to each transaction from the sorted multiple operation messages.

[0155] The execution process of S706-S708 can be found in the execution process of S202-S204, and will not be repeated here.

[0156] S709. For any transaction, if the memory occupied by at least one operation information corresponding to the transaction is greater than or equal to the second threshold, then the at least one operation information corresponding to the transaction is stored on the disk in the form of a file to obtain the transaction file corresponding to the transaction.

[0157] For example, assuming the second threshold can be 100M, if the memory occupied by operation information 1 and operation information 2 corresponding to transaction 1 is 80M and the memory occupied by operation information 2 is 50M, and the memory occupied by operation information corresponding to transaction 1 is greater than the second threshold, then operation information 1 and operation information 2 corresponding to transaction 1 will be directly stored to disk.

[0158] If the memory occupied by at least one operation information corresponding to a transaction is greater than or equal to the second threshold, then the transaction is a large transaction, and at least one operation information corresponding to a large transaction can be stored on the disk in the form of a large transaction file.

[0159] Zero-copy technology can also be used to copy the large transaction file to a transaction file to improve data transfer performance.

[0160] A transaction file may include at least one operation information corresponding to multiple transactions.

[0161] S710. If the memory occupied by at least one operation information corresponding to a transaction is less than the second threshold, then at least one operation information corresponding to the transaction is stored in the cache until the amount of data in the cache is greater than or equal to the third threshold. Then, the operation information in the cache is stored in the form of a file on the disk to obtain the corresponding transaction file.

[0162] The third threshold is greater than or equal to the second threshold.

[0163] For example, assuming the second threshold is 100M and the third threshold is 300M, transaction 1 corresponds to operation information 1 and operation information 2. The memory required for operation information 1 is 20M and the memory required for operation information 2 is 40M. The memory occupied by transaction 1 is less than the second threshold, so transaction 1 is stored in the cache. Transaction 2 corresponds to operation information 3 and operation information 4. The memory required for operation information 3 is 30M and the memory required for operation information 4 is 25M. The memory occupied by transaction 2 is less than the second threshold, so transaction 1 is stored in the cache. At this time, the memory occupied by transaction 1 and transaction 2 is greater than the third threshold, so transaction 1 and transaction 2 can be stored in the form of transaction file 2.

[0164] S711. Generate a transaction file based on at least one operation information corresponding to each transaction.

[0165] S712, Send transaction file to target device.

[0166] The execution process of S711-S712 can be found in the execution process of S205-206, and will not be repeated here.

[0167] The data migration method provided in this application embodiment can download multiple data blocks according to the download progress of the source log file, sort the multiple data blocks, store them on the disk in the form of multiple log files, parse and process the multiple log files to obtain multiple operation information, sort the multiple operation information, determine at least one operation information corresponding to each transaction, and store at least one operation information corresponding to the transaction on the disk in the form of a transaction file, thereby improving the migration efficiency of log files.

[0168] Figure 10 This is a flowchart illustrating another data migration method provided in an embodiment of this application. Please refer to... Figure 10 The method may include:

[0169] S1001, Determine the first transaction to be rolled back.

[0170] After determining at least one operation information corresponding to each transaction, the first transaction to be rolled back can be determined based on the transaction.

[0171] If at least one operation information corresponding to any transaction is undone or erroneous, then the transaction needs to be rolled back.

[0172] S1002. If the first transaction is in the cache, then delete the first transaction from the cache.

[0173] Optionally, at least one operation information corresponding to the first transaction may be stored in a cache, with some operation information stored on disk.

[0174] S1003. If the first transaction is located on the disk, then delete the first transaction on the disk.

[0175] If the first transaction is located on the disk, then after reading the operation information corresponding to the first transaction from the disk from the end to the beginning, the first transaction is deleted.

[0176] The storage format for operation information can be a symmetrical format, for example, {operation information length: operation information: operation information length}.

[0177] Transaction files can be read from front to back, and rollback can be done from back to front. The storage format of operation information adopts a symmetrical format, which allows complete operation information to be read during both reading and rollback.

[0178] Below, in conjunction with Figure 11 The sorting process of the data migration method provided in the embodiments of this application will be further explained.

[0179] Figure 11 This is a flowchart illustrating the sorting process provided in an embodiment of this application. The minimum progress of the parsing thread can be determined, a memory lock can be acquired, and multiple operation information exceeding the minimum progress can be obtained. These multiple operations are not sorted. Based on the transaction identifier, at least one operation information corresponding to the transaction can be determined. For a given transaction, it can be determined whether it is a transaction to be rolled back. If it is, and the transaction is in the cache, the operation information corresponding to that transaction is directly deleted. If the transaction is on disk, the operation information corresponding to that transaction is read from the disk and then deleted.

[0180] If the transaction is not pending rollback, it is determined whether the transaction exceeds the second threshold. If it exceeds the second threshold, the transaction is persisted to disk and zero-copied to the transaction file. If it is less than the second threshold, the transaction is cached, and it is determined whether the memory occupied by the cache exceeds the third threshold. If it is less than the third threshold, the next transaction is retrieved. If it exceeds the third threshold, the cached transaction is persisted to disk and stored with the information in the transaction file. After clearing the cache, the memory lock is released, and the next transaction is retrieved.

[0181] The data migration method provided in this application embodiment can determine the first transaction to be rolled back based on the transaction. If the first transaction is in the cache, the first transaction is deleted from the cache; if the first transaction is in the disk, the first transaction is deleted from the disk. This can delete the undo or erroneous operation process in the source database and improve the migration efficiency of log files.

[0182] Figure 12 This is a schematic diagram of a data migration device provided in an embodiment of this application. Please refer to... Figure 12 The data migration device 10 includes an acquisition module 11, a first processing module 12, a second processing module 13, a first determination module 14, a generation module 15, and a sending module 16, wherein:

[0183] The acquisition module 11 is used to acquire multiple log files that are used to operate on the source database;

[0184] The first processing module 12 is used to parse and process multiple log files to obtain multiple operation information, including operation statements, as well as the identifier and global sequence number of the transaction corresponding to the operation statement;

[0185] The second processing module 13 is used to sort multiple operation information entries according to their global sequence numbers, thereby obtaining sorted operation information entries.

[0186] The first determining module 14 is used to determine at least one operation information corresponding to each transaction from the sorted multiple operation information;

[0187] The generation module 15 is used to generate a transaction file based on at least one operation information corresponding to each transaction;

[0188] The sending module 16 is used to send transaction files to the target device.

[0189] In one possible implementation, the acquisition module 11 is specifically used for:

[0190] Get the log file download progress;

[0191] Based on the log file download progress, multiple data blocks are downloaded from the source database and stored on the disk as log files to obtain multiple log files.

[0192] In one possible implementation, the acquisition module 11 is specifically used for:

[0193] Multiple data blocks are downloaded in parallel using multiple download threads, and these data blocks are stored in a data block queue.

[0194] The sorting thread retrieves multiple data blocks from the data block queue and sorts them according to their data block information to obtain multiple ordered data blocks. The data block information includes the identifier of the log file corresponding to the data block and the sequence number of the data block in the corresponding log file.

[0195] Multiple ordered data blocks are stored on the disk as log files to obtain multiple log files.

[0196] In one possible implementation, the acquisition module 11 is specifically used for:

[0197] Multiple data blocks are stored in the cache in sequence, and when the amount of data in the cache is greater than or equal to the first threshold, the data blocks in the cache are stored to the disk, until multiple log files are obtained.

[0198] In one possible implementation, the data migration device 10 further includes a first storage module and a second storage module, wherein:

[0199] The first storage module is specifically used to, for any transaction, if the memory occupied by at least one operation information corresponding to the transaction is greater than or equal to the second threshold, store at least one operation information corresponding to the transaction to the disk in the form of a file to obtain the transaction file corresponding to the transaction.

[0200] The second storage module is specifically used to store at least one operation information corresponding to a transaction in a cache if the memory occupied by at least one operation information is less than a second threshold, until the amount of data in the cache is greater than or equal to a third threshold, and then store the operation information in the cache in the form of a file to the disk to obtain the corresponding transaction file.

[0201] In one possible implementation, the data migration apparatus 10 further includes a second determining module, a first deleting module, and a second deleting module, wherein:

[0202] The second determination module is specifically used to determine the first transaction to be rolled back;

[0203] The first deletion module is specifically used to delete the first transaction from the cache if the first transaction is in the cache.

[0204] The second deletion module is specifically used to delete the first transaction from the disk if the first transaction is located on the disk.

[0205] The data migration apparatus provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0206] Figure 13For a schematic diagram of the structure of an electronic device provided in this application embodiment, please refer to [link / reference]. Figure 13 The electronic device 20 may include a processor 21 and a memory 22. Exemplarily, the processor 21 and the memory 22 are interconnected via a bus 23.

[0207] Memory 22 stores instructions executed by the computer;

[0208] The processor 21 executes computer execution instructions stored in the memory 22, causing the processor 21 to perform the data migration method as shown in the above method embodiment.

[0209] Accordingly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data migration method of the above-described method embodiments.

[0210] Accordingly, embodiments of this application may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the data migration method shown in the above method embodiments.

[0211] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0212] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0213] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0214] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0215] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0216] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0217] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0218] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0219] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0220] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data migration method, characterized in that, include: Get the log file download progress; Based on the log file download progress, multiple data blocks are downloaded in parallel from the source database using multiple download threads, and the multiple data blocks are stored in a data block queue. The sorting thread retrieves the multiple data blocks from the data block queue and sorts them according to their data block information to obtain multiple ordered data blocks. The data block information includes the identifier of the log file corresponding to the data block and the sequence number of the data block in the corresponding log file. The multiple ordered data blocks are stored on the disk in the form of log files to obtain multiple log files; The multiple log files are parsed to obtain multiple operation information entries. The operation information includes operation statements, as well as the identifier and global sequence number of the transaction corresponding to the operation statement; the global sequence number is the number of the data block's arrangement order in the log file. The multiple operation information is sorted according to the global sequence number to obtain sorted multiple operation information, and at least one operation information corresponding to each transaction is determined in the sorted multiple operation information. A transaction file is generated based on at least one operation information corresponding to each transaction, and the transaction file is sent to the target device.

2. The method according to claim 1, characterized in that, The plurality of ordered data blocks are stored on disk in the form of log files to obtain the plurality of log files, including: The multiple data blocks are stored in a cache in the order they are ordered. When the amount of data in the cache is greater than or equal to a first threshold, the data blocks in the cache are stored on the disk until the multiple log files are obtained.

3. The method according to claim 1, characterized in that, After determining at least one operation message corresponding to each transaction from the sorted multiple operation messages, the method further includes: For any given transaction, if the memory occupied by at least one operation information corresponding to the transaction is greater than or equal to a second threshold, then at least one operation information corresponding to the transaction is stored on the disk in the form of a file to obtain the transaction file corresponding to the transaction. If the memory occupied by at least one operation information corresponding to the transaction is less than the second threshold, then at least one operation information corresponding to the transaction is stored in the cache until the amount of data in the cache is greater than or equal to the third threshold. Then, the operation information in the cache is stored in the form of a file on the disk to obtain the corresponding transaction file.

4. The method according to claim 1, characterized in that, The method further includes: Identify the first transaction to be rolled back; If the first transaction is in the cache, then delete the first transaction from the cache; If the first transaction is located on a disk, then delete the first transaction on that disk.

5. A data migration device, characterized in that, It includes an acquisition module, a first processing module, a second processing module, a first determination module, a generation module, and a sending module, wherein: The acquisition module is used to acquire multiple log files that are used to operate on the source database; The first processing module is used to parse and process the multiple log files to obtain multiple operation information, the operation information including operation statements, and the identifier and global sequence number of the transaction corresponding to the operation statement; The second processing module is used to sort the multiple operation information entries according to the global sequence number to obtain sorted multiple operation information entries. The first determining module is used to determine at least one operation information corresponding to each transaction from the sorted multiple operation information; The generation module is used to generate a transaction file based on at least one operation information corresponding to each transaction. The sending module is used to send the transaction file to the target device; The acquisition module is specifically used to acquire the download progress of the log file; Based on the log file download progress, multiple data blocks are downloaded in parallel from the source database using multiple download threads, and the multiple data blocks are stored in a data block queue. The sorting thread retrieves the multiple data blocks from the data block queue and sorts them according to their data block information to obtain multiple ordered data blocks. The data block information includes the identifier of the log file corresponding to the data block and the sequence number of the data block in the corresponding log file. The multiple ordered data blocks are stored on the disk in the form of log files to obtain the multiple log files; the global sequence number is the number of the order in which the data blocks are arranged in the log files.

6. An electronic device, characterized in that, include: Memory and processor The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the data migration method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data migration method according to any one of claims 1 to 4.

8. A computer program product comprising a computer program that, when executed by a processor, implements the data migration method according to any one of claims 1 to 4.

Images