A Method and Device for Continuable Transmission of Full and Incremental Migration Data Connection

Abstract

The present application provides a database migration method, in particular a method for seamless connection of resumable full-incremental migration data from an Oracle database to a GaussDB database. This method includes steps such as creating a data migration task, running the data migration task, collecting incremental data, performing full-scale data migration, applying incremental data, creating an application site table, and handling resumption of transmission. By optimizing the performance of full-scale migration, achieving seamless connection from full-scale to incremental migration, and the breakpoint resumption function, this method can ensure the accuracy, integrity, and consistency of data during the migration to the GaussDB database, realizing accurate, non-lost, and non-duplicate data during the connection process of full-scale migration and incremental migration, and at the same time having strong anti-interference ability and breakpoint resumption ability.

Description

Technical Field

[0001] This application belongs to the technical field of data migration, and particularly relates to a method, apparatus, computer-readable storage medium, and electronic device for seamless connection of resumable full-increment data migration. Background Art

[0002] With the development of domestic database technology and the maturity of products, many institutions and enterprises have begun to consider migrating Oracle business data to domestic databases. During the migration process, due to the very short downtime window for important services, when the volume of business data is large, it is very difficult to complete the migration within the limited downtime only through full-volume migration. Therefore, a technical solution combining full-volume migration and incremental migration is needed to solve this problem.

[0003] Full-volume migration refers to the process of migrating all data from the source database to the target database at one time, while incremental migration refers to the process of continuously synchronizing the newly added or changed data in the source database to the target database after the full-volume migration is completed. The connection between these two migration methods is a technical difficulty, which not only needs to ensure the consistency and integrity of the data, but also needs to consider the continuity of the business. Therefore, developing a data migration method that can ensure the accurate connection between full-volume migration and incremental migration has become an urgent need in this industry. Summary of the Invention

[0004] To solve the above problems, this application proposes a new method for seamless connection of resumable full-increment data migration from Oracle to GaussDB.

[0005] Specifically, this application provides the following technical solutions:

[0006] In the first aspect of this application, a method for seamless connection of resumable full-increment data migration is provided, and the method includes:

[0007] Create a data migration task for the source database and the target database;

[0008] Run the data migration task, first perform the migration of the database table structure and the full-volume data migration, and obtain the starting point for incremental data collection before migrating the full-volume data;

[0009] The incremental collector starts running from the starting point, collects the incremental data changes in the source database during the full-volume data migration, and sends the collected data to the message queue;

[0010] Perform the full-volume data migration in the way of one table per thread. For each table, obtain the current snapshot position before migration and use this snapshot position for snapshot reading;

[0011] After all tables have completed the full - volume data migration, start the incremental data applicator, pull incremental data from the message queue for application, and perform corresponding processing according to the relationship between the commit position and the snapshot position of the incremental data;

[0012] Before applying incremental data, create an application position table to record the commit position and transaction number of the currently applied data, so as to filter duplicate data and only apply data that has not been applied before;

[0013] When the data migration component crashes and restarts, continue to collect incremental data starting from the recorded commit position to ensure that data is not lost;

[0014] When applying incremental data, only apply eligible incremental data according to the commit position and transaction number recorded in the application position table.

[0015] Optionally, in the method of this application, the source database is an Oracle database, the target database is a GaussDB database; the message queue is Kafka.

[0016] Optionally, in the method of this application, the starting point of incremental data collection is the minimum transaction start point of all uncommitted transactions in the source database to ensure that data of uncommitted transactions is not lost.

[0017] Optionally, in the method of this application, after the incremental collector receives the signal that the message queue has successfully received incremental data, record the latest start point of this incremental data as the collection start position point. If the program is abnormally interrupted, incremental collection will start again from this collection start position point.

[0018] Optionally, in the method of this application, during the full - volume data migration process, each table obtains the current snapshot position before starting the migration to ensure that the snapshot of migrating a certain table does not become invalid.

[0019] Optionally, in the method of this application, when the incremental data applicator applies incremental data, if the commit position of the incremental data is less than or equal to the snapshot position, discard the data; if the commit position of the incremental data is greater than the snapshot position, send it to the target database for application execution.

[0020] Optionally, in the method of this application, the application position table is used to record the commit position and transaction number of the currently applied transaction, and its update event is executed in the same transaction as the incremental data application to ensure that the commit position and transaction number recorded in the application position table must be those of successful applications;

[0021] When the data migration component crashes and restarts, only apply incremental data that meets the conditions according to the commit position and transaction number of the currently applied data strictly recorded in the application position table of the target database.

[0022] Optionally, in the method of the present application, after the data migration component sends the increment data collected to the message queue and receives the response, it records the commit position of the last commit event of this batch of increment data in a file, so that after the component crashes and restarts, it continues to collect starting from the commit position recorded in the file.

[0023] Optionally, in the method of the present application, when applying the increment data, the application condition is that the commit position of the increment data is greater than the commit position of the last applied data. When the two commit positions are the same, only the data with different transaction numbers is applied.

[0024] The second aspect of the present application provides a continuous transfer full-increment migration data connection device, and the device includes:

[0025] A creation module for creating data migration tasks for the source library and the target library;

[0026] A migration module for running data migration tasks, including performing library table structure migration and full amount data migration, and obtaining the starting point of increment data collection before migrating the full amount of data;

[0027] An increment collection module for starting to run from the starting point, collecting increment data changes in the source database during the full amount data migration, and sending the collected data to the message queue;

[0028] A full amount migration module for performing full amount data migration in the manner of one table per thread. Before migrating each table, it obtains the current snapshot position and uses this snapshot position for snapshot reading;

[0029] An increment application module for starting the increment data applicator after all tables have completed full amount data migration, pulling increment data from the message queue for application, and performing corresponding processing according to the relationship between the commit position and the snapshot position of the increment data;

[0030] A position recording module for creating an application position table to record the commit position and transaction number of the currently applied data before applying the increment data, so as to filter duplicate data and only apply the data that has not been applied;

[0031] A continuous transfer processing module for continuing to collect increment data starting from the recorded commit position after the data migration component crashes and restarts, ensuring that data is not lost;

[0032] A condition application module for only applying eligible increment data according to the commit position and transaction number recorded in the application position table when applying increment data.

[0033] When the device runs, it implements the steps of the aforementioned continuous transfer full-increment migration data connection method.

[0034] A third aspect of the present application provides an electronic device, including: a memory and a processor;

[0035] The memory: is used for storing a computer program;

[0036] The processor: is used for executing the computer program to implement the steps of the aforementioned method for seamless connection of continuously transferable full-increment migration data.

[0037] A fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the aforementioned method for seamless connection of continuously transferable full-increment migration data are implemented.

[0038] In summary, the method for seamless connection of continuously transferable full-increment migration data from Oracle to GaussDB proposed in the present application can ensure the accuracy, integrity and consistency of data during the migration to the GaussDB database by optimizing the full-increment migration performance, realizing the seamless connection from full-increment to incremental migration, and the breakpoint continuation function, so as to achieve accurate, non-lost and non-duplicated data during the connection process of full-increment migration and incremental migration, and at the same time have strong anti-interference ability and breakpoint continuation ability.

[0039] Other features and advantages of the present application will be elaborated in detail in the subsequent description, or can be understood by implementing the relevant technical solutions of the present application. The objectives and other advantages of the present application can be achieved by the technical features and technical means clearly pointed out in the description, claims and drawings, and obtained through the implementation process of these technical contents. Description of the Drawings

[0040] In order to more clearly elaborate the technical solutions of the embodiments of the present application, the drawings involved in the description of the embodiments will be briefly introduced below. It should be noted that the drawings only show some embodiments of the present application. For those skilled in the art, other relevant drawings can be deduced based on these drawings without creative labor.

[0041] Figure 1 It is the overall implementation flowchart of the method for seamless connection of continuously transferable full-increment migration data from Oracle to GaussDB of the present application.

[0042] Figure 2 It is a schematic diagram of the connection process between full-increment migration and incremental migration in the method of the present application.

[0043] Figure 3 It is a schematic diagram of the breakpoint continuation process in the method of the present application.

[0044] Figure 4 It is the composition structure diagram of the device for seamless connection of continuously transferable full-increment migration data from Oracle to GaussDB of the present application.

[0045] Figure 5 It is a schematic structural diagram of the electronic device provided by the embodiment of the present application. Specific embodiments

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0047] In this document, the term "including" and any form of its deformation (such as "including", "including") are open-ended expressions and should be understood as "including but not limited to", that is, the listed content is not an exhaustive list and may also include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on", that is, the referred basis or condition may not be the only factor and may also involve other relevant factors. The term "an embodiment" should be understood as "at least one embodiment", that is, the described embodiment is not the only possible implementation and there may be other similar embodiments.

[0048] In the present application, when the terms "one" and "multiple" are used to modify relevant elements or features, their expressions are illustrative rather than restrictive. Unless otherwise clearly stated in the context, "one" should be understood as "at least one", and "multiple" should be understood as "at least two". Those skilled in the art should reasonably interpret these terms according to the semantic and logical relationships in the context to ensure that they cover the possibility of "one or more".

[0049] Figure 1 The following shows the overall implementation process of the continuous full-increment migration data connection method provided by the present application, including the following steps:

[0050] Create a data migration task for the source library and the target library;

[0051] Run the data migration task, first perform the migration of the database table structure and the full amount of data, and obtain the starting point of incremental data collection before migrating the full amount of data;

[0052] The incremental collector starts running from the starting point, collects the incremental data changes of the source library during the full amount of data migration, and sends the collected data to the message queue;

[0053] Perform the full amount of data migration in the way of one thread per table. Before migrating each table, obtain the current snapshot site and use this snapshot site for snapshot reading;

[0054] After all tables have completed the full data migration, start the incremental data applicator to pull incremental data from the message queue for application, and perform corresponding processing according to the relationship between the commit position and the snapshot position of the incremental data;

[0055] Before applying the incremental data, create an application position table to record the commit position and transaction number of the currently applied data to filter duplicate data and only apply the data that has not been applied;

[0056] When the data migration component crashes and restarts, continue to collect incremental data starting from the recorded commit position to ensure that data is not lost;

[0057] When applying incremental data, only apply the incremental data that meets the conditions according to the commit position and transaction number recorded in the application position table.

[0058] To more clearly illustrate the technical solution of this application, the following will be further described through embodiments of specific scenarios.

[0059] The following takes the application of this solution in the exBase database migration system (referred to as exBase) as an example for illustration, as Figure 2 and Figure 3 shown, including the following steps:

[0060] 1) exBase creates a data migration task with the source library as Oracle and the target library as GaussDB.

[0061] 2) Run the data migration task, first perform the library table structure migration and full data migration. Before migrating the full data, it is necessary to obtain the SCN starting point for incremental data collection (referred to as the collection starting point). Since the events of uncommitted transactions in Oracle are written to the REDO log in advance, the collection starting point cannot directly obtain the current SCN. It is necessary to give priority to using the minimum transaction start SCN of all uncommitted transactions in Oracle as the collection starting point to ensure that the data of uncommitted transactions is not lost.

[0062] 3) The incremental collector starts running from the collection starting point obtained in step 2), collects the incremental data changes in Oracle during the full data migration, and sends the collected data to Kafka. After the incremental collector receives the signal that Kafka has successfully received the incremental data, it records the latest SCN of this incremental data as the collection SCN position. If the program is abnormally interrupted, the incremental collection will start collecting and sending again from this collection SCN position.

[0063] 4) exBase migrates all data by one table per thread (since there may be many tables in Oracle, to ensure that the snapshot SCN of a table being migrated does not become invalid, the current SCN needs to be obtained as the snapshot SCN before starting to migrate each table). Before data migration, each table records the current SCN of Oracle (snapshot SCN). When Oracle queries data, snapshot reads are performed using the snapshot SCN.

[0064] 5) After all tables in the entire database have completed full - volume data migration, exBase starts the incremental data applicator and begins to pull incremental data from Kafka for application. The incremental data of a table will carry the SCN at the time of transaction commit (abbreviation: CommitSCN). When the CommitSCN is less than or equal to the snapshot SCN, it is considered data before full - volume data migration and is discarded; when the CommitSCN is greater than the snapshot SCN, it is sent to GaussDB for application execution.

[0065] 6) There is a key process before step 5) during application execution. To avoid collecting duplicate incremental data, an application position table is created in GaussDB in advance by the incremental application (the update of this table is carried in the transaction of applying incremental data, enabling the incremental applicator to filter incremental data and only apply data that has not been applied). This table is used to record the CommitSCN and transaction ID TxId of the current application transaction. The application in step 5) is also executed in transactions. exBase will add an additional record to update the application position table in the transaction, ensuring that the CommitSCN and TxId recorded in the application position table are definitely those of a successfully applied transaction.

[0066] 7) The above basically describes the entire process of full - volume and incremental migration. Next, it explains how to perform resume - transfer processing. After exBase sends incremental data collection to Kafka and Kafka responds with successful reception, it will record the CommitSCN of the last commit event of this batch of incremental data in a file. When exBase crashes and restarts due to various factors, exBase will start collecting from the CommitSCN recorded in the file as the starting point. In this way, even if the CommitSCN is not recorded in the file in time, data will not be lost, only some additional data will be collected.

[0067] 8) When exBase crashes and restarts, since the application position table in GaussDB strictly records the CommitSCN and TxId of the currently applied data, only incremental data that meets the conditions will be applied. The application conditions are: the CommitSCN of the incremental data is greater than the CommitSCN of the previously applied data. When the two CommitSCNs are the same, only data with different TxIds will be applied.

[0068] Figure 4 The following shows a data connection device for resumable full-increment migration proposed by this application. The device includes:

[0069] A creation module, used to create data migration tasks for the source library and the target library;

[0070] A migration module, used to run data migration tasks, including performing library table structure migration and full-volume data migration, and obtaining the starting point of incremental data collection before migrating full-volume data;

[0071] An incremental collection module, used to start running from the starting point, collect incremental data changes in the source database during full-volume data migration, and send the collected data to the message queue;

[0072] A full-volume migration module, used to perform full-volume data migration in the way of one thread per table. Each table obtains the current snapshot site before migration and uses this snapshot site for snapshot reading;

[0073] An incremental application module, used to start an incremental data applicator after all tables have completed full-volume data migration, pull incremental data from the message queue for application, and perform corresponding processing according to the relationship between the commit site and the snapshot site of the incremental data;

[0074] A site recording module, used to create an application site table to record the commit site and transaction number of the currently applied data before applying incremental data, so as to filter duplicate data and only apply unapplied data;

[0075] A resumption processing module, used to continue collecting incremental data starting from the recorded commit site after the data migration component crashes and restarts, ensuring that data is not lost;

[0076] A conditional application module, used to only apply eligible incremental data according to the commit site and transaction number recorded in the application site table during incremental data application.

[0077] When the above device runs, it implements the steps of the resumable full-increment migration data connection method disclosed in this application.

[0078] The flowcharts and block diagrams in the accompanying drawings show possible implementation manners of devices, methods, and computer program products according to various embodiments of this application, including system architectures, functions, and operations. In these figures, each box may represent a module, a program segment, or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should be noted that each box in the block diagram and / or flowchart, as well as combinations of these boxes, can be implemented by a dedicated hardware-based system to implement the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0079] AsFigure 5 As shown, an embodiment of the present application further discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing computer programs executable by the processor, and a communication bus 340. Among them, the processor 310, the communication interface 320, and the memory 330 complete communication with each other through the communication bus 340. The processor 310 runs the executable computer program to implement the steps of the above-mentioned method for seamless connection of continuously transferable full-increment migration data.

[0080] It can be understood that in addition to including a memory and a processor, the electronic device may further include an input device (such as a keyboard), an output device (such as a display), and other communication modules. These input devices, output devices, and other communication modules communicate with the processor through an I / O interface (i.e., an input / output interface).

[0081] The operations of the present application can be implemented by writing computer program code using one or more programming languages or combinations thereof. The programming languages include but are not limited to the following types:

[0082] Object-oriented programming languages, such as Java, Smalltalk, C++, etc.;

[0083] Conventional procedural programming languages, such as the "C" language or similar programming languages.

[0084] The execution modes of the program code include but are not limited to:

[0085] Fully executed on the user's computer;

[0086] Partially executed on the user's computer and partially executed on a remote computer;

[0087] Executed as an independent software package;

[0088] Fully executed on a remote computer or server.

[0089] In scenarios involving a remote computer, the remote computer can be connected to the user's computer through any type of network, and the network includes but is not limited to a local area network (LAN) or a wide area network (WAN). In addition, the remote computer can also be connected to an external computer through an Internet service provider, for example, by using the Internet for connection.

[0090] Furthermore, the present application also discloses a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can execute each step of the method for seamless connection of continuously transferable full-increment migration data disclosed in the present application.

[0091] In the context of the present application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following:

[0092] (1) Portable computer disk: A removable magnetic storage medium such as a floppy disk.

[0093] (2) Hard disk: A fixed storage device including a mechanical hard disk and a solid-state drive, etc.

[0094] (3) Random Access Memory (RAM): A volatile storage medium for temporarily storing data and program code.

[0095] (4) Read-Only Memory (ROM): A non-volatile storage medium for storing fixed programs and data.

[0096] (5) Erasable Programmable Read-Only Memory (EPROM) or Flash Memory: A non-volatile storage medium that supports multiple erasures and programming.

[0097] (6) Fiber optic storage device: A storage medium based on fiber optic technology.

[0098] (7) Portable Compact Disc Read-Only Memory (CD-ROM): A read-only medium for storing data in the form of an optical disc.

[0099] (8) Optical storage device: A storage medium based on optical principles such as DVD, Blu-ray Disc, etc.

[0100] (9) Magnetic storage device: A storage medium based on magnetic principles such as magnetic tape, magnetic disk, etc.

[0101] (10) Any suitable combination of the above: For example, multiple storage media are combined to meet different storage requirements.

[0102] These computer-readable storage media can be used to store the program code and related data described in the present application to support the operation of the program and the persistent storage of data.

[0103] In particular, according to the embodiments of the present application, the processes described in the flowchart can be implemented as computer software programs. For example, the embodiments of the present application relate to a computer program product that includes a computer program carried on a non-transitory computer-readable medium. The computer program contains program code for executing the data connection method of continuous full incremental migration disclosed in the present application. When the computer program is executed by a processing device, the above functions defined in the embodiments of the present application can be realized.

[0104] Although the above description contains several specific implementation details, these details should not be construed as limiting the scope of the present application. The above description is only a preferred embodiment of the present application and an explanation of the applied technical principles. Those skilled in the art should understand that the disclosed scope of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features. At the same time, the present application should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept.

[0105] Those skilled in the art should also understand that they can modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features without departing from the spirit and scope of the technical solutions of the embodiments of the present application. These modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for connecting data in full-incremental migration with resume function, characterized in that Including the following steps: Create a data migration task for the source library and the target library; Run the data migration task, first perform the migration of the database table structure and the full - volume data migration, and obtain the starting point of incremental data collection before migrating the full - volume data; The incremental collector starts running from the starting point, collects the incremental data changes of the source library during the full - volume data migration, and sends the collected data to the message queue; Perform the full - volume data migration in the way of one thread per table. Before migrating each table, obtain the current snapshot site and use this snapshot site for snapshot reading; After all tables complete the full - volume data migration, start the incremental data applicator, pull the incremental data from the message queue for application, and perform corresponding processing according to the relationship between the commit site and the snapshot site of the incremental data; Before applying the incremental data, create an application site table to record the commit site and transaction number of the currently applied data to filter duplicate data and only apply the data that has not been applied; When the data migration component crashes and restarts, continue to collect incremental data starting from the recorded commit site; When applying incremental data, according to the commit site and transaction number recorded in the application site table, only apply the incremental data that meets the conditions.

2. The method according to claim 1, wherein The source library is an Oracle database, and the target library is a GaussDB database; the message queue is Kafka.

3. The method according to claim 1, wherein The starting point of the incremental data collection is the minimum transaction start starting point of all uncommitted transactions in the source library to ensure that the data of uncommitted transactions is not lost.

4. The method according to claim 1, wherein After receiving the signal that the message queue has successfully received the incremental data, the incremental collector records the latest starting point of this incremental data as the collection starting point site. If the program is abnormally interrupted, the incremental collection will start collecting and sending again from this collection starting point site.

5. The method according to claim 1, characterized in that During the full - volume data migration process, each table obtains the current snapshot site before starting the migration to ensure that the snapshot of a certain table does not become invalid.

6. The method according to claim 1, wherein When the incremental data applicator applies incremental data, if the commit site of the incremental data is less than or equal to the snapshot site, discard the data; if the commit site of the incremental data is greater than the snapshot site, send it to the target library for application execution.

7. The method according to claim 1, characterized in that The application site table is used to record the commit site and transaction number of the currently applied transaction, and its update event is executed in the same transaction as the incremental data application to ensure that the commit site and transaction number recorded in the application site table must be successfully applied; When the data migration component crashes and restarts, according to the commit site and transaction number of the currently applied data strictly recorded in the application site table of the target database, only apply the incremental data that meets the conditions.

8. The method according to claim 1, characterized in that, After the data migration component sends the incremental data collection to the message queue and receives the response, record the commit site of the last commit event of this batch of incremental data in a file, so that after the component crashes and restarts, continue to collect starting from the commit site recorded in the file.

9. The method according to claim 1, wherein When applying incremental data, the application condition is: the commit site of the incremental data is greater than the commit site of the last applied data. When the two commit sites are the same, only apply the data with different transaction numbers.

10. A data connection device for continuous transfer of full and incremental migration, characterized in that, The device includes: A creation module for creating a data migration task for the source library and the target library; Migration module, which is used to run data migration tasks, including performing library table structure migration and full-volume data migration, and obtaining the starting point of incremental data collection before migrating full-volume data; Incremental collection module, which is used to start running from the starting point, collect incremental data changes in the source database during full-volume data migration, and send the collected data to the message queue; Full-volume migration module, which is used to perform full-volume data migration in the way of one thread per table. Each table obtains the current snapshot position before migration and uses this snapshot position for snapshot reading; Incremental application module, which is used to start the incremental data applicator after all tables have completed full-volume data migration, pull incremental data from the message queue for application, and perform corresponding processing according to the relationship between the commit position and the snapshot position of the incremental data; Position recording module, which is used to create an application position table to record the commit position and transaction number of the currently applied data before applying incremental data, so as to filter duplicate data and only apply the data that has not been applied; Resume transfer processing module, which is used to continue collecting incremental data from the recorded commit position as the starting point after the data migration component crashes and restarts, ensuring that data is not lost; Condition application module, which is used to only apply incremental data that meets the conditions according to the commit position and transaction number recorded in the application position table during incremental data application.

Images