Continuous transmission full-increment migration data connection method and device

Abstract

The invention provides a database migration method, and particularly relates to a continuous transmission full increment migration data connection method for migrating an Oracle database to a GaussDB database. The method comprises the steps of creating a data migration task, operating the data migration task, collecting incremental data, carrying out total data migration, applying the incremental data, creating an application site table, processing continuous transmission and the like. According to the method, by optimizing the full migration performance, seamless connection from full migration to incremental migration and a breakpoint resume function are achieved, the accuracy, integrity and consistency of data in the process of migration to a GaussDB database can be ensured, data are accurate, not lost and not repeated in the connection process of full migration and incremental migration, and the data migration efficiency is improved. And meanwhile, the anti-interference capability and the breakpoint resume capability are relatively high.

Description

Technical Field

[0001] The present application belongs to the technical field of data migration, and in particular, relates to a method, device, computer-readable storage medium and electronic device for connecting data with resumable full-incremental migration. Background Art

[0002] With the development of domestic database technology and the maturity of products, many institutions and enterprises have begun to consider migrating Oracle business data to domestic databases. During the migration process, since the downtime cutover window for important businesses is very short, when the amount of business data is large, it is difficult to complete the migration within the limited downtime by only using full migration. Therefore, a technical solution combining full migration and incremental migration is needed to solve this problem.

[0003] Full migration refers to the process of migrating all data from the source database to the target database at one time, while incremental migration refers to the process of continuously synchronizing the newly added or changed data in the source database to the target database after the full migration is completed. The connection between these two migration methods is a technical difficulty, which requires ensuring the consistency and integrity of the data while also considering business continuity. Therefore, developing a data migration method that can ensure the precise connection between full migration and incremental migration has become an urgent need in this industry. Summary of the invention

[0004] In order to solve the above problems, this application proposes a new method for connecting data from Oracle to GaussDB with resumable full-incremental migration.

[0005] Specifically, this application provides the following technical solutions: A first aspect of the present application provides a method for connecting data with resumable full-incremental migration, the method comprising: Create data migration tasks for source and target databases; Run the data migration task, first migrate the database table structure and migrate all the data, and obtain the starting point for incremental data collection before migrating all the data; The incremental collector starts running from the starting point, collects incremental data changes of the source database during the full data migration, and sends the collected data to the message queue; Full data migration is performed in one thread per table. Before migration, each table obtains the current snapshot location and uses the snapshot location for snapshot reading. After all tables have completed full data migration, start the incremental data applier to pull incremental data from the message queue for application, and perform corresponding processing based on the relationship between the incremental data submission point and the snapshot point. Before applying incremental data, create an application site table to record the submission site and transaction number of the current application data to filter duplicate data and apply only the data that has not been applied; When the data migration component crashes and restarts, it continues to collect incremental data using the recorded submission location as the starting point to ensure that data is not lost; When incremental data is applied, only the incremental data that meets the conditions is applied according to the submission site and transaction number recorded in the application site table.

[0006] Optionally, in the method of the present application, the source database is an Oracle database, the target database is a GaussDB database; and the message queue is Kafka.

[0007] Optionally, in the method of the present application, the starting point of the incremental data collection is the minimum transaction starting point of all uncommitted transactions in the source database to ensure that the data of the uncommitted transactions is not lost.

[0008] Optionally, in the method of the present application, after receiving the signal from the message queue that the incremental data has been successfully received, the incremental collector records the latest starting point of the incremental data as the collection starting point. If the program is abnormally interrupted, the incremental collection will start collecting and sending again from the collection starting point.

[0009] Optionally, in the method of the present application, during the full data migration process, each table obtains a current snapshot location before starting the migration to ensure that the snapshot of a certain table is not invalidated.

[0010] Optionally, in the method of the present application, when the incremental data applicator applies the incremental data, if the submission site of the incremental data is less than or equal to the snapshot site, the data is discarded; if the submission site of the incremental data is greater than the snapshot site, the data is sent to the target library for application execution.

[0011] Optionally, in the method of the present application, the application site table is used to record the commit site and transaction number of the currently applied transaction, and its update event and incremental data application are executed in the same transaction to ensure that the commit site and transaction number recorded in the application site table must be successfully applied; When the data migration component crashes and restarts, only the incremental data that meets the conditions is applied according to the commit site and transaction number of the current application data strictly recorded in the application site table of the target database.

[0012] Optionally, in the method of the present application, after the data migration component sends the incremental data collection to the message queue and receives a response, the data migration component records the submission location of the last submission event of this batch of incremental data into a file, so that after the component crashes and restarts, the collection can continue with the submission location recorded in the file as the starting point.

[0013] Optionally, in the method of the present application, when the incremental data is applied, the application condition is: the commit site of the incremental data is greater than the commit site of the last applied data, and when the two commit sites are the same, only data with different transaction numbers are applied.

[0014] A second aspect of the present application provides a device for connecting data with resumable full-incremental migration, the device comprising: Create a module to create data migration tasks for source and target libraries; The migration module is used to run data migration tasks, including database and table structure migration and full data migration, and obtain the starting point of incremental data collection before full data migration; An incremental collection module, configured to start running from the starting point, collect incremental data changes of the source database during the full data migration, and send the collected data to a message queue; The full migration module is used to perform full data migration in a one-table-one-thread manner. Before migration, each table obtains the current snapshot location and uses the snapshot location for snapshot reading. The incremental application module is used to start the incremental data applier after all tables have completed full data migration, pull incremental data from the message queue for application, and perform corresponding processing based on the relationship between the commit point and snapshot point of the incremental data; The site record module is used to create an application site table to record the submission site and transaction number of the current application data before applying the incremental data, so as to filter duplicate data and only apply the data that has not been applied; The resume processing module is used to continue collecting incremental data using the recorded submission location as the starting point after the data migration component crashes and restarts, ensuring that data is not lost; The conditional application module is used for incremental data application. According to the submission site and transaction number recorded in the application site table, only the incremental data that meets the conditions is applied.

[0015] The device implements the steps of the aforementioned method for connecting data with resumable full-incremental migration when running.

[0016] A third aspect of the present application provides an electronic device, comprising: a memory and a processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the aforementioned method for connecting data with resumable full-incremental migration.

[0017] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the aforementioned method for connecting data with resumable full-incremental migration are implemented.

[0018] To summarize, the Oracle to GaussDB resumable full-incremental migration data connection method proposed in this application can ensure the accuracy, integrity and consistency of data during migration to the GaussDB database by optimizing the full migration performance, realizing seamless connection from full to incremental migration and breakpoint resume function, and ensure that the data is accurate, without loss or duplication during the connection process of full migration and incremental migration, and at the same time has strong anti-interference ability and breakpoint resume capability.

[0019] Other features and advantages of the present application will be described in detail in the subsequent description, or can be understood by implementing the relevant technical solutions of the present application. The purpose and other advantages of the present application can be achieved through the technical features and technical means clearly indicated in the description, claims and drawings, and obtained through the implementation process of these technical contents. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings involved in the description of the embodiments. It should be noted that the drawings only show some embodiments of the present application. For those skilled in the art, other related drawings can be derived from these drawings without creative work.

[0021] Figure 1 This is the overall implementation flowchart of the method for connecting data from Oracle to GaussDB with resumable full-incremental migration for this application.

[0022] Figure 2 This is a schematic diagram of the connection process between full migration and incremental migration in the present application method.

[0023] Figure 3 This is a schematic diagram of the breakpoint resume process in the method of this application.

[0024] Figure 4 This is a structural diagram of the composition of the data connection device for resumable full-incremental migration from Oracle to GaussDB in this application.

[0025] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be clear that the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present application.

[0027] In this document, the term "including" and any of its variations (such as "including", "including", etc.) are open expressions and should be understood as "including but not limited to", that is, the listed contents are not exhaustive and may also include other contents not explicitly mentioned. The term "based on" should be understood as "based at least in part", that is, the basis or condition referred to may not be the only factor, and other relevant factors may also be involved. The term "one embodiment" should be understood as "at least one embodiment", that is, the described embodiment is not the only possible implementation method, and there may be other similar embodiments.

[0028] In this application, when the terms "one" and "multiple" are used to modify related elements or features, their expressions are illustrative rather than restrictive. Unless otherwise clearly stated in the context, "one" should be understood as "at least one" and "multiple" should be understood as "at least two". Those skilled in the art should reasonably interpret these terms based on the semantics and logical relationship of the context to ensure that they cover the possibility of "one or more".

[0029] Figure 1 The overall implementation process of the method for connecting data with full incremental migration with resumable transfer provided by this application is shown, including the following steps: Create data migration tasks for source and target databases; Run the data migration task, first migrate the database table structure and migrate all the data, and obtain the starting point for incremental data collection before migrating all the data; The incremental collector starts running from the starting point, collects incremental data changes of the source database during the full data migration, and sends the collected data to the message queue; Full data migration is performed in one thread per table. Before migration, each table obtains the current snapshot location and uses the snapshot location for snapshot reading. After all tables have completed full data migration, start the incremental data applier to pull incremental data from the message queue for application, and perform corresponding processing based on the relationship between the incremental data submission point and the snapshot point. Before applying incremental data, create an application site table to record the submission site and transaction number of the current application data to filter duplicate data and apply only the data that has not been applied; When the data migration component crashes and restarts, it continues to collect incremental data using the recorded submission location as the starting point to ensure that data is not lost; When incremental data is applied, only the incremental data that meets the conditions is applied according to the submission site and transaction number recorded in the application site table.

[0030] In order to more clearly illustrate the technical solution of the present application, further explanation will be given below through embodiments of specific scenarios.

[0031] The following is an example of how this solution is applied in the exBase database migration system (exBase for short). Figure 2 and Figure 3 As shown, the following steps are included: 1) Create a data migration task in exBase with Oracle as the source database and GaussDB as the target database.

[0032] 2) Run the data migration task, first migrate the database table structure and migrate all the data. Before fully migrating the data, you need to obtain the SCN starting point for incremental data collection (referred to as the collection starting point). Because Oracle's uncommitted transaction events will be written to the REDO log in advance, the collection starting point cannot directly obtain the current SCN. You must first use the minimum transaction start SCN of all Oracle's uncommitted transactions as the collection starting point to ensure that the data of uncommitted transactions is not lost.

[0033] 3) The incremental collector starts running from the collection starting point obtained in step 2), collects the incremental data changes of Oracle during the full data migration, and sends the collected data to Kafka. After the incremental collector receives the signal from Kafka that the incremental data has been successfully received, it records the latest SCN of the incremental data as the collection SCN site. If the program is abnormally interrupted, the incremental collection will start collecting and sending from the collection SCN site again.

[0034] 4) exBase performs full data migration on a per-table-per-thread basis (because Oracle may have many tables, in order to ensure that the snapshot SCN of a table is not invalidated, it is necessary to obtain the current SCN as the snapshot SCN before starting to migrate each table). Before data migration, each table will record Oracle's current SCN (snapshot SCN). When Oracle queries data, the snapshot SCN is used for snapshot reading.

[0035] 5) After all tables in the entire database have completed full data migration, exBase starts the incremental data applicator and starts pulling incremental data from Kafka for application. The incremental data of the table will carry the SCN when the transaction is committed (CommitSCN for short). When CommitSCN is less than or equal to the snapshot SCN, it is considered to be data before the full data migration and is discarded; when CommitSCN is greater than the snapshot SCN, it is sent to GaussDB for application execution.

[0036] 6) Step 5) There is another key process before the application is executed. In order to avoid duplicate data from being collected incrementally, the incremental application creates an application site table in GaussDB in advance (the transaction that applies the incremental data carries the update of this table, so that the incremental applicator can filter the incremental data and only apply the data that has not been applied) to record the CommitSCN and transaction number TxId of the current application transaction. Step 5) Sending the application is also executed as a transaction. ExBase will add an additional record to update the application site table in the transaction, thus ensuring that the CommitSCN and TxId recorded in the application site table must be successfully applied.

[0037] 7) The above basically explains the entire process of full incremental migration. Next, we will explain how to resume the transfer. When exBase collects incremental data and sends it to Kafka, after Kafka responds to successful reception, it will record the CommitSCN of the last commit event of this batch of incremental data into the file. When exBase crashes and restarts due to various factors, exBase will continue to collect data with the CommitSCN recorded in the file as the starting point. In this way, even if the CommitSCN is not recorded in the file in time, the data will not be lost, but some extra data will be collected.

[0038] 8) When exBase crashes and restarts, since the application site table of GaussDB strictly records the CommitSCN and TxId of the current application data, only the incremental data that meets the conditions will be applied. The application condition is: the incremental data CommitSCN is greater than the last application data CommitSCN, and when the two CommitSCNs are the same, only the data with different TxIds will be applied.

[0039] Figure 4 The present application shows a device for connecting data with full incremental migration that can be resumed, the device comprising: Create a module to create data migration tasks for source and target libraries; The migration module is used to run data migration tasks, including database and table structure migration and full data migration, and obtain the starting point of incremental data collection before full data migration; An incremental collection module, configured to start running from the starting point, collect incremental data changes of the source database during the full data migration, and send the collected data to a message queue; The full migration module is used to perform full data migration in a one-table-one-thread manner. Before migration, each table obtains the current snapshot location and uses the snapshot location for snapshot reading. The incremental application module is used to start the incremental data applier after all tables have completed full data migration, pull incremental data from the message queue for application, and perform corresponding processing based on the relationship between the commit point and snapshot point of the incremental data; The site recording module is used to create an application site table to record the submission site and transaction number of the current application data before applying the incremental data, so as to filter duplicate data and only apply the data that has not been applied; The resume processing module is used to continue collecting incremental data using the recorded submission location as the starting point after the data migration component crashes and restarts, ensuring that data is not lost; The conditional application module is used for incremental data application. According to the submission site and transaction number recorded in the application site table, only the incremental data that meets the conditions is applied.

[0040] When the above-mentioned device is running, the steps of the method for connecting data with resumable full-incremental migration disclosed in this application are implemented.

[0041] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of the apparatus, methods, and computer program products according to various embodiments of the present application, including architecture, functions, and operations. In these figures, each box may represent a module, a program segment, or a portion of a code, which contains one or more executable instructions for implementing a specified logical function. It should be noted that each box in the block diagram and / or flowchart, as well as the combination of these boxes, can be implemented using a dedicated hardware-based system to implement the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0042] like Figure 5 As shown, an embodiment of the present application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a computer program executable by the processor, and a communication bus 340. The processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 runs the executable computer program to implement the steps of the above-mentioned method for connecting data with full incremental migration that can be resumed.

[0043] It is understandable that, in addition to the memory and the processor, the electronic device may also include an input device (such as a keyboard), an output device (such as a display) and other communication modules. These input devices, output devices and other communication modules communicate with the processor through an I / O interface (i.e., an input / output interface).

[0044] The operation of the present application can be implemented by writing computer program codes using one or more programming languages ​​or a combination thereof. The programming languages ​​include but are not limited to the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc.; A conventional procedural programming language, such as "C" or a similar programming language.

[0045] The execution methods of program code include but are not limited to: Executes entirely on the user's computer; Partial execution on the user's computer and part execution on a remote computer; Implemented as a standalone package; Executes entirely on the remote computer or server.

[0046] In scenarios involving remote computers, the remote computer can be connected to the user's computer through any type of network, including but not limited to a local area network (LAN) or a wide area network (WAN). In addition, the remote computer can also be connected to an external computer through an Internet service provider, such as using the Internet.

[0047] Furthermore, the present application also discloses a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute each step of the method for resumable full-incremental migration data connection disclosed in the present application.

[0048] In the context of this application, computer-readable storage media refers to tangible media that can store computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: A removable magnetic storage medium such as a floppy disk.

[0049] (2) Hard disk: includes fixed storage devices such as mechanical hard disks and solid-state hard disks.

[0050] (3) Random Access Memory (RAM): Volatile storage medium used for temporary storage of data and program code.

[0051] (4) Read-only memory (ROM): A non-volatile storage medium used to store fixed programs and data.

[0052] (5) Erasable Programmable Read-Only Memory (EPROM) or Flash Memory: A non-volatile storage medium that supports multiple erasing and programming.

[0053] (6) Fiber optic storage device: storage medium based on fiber optic technology.

[0054] (7) CD-ROM: A read-only medium that stores data in the form of an optical disc.

[0055] (8) Optical storage devices: storage media based on optical principles, such as DVDs and Blu-ray discs.

[0056] (9) Magnetic storage devices: storage media based on magnetic principles, such as magnetic tapes and disks.

[0057] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage requirements.

[0058] These computer-readable storage media can be used to store the program code and related data described in this application to support the operation of the program and the persistent storage of data.

[0059] In particular, according to an embodiment of the present application, the process described in the flowchart can be implemented as a computer software program. For example, an embodiment of the present application relates to a computer program product, which includes a computer program carried on a non-transitory computer-readable medium. The computer program includes program code for executing the method for connecting data with resumable full-incremental migration disclosed in the present application. When the computer program is executed by a processing device, the above-mentioned functions defined in the embodiments of the present application can be implemented.

[0060] Although the above discussion contains some specific implementation details, these details should not be interpreted as limiting the scope of this application. The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in this application is not limited to the technical solutions formed by the specific combination of the above technical features. At the same time, this application should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above public concept.

[0061] Those skilled in the art should also understand that they can modify the technical solutions described in the above embodiments without departing from the spirit and scope of the technical solutions of the embodiments of the present application, or replace some of the technical features therein by equivalents. These modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for connecting data with full incremental migration that can be resumed, characterized in that: The following steps are involved: Create data migration tasks for source and target databases; Run the data migration task, first migrate the database table structure and migrate all the data, and obtain the starting point for incremental data collection before migrating all the data; The incremental collector starts running from the starting point, collects incremental data changes of the source database during the full data migration, and sends the collected data to the message queue; Full data migration is performed in one thread per table. Before migration, each table obtains the current snapshot location and uses the snapshot location for snapshot reading. After all tables have completed full data migration, start the incremental data applier to pull incremental data from the message queue for application, and perform corresponding processing based on the relationship between the incremental data submission point and the snapshot point. Before applying incremental data, create an application site table to record the submission site and transaction number of the current application data to filter duplicate data and apply only the data that has not been applied; When the data migration component crashes and restarts, it continues to collect incremental data using the recorded commit location as the starting point; When incremental data is applied, only the incremental data that meets the conditions is applied according to the submission site and transaction number recorded in the application site table.

2. The method according to claim 1, characterized in that The source database is an Oracle database, the target database is a GaussDB database, and the message queue is Kafka.

3. The method according to claim 1, characterized in that The starting point of the incremental data collection is the minimum transaction starting point of all uncommitted transactions in the source database to ensure that the data of uncommitted transactions is not lost.

4. The method according to claim 1, characterized in that: After receiving the signal that the message queue has successfully received the incremental data, the incremental collector records the latest starting point of the incremental data as the collection starting point. If the program is abnormally interrupted, the incremental collection will start collecting and sending again from the collection starting point.

5. The method according to claim 1, characterized in that: During the full data migration process, each table obtains the current snapshot location before starting the migration to ensure that the snapshot of a certain table is not invalidated.

6. The method according to claim 1, characterized in that When applying incremental data, the incremental data applicator discards the data if the commit site of the incremental data is less than or equal to the snapshot site; if the commit site of the incremental data is greater than the snapshot site, the data is sent to the target library for application execution.

7. The method according to claim 1, characterized in that The application site table is used to record the commit site and transaction number of the currently applied transaction, and its update event and incremental data application are executed in the same transaction to ensure that the commit site and transaction number recorded in the application site table must be successfully applied; When the data migration component crashes and restarts, only the incremental data that meets the conditions is applied according to the commit site and transaction number of the current application data strictly recorded in the application site table of the target database.

8. The method according to claim 1, characterized in that After sending the incremental data collection to the message queue and receiving a response, the data migration component records the submission location of the last submission event of this batch of incremental data into a file, so that after the component crashes and restarts, the collection can continue with the submission location recorded in the file as the starting point.

9. The method according to claim 1, characterized in that: When the incremental data is applied, the application condition is: the commit site of the incremental data is greater than the commit site of the last applied data, and when the two commit sites are the same, only data with different transaction numbers are applied.

10. A data connection device for continuous full-increment migration, characterized in that: The device comprises: Create a module to create data migration tasks for source and target libraries; The migration module is used to run data migration tasks, including database and table structure migration and full data migration, and obtain the starting point of incremental data collection before full data migration; An incremental collection module, configured to start running from the starting point, collect incremental data changes of the source database during the full data migration, and send the collected data to a message queue; The full migration module is used to perform full data migration in a one-table-one-thread manner. Before migration, each table obtains the current snapshot location and uses the snapshot location for snapshot reading. The incremental application module is used to start the incremental data applier after all tables have completed full data migration, pull incremental data from the message queue for application, and perform corresponding processing based on the relationship between the commit point and snapshot point of the incremental data; The site recording module is used to create an application site table to record the submission site and transaction number of the current application data before applying the incremental data, so as to filter duplicate data and only apply the data that has not been applied; The resume processing module is used to continue collecting incremental data using the recorded submission location as the starting point after the data migration component crashes and restarts, ensuring that data is not lost; The conditional application module is used for incremental data application. According to the submission site and transaction number recorded in the application site table, only the incremental data that meets the conditions is applied.

Images