Method and device for realizing synchronization of oversized transaction to openGauss database based on Debezium

Abstract

The invention relates to a method and a device for realizing synchronization of an oversized transaction to an openGauss database based on Debezium. The method comprises the following steps: configuring the size of a file cache slice; starting an incremental synchronization service, and obtaining a data change record (EVENT) by analyzing an Oracle source library log; dividing the EVENT into an independent file cache queue according to the transaction ID, and triggering dynamic segmentation (the storage capacity exceeds the standard or is overtime); when the service is abnormal, positioning a site through an index to realize breakpoint resume; after the transactions are submitted, the file cache queue is read in sequence and pushed to the downstream, and incomplete transactions are rolled back when the transactions are abnormal; events containing LOB / XML columns are combined into DML statements, and the DML statements are issued according to a primary key strategy; and converting the downstream system into a standard format for the consumer to subscribe. According to the scheme, memory caching is replaced by file caching, OOM is avoided, dynamic segmentation and breakpoint resume improve the large transaction processing capacity, the LOB / XML merging strategy guarantees the data consistency, general transaction optimization is supported, and the oversized transaction synchronization efficiency and stability are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of data synchronization technology, and in particular to a method, device, computer-readable storage medium, and electronic device for synchronizing ultra-large transactions to an openGauss database based on Debezium. Background Art

[0002] With the continuous improvement of domestic database technology and product systems, more and more domestic institutions and enterprises are gradually promoting the process of domestic database replacement. Among them, the demand for database migration based on openGauss is particularly significant. During the migration process from traditional commercial databases (such as Oracle) to openGauss, the source database data needs to be completely migrated to the new database. To ensure business continuity and data consistency, it is usually necessary to go through an incremental synchronization phase before the business is cut over to the new database. The core goal of this phase is to ensure that the target database (openGauss) and the source database remain synchronized in terms of data status by capturing and synchronizing incremental changes (including DML operations) of the source database (such as Oracle) in real time. At the same time, it provides the business side with a verification window for the parallel operation, so that key indicators such as synchronization delay and data consistency can be observed, thereby reducing migration risks.

[0003] However, during actual migrations, the source database often faces the challenge of extremely large transactions. These transactions typically involve over 8 million DML operations per transaction, typically triggered by scheduled tasks that process batch data (such as bulk deletes and updates). While these operations may appear as a single SQL statement from a business perspective, the amount of incremental data generated (such as row-level change logs) can reach gigabytes or even larger. Against this backdrop, the existing mainstream Debezium incremental synchronization pipeline faces significant technical bottlenecks when handling these extremely large transactions: Debezium typically uses an in-memory map structure to temporarily store transaction DML records from different channels (such as Oracle log channels) and then outputs them to the target database at the transaction granularity after the transaction is fully committed. When encountering extremely large transactions, the number and volume of transaction DML records far exceed the memory capacity of the service process, directly leading to out-of-memory (OOM) errors and ultimately causing the migration service to terminate abnormally. This can lead to serious incidents such as business interruptions and data inconsistencies in the production environment.

[0004] In addition, although some data synchronization methods based on Debezium have been developed one after another, for example, the patent application "A method for synchronizing Kingbase ES database data" (CN119961352A) discloses a method for synchronizing Kingbase ES database data based on Debezium; the Chinese patent "A data synchronization method based on Debezium embedded engine" (CN118410109A) discloses a data synchronization method based on Debezium embedded engine; the patent application "A componentized incremental data synchronization method based on Debezium" (CN117332013A) discloses a componentized incremental data synchronization method based on Debezium; the patent application "A method and tool for synchronizing Oracle data to GreenPlum based on Debezium" (CN115344645A) discloses a method and tool for synchronizing Oracle data to GreenPlum based on Debezium, etc., none of the above solutions proposes optimization and improvement solutions for the synchronization of ultra-large transactions.

[0005] To address the above technical pain points, there is an urgent need to design an ultra-large transaction synchronization optimization solution that is compatible with the Debezium framework, focusing on solving the memory cache bottleneck problem of large transactions under common source channels such as Oracle. Summary of the Invention

[0006] To address the aforementioned issues, this application proposes a new Debezium-based method for synchronizing very large transactions to an openGauss database. This method combines the characteristics of different channels (such as Oracle log channels and other relational database channels) to propose differentiated file caching application strategies. This method can overflow transaction data to local disks through file caching without changing the existing architecture. Furthermore, it analyzes the feasibility of splitting storage for Oracle large LOB (large object) / XML data. While ensuring transaction atomicity, this method replaces memory caching with file caching, effectively reducing service process memory pressure and avoiding memory overflows. This ensures stable, reliable, and efficient incremental synchronization of very large transactions to the openGauss database.

[0007] In order to achieve the above objectives, this application adopts the following technical strategies: In summary, this solution establishes a file cache queue (component), uses memory-mapped files for persistence, performs large transaction memory optimization for Oracle's hybrid transaction scenarios, and performs large transaction memory optimization for Debezium-Server's handleBatch general transaction processing, ensuring that all Debezium scenarios can support large transaction scenarios on the business side.

[0008] Specifically, when Oracle collects DMLs from different transactions at the same time, it will store the DMLs in a Map by transaction ID. At this time, the file cache queue is used to write the data directly to the disk page cache through the memory-mapped file. The OS asynchronously flushes the disk, which greatly improves the persistence performance. The single-machine write throughput can reach 10 million records per second, and the performance is close to that of memory operations because the data is directly mapped to the user space through the OS page cache. <trxId,List <event>>Maintain transaction table ( Figure 2 ), when mixed transactions are collected, a list is used to store the order of the same transactions. This is achieved using file caching and indexing, with each transaction corresponding to a file cache queue. Data changes are written sequentially. If a file is full or the write time is too long, a new file is generated, using rolling partitioning to control file read and write granularity.

[0009] After the complete transaction is collected (collected to COMMIT), the files are read in order, read and merged, and then sent in order. When the size of the entire transaction is larger than the free memory, you can configure the adaptive file segmentation size ( Figure 3 ), ensuring that memory only needs to store transaction information for one file at a time, preventing out-of-memory (OOM) and maintaining data order. However, when LOB / XML changes are included, multiple events must be merged into a single DML transaction. The size of the LOB / XML changes within the transaction must be less than the available memory.

[0010] If the service crashes during a write process, when the service is restarted, the latest location in the file cache can be found based on the index, and data can be collected and written from the previous location, achieving breakpoint-resume transmission. This ensures that data collection is not repeated during large transaction anomalies, speeding up service recovery and avoiding issues such as insufficient cutover windows or slow online fault recovery.

[0011] If the service crashes during reading, a ROLLBACK is sent when the service is restarted to roll back the partially issued transaction (incomplete transaction) and restart the reading from the beginning to ensure data consistency.

[0012] In the same general design, this solution also implements the same file cache design for the Debezium-Server side of handleBatch, achieving the same effect and ensuring the large transaction file cache optimization of the general design.

[0013] In summary, this solution optimizes Debezium's Oracle and general large transaction cache through file cache queues (memory-mapped files, sequential append, and zero GC pressure).

[0014] More specifically, this application provides the following technical solutions: A first aspect of the present application provides a method for synchronizing a large transaction to an openGauss database based on Debezium, the method comprising: S1: Configure the cache slice size on the Debezium side based on the resource capacity and transaction scale of the target machine; S2: Start the incremental synchronization service and create a synchronization task with Oracle as the source database and openGauss as the target database. The collector obtains data change records (EVENTs) by parsing the source database log based on the specified starting SCN. S3: The collected data change records are divided into independent file cache queues according to the transaction ID. Each file cache queue corresponds to one transaction. When the storage capacity of a single file cache queue reaches the preset threshold, dynamic file segmentation is triggered to generate a new cache file. S4: After capturing a transaction commit event, locate the corresponding file cache queue based on the transaction ID, read the entire batch of change events in their original order, and push them to the downstream Debezium-Server at the file level. S5: When reading events from the file cache queue, if the table to which the event belongs contains LOB / XML columns, multiple data change records are merged into a single DML statement and delivered according to the primary key change policy. If the table does not contain LOB / XML columns, the event is delivered directly. S6: The downstream system converts the received data change records into a standard format for on-demand subscription by the Debezium-Server consumer.

[0015] Furthermore, in the method of the present application, the file cache slice size in step S1 is dynamically adjusted according to the resource capacity and transaction scale of the target machine; Step S2 also includes: the collector uses Oracle LogMiner to parse the source library log based on the specified starting SCN, load the online data dictionary and obtain data change records in the archive / REDO log.

[0016] Furthermore, in the method of the present application, step S3 also includes: if the service crashes abnormally during the collection process, the latest file cache location is located through the index after restart, and data is continued to be collected and written from the historical location to achieve breakpoint resumption.

[0017] Furthermore, in the method of the present application, step S3 also includes: the file cache queue is persisted in a memory-mapped file manner, and the persistence performance is improved through the asynchronous disk flushing mechanism of the operating system.

[0018] Furthermore, in the method of the present application, the triggering conditions for the dynamic file segmentation in step S3 include: the storage capacity of a single file cache queue reaches a preset segment size threshold or exceeds the maximum time window (such as the transaction duration exceeds the set threshold).

[0019] Furthermore, in the method of the present application, step S4 also includes: if the service crashes abnormally during the reading process, a ROLLBACK flag is sent to roll back the incomplete transaction that has been issued, and the current file cache queue is read from the beginning after restart to ensure data consistency.

[0020] Furthermore, in the method of the present application, step S5 further includes: when reading events from the file cache queue, if the table to which the events belong contains LOB / XML columns, merging multiple data change records into one DML statement only when the total size of the LOB / XML changes within the transaction is less than the free memory capacity to avoid memory overflow (OOM); The primary key change strategy includes: if the primary key column of the table to which the event belongs has not changed, multiple data change records are merged into the original DML statement and then issued; if the primary key column has changed, the previous merged DML statement is issued first, and then subsequent data change records are spliced ​​in memory, and after reading is completed, the spliced ​​DML statement in memory is issued.

[0021] Furthermore, the present application method also includes general transaction optimization for handleBatch processing on the Debezium-Server side: integrating file cache queues and dynamic segmentation mechanisms into the general transaction processing flow to achieve the same large transaction synchronization effect as the Oracle hybrid transaction scenario.

[0022] A second aspect of the present application provides a device for synchronizing ultra-large transactions to an openGauss database based on Debezium, the device comprising: Configuration module, used to configure the file cache slice size according to the resource capacity and transaction scale of the target machine; The log collection module is used to obtain data change records by parsing the source database log based on the starting SCN; The transaction cache module is used to divide data change records into independent file cache queues according to transaction IDs. Each file cache queue corresponds to one transaction. When the storage capacity of a single file cache queue reaches a preset threshold, it triggers dynamic file segmentation and generates a new cache file. The data delivery module is used to locate the corresponding file cache queue according to the transaction ID after capturing the transaction commit event, read the entire batch of change events in the original order of the events, and push them to the downstream Debezium-Server at the file granularity; The LOB / XML processing module is used to merge multiple data change records into a single DML statement when reading events from the file cache queue. If the table to which the event belongs contains a LOB / XML column, the statement is issued according to the primary key change policy. The format conversion module is used by the downstream system to convert the received data change records into a standard format for on-demand subscription by the Debezium-Server consumer.

[0023] The device implements the steps of the aforementioned method for synchronizing ultra-large transactions to an openGauss database based on Debezium when running.

[0024] A third aspect of the present application provides an electronic device, comprising: a memory and a processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the aforementioned method for synchronizing ultra-large transactions to the openGauss database based on Debezium.

[0025] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the aforementioned method for synchronizing ultra-large transactions to an openGauss database based on Debezium are implemented.

[0026] This application proposes a Debezium-based method for synchronizing very large transactions to an openGauss database. By innovatively replacing the traditional memory cache mechanism with a file cache, this method systematically addresses the out-of-memory (OOM) risk in very large transaction synchronization scenarios, while also balancing data persistence performance and synchronization efficiency. Ultimately, this method achieves a dual improvement in synchronization capability and reliability, as embodied in the following aspects: (1) Effectively avoid memory overflow risks and enhance service stability: Traditional Debezium pipelines rely on memory map structures to temporarily store transaction DML records. When processing ultra-large transactions containing over 8 million DML operations, memory usage can easily exceed the service process limit, resulting in OOM exceptions. This method replaces memory cache with file cache, persistently storing DML records of ultra-large transactions in the local file system. This fundamentally breaks through the limitation of memory capacity on transaction scale, significantly reduces the risk of service downtime caused by insufficient memory, and ensures the continuous operation of the migration service.

[0027] (2) Optimizing synchronization latency and throughput to improve migration efficiency: The file cache mechanism balances persistence performance and synchronization latency through reasonable cache read and write strategies (such as sequential writes and batch flushing) while ensuring storage reliability. Taking the default 8GB memory allocation scenario as an example, the maximum number of transactions supported by this method for a single service has increased from the traditional approximately 2 million large transactions to at least 8 million large transactions. In particular, when processing scenarios where the scale of a single transaction exceeds 100,000 DMLs, the migration rate is significantly improved compared to traditional solutions, effectively shortening the incremental synchronization cycle and providing a more efficient time window for business-side parallel operation verification.

[0028] (3) Fully guarantee transaction atomicity and data consistency: The file cache design strictly adheres to the principle of transaction boundary demarcation. Through the transaction ID index and the global unique identifier (source database location), all DML records of the same transaction are stored continuously in the file in the order of submission. When synchronizing to the target database, the transaction-granular atomic write mechanism is adopted to avoid transaction incompleteness caused by partial record loss or disorder. It fully guarantees data consistency between the source and target databases and meets the stringent data integrity requirements of enterprise-level migration scenarios.

[0029] In summary, this method solves the bottleneck of ultra-large transaction memory through the innovative application of file caching, while achieving coordinated optimization of synchronization efficiency, stability and data consistency, providing a highly reliable and high-performance solution for the migration project of domestic databases based on Debezium (such as openGauss).

[0030] Other features and advantages of this application will be described in detail in the following description, or will be understood through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and technical means clearly indicated in the description, claims, and drawings, and obtained through the implementation of these technical contents. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] To more clearly illustrate the technical solution of this application, the following is a brief introduction to the drawings involved in the description of this solution. It should be noted that the drawings only illustrate some embodiments of this application. Those skilled in the art can deduce other relevant drawings based on these drawings without engaging in creative work.

[0032] Figure 1 This is the overall implementation flow chart of the method for synchronizing ultra-large transactions to the openGauss database based on Debezium in this application.

[0033] Figure 2 This is the Debezium data extraction process flow chart.

[0034] Figure 3 This diagram shows how DML in an Oracle log is split into multiple events.

[0035] Figure 4 Flowchart of merging EVENT for consumption.

[0036] Figure 5 This is a structural diagram of the device of this application.

[0037] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] In this document, the term "including" and any variations thereof (such as "including," "comprising," etc.) are open-ended expressions and should be understood to mean "including but not limited to," meaning that the listed items are not exhaustive and may include other items not explicitly mentioned. The term "based on" should be understood to mean "based, at least in part, on," meaning that the basis or condition referred to may not be the only factor and may also involve other relevant factors. The term "one embodiment" should be understood to mean "at least one embodiment," meaning that the described embodiment is not the only possible implementation and that other similar embodiments may exist.

[0040] In this application, the terms "a" and "a plurality" are used to modify elements or features in an illustrative, non-restrictive manner. Unless the context clearly indicates otherwise, "a" should be understood as meaning "at least one," and "a plurality" should be understood as meaning "at least two." Those skilled in the art should interpret these terms appropriately based on the semantics and logical relationships of the context to ensure they encompass the possibility of "one or more."

[0041] Figure 1 The following is the overall implementation process of the method for synchronizing ultra-large transactions to the openGauss database based on Debezium provided by this application, including the following steps: S1: Configure the cache slice size on the Debezium side based on the resource capacity and transaction scale of the target machine; S2: Start the incremental synchronization service and create a synchronization task with Oracle as the source database and openGauss as the target database. The collector obtains data change records (EVENTs) by parsing the source database log based on the specified starting SCN. S3: The collected data change records are divided into independent file cache queues according to the transaction ID. Each file cache queue corresponds to one transaction. When the storage capacity of a single file cache queue reaches the preset threshold, dynamic file segmentation is triggered to generate a new cache file. S4: After capturing a transaction commit event, locate the corresponding file cache queue based on the transaction ID, read the entire batch of change events in their original order, and push them to the downstream Debezium-Server at the file level. S5: When reading events from the file cache queue, if the table to which the event belongs contains LOB / XML columns, multiple data change records are merged into a single DML statement and delivered according to the primary key change policy. If the table does not contain LOB / XML columns, the event is delivered directly. S6: The downstream system converts the received data change records into a standard format for on-demand subscription by the Debezium-Server consumer.

[0042] In order to more clearly illustrate the technical solution of the present application, the following will further illustrate it through embodiments of specific scenarios.

[0043] This method comprises the following steps: (1) First, set the file cache slice size that suits the environment in the Debezium configuration.

[0044] (2) Then start the incremental synchronization service and create a synchronization task with Oracle as the source database and openGauss as the target database. Based on the specified starting SCN, the collector uses the Oracle LogMiner component to parse the database log, load the online data dictionary, and obtain the data change records in the archive / REDO log.

[0045] (3) Based on each acquired data change record (EVENT), the change events of multiple transactions are read in SCN order. The events of each transaction are persisted in an independent file cache queue. When a single file reaches the shard size threshold or exceeds the maximum time window, dynamic file splitting is triggered. If the service crashes during reading, the latest location of the file cache can be found based on the index when the service is restarted. Data can be collected and written from the historical location to achieve breakpoint resumption, avoid repeated collection, and catch up with the migration progress faster.

[0046] (4) When a transaction commit (COMMIT) event is captured, the system locates the corresponding file cache queue based on the transaction ID, reads the entire batch of change events in their original order, and pushes them to the downstream server at the file level. If the service crashes during reading, a ROLLBACK flag is sent, rolling back the incomplete transaction sent previously and restarting the read from the beginning to ensure data consistency.

[0047] (5) When reading, if the table containing the event does not contain a LOB / XML column, it will be sent directly. If a LOB / XML column exists, multiple events will need to be merged into a single DML statement. At this time, it is necessary to determine whether the same column changes are made based on the primary key column. If so, the changes are merged into the original DML statement and sent last. If a primary key change occurs, the previous merged DML statement will be sent first, and then the subsequent events will be spliced ​​in memory. After reading all events, the spliced ​​DML statements in memory will be sent in order.

[0048] (6) The downstream system converts the change records into a standard format for on-demand subscription by customized Debezium-Server consumers ( Figure 4 ).

[0049] Note: In the actual implementation of this method, it is necessary to reasonably configure the large transaction cache file slice size based on machine configuration information such as memory capacity, disk capacity, etc.

[0050] Figure 5 The figure shows a device proposed in this application for synchronizing ultra-large transactions to an openGauss database based on Debezium, the device comprising: Configuration module, used to configure the file cache slice size according to the resource capacity and transaction scale of the target machine; The log collection module is used to obtain data change records by parsing the source database log based on the starting SCN; The transaction cache module is used to divide data change records into independent file cache queues according to transaction IDs. Each file cache queue corresponds to one transaction. When the storage capacity of a single file cache queue reaches a preset threshold, it triggers dynamic file segmentation and generates a new cache file. The data delivery module is used to locate the corresponding file cache queue according to the transaction ID after capturing the transaction commit event, read the entire batch of change events in the original order of the events, and push them to the downstream Debezium-Server at the file granularity; The LOB / XML processing module is used to merge multiple data change records into a single DML statement when reading events from the file cache queue. If the table to which the event belongs contains a LOB / XML column, the statement is issued according to the primary key change policy. The format conversion module is used by the downstream system to convert the received data change records into a standard format for on-demand subscription by the Debezium-Server consumer.

[0051] When the above device is running, the steps of the method disclosed in this application for synchronizing ultra-large transactions to the openGauss database based on Debezium are implemented.

[0052] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of the apparatus, methods, and computer program products according to various embodiments of the present application, including architecture, functions, and operations. In these figures, each box may represent a module, a program segment, or a portion of a code, which contains one or more executable instructions for implementing a specified logical function. It should be noted that each box in the block diagram and / or flowchart, and the combination of these boxes, can be implemented using a dedicated hardware-based system to implement the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0053] like Figure 6 As shown, an embodiment of the present application further discloses an electronic device, comprising: a processor 310, a communication interface 320, a memory 330 for storing a computer program executable by the processor, and a communication bus 340. The processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the above-mentioned method for synchronizing very large transactions to an openGauss database based on Debezium.

[0054] It is understood that, in addition to the memory and processor, the electronic device may also include an input device (e.g., a keyboard), an output device (e.g., a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor via an I / O interface (i.e., an input / output interface).

[0055] The operation of the present application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include but are not limited to the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc.; A conventional procedural programming language, such as "C" or a similar programming language.

[0056] The execution methods of the program code include but are not limited to: Executes entirely on the user's computer; Partially executed on the user's computer and partially on a remote computer; Executed as a standalone software package; Executes entirely on the remote computer or server.

[0057] In scenarios involving a remote computer, the remote computer can be connected to the user's computer via any type of network, including but not limited to a local area network (LAN) or a wide area network (WAN). Additionally, the remote computer can be connected to an external computer via an Internet service provider, such as the Internet.

[0058] Furthermore, the present application also discloses a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by the processor of an electronic device, the electronic device can execute the various steps of the method disclosed in the present application for synchronizing ultra-large transactions to an openGauss database based on Debezium.

[0059] In the context of this application, computer-readable storage media refers to tangible media that can store computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: A removable magnetic storage medium such as a floppy disk.

[0060] (2) Hard disk: includes fixed storage devices such as mechanical hard disks and solid-state hard disks.

[0061] (3) Random Access Memory (RAM): Volatile storage medium used for temporary storage of data and program code.

[0062] (4) Read-only memory (ROM): A non-volatile storage medium used to store fixed programs and data.

[0063] (5) Erasable Programmable Read-Only Memory (EPROM) or Flash Memory: A non-volatile storage medium that supports multiple erasing and programming.

[0064] (6) Fiber optic storage device: storage medium based on fiber optic technology.

[0065] (7) Compact Disc Read-Only Memory (CD-ROM): A read-only medium that stores data in the form of an optical disc.

[0066] (8) Optical storage devices: storage media based on optical principles, such as DVDs and Blu-ray discs.

[0067] (9) Magnetic storage devices: storage media based on magnetic principles, such as magnetic tapes and disks.

[0068] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage requirements.

[0069] These computer-readable storage media can be used to store the program code and related data described in this application to support the operation of the program and the persistent storage of data.

[0070] In particular, according to embodiments of the present application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of the present application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. The computer program includes program code for executing the method disclosed in the present application for implementing the synchronization of ultra-large transactions to an openGauss database based on Debezium. When the computer program is executed by a processing device, the above-mentioned functions defined in the embodiments of the present application can be implemented.

[0071] Although the above discussion contains several specific implementation details, these details should not be interpreted as limiting the scope of this application. The above description is only a preferred embodiment of the present application and an illustration of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in this application is not limited to the technical solutions formed by the specific combination of the above technical features. At the same time, this application should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concepts.

[0072] Those skilled in the art should also understand that they may modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents, without departing from the spirit and scope of the technical solutions of the embodiments of the present application. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of the present application.< / event>

Claims

1. A method for synchronizing ultra-large transactions to an openGauss database based on Debezium, characterized in that: The method comprises: S1: Configure the cache slice size on the Debezium side based on the resource capacity and transaction scale of the target machine; S2: Start the incremental synchronization service and create a synchronization task with Oracle as the source database and openGauss as the target database. The collector obtains data change records by parsing the source database log based on the specified starting SCN. S3: The collected data change records are divided into independent file cache queues according to the transaction ID. Each file cache queue corresponds to one transaction. When the storage capacity of a single file cache queue reaches the preset threshold, dynamic file segmentation is triggered to generate a new cache file. S4: After capturing a transaction commit event, locate the corresponding file cache queue based on the transaction ID, read the entire batch of change events in the original order of the events, and push them to the downstream Debezium-Server at the file granularity; S5: When reading events from the file cache queue, if the table to which the event belongs contains LOB / XML columns, multiple data change records are merged into a single DML statement and delivered according to the primary key change policy. If the table does not contain LOB / XML columns, the event is delivered directly. S6: The downstream system converts the received data change records into a standard format for on-demand subscription by the Debezium-Server consumer.

2. The method according to claim 1, characterized in that The file cache slice size in step S1 is dynamically adjusted according to the resource capacity and transaction scale of the target machine; Step S2 also includes: the collector uses Oracle LogMiner to parse the source library log based on the specified starting SCN, load the online data dictionary and obtain data change records in the archive / REDO log.

3. The method according to claim 1, characterized in that Step S3 also includes: if the service crashes during the collection process, after restarting, the latest file cache location is located through the index, and data is collected and written from the historical location to achieve breakpoint resumption.

4. The method according to claim 1, wherein Step S3 also includes: the file cache queue is persisted in a memory-mapped file manner, and the persistence performance is improved through the operating system's asynchronous disk flushing mechanism.

5. The method according to claim 1, wherein The triggering conditions for the dynamic file segmentation in step S3 include: the storage capacity of a single file cache queue reaches a preset segment size threshold or exceeds a maximum time window.

6. The method according to claim 1, characterized in that Step S4 also includes: if the service crashes during the reading process, a ROLLBACK flag is sent to roll back the incomplete transaction that has been issued, and the current file cache queue is read from the beginning after restart to ensure data consistency.

7. The method according to claim 1, characterized in that Step S5 further includes: when reading events from the file cache queue, if the table to which the events belong contains LOB / XML columns, merging multiple data change records into one DML statement only when the total size of the LOB / XML changes in the transaction is less than the free memory capacity to avoid memory overflow; The primary key change strategy includes: if the primary key column of the table to which the event belongs has not changed, multiple data change records are merged into the original DML statement and then issued; if the primary key column has changed, the previous merged DML statement is issued first, and then subsequent data change records are spliced ​​in memory, and after reading is completed, the spliced ​​DML statement in memory is issued.

8. The method according to claim 1, characterized in that The method also includes general transaction optimization for handleBatch processing on the Debezium-Server side: integrating a file cache queue and a dynamic splitting mechanism into the general transaction processing flow to achieve the same large transaction synchronization effect as the Oracle hybrid transaction scenario.

9. A device for synchronizing ultra-large transactions to an openGauss database based on Debezium, characterized in that: The device comprises: Configuration module, used to configure the file cache slice size according to the resource capacity and transaction scale of the target machine; The log collection module is used to obtain data change records by parsing the source database log based on the starting SCN; The transaction cache module is used to divide data change records into independent file cache queues according to transaction IDs. Each file cache queue corresponds to one transaction. When the storage capacity of a single file cache queue reaches a preset threshold, it triggers dynamic file segmentation and generates a new cache file. The data delivery module is used to locate the corresponding file cache queue according to the transaction ID after capturing the transaction commit event, read the entire batch of change events in the original order of the events, and push them to the downstream Debezium-Server at the file granularity; The LOB / XML processing module is used to merge multiple data change records into a single DML statement when reading events from the file cache queue. If the table to which the event belongs contains a LOB / XML column, the statement is issued according to the primary key change policy. The format conversion module is used by the downstream system to convert the received data change records into a standard format for on-demand subscription by the Debezium-Server consumer.

10. An electronic device, characterized in that: include: memory and processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the method for synchronizing ultra-large transactions to an openGauss database based on Debezium as described in any one of claims 1 to 8.

Images