Method and device for realizing transaction consistency, equipment and medium

Abstract

The invention belongs to the technical field of databases, and particularly relates to a method and device for achieving transaction consistency, equipment and a medium. The method comprises the following steps: capturing change event data from a transaction log of an Oracle database by taking a transaction as a unit; based on the change event data, extracting an operation type, a table name, a primary key of a row, a native transaction ID and a change value, and packaging into a structured message; the Kafka cluster routes the structured message corresponding to the same native transaction ID or the primary key in the same row to the same Kafka partition through a preset partition; the Storm cluster consumes the structured message from the Kafka partition, pulls the structured message according to a transaction batch, allocates a unique transaction ID, performs data conversion on the structured message, and submits the structured message cooperatively with a target non-relational database to realize idempotent writing; and performing inverse normal aggregation design on the table structure of the non-relational database, and aggregating the business data of the plurality of association tables in the Oracle database into a single-row data structure in the non-relational database.

Description

Technical Field

[0001] This invention belongs to the field of database technology, and specifically relates to a method, apparatus, device and medium for achieving transaction consistency. Background Technology

[0002] In the era of big data, with the continuous expansion of data scale in various application systems and the constant evolution of business models, data processing needs are becoming increasingly diversified and complex. Traditional relational databases, such as Oracle, have long provided robust and reliable data storage and transaction processing guarantees for enterprise-level applications in key industries such as power, finance, telecommunications, and government, thanks to their mature and stable architecture and strict adherence to the ACID properties of transactions—Atomicity, Consistency, Isolation, and Durability. However, with the explosive growth of internet and mobile internet services, the speed of data generation has accelerated dramatically, and data formats have become increasingly diverse. Non-relational databases (NoSQL), due to their significant advantages in massive data storage, high-concurrency read / write throughput, and flexible and scalable data models, have gradually become an indispensable component of big data platforms.

[0003] Among numerous NoSQL databases, HBase, an open-source project under Apache, is a high-quality implementation of the Google Bigtable architecture, possessing high reliability, high performance, column-oriented storage, and highly scalable distributed characteristics. It is widely used in big data scenarios such as user behavior log storage, real-time querying, and recommendation systems in internet enterprises, effectively handling application environments with massive data volumes and intensive read / write requests. However, HBase was initially designed with scalability and high throughput in mind, resulting in relatively limited transaction processing capabilities, particularly lacking cross-row and cross-table transaction guarantees, making it difficult to support complex transaction operations with strict consistency like those in Oracle. Although later versions of HBase introduced some transaction mechanisms, such as limited row-level transactions, they still lag significantly behind the strong consistency transaction guarantees provided by Oracle.

[0004] This deficiency in transaction capabilities is particularly pronounced in business scenarios with extremely stringent requirements for data accuracy, integrity, and transaction consistency, such as power system monitoring, electricity billing, financial transactions, and telecommunications billing. In these areas, even minor data inconsistencies or lost transactions can lead to serious business errors or even systemic risks. Therefore, although HBase has significant advantages in scalability and performance, its limitations in critical transaction scenarios restrict its further application expansion. Currently, while there are some technical solutions for data synchronization between Oracle and HBase, they often struggle to simultaneously meet the dual requirements of low latency and strong consistency in a distributed environment. Common problems such as data loss, out-of-order data, and high synchronization latency make maintaining the same level of strict transaction consistency as Oracle in a distributed architecture a highly challenging and urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, device, and medium for achieving transaction consistency, thereby solving the problem in the prior art of simultaneously satisfying low latency and strong consistency in a distributed environment.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for achieving database transaction consistency, comprising: Change event data is captured from the transaction log of the relational database on a transaction-by-transaction basis; the change event data includes operation records of all related tables involved in the transaction; Based on the change event data, extract the operation type, table name, row primary key, native transaction ID and change value corresponding to each operation record, and encapsulate them into a structured message; The Kafka cluster routes the structured messages using a preset partitioning rule to obtain multiple Kafka partitions. The preset partitioning rule is to route structured messages with the same native transaction ID or the same row primary key to the same Kafka partition. The Storm cluster consumes the standardized structured messages from each Kafka partition, divides the structured messages into transaction batches according to the native transaction ID, and assigns a unique global transaction ID to each transaction batch; based on the preset single-row aggregated data structure, it performs data transformation and aggregation on the field values ​​before and after the changes of all operation records in the transaction batch to obtain single-row aggregated data; Based on the global transaction ID, perform an atomic write operation on a single row of aggregated data to a non-relational database.

[0007] Preferred options also include: Configure a data loss compensation mechanism based on the synchronization of data between Oracle and a non-relational database. The data loss compensation mechanism includes two methods: manual synchronization or timed automatic comparison and synchronization.

[0008] Preferably, in the step of capturing change event data from the transaction log of the relational database on a transaction-by-transaction basis, OGG is used to capture data changes from the transaction log of the relational database on a transaction-by-transaction basis, and the order of the data changes is maintained according to the transaction commit order of the relational database.

[0009] Preferably, the step of extracting the operation type, table name, row primary key, native transaction ID, and change value based on the change event data, and encapsulating them into a structured message, includes: Based on change event data, the operation type, table name, row primary key, native transaction ID and change value are extracted through a custom UserExit program and encapsulated into a structured message in JSON or Avro format; The UserExit program supports specifying the Oracle table names and field lists to be extracted. When the amount of change event data exceeds a preset threshold, it automatically unpacks the data and marks the unpacking status with unpacking identifier, unpacking sequence number, and last unpacking identifier.

[0010] Preferably, the step of routing the structured messages using preset partitioning rules to obtain multiple Kafka partitions in the Kafka cluster includes: Obtain the encapsulated structured message, and extract the native transaction ID or row primary key from the structured message as the partition key; Based on the partition key, a preset partitioning strategy is loaded, and a mapping calculation is performed on the partition key to obtain the target partition number; Based on the target partition number, the structured message is sent to the target partition of the corresponding topic in the Kafka cluster, completing the directed routing of primary key messages with the same native transaction ID or the same row.

[0011] Preferably, the step of performing data transformation on the field change values ​​in the transaction batch based on the single-row aggregated data structure to obtain single-row aggregated data includes: Based on structured messages, the Storm cluster and the target non-relational database perform a collaborative commit operation to verify whether the structured message corresponding to the native transaction ID has been written. If it has not been written, the full data commit is completed; if it has been written, the commit is skipped, thus achieving idempotent write.

[0012] Preferably, it also includes denormalizing and aggregating the table structure of the non-relational database, including: Obtain the structured message with a unique transaction ID output after data transformation in the Storm cluster, and at the same time obtain the core association fields and data association relationships of multiple related business tables in the Oracle database to obtain the table association rules; Based on the table association rules, aggregation mapping rules for single-row data structures in non-relational databases are formulated, and the core identifier fields of single-row data structures are determined. Based on the aggregation mapping rules and core identifier fields, structured messages corresponding to multiple Oracle related tables under the same native transaction ID are mapped and merged into single-row aggregated data in a non-relational database. Based on the single-row aggregated data, the single-row atomic operation interface of the non-relational database is called to write the single-row data into the non-relational database at once, thereby achieving transaction consistency by utilizing the characteristics of single-row atomic operations.

[0013] A second aspect of the present invention provides an apparatus for achieving transaction consistency, comprising: The capture module is used to capture change event data from the transaction log of a relational database on a transaction-by-transaction basis; the change event data includes operation records of all related tables involved in the transaction; The extraction module is used to extract the operation type, table name, row primary key, native transaction ID and change value corresponding to each operation record based on the change event data, and encapsulate them into a structured message; The partitioning module is used by the Kafka cluster to route the structured messages using a preset partitioning rule to obtain multiple Kafka partitions. The preset partitioning rule is to route structured messages with the same native transaction ID or the same row primary key to the same Kafka partition. The aggregation module is used by the Storm cluster to consume the standardized structured messages from each Kafka partition, divide the messages into transaction batches according to the original transaction ID, and assign a unique global transaction ID to each transaction batch; based on the preset single-row aggregated data structure, it performs data transformation and aggregation on the field values ​​before and after the changes of all operation records in the transaction batch to obtain single-row aggregated data. The write module is used to perform atomic write operations on single-row aggregated data to a non-relational database based on the global transaction ID.

[0014] In a third aspect, the present invention provides an electronic device including a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the method for achieving transaction consistency.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium storing at least one instruction that, when executed by a processor, implements the method for achieving transaction consistency.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: By leveraging OGG to capture Oracle database changes at the transaction level and serialize them in commit order, UserExit to encapsulate structured messages, Kafka to route data to the same partition based on native transaction ID / primary key, Storm to assign unique transaction IDs in batches and coordinate commits, and HBase's denormalized aggregation design for seamless integration, this system accurately transmits transaction metadata, avoids the shortcomings of HBase cross-row transactions, and achieves atomicity and consistency in Oracle and HBase data synchronization. Furthermore, by fully identifying transaction boundaries with SCN and utilizing the FIFO (First-In, First-Out) message characteristic within Kafka partitions, it ensures that the order of data changes aligns with Oracle's database structure. The source database is completely consistent with the Oracle database, ensuring that the transaction order is not disordered during distributed transmission and guaranteeing the continuity of business logic. Through Storm's out-of-order buffering, transaction batch association, and idempotent write verification, it can aggregate scattered messages of the same transaction, filter duplicate data, and achieve transaction semantics of either all successes or all rollbacks, eliminating intermediate inconsistent states. By aggregating data from multiple related tables in the Oracle database into a single-row structure through the HBase database and calling the single-row atomic operation interface for writing, it can transform multi-table transaction updates into single-row atomic operations, avoiding the deficiencies of HBase's native transaction capabilities and achieving a unification of transaction consistency and storage adaptation.

[0017] By configuring the Checkpoint mechanism in OGG to persistently capture positions, data can be resumed from the last successful position after a process or server restart, ensuring no data loss at the source and guaranteeing the reliability of the synchronization link. Configuring a multi-replica mechanism in the Kafka cluster automatically switches partition replicas when a single Broker node fails, ensuring uninterrupted message transmission and no data loss, improving the availability of the intermediate transport layer. Storm's Acker message acknowledgment mechanism and Worker process fast failure and automatic restart features enable tracking the entire message processing flow and triggering retransmissions upon failure, ensuring that a single node failure does not affect the overall computing task and guaranteeing stable operation of the data processing layer. A compensation mechanism combining manual synchronization and scheduled automatic batch comparison synchronization accurately identifies and corrects data differences between Oracle and HBase databases, providing a backup in extreme scenarios and ensuring final data consistency, further enhancing system fault tolerance.

[0018] By enabling the horizontal scaling of topic partitions using Kafka, data transmission parallelism can be improved, achieving linear scaling when data table throughput surges and adapting to high-concurrency data transmission needs. By dynamically adjusting the number of Workers, Executors, and Tasks in a Topology using Storm, computing resources can be elastically scaled according to data pressure, enabling on-demand improvement of data processing capabilities during peak business periods. By supporting the addition of data nodes through the distributed architecture of the target HBase database, storage and read / write pressure can be distributed, achieving long-term storage and synchronization of massive amounts of data without scaling bottlenecks, supporting the continuous growth of business data scale.

[0019] By configuring filtering and transformation rules in the Extract or Replicat process using OGG, data can be filtered and its format initially cleaned based on table name, column value, or operation type, achieving accurate synchronization of core business data and adapting to diverse data needs. Through the loosely coupled integration design of Kafka and Storm, data extraction and processing modules can be independently maintained and upgraded, enabling multiple OGG processes to send data to different Kafka topics and multiple StormTopologies to consume data separately, supporting multi-tenant and multi-structured message synchronization scenarios. By custom-developing the core transformation and write logic Bolt for StormTopology, it can adapt to different structured message libraries such as HBase, MongoDB, and Elasticsearch, achieving a unified real-time data channel to connect to multiple storage systems, improving business adaptability and system reusability. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a method for achieving transaction consistency according to Embodiment 1 of the present invention; Figure 2 This is a logical block diagram of a method for achieving transaction consistency according to Embodiment 1 of the present invention; Figure 3 This is a structural block diagram of an apparatus for achieving transaction consistency according to Embodiment 2 of the present invention; Figure 4 This is a structural block diagram of an electronic device according to Embodiment 3 of the present invention. Detailed Implementation

[0021] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0022] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0023] Example 1 like Figures 1-2 As shown, a method for achieving database transaction consistency includes: S1. Capture change event data from the Oracle database transaction log on a transaction-by-transaction basis; the change event data includes operation records of all related tables involved in the transaction; Specifically, change event data of type INSERT, UPDATE, and DELETE are captured from the online redo logs or archive logs of the Oracle database on a transaction-by-transaction basis to ensure that all change operations within the same transaction are fully captured without missing any single DML statement; the transaction commit sequence identifier (Commit SCN) is recorded during the capture process to provide a basis for subsequent data ordering guarantees.

[0024] Preferably, Oracle GoldenGate (OGG) is used to perform the above capture operation. OGG parses the Oracle transaction log through the Extract process, maintaining the order of data changes according to the Oracle database transaction commit order. The OGG Checkpoint mechanism is configured to persist the data capture position. When the OGG process restarts, it can resume from the last successfully captured position to avoid data loss. At the same time, through the OGG additional log configuration, the primary key of the row and the values ​​before and after the change are captured to ensure complete data traceability of the changed fields in the update transaction.

[0025] OGG guarantees the atomicity of transactions when capturing changes to an Oracle database. This means that all changes within a transaction are either captured and sent, or none are sent. Through OGG's log reading mechanism, the sequential nature of changed data is ensured to match the transaction execution order in the Oracle database.

[0026] S2. Based on the change event data, extract the operation type, table name, row primary key, native transaction ID and change value corresponding to each operation record, and encapsulate them into a structured message; Specifically, including: Based on change event data, the operation type, table name, row primary key, native transaction ID and change value are extracted by a custom UserExit program. The original Trail file message output by OGG is parsed and encapsulated into a structured message in JSON or Avro format. Preferably, the UserExit program supports specifying the Oracle table names and field lists to be extracted, synchronizing only core business data, and automatically splitting structured messages into multiple packets when the amount of change event data exceeds a preset threshold. After splitting, the splitting status is marked by the splitting identifier (is_split), splitting sequence number (split_no), and last splitting identifier (is_last_split) to ensure that downstream transaction data can be completely reassembled.

[0027] S3, the Kafka cluster uses a preset partitioning rule to route the structured messages, resulting in multiple Kafka partitions; the preset partitioning rule is: to route structured messages with the same native transaction ID or the same row primary key to the same Kafka partition; Specifically, including: Obtain the encapsulated structured message, and extract the transaction ID or row primary key from the structured message as the partition key; Based on the partition key, a preset partitioning strategy is loaded, and a mapping calculation is performed on the partition key to obtain the target partition number; Based on the target partition number, the structured message is sent to the target partition of the corresponding topic in the Kafka cluster, completing the directed routing of primary key messages with the same native transaction ID or the same row.

[0028] Kafka clusters are used to store structured messages. Based on the identifier field in the structured message, targeted routing rules are formulated to ensure that messages of the same transaction or the same business entity are transmitted in a centralized manner, guaranteeing message transmission order and persistence. During the routing process, Kafka's high throughput characteristics are used to decouple the upstream encapsulation module and the downstream processing module, thereby achieving peak shaving and valley filling.

[0029] By using pre-defined partitioning rules, modified data from the same transaction is sent to the same partition, ensuring the order of messages within the same partition. Kafka's replication mechanism ensures data reliability and prevents data loss.

[0030] The S4 and S Storm clusters consume the standardized structured messages from each Kafka partition, divide the structured messages into transaction batches according to the native transaction ID, and assign a unique global transaction ID to each transaction batch; based on the preset single-row aggregated data structure, the data transformation and aggregation are performed on the field values ​​before and after the changes of all operation records in the transaction batch to obtain single-row aggregated data; Specifically, including: The Storm cluster obtains structured messages from Kafka partitions through the Spout component, divides the structured messages into batches according to the native transaction ID or the unpacking identifier, pulls the structured messages of the same native transaction ID batch and assigns a unique transaction ID to the batch; Storm's Bolt component receives structured messages in batches with unique transaction IDs. It first performs out-of-order buffering on the structured messages, temporarily storing scattered messages belonging to the same transaction but from different Kafka partitions, until it is confirmed that all messages of the transaction have been received. Then, it performs data cleaning, format conversion, and business association processing to obtain structured messages adapted to the target non-relational database storage format. If a failure occurs during the commit process, the entire transaction batch is rolled back and reprocessed after the failure is resolved, ensuring that all data changes within the same transaction either succeed completely or are rolled back completely.

[0031] Storm's message processing mechanism guarantees at-least-once semantics for messages, and combined with Kafka's offset management, ensures no data loss. Furthermore, Storm's transaction topology or custom transaction processing logic further guarantees transaction consistency.

[0032] S5. Based on the global transaction ID, perform a single-row aggregated data write operation to the non-relational database, and achieve idempotent writing through the uniqueness verification of the global transaction ID.

[0033] Based on structured messages, the Storm cluster and the target non-relational database perform a collaborative commit operation: the atomic operation interface of the target non-relational database is called, and the preset transaction version number table in the target database is queried to verify whether the structured message corresponding to the transaction ID has been written; if it has not been written, the full data commit is completed; if it has been written, it is skipped, thus achieving idempotent write. Specifically, including: Obtain the structured message with a unique transaction ID output after data transformation from the Storm cluster, and at the same time obtain the core association fields and data association relationships of multiple related business tables in Oracle to obtain the table association rules; Based on the table association rules, aggregation mapping rules for single-row data structures in non-relational databases are formulated, and the core identifier fields of single-row data structures are determined. Based on the aggregation mapping rules and core identifier fields, structured messages corresponding to multiple Oracle related tables under the same transaction ID are mapped and merged into a single row of data in a non-relational database. Based on the single row of data, the single row atomic operation interface of the non-relational database is called to write the single row of data into the non-relational database at once, thereby achieving transaction consistency by utilizing the characteristics of single row atomic operation.

[0034] As a specific example of the above embodiments, the target non-relational database is an HBase database. The structured message with a unique transaction ID is obtained after the data transformation of the Storm cluster. At the same time, the core association fields and data association relationships of multiple related business tables in the Oracle database are obtained to obtain the table association rules. The related business tables are customer basic information tables, metering point tables, or metering reading tables. Based on the table association rules, the aggregation mapping rules for HBase single-row data structures are formulated: the core association field of the business transaction in the Oracle side or the combination of the meter reading segment flip and the business primary key is used as the RowKey of the HBase table; the fields of multiple association tables in Oracle are divided into different column families in HBase according to the business type, and the column qualifiers under each column family correspond to the specific fields of the Oracle association table, forming a nested composite data structure and creating an aggregation table. For historical data migration scenarios, aggregation is achieved using MapReduce programs: First, 33 mapping tables are created in HBase to initialize the historical data of the Oracle related tables; then, partition tables are created with the meter reading segment number flipped as the partition condition, the mapping table data is queried according to the meter reading segment range, and the data is assembled into a hierarchical structure and then migrated to the HBase aggregation table in multiple processes. Based on the aggregation mapping rules and RowKey design, structured messages corresponding to multiple Oracle related tables under the same unique transaction ID are mapped and merged into a single row of data corresponding to the RowKey in HBase. Based on the single row of data, the HBase single-row atomic operation interface is called to write the single row of data into the HBase aggregate table at once, thus ensuring transaction consistency by utilizing the atomicity of HBase single-row operations.

[0035] As a preferred example of the above embodiments, it further includes: Configure a data loss compensation mechanism based on the synchronization of data between Oracle and a non-relational database. The data loss compensation mechanism includes two methods: manual synchronization or timed automatic comparison and synchronization.

[0036] Example 2 like Figure 3 As shown, based on the same inventive concept as the above embodiments, the present invention also provides an apparatus for achieving database transaction consistency, comprising: The capture module is used to capture change event data from the transaction log of a relational database on a transaction-by-transaction basis; the change event data includes operation records of all related tables involved in the transaction; The extraction module is used to extract the operation type, table name, row primary key, native transaction ID and change value corresponding to each operation record based on the change event data, and encapsulate them into a structured message; The partitioning module is used by the Kafka cluster to route the structured messages using a preset partitioning rule to obtain multiple Kafka partitions. The preset partitioning rule is to route structured messages with the same native transaction ID or the same row primary key to the same Kafka partition. The aggregation module is used by the Storm cluster to consume the standardized structured messages from each Kafka partition, divide the messages into transaction batches according to the original transaction ID, and assign a unique global transaction ID to each transaction batch; based on the preset single-row aggregated data structure, it performs data transformation and aggregation on the field values ​​before and after the changes of all operation records in the transaction batch to obtain single-row aggregated data. The write module is used to perform atomic write operations on single-row aggregated data to a non-relational database based on the global transaction ID.

[0037] Example 3 like Figure 4 As shown, the present invention also provides an electronic device 100 for implementing a method for achieving database transaction consistency; The electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on at least one processor 102, and at least one communication bus 104.

[0038] The memory 101 can be used to store computer program 103. The processor 102 implements the steps of a method for achieving transaction consistency in Embodiment 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0039] The memory 101 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0040] At least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 102 may be a microprocessor or any conventional processor. Processor 102 is the control center of electronic device 100, connecting various parts of electronic device 100 via various interfaces and lines.

[0041] The memory 101 in the electronic device 100 stores multiple instructions to implement a method for achieving transaction consistency, and the processor 102 can execute multiple instructions to achieve: Change event data is captured from the transaction log of the relational database on a transaction-by-transaction basis; the change event data includes operation records of all related tables involved in the transaction; Based on the change event data, extract the operation type, table name, row primary key, native transaction ID and change value corresponding to each operation record, and encapsulate them into a structured message; The Kafka cluster routes the structured messages using a preset partitioning rule to obtain multiple Kafka partitions. The preset partitioning rule is to route structured messages with the same native transaction ID or the same row primary key to the same Kafka partition. The Storm cluster consumes the standardized structured messages from each Kafka partition, divides the structured messages into transaction batches according to the native transaction ID, and assigns a unique global transaction ID to each transaction batch; based on the preset single-row aggregated data structure, it performs data transformation and aggregation on the field values ​​before and after the changes of all operation records in the transaction batch to obtain single-row aggregated data; Based on the global transaction ID, perform an atomic write operation on a single row of aggregated data to a non-relational database.

[0042] Example 4 If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0043] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0044] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0045] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0046] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0047] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for achieving database transaction consistency, characterized in that, include: Capture change event data from the transaction log of a relational database, on a transaction-by-transaction basis; The change event data includes operation records for all related tables involved in the transaction; Based on the change event data, extract the operation type, table name, row primary key, native transaction ID and change value corresponding to each operation record, and encapsulate them into a structured message; The Kafka cluster routes the structured messages using a preset partitioning rule to obtain multiple Kafka partitions. The preset partitioning rule is to route structured messages with the same native transaction ID or the same row primary key to the same Kafka partition. The Storm cluster consumes the standardized structured messages from each Kafka partition, divides the structured messages into transaction batches according to the native transaction ID, and assigns a unique global transaction ID to each transaction batch. Based on the preset single-row aggregated data structure, the values ​​of the fields before and after the changes in all operation records in the transaction batch are transformed and aggregated to obtain single-row aggregated data. Based on the global transaction ID, perform an atomic write operation on a single row of aggregated data to a non-relational database.

2. The method for achieving database transaction consistency as described in claim 1, characterized in that, Also includes: Configure a data loss compensation mechanism based on the synchronization of data between Oracle and a non-relational database. The data loss compensation mechanism includes two methods: manual synchronization or timed automatic comparison and synchronization.

3. The method for achieving database transaction consistency as described in claim 1, characterized in that, In the step of capturing change event data from the transaction log of the relational database on a transaction-by-transaction basis, OGG is used to capture data changes from the transaction log of the relational database on a transaction-by-transaction basis, and the order of the data changes is maintained according to the transaction commit order of the relational database.

4. The method for achieving database transaction consistency as described in claim 1, characterized in that, The step of extracting the operation type, table name, row primary key, native transaction ID, and change value based on the change event data, and encapsulating them into a structured message, includes: Based on change event data, the operation type, table name, row primary key, native transaction ID and change value are extracted through a custom UserExit program and encapsulated into a structured message in JSON or Avro format; The UserExit program supports specifying the Oracle table names and field lists to be extracted. When the amount of change event data exceeds a preset threshold, it automatically unpacks the data and marks the unpacking status with unpacking identifier, unpacking sequence number, and last unpacking identifier.

5. The method for achieving database transaction consistency as described in claim 4, characterized in that, The Kafka cluster uses preset partitioning rules to route the structured messages, resulting in multiple Kafka partitions. This process includes: Obtain the encapsulated structured message, and extract the native transaction ID or row primary key from the structured message as the partition key; Based on the partition key, a preset partitioning strategy is loaded, and a mapping calculation is performed on the partition key to obtain the target partition number; Based on the target partition number, the structured message is sent to the target partition of the corresponding topic in the Kafka cluster, completing the directed routing of primary key messages with the same native transaction ID or the same row.

6. The method for achieving database transaction consistency as described in claim 1, characterized in that, The step of performing data transformation and aggregation on the field values ​​before and after changes of all operation records in the transaction batch based on the preset single-row aggregated data structure to obtain single-row aggregated data includes: The Storm cluster performs a collaborative commit operation with the target NoSQL database, verifying whether the structured message corresponding to the native transaction ID has been written. If it has not been written, the full data commit is completed; otherwise, it is skipped.

7. The method for achieving transaction consistency as described in claim 1, characterized in that, This also includes denormalizing and aggregating the table structures of non-relational databases, including: Obtain the structured message with a unique transaction ID output after data transformation in the Storm cluster, and at the same time obtain the core association fields and data association relationships of multiple related business tables in the Oracle database to obtain the table association rules; Based on the table association rules, aggregation mapping rules for single-row data structures in non-relational databases are formulated, and the core identifier fields of single-row data structures are determined. Based on the aggregation mapping rules and core identifier fields, structured messages corresponding to multiple Oracle related tables under the same native transaction ID are mapped and merged into single-row aggregated data in a non-relational database. Based on the single-row aggregated data, the single-row atomic operation interface of the non-relational database is called to write the single-row data into the non-relational database at once, thereby achieving transaction consistency by utilizing the characteristics of single-row atomic operations.

8. An apparatus for achieving transaction consistency, characterized in that, include: The capture module is used to capture change event data from the transaction log of a relational database on a transaction-by-transaction basis. The change event data includes operation records for all related tables involved in the transaction; The extraction module is used to extract the operation type, table name, row primary key, native transaction ID and change value corresponding to each operation record based on the change event data, and encapsulate them into a structured message; The partitioning module is used by the Kafka cluster to route the structured messages using a preset partitioning rule to obtain multiple Kafka partitions. The preset partitioning rule is to route structured messages with the same native transaction ID or the same row primary key to the same Kafka partition. The aggregation module is used by the Storm cluster to consume the standardized structured messages from each Kafka partition, divide the messages into transaction batches according to the native transaction ID, and assign a unique global transaction ID to each transaction batch; Based on the preset single-row aggregated data structure, the values ​​of the fields before and after the changes in all operation records in the transaction batch are transformed and aggregated to obtain single-row aggregated data. The write module is used to perform atomic write operations on single-row aggregated data to a non-relational database based on the global transaction ID.

9. An electronic device, characterized in that, It includes a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the method for achieving transaction consistency as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the method for achieving transaction consistency as described in any one of claims 1 to 7.

Images