Method, equipment and computer program product for realizing distributed transaction consistency

Abstract

The invention relates to a method, equipment and a computer program product for realizing distributed transaction consistency. The method comprises the following steps: starting an initiator local database transaction by a database agent based on a user operation obtained by a transaction initiator, wherein the starting of the initiator local database transaction comprises the following steps: executing initiator modification corresponding to the user operation in an initiator database associated with the transaction initiator; and creating an initiator transaction record associated with the initiator modification in the initiator database; generating, by the database agent, an initiator transaction message based on an initiator log event captured in a log file of the initiator database, the initiator log event being generated and recording an initiator modification in response to successful creation of an initiator transaction record; sending, by the database agent, the initiator transaction message to a message queue; and obtaining, by one or more transaction participants, the initiator transaction message from the message queue and processing the initiator transaction message.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to distributed transactions, and in particular, to a method, device and computer program product for implementing consistency of distributed transactions. BACKGROUND

[0002] The CAP theorem is a core theory in the design of distributed systems, used to guide the architecture design of distributed systems. The theory points out that in a distributed system, the three characteristics of consistency (Consistency), availability (Availability) and partition tolerance (Partition Tolerance) cannot be met at the same time, and must be balanced according to business needs.

[0003] Distributed microservice architecture is a software development pattern based on distributed architecture, which splits a single large application into a series of small, independently running microservices. Each microservice is independently deployed, independently running, and communicates through the network. In the distributed microservice architecture, each microservice is responsible for handling a specific business function, and they can be independently developed, tested, deployed and expanded, each maintaining its own database or data storage.

[0004] Distributed microservice architecture realizes high availability, scalability and fault tolerance. However, this architecture, while bringing advantages, also brings challenges and complexity. In a distributed microservice system, a business request is usually processed by multiple microservice nodes, and it is usually necessary to guarantee consistency in the CAP theorem, i.e. it is necessary to guarantee that multiple microservices succeed or fail at the same time. SUMMARY

[0005] The present disclosure provides a method, device and computer program product for implementing consistency of distributed transactions, which can use the mechanism of database local transactions and database own log files to guarantee that the successful opening of a local database transaction at a transaction initiator can cause the transaction participants to be notified to perform related processing, thereby realizing consistency of distributed transactions.

[0006] According to one aspect of this disclosure, a method for achieving distributed transaction consistency is provided. The method may include: a database broker initiating an initiator-local database transaction based on a user operation obtained by a transaction initiator, wherein initiating the initiator-local database transaction includes: performing an initiator modification corresponding to the user operation in an initiator database associated with the transaction initiator; and creating an initiator transaction record associated with the initiator modification in the initiator database; the database broker generating an initiator transaction message based on an initiator log event captured in a log file of the initiator database, the initiator log event being generated in response to the successful creation of the initiator transaction record and recording the initiator modification; the database broker sending the initiator transaction message to a message queue; and one or more transaction participants retrieving the initiator transaction message from the message queue and processing the initiator transaction message.

[0007] In some embodiments, the processing of the initiator transaction message by the one or more transaction participants may include the one or more transaction participants initiating a participant-local database transaction for each transaction participant via the database broker, wherein initiating a participant-local database transaction for each transaction participant may include: performing a participant modification corresponding to the initiator modification in the participant database associated with the transaction participant; and creating a participant transaction record associated with the participant modification in the participant database.

[0008] In some embodiments, the method may further include: the database agent performing reconciliation based on the initiator log event and participant log events in the log files of each participant database, wherein the participant log events in the log files of a participant database are generated in response to the successful creation of a participant transaction record in the participant database and record the participant modifications performed in the participant database.

[0009] In some embodiments, the method may further include: in response to the reconciliation result indicating that the initiator log event does not match the participant log event in the log file of the participant database of the first transaction participant among the one or more transaction participants, the database agent triggers the initiator transaction message to be sent to the message queue again, so that the message queue can send the initiator transaction message to the first transaction participant again.

[0010] In some embodiments, the database agent triggering the resending of the initiator transaction message to the message queue may include: the database agent sending an exception notification message to the message queue; the message queue sending a trigger message to the transaction initiator in response to receiving the exception notification message; and the transaction initiator sending the initiator transaction message to the message queue in response to the trigger message.

[0011] In some embodiments, the method may further include: in response to a second transaction participant among the one or more transaction participants failing to process the initiator transaction message, the message queue resends the initiator transaction message to the second transaction participant based on a retry mechanism.

[0012] In some embodiments, the method may further include: in response to the retry count reaching a retry threshold, the message queue stops sending the initiator transaction message to the second transaction participant, and sends the initiator transaction message to a dead-letter queue.

[0013] In some embodiments, the method may further include: the transaction initiator sending the initiator transaction message to the message queue in response to detecting the initiator transaction message in the dead-letter queue.

[0014] In some embodiments, the method may further include: the transaction initiator issuing an alarm signal in response to detecting the initiator transaction message in the dead-letter queue, so that maintenance personnel send the initiator transaction message to the message queue in response to the alarm signal.

[0015] In some embodiments, the reconciliation is performed after a predetermined time period has elapsed since the initiator log event or the participant log event was generated.

[0016] In some embodiments, the method may further include: after capturing the initiator log event, the database agent deletes the initiator transaction record in the initiator database.

[0017] In some embodiments, the method may further include: in response to the message queue not receiving the initiator transaction message, the database broker resends the initiator transaction message to the message queue.

[0018] In some embodiments, the method may further include: the transaction initiator issuing an alarm signal in response to the trigger message, so that maintenance personnel send the initiator transaction message to the message queue in response to the alarm signal.

[0019] In some embodiments, the transaction initiator and the transaction participant are microservices in a distributed microservice system.

[0020] According to another aspect of this disclosure, an apparatus for implementing distributed transaction consistency is provided. The apparatus may include: one or more processors; a memory coupled to at least one of the processors; and a set of computer program instructions stored in the memory, the set of computer program instructions performing the methods described above when executed by at least one of the processors.

[0021] According to another aspect of this disclosure, a computer program product for implementing distributed transaction consistency is provided. This computer program product includes a computer program that, when executed by a processor, implements the above-described method.

[0022] Based at least on the above embodiments of this disclosure, the mechanism of local database transactions and the database's own log files can be used to ensure that the successful initiation of a local database transaction at the transaction initiator can lead to notification of transaction participants to perform relevant processing, thereby achieving distributed transaction consistency. Attached Figure Description

[0023] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to offer a further understanding of the embodiments of this disclosure and form part of the specification. The drawings, together with the embodiments of this disclosure, are used to explain this disclosure but do not constitute a limitation thereof. In the drawings, unless explicitly stated otherwise, the same reference numerals denote the same parts, steps, or elements.

[0024] Figure 1 A schematic diagram illustrating the various components involved in embodiments of this disclosure and their interactions is shown;

[0025] Figure 2 A flowchart illustrating a method for implementing distributed transaction consistency according to embodiments of the present disclosure is shown;

[0026] Figure 3 An embodiment according to this disclosure is shown. Figure 2 Sub-steps of the method shown;

[0027] Figure 4 A schematic diagram of an apparatus for implementing distributed transaction consistency according to an embodiment of the present disclosure is shown.

[0028] Those skilled in the art will understand that the elements in the accompanying drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements in the illustrations, block diagrams, or flowcharts may be exaggerated relative to other elements to aid in accurate understanding of this embodiment. Detailed Implementation

[0029] The technical solutions of this disclosure will now be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are part of, but not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without any creative effort fall within the protection scope of this disclosure.

[0030] In the description of this disclosure, it should be noted that terms such as “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Similarly, words such as “a,” “an,” or “the” do not indicate a quantity limitation but rather indicate the presence of at least one. Words such as “including” or “comprising” mean that the element or object preceding the word encompasses those elements or objects listed following the word and their equivalents, without excluding other elements or objects.

[0031] Furthermore, the technical features involved in the different embodiments of this disclosure described below can be combined with each other, as long as there is no conflict between them.

[0032] As mentioned above, in distributed systems, especially distributed microservice systems, it is typically necessary to ensure consistency as defined in the CAP theorem. Taking a distributed microservice system as an example, it is necessary to guarantee that multiple microservices succeed or fail simultaneously. For instance, in an application, if a user cancels their account, the user's account information must be deleted, along with all their personal data, such as products, orders, and points associated with that user. In a distributed microservice system, different data is maintained by different business teams and different microservices. When the account service (which is a microservice, acting as, for example, a transaction initiator) deletes the user's account, other microservices (such as the product service and order service, acting as, for example, transaction participants) must also perform corresponding operations. For example, the product service needs to delete products associated with that user, the order service needs to delete orders associated with that user, and the points service needs to delete points associated with that user, etc. Otherwise, a situation might arise where the user has canceled their account, but their personal data is still retained in the system, potentially leading to information leaks and privacy violations.

[0033] To this end, this disclosure provides a method, apparatus, and computer program product for achieving distributed transaction consistency in a lightweight manner. It can utilize the mechanisms of local database transactions and the database's own log files to ensure that the successful initiation of a local database transaction at the transaction initiator can lead to notification of transaction participants to perform relevant processing, thereby achieving distributed transaction consistency.

[0034] The term "database" as used in this disclosure can refer to any mainstream database, including but not limited to MySQL, PostgreSQL, Oracle Database, Microsoft SQL Server, MongoDB, etc. The above list is merely illustrative and not exhaustive. As technology advances, any existing or future database product that possesses the mechanisms for local database transactions and log files mentioned in this disclosure will fall within the scope of "database" as described in this disclosure.

[0035] In this disclosure, a "transaction" refers to a set of operations performed on business data in a database that satisfy ACID (Atomic, Consistent, Isolated, Durable) constraints. Furthermore, a "local database transaction" in this disclosure refers to a set of operations that involves only the resources of a single database.

[0036] In the following description, embodiments of this disclosure are illustrated using a distributed microservice system as a typical application scenario. However, it is understood that the methods, apparatus, and computer program products for achieving distributed transaction consistency proposed in this disclosure can be applied to various distributed systems across nodes and resources, and are not limited to distributed microservice systems.

[0037] Figure 1 A schematic diagram illustrating the various components involved in embodiments of this disclosure and their interactions is shown. For example... Figure 1 As shown, user 110 can perform user operations, such as placing an order. In response, on the transaction initiator side, transaction initiator 120 (e.g., the order service, which is a microservice) can initiate an initiator-local database transaction via database broker 130. Figure 1 In this context, each of the transaction initiator 120 and transaction participants 140-1 (e.g., inventory service, which is a microservice) and 140-2 (e.g., account service, which is a microservice) can have its own database and can interact with its respective database through database proxy 130. That is, transaction initiator 120 can interact with initiator database 170 through database proxy 130, transaction participant 140-1 can interact with participant database 180-1 through database proxy 130, and transaction participant 140-2 can interact with participant database 180-2 through database proxy 130. Furthermore, Figure 1The diagram also shows message queue 150, used for interaction between transaction initiator 120 and transaction participants 140-1 and 140-2. If a message in message queue 150 reaches its maximum retry threshold or lifespan without being successfully processed (or consumed) by a transaction participant, the message can be routed to dead-letter queue 160 to ensure that important messages are not lost and subsequent manual or automatic retries are possible. Additionally, as shown, transaction initiator 120 can also communicate with alarm system 190 to alert staff in case of anomalies, enabling rapid detection and timely response to anomalies.

[0038] Transaction initiator 120 and transaction participants 140-1 and 140-2 (collectively referred to as transaction participant 140) are all microservices in a distributed microservice system. This is understandable, although... Figure 1 The example shows two transaction participants (140), but there can be more or fewer, depending on the specific business rules. For example, in the example where transaction initiator 120 serves orders, transaction participants could further include coupon services, points services, etc. Furthermore, it's important to understand that the roles of transaction initiator and transaction participant are not fixed but interchangeable. A transaction initiator responsible for starting a transaction in one scenario might be invoked by another microservice in another scenario and join the latter's transaction, thus becoming a transaction participant.

[0039] Database proxy 130, message queue 150, and dead-letter queue 160 can all be implemented as middleware or software modules. Additionally, Figure 1 The database proxy 130 appearing in multiple places can represent the same database proxy instance, which provides a unified database access interface to the transaction initiator and transaction participants to complete the coordination and data interaction of distributed transactions.

[0040] Now for reference Figure 2 . Figure 2 A flowchart of a method 200 for implementing distributed transaction consistency according to an embodiment of the present disclosure is shown. Figure 2 Method 200 can be executed by the processor. See below for reference. Figure 1 and Figure 2 To describe the steps of method 200.

[0041] In step 210, the database agent 130 can initiate a local database transaction based on the user operation obtained by the transaction initiator 120. As mentioned above, a "transaction" refers to a set of operations performed on business data in a database, and a "local database transaction" refers to a set of operations that only involves the resources of a single database. Thus, within the scope of this single database, the local database transaction is guaranteed to have strong consistency by the database engine through the ACID mechanism. Specifically, initiating a local database transaction can include: (1) performing an initiator modification corresponding to the user operation in the initiator database 170 associated with the transaction initiator 120; and (2) creating an initiator transaction record associated with the initiator modification in the initiator database 170. In a database, there are usually business tables for storing business data and transaction tables for recording the status, participants, stages, rollback information, etc. of distributed transactions. The aforementioned initiator modification can be a modification of business data in the business table, and can include operations such as insert and update. The aforementioned initiator transaction record is a record of such initiator modification and can be written to the aforementioned transaction table. For example, taking the user operation as an order placement operation, the aforementioned transaction initiator 120 can be an order service. The aforementioned initiator modification could be the insertion of an order record, including business data such as order number, product, amount, and user information. The aforementioned initiator transaction record could be a transaction record associated with this initiator modification, "inserting an order record," used to record metadata about the information modified by the initiator.

[0042] Because it's a local database transaction, strong consistency is guaranteed. Therefore, the two operations mentioned above—the execution of the initiator's modification and the creation of the initiator's transaction record—can either succeed simultaneously or fail simultaneously. That is, if the execution of the initiator's modification succeeds, the initiator's transaction record will definitely be created successfully; conversely, if the execution of the initiator's modification fails, the initiator's transaction record will definitely not be created successfully. Thus, step 210 implements a guarantee mechanism during transaction creation: the transaction initiator starts a local database transaction through the database proxy, and while modifying the database (and its business data), creates a transaction record. This local database transaction ensures that the transaction initiator's processing logic will either succeed or be rolled back.

[0043] In step 220, database agent 130 may generate an initiator transaction message based on initiator log events captured in the log file of initiator database 170. These initiator log events are generated in response to the successful creation of the initiator transaction record mentioned in step 210 and record the initiator modifications mentioned in step 210.

[0044] As mentioned above, the two operations mentioned in step 210—executing the initiator modification and creating the initiator transaction record—may succeed simultaneously or fail simultaneously. Therefore, if the initiator transaction record is successfully created, it means that the initiator modification operation must have succeeded. In this case, an initiator transaction message can be generated to notify the transaction participants to perform the corresponding processing. For example, taking a user operation as an order placement operation, the transaction initiator can correspond to the order service, the initiator modification can correspond to creating an order, and the corresponding transaction participants can include, for example, the inventory service, the account service, the points service, etc., which need to perform corresponding processing. For example, the inventory service needs to perform the operation of deducting product inventory, the account service needs to perform the operation of deducting payment, the points service needs to perform the operation of increasing user points, and so on.

[0045] In the embodiments of this disclosure, the database itself utilizes the following mechanism: the successful creation of a transaction record will always result in the generation of a log event in the database log file recording the corresponding initiator's modification. Therefore, if the log event is captured, it can be determined that the transaction record was successfully created, thus confirming the successful execution of the initiator's modification. In this case, an initiator transaction message can be generated based on such a log event to subsequently notify transaction participants to perform the corresponding processing.

[0046] In the embodiments of this disclosure, by utilizing the mechanism of local database transactions and the database's own log files, it can be ensured that the successful initiation of a local database transaction at the transaction initiator can lead to notification of transaction participants to perform relevant processing, thereby achieving distributed transaction consistency.

[0047] The log files and log events described above can be implemented in different forms in different databases. For example, in a MySQL database, the log file can be a binary log (binlog) file, and the log events can be binlog events. A binlog file records multiple binlog events sequentially, and each binlog event can record the initiator modification made to business data, such as in a before-image and after-image format. The before-image and after-image can be used to describe the complete state of a data row before and after modification, thereby supporting master-slave replication, point-in-time recovery, and compensated rollback of distributed transactions. As another example, in an Oracle database, the log file can be an archive log, and the log events can correspond to redo records used to store change vectors for data blocks. As yet another example, in PostgreSQL, the log file can be a write-ahead log (WAL), and the log events can be WAL records used to store page-level physical changes or logically decoded data. It should be noted that the specific examples of log files and log events in different databases described above are merely illustrative examples for ease of understanding and are not intended to limit the scope of application of the embodiments disclosed herein. The embodiments of this disclosure are not limited to being combined with a particular type of database, nor do they depend on a particular form of log event. In fact, as long as a log event can meet the above conditions, namely, on the one hand, it is generated in response to the successful creation of the initiator transaction record as described in step 210; and on the other hand, it can record the initiator modification as described in step 210, then regardless of the type of database or the technical implementation, the log event falls within the protection and application scope of the embodiments of this disclosure.

[0048] Initiator transaction messages are generated based on initiator log events, which record modifications made by the initiator. Therefore, initiator transaction messages can include key information about the initiator's local database transaction, namely, the core business data and transaction metadata that need to be passed to downstream services (transaction participants), serving as the basis for downstream services to execute subsequent operations. For example, continuing with the example above, initiator transaction messages can include, for instance, the primary key of the order record (such as order ID, user ID, product ID, amount, quantity, etc.); the operation type (here, "insert order"); and the operation time (the timestamp of the order record creation). Downstream services can then use initiator transaction messages to complete corresponding business logic (such as inventory deduction or account balance update).

[0049] As described above, the initiator transaction record mentioned in step 210 can be stored, for example, in a transaction table in a database. Since the initiator log event itself already carries key information about the transaction execution (recording the initiator's modifications), there is no need to retain the initiator transaction record in the initiator database 170 for an extended period. In some embodiments, method 200 may further include deleting the initiator transaction record in the initiator database 170 by the database agent 130 after the database agent 130 captures the initiator log event. That is, the initiator transaction record can be deleted after the log event is generated and captured, thereby reducing the storage space occupied by the initiator database.

[0050] After generating the initiator transaction message, in step 230, the database broker 130 can send the initiator transaction message to the message queue 150, and in step 240, one or more transaction participants 140 retrieve the initiator transaction message from the message queue 150 and process it. In some embodiments, as described above, the transaction initiator 120 and the transaction participant 140 can each be a microservice in a distributed microservice system. Here, a microservice refers to a small, independent service decomposed from a complex application, and each microservice can be developed, tested, deployed, and scaled independently. The transaction participant 140 retrieves the initiator transaction message from the message queue 150, either by the message queue 150 sending the initiator transaction message to the transaction participant 140, or by the transaction participant 140 listening to the message queue 150 and pulling the initiator transaction message from it.

[0051] In the embodiments of this disclosure, message queue 150 is used as an intermediary to send initiator transaction messages to transaction participants. This design enables decoupling between components. Database broker 130 and transaction participant 140 only need to interact with message queue 150 respectively—database broker 130 only needs to send initiator transaction messages to message queue 150, without needing to know when the transaction participant will process the initiator transaction messages subsequently; transaction participant 140 only needs to retrieve initiator transaction messages from message queue 150, without needing to know the specific source and sending time of the initiator transaction messages. Secondly, message queues inherently support message ordering and broadcasting characteristics, which can meet the diverse needs of distributed transaction scenarios. Through the ordered storage mechanism of message queues, it can be ensured that transaction participants process messages in the order in which the transactions occur, avoiding data inconsistency caused by out-of-order delivery; and with the help of broadcasting characteristics, the same initiator transaction message can be distributed to multiple related transaction participants simultaneously, efficiently supporting business scenarios with multi-node collaboration.

[0052] As described above, in the embodiments of this disclosure, both the transaction initiator and the transaction participant interact with their respective local databases through the same database proxy. Therefore, all database interactions pass through this database proxy, facilitating centralized collection of log files. Furthermore, the upper-layer business (transaction initiator and participant) is decoupled from the underlying database through this database proxy. The transaction initiator and participant do not need to be concerned with the type of the underlying database; they only need to use the standardized interface provided by the database proxy to complete operations, which reduces the development complexity caused by database heterogeneity. In addition, the database proxy mentioned in this disclosure, besides providing the functions described herein, can also provide load balancing, multi-datacenter routing, disaster recovery strategies, and other capabilities.

[0053] In step 230 above, the database broker 130 sends the initiator transaction message to the message queue 150. However, in some cases, the message queue 150 may fail to receive the initiator transaction message, for example, due to network failure, momentary connection interruption, packet loss, or response timeout caused by excessive message queue load. Therefore, in some embodiments, method 200 may further include, in response to the message queue 150 not receiving the initiator transaction message, the database broker 130 resends the initiator transaction message to the message queue 150, thereby ensuring reliable delivery of the initiator transaction message to the message queue and thus ensuring the consistency of the entire distributed transaction. In implementation, the message queue 150 may, for example, send feedback to the database broker 130 after receiving the initiator transaction message, so that the database broker 130 can confirm that the initiator transaction message has been successfully sent to the message queue 150, and that if the database broker 130 does not receive feedback (e.g., within a predetermined time period from the start of the database broker 130 sending the initiator transaction message), it can confirm that the initiator transaction message has not been successfully sent to the message queue 150.

[0054] In step 240 above, processing the initiator transaction message by one or more transaction participants 140 may include performing corresponding operations on the business data in the participant 140's own database based on the initiator transaction message. Continuing with the example above where a user operation is an order placement operation, the transaction initiator may correspond to an order service, and the transaction participants may include, for example, an inventory service, an account service, a points service, etc. After receiving the initiator transaction message, each transaction participant needs to perform a corresponding operation. For example, the inventory service needs to perform an operation to deduct product inventory, the account service needs to perform a deduction operation, the points service needs to perform an operation to increase user points, and so on.

[0055] In step 240 above, one or more transaction participants 140 retrieve and process the initiator transaction message from message queue 150. However, in some cases, transaction participant 140 may fail to retrieve the initiator transaction message from message queue 150 or fail to process it successfully due to various reasons such as network failure, memory overflow, or process crash, resulting in a suspended transaction, conflicting final states of transaction initiator 120 and transaction participant 140, and compromising transaction consistency. To solve this problem, the concept of reconciliation in distributed microservice systems and the retry mechanism of message queues can be considered. Reconciliation refers to the process of periodically or in real-time comparing and calibrating data between multiple microservices to ensure the consistency of records and to identify and repair discrepancies. It is the core mechanism for ensuring eventual consistency of data in distributed systems and an important component of compensation transactions. The retry mechanism of message queues refers to the automatic retransmission of messages according to preset rules (such as the number of retries and intervals) when a message is not successfully processed (or consumed) by a transaction participant. These two aspects will be explained below.

[0056] As described above, by initiating a local database transaction, the successful execution of the initiator's modification will inevitably lead to the successful creation of the initiator's transaction record. The successful creation of the initiator's transaction record will then generate an initiator log event in the initiator's database log file, recording the corresponding initiator modification. Conversely, if the initiator's modification is not successfully executed, the initiator's transaction record will not be created, and therefore, the aforementioned initiator log event will not be generated. In other words, the initiator's database log file will lack the initiator log event corresponding to the aforementioned initiator modification. To perform distributed transaction tracking and reconciliation, a similar mechanism can be used at the transaction participants to generate participant log events recording participant modifications. By comparing the modifications recorded in the initiator's log event and the participant's log event, it is possible to detect whether a transaction has been suspended or transaction consistency has been compromised.

[0057] Therefore, the processing of the initiator transaction message by one or more transaction participants 140 mentioned in step 240 above may include one or more transaction participants 140 initiating a participant-local database transaction for each transaction participant 140 via the database broker 130. Initiating a participant-local database transaction for each transaction participant 140 includes: (1) performing a participant modification corresponding to the initiator modification in the participant database 180 associated with the transaction participant 140; and (2) creating a participant transaction record associated with the participant modification in the participant database 180. These operations are similar to those described above in conjunction with step 210, except that they are performed on the transaction participant side, and therefore will not be described again.

[0058] Therefore, the two operations of executing participant modification and creating participant transaction records can either succeed simultaneously or fail simultaneously. If the participant transaction record is successfully created, it means that the participant modification operation is guaranteed to succeed, and the successful creation of the participant transaction record will definitely result in the generation of a participant log event in the participant database's log file, which records the corresponding participant modification. Conversely, if the participant modification is not successfully executed, the participant transaction record will not be successfully created, and the aforementioned participant log event will not be generated. That is, the participant database's log file will lack the participant log event corresponding to the aforementioned participant modification. Therefore, reconciliation can be performed based on the initiator log event and the participant log event. Thus, in some embodiments, method 200 may further include step 250. In step 250, database agent 130 can perform reconciliation based on the initiator log event mentioned in step 220 and the participant log event in the log file of each participant database 180, where the participant log event in the log file of a participant database is generated in response to the successful creation of a participant transaction record in that participant database 180 and records the participant modification executed in that participant database 180. Continuing with the example of the user operation of placing an order. The order service, acting as the transaction initiator, performs an order creation operation in its database (initiator database) and generates an initiator log event. This initiator log event records modifications made by the initiator, such as "User U100 purchased 2 jin of apples for 10 yuan and the order status is confirmed complete." The inventory service, acting as a transaction participant, responds to the order by performing an inventory deduction operation in its database (participant database) and generating a participant log event. This participant log event records modifications made by the participant, such as "Apple inventory decreased by 2 jin." The account service, acting as another transaction participant, responds to the order by performing a deduction operation in its database (participant database) and generating a participant log event. This participant log event records modifications made by the participant, such as "User account deducted 10 yuan." Database agent 130 can obtain these initiator and participant log events to perform reconciliation.

[0059] The reconciliation trigger time can be configured to occur some time after the initiator log event and / or participant log event are generated, to avoid reconciliation occurring while transaction participants are processing transactions. Therefore, in some embodiments, step 250 can be executed after a predetermined time period has elapsed since the generation of the initiator log event or participant log event.

[0060] If reconciliation is successful, meaning the initiator log event matches the participant log event in the log file of each participant's database, it can be determined that no transaction is suspended, and transaction consistency is achieved. If reconciliation fails, it can be determined that a transaction is suspended, and transaction recovery can be performed, for example, by re-delivering the initiator transaction message to the transaction participant associated with the participant log event that failed to match the initiator log event, so that it can perform the relevant processing. In some embodiments, method 200 may further include step 260. In step 260, in response to the result of the reconciliation performed in step 250 indicating that the initiator log event does not match the participant log event in the log file of the participant database of a first transaction participant among one or more transaction participants, database agent 130 may trigger the re-sending of the initiator transaction message to message queue 150, so that message queue 150 re-sends the initiator transaction message to the first transaction participant. In this way, the first transaction participant can process the initiator transaction message again, thereby subsequently generating the corresponding participant log event and performing reconciliation again. Here, the first transaction participant is not intended to refer to a specific transaction participant, but rather indicates any transaction participant whose associated participant log event failed to match the initiator log event.

[0061] The aforementioned database proxy 130 triggers the resending of the initiator transaction message to the message queue 150. This can be achieved by (1) the database proxy 130 directly sending the initiator transaction message to the message queue 150, or by (2) the database proxy layer sending an exception notification to the message queue 150, which then sends a notification to the transaction initiator 120, which in turn sends the initiator transaction message to the message queue 150. Compared to the first method, the second method adds several steps to the operation process, but its advantage is that the transaction initiator 120 can perceive the retry status of the initiator transaction message in the message queue 150 in real time or periodically, including key information such as the number of retries and the time node of each retrieval. Therefore, in some embodiments, such as Figure 3As shown, step 260 may further include sub-steps 261 to 263. In sub-step 261, the database broker 130 may send an exception notification message to the message queue 150. In sub-step 262, in response to receiving the exception notification message, the message queue 150 may send a trigger message to the transaction initiator 120. In sub-step 263, in response to the trigger message, the transaction initiator 120 may send the initiator transaction message generated in step 220 to the message queue 150. It can be understood that since the initiator transaction message has already been generated in step 220, the transaction initiator 120 does not need to regenerate the initiator transaction message through the database broker 130, but can directly obtain the initiator transaction message generated in step 220 and send it to the message queue 150.

[0062] For transactions that fail to reconcile, in addition to automatically resending the initiator transaction message by sending it to message queue 150 via transaction initiator 120 in sub-step 263, transaction initiator 120 can also link with alarm system 190 to trigger a manual intervention mechanism. By sending an alarm signal to alarm system 190, maintenance personnel can manually resend the initiator transaction message. Therefore, as a supplement or alternative to the automatic resend scheme in sub-step 263, in some embodiments, method 200 may further include issuing an alarm signal by transaction initiator 120 in response to the trigger message mentioned in sub-step 262, so that maintenance personnel can send the initiator transaction message to message queue 150 in response to the alarm signal. For complex faults that automatic resend cannot resolve (such as configuration errors, logical defects, etc.), manual resend can provide a flexible intervention method to avoid long-term accumulation of dead-letter messages leading to business losses.

[0063] On the other hand, as described above, the retry mechanism of a message queue can also be used to resolve transaction hanging. Therefore, in some embodiments, method 200 may further include step 270. In step 270, in response to a second transaction participant (e.g., 140-1 or 140-2) among one or more transaction participants 140 failing to process the initiator transaction message, message queue 150 may resend the initiator transaction message to the second transaction participant based on the retry mechanism. Here, "second transaction participant" does not refer to a specific transaction participant, but rather indicates any transaction participant that failed to process the initiator transaction message. The reason for failing to process the initiator transaction message may be that transaction participant 140 did not receive the initiator transaction message, or that transaction participant 140 received the initiator transaction message but failed to process it. In implementation, transaction participant 140 may, for example, send feedback to message queue 150 after successfully processing the initiator transaction message, thereby enabling message queue 150 to confirm that the initiator transaction message has been successfully processed, and enabling message queue 150 to confirm that the initiator transaction message has not been successfully processed if no feedback is received (e.g., within a predetermined time period). By leveraging the automatic retry capabilities of message queues, occasional transient failures in a distributed environment can be effectively mitigated, improving the reachability of transactional messages and thus ensuring the eventual consistency of distributed transactions. Simultaneously, this approach avoids the performance overhead and resource waste caused by the failure of some transaction participants triggering a global transaction rollback.

[0064] In the design of the retry mechanism of a message queue, the number of retries must be set to a finite value, rather than an unlimited loop of retries. Without a limit on the number of retries, when a second transaction participant is unable to receive or process the initiator's transaction message due to persistent failures such as service downtime, network interruption, or resource exhaustion, the message queue will fall into an infinite loop of retries. This will not only continuously consume the message queue's thread resources, network bandwidth, and storage capacity, but may also cause message backlog, leading to a decrease in queue processing capacity and consequently affecting the delivery efficiency of other messages. To solve this problem, the message queue can pre-set a retry threshold (this threshold can be dynamically adjusted according to the business scenario). When the number of retries by the message queue to send the initiator's transaction message to the second transaction participant reaches this retry threshold, the message queue will terminate the retries and send the initiator's transaction message to the Dead-Letter Queue (DLQ). The Dead-Letter Queue is an important component of the message queue's high availability and fault tolerance mechanism, a special queue specifically used to store messages that "cannot be processed normally," used to isolate abnormal messages from normal messages. Therefore, in some embodiments, method 200 may also include step 280. In step 280, in response to the retry count reaching the retry threshold, message queue 150 can stop sending the initiator transaction message to the second transaction participant and send the initiator transaction message to dead-letter queue 160. The dead-letter queue isolates and stores abnormal messages, avoiding resource waste and system performance degradation caused by infinite retries, and preventing message loss. After the fault is cleared, messages in the dead-letter queue can be re-delivered or manually processed, ensuring the integrity of the business process.

[0065] To support the redelivery of messages in the dead-letter queue after fault clearance, the transaction initiator 120 can monitor the dead-letter queue 160 to capture and process newly added pending initiator transaction messages in the dead-letter queue. In some embodiments, method 200 may further include step 290. In step 290, in response to detecting the initiator transaction message in the dead-letter queue 160, the transaction initiator 120 may send the initiator transaction message to the message queue 150, so that the message queue 150 can resend the initiator transaction message to a transaction participant that did not successfully process the initiator transaction message, such as the second transaction participant mentioned above. It is understood that the monitoring frequency of the transaction initiator 120 on the dead-letter queue 160 needs to be reasonably configured, and is usually lower than the message retry sending frequency of the message queue 150. This is because the initial retries in a message queue are designed for short-term, recoverable faults such as network jitter or temporary service unavailability, requiring high-frequency retries to ensure real-time performance. However, the generation of messages in a dead-letter queue often indicates a persistent fault (such as service downtime or interface logic anomalies), meaning that retrying within a short period is likely to fail again. By reducing the monitoring frequency for the dead-letter queue, ineffective retries can be avoided from consuming system resources, while also allowing maintenance personnel sufficient time to troubleshoot and repair underlying faults, thus improving the success rate of retrying.

[0066] It's understandable that once an initiator transaction message enters the dead-letter queue 160, it means that the message failed to be processed at at least one transaction participant. Therefore, the reconciliation performed in step 250 will inevitably fail or mismatch. In this case, steps 260 and 290 will both be executed. That is, reconciliation and monitoring the dead-letter queue can serve as complementary means to ensure transaction consistency. It's also understandable that in distributed transaction scenarios, transaction participants typically implement idempotency designs through various existing methods. Therefore, even if the initiator transaction message is resubmitted multiple times to the same participant, that participant will identify duplicate requests through idempotency checks and perform only one valid processing, thus ensuring the correctness of the execution result.

[0067] Furthermore, it's understandable that the listening frequency of the transaction initiator 120 to the dead-letter queue 160 can be configured to be higher than the frequency at which the database agent 130 performs reconciliation operations, depending on business needs. This is because the dead-letter queue stores abnormal transaction messages that have reached the retry threshold and cannot be processed through the regular delivery process. By setting the listening frequency of the dead-letter queue to a higher level, the system can promptly detect and process unsuccessfully processed transaction messages in the dead-letter queue before the reconciliation cycle arrives, without waiting for the reconciliation operation to be triggered to know about the transaction execution anomaly. This allows for proactive problem-solving through automatic resubmission or manual intervention, shortening the processing cycle of abnormal transactions, reducing business risks caused by prolonged non-closed-loop transactions (such as user order delays, unreleased locked inventory, etc.), and also reducing the amount of abnormal data that needs to be processed during reconciliation, thus improving reconciliation efficiency and accuracy.

[0068] In addition to automatically resending the initiator transaction message by sending it to message queue 150 via the transaction initiator 120, the transaction initiator 120 can also link with the alarm system 190 to trigger a manual intervention mechanism. By sending an alarm signal to the alarm system 190, maintenance personnel can manually resend the initiator transaction message. Therefore, as a supplement or alternative to the automatic resend scheme in step 290, in some embodiments, method 200 may further include the transaction initiator 120 issuing an alarm signal in response to detecting the initiator transaction message in dead-letter queue 160, so that maintenance personnel can send the initiator transaction message to message queue 150 in response to the alarm signal. For complex faults that automatic resend cannot resolve (such as configuration errors, logical defects, etc.), manual resend can provide a flexible intervention method to avoid long-term accumulation of dead-letter messages leading to business losses.

[0069] The method 200 for achieving distributed transaction consistency according to embodiments of this disclosure utilizes the mechanisms of local database transactions and the database's own log files to ensure that the successful initiation of a local database transaction at the transaction initiator can lead to notification of transaction participants for relevant processing, thereby achieving distributed transaction consistency. It achieves distributed transaction consistency in a lightweight manner, without requiring additional software development kits (SDKs) and middleware platforms, significantly reducing the usage and maintenance costs of traditional distributed transaction frameworks. Furthermore, the method 200 for achieving distributed transaction consistency according to embodiments of this disclosure can also track, reconcile, and recover suspended transactions through the database's own log file mechanism, ensuring timely detection and recovery of abnormal transactions in various abnormal scenarios. In addition, as a complement to the reconciliation mechanism, a message queue retry mechanism and a dead-letter queue are used to ensure transaction consistency. Method 200 also completely decouples the transaction itself from business logic through a database proxy, freeing business developers from excessive focus on the transaction's execution. Finally, method 200 introduces a robust alerting mechanism to facilitate timely detection and resolution of problems through automatic and manual methods.

[0070] Embodiments of this disclosure also provide an apparatus for implementing distributed transaction consistency. Figure 4 A schematic diagram of an apparatus 400 for implementing distributed transaction consistency according to an embodiment of the present disclosure is shown. The apparatus 400 can be used to perform, for example, the method 200 described above.

[0071] like Figure 4 As shown, device 400 may include one or more processors 410 and memory 420. Processor 410 is communicatively coupled to memory and configured to perform the method 200 discussed above.

[0072] Examples of processor 410 include microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described in this disclosure.

[0073] Processor 410 can execute software. Software should be broadly interpreted as instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, application programs, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether it is referred to as software, firmware, middleware, microcode, hardware description languages, or otherwise. The software may reside on memory 420.

[0074] Memory 420 may be a non-transitory computer-readable medium. Non-transitory computer-readable media include, for example, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical disks (e.g., optical discs (CDs) or digital versatile optical discs (DVDs)), smart cards, flash memory devices (e.g., cards, memory cards, or key drives), random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), registers, removable disks, and any other suitable medium for storing software and / or instructions that can be accessed and read by a computer. Memory 420 may reside in, be external to, or be distributed across multiple entities including processor 410. Memory 420 may be embodied in a computer program product. For example, a computer program product may include a computer-readable medium in packaging material. Those skilled in the art will recognize how the functionality described throughout this disclosure can be implemented based on the specific application and overall design constraints imposed on the overall system.

[0075] Furthermore, according to another embodiment of this disclosure, a computer program product for implementing distributed transaction consistency is disclosed. As an example, the computer program product includes a non-transitory computer-readable storage medium containing program instructions executable by a processor. When executed, the program instructions cause the processor to perform one or more of the above-described processes; details are omitted here for brevity.

[0076] This invention can be a system, method, and / or computer program product at any possible level of integration technical detail. The computer program product may include computer-readable program instructions for causing a processor to perform various aspects of this disclosure.

[0077] Unless otherwise stated, an element mentioned in the singular is not intended to mean "one and only one," but rather "one or more." Similarly, a plural reference to an element does not mean "more than one," but rather "one or more," unless otherwise stated or contradicting description elsewhere. Terms such as "if," "when," and "although" should be interpreted as "under the condition of," rather than implying an immediate temporal relationship or response. That is, these phrases, such as "when," do not imply an immediate action in response to an action occurring or during an action, but merely imply that an action will occur if the condition is met, but does not require a specific or immediate time constraint for the action to occur. Combinations, such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof," include any combination of A, B, and / or C, and may include multiple A, multiple B, or multiple C. Combinations such as “at least one of A, B or C”, “one or more of A, B or C”, “at least one of A, B and C”, “one or more of A, B and C” and “A, B, C or any combination thereof” can be only A, only B, only C, A and B, A and C, B and C or A, B and C, wherein any such combination may contain one or more members of A, B or C.

[0078] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible structures, functions, and operations of the methods and apparatus according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, a program segment, or a portion of code containing at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative embodiments, the functions described in a block may occur in a different order than those described in the accompanying drawings. For example, two blocks shown consecutively may actually be executed in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.

[0079] The various embodiments described in this disclosure are for illustrative purposes and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or improvements to techniques found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0080] Throughout the description and claims of this specification, the word “comprising” and variations thereof, such as “comprising” and “including,” means “including, but not limited to,” and are not intended to exclude, for example, other additives, components, integers, or steps. “Exemplary” means “an example of a preferred or ideal implementation and is not intended to convey its indication.” “Like” is not used in a limiting sense but for interpretative purposes.

[0081] As used in this disclosure, the term "determine" can include a variety of operations. For example, "determine," calculation, operation, processing, derivation, investigation, search (e.g., searching in a table, database, or other data structure), and ascertainment are all considered "determine." Additionally, "determine" also refers to receiving (e.g., receiving information), sending (e.g., sending information), inputting, outputting, and accessing (e.g., accessing data in memory). Furthermore, "determine" can also refer to parsing, selecting, picking, opening, and comparing. In other words, several actions can be considered "determine."

[0082] As used in this disclosure, terms such as “connection,” “coupling,” or any variations thereof refer to any direct or indirect connection or combination between two or more units, which may include situations where one or more intermediate units exist between two units that are “connected” or “coupled” to each other. The coupling or connection between units may be physical or logical, or a combination of both. As used in this disclosure, two units may be considered electrically connected by means of one or more wires, cables, and / or printing, and as numerous non-limiting and non-exhaustive examples, may be “connected” or “coupled” to each other by means of electromagnetic energy in the radio frequency region, microwave region, and / or light (visible and invisible) region, etc.

[0083] The present disclosure has been described in detail above; however, it will be apparent to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure may be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims.

Claims

1. A method for achieving distributed transaction consistency, comprising: The database broker initiates a local database transaction for the initiator based on user operations obtained from the transaction initiator, wherein initiating the local database transaction for the initiator includes: Perform the initiator modification corresponding to the user operation in the initiator database associated with the transaction initiator; and Create an initiator transaction record associated with the initiator modification in the initiator database; The database agent generates an initiator transaction message based on initiator log events captured in the log file of the initiator database. The initiator log events are generated in response to the successful creation of the initiator transaction record and record the initiator modifications. The database agent sends the initiator transaction message to the message queue; and One or more transaction participants retrieve the initiator transaction message from the message queue and process the initiator transaction message.

2. The method according to claim 1, wherein, The processing of the initiator transaction message by the one or more transaction participants includes the one or more transaction participants initiating a participant-local database transaction for each transaction participant via the database broker, wherein initiating a participant-local database transaction for each transaction participant includes: Perform participant modifications corresponding to the initiator's modifications in the participant database associated with the transaction participants; and Create a participant transaction record in the participant database that is associated with the participant's modification.

3. The method according to claim 2, further comprising: The database agent performs reconciliation based on the initiator log event and the participant log event in the log file of each participant database. The participant log event in the log file of a participant database is generated in response to the successful creation of a participant transaction record in the participant database and records the participant modifications performed in the participant database.

4. The method according to claim 3, further comprising: In response to the reconciliation result indicating that the initiator log event does not match the participant log event in the participant database log file of the first participant among the one or more transaction participants, the database agent triggers the initiator transaction message to be sent to the message queue again, so that the message queue can resend the initiator transaction message to the first transaction participant.

5. The method according to claim 4, wherein, The database agent triggering the resending of the initiator transaction message to the message queue includes: The database agent sends an exception notification message to the message queue; In response to receiving the exception notification message, the message queue sends a trigger message to the transaction initiator; and In response to the trigger message, the transaction initiator sends the initiator transaction message to the message queue.

6. The method according to claim 1, further comprising: If a second transaction participant among the one or more transaction participants fails to process the initiator transaction message, the message queue resends the initiator transaction message to the second transaction participant based on a retry mechanism.

7. The method according to claim 6, further comprising: In response to the retry count reaching the retry threshold, the message queue stops sending the initiator transaction message to the second transaction participant and sends the initiator transaction message to the dead-letter queue.

8. The method according to claim 7, further comprising: In response to detecting the initiator transaction message in the dead-letter queue, the transaction initiator sends the initiator transaction message to the message queue.

9. The method according to claim 8, further comprising: In response to the detection of the initiator transaction message in the dead-letter queue, the transaction initiator issues an alarm signal, which in turn prompts maintenance personnel to send the initiator transaction message to the message queue.

10. The method according to claim 3, wherein, The reconciliation is performed after a predetermined time period has elapsed since the initiator log event or the participant log event was generated.

11. The method according to claim 1, further comprising: After capturing the initiator log event, the database agent deletes the initiator transaction record from the initiator database.

12. The method according to claim 1, further comprising: If the message queue does not receive the initiator transaction message, the database agent will resend the initiator transaction message to the message queue.

13. The method of claim 5, further comprising: In response to the trigger message, the transaction initiator issues an alarm signal, which in turn prompts maintenance personnel to send the initiator's transaction message to the message queue.

14. The method according to claim 1, wherein, The transaction initiator and the transaction participant are microservices in a distributed microservice system.

15. A device for implementing distributed transaction consistency, comprising: One or more processors; Memory coupled to at least one of the processors in the processor; and A set of computer program instructions stored in the memory, which, when executed by at least one of the processors, perform the method according to any one of claims 1-14.

16. A computer program product for implementing distributed transaction consistency, comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-14.

Images