A method for achieving data consistency across multiple data sources using distributed locks and message middleware

Abstract

This invention relates to the field of data processing methods, and discloses a method for achieving data consistency across multiple data sources using distributed locks and message middleware. The method includes: receiving a business request; acquiring a Tair distributed lock using the business ID as the key; handling lock acquisition success or failure; returning a custom error message and ending the process if lock acquisition fails; executing a local database transaction if lock acquisition is successful; rolling back the transaction and releasing the Tair lock if the transaction commit fails; releasing the Tair lock and asynchronously sending a message to RocketMQ; handling message sending success or failure; executing cross-database transaction processing and ending the process if message sending fails; and handling exception handling, such as retrying or circuit breaker mechanisms, and ending the process if message sending fails. This invention effectively solves the problems of cross-database transaction failure, concurrent operation conflicts, and data inconsistency across multiple databases.

Description

Technical Field

[0001] This invention relates to the field of data processing methods, specifically to a method for achieving data consistency across multiple data sources using distributed locks and message middleware. Background Technology

[0002] In recent years, with the rapid development of distributed system architecture, business data is often stored in multiple heterogeneous databases, such as MySQL, Oracle, SQL Server, MongoDB, and Elasticsearch. In this case, ensuring the consistency of business operations across multiple data sources has become a key challenge.

[0003] Traditional solutions suffer from the following drawbacks: performance and consistency are at odds; two-phase commit (2PC) of transactions relies on global locks, which leads to reduced throughput, and single points of failure can cause system paralysis; the reliability and complexity of asynchronous communication are inverted; local message tables need to be polled in a business-coupled manner, messages are easily lost under tens of thousands of concurrent requests, traditional message queues cannot guarantee atomic delivery of transactional messages, and consumption failures require manual compensation; cross-heterogeneous data source collaboration fails, and existing solutions cannot uniformly coordinate SQL and NoSQL operations, such as: MySQL succeeds, but Elasticsearch updates fail. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a method for achieving data consistency across multiple data sources using distributed locks and message middleware, effectively solving the problems of cross-database transaction failures, concurrent operation conflicts, and data inconsistencies across multiple data sources.

[0005] To achieve the above-mentioned objectives, the technical solution of the present invention is as follows:

[0006] A method for achieving data consistency across multiple data sources using distributed locks and message middleware is provided, comprising the following steps:

[0007] Create TairManagerImpl and DefaultMQProducer objects, and initialize the Tair distributed lock and RocketMQ message producer;

[0008] Upon receiving a business request, the `setnx(business ID, "locked", 30)` method of `TairManagerImpl` is called to acquire a Tair distributed lock using the business ID as the key. The lock expires in 30 seconds, and the process is divided into two cases: success and failure, depending on the lock acquisition outcome.

[0009] If acquiring the lock fails, return a custom error message and end the process;

[0010] Once the lock is successfully acquired, execute local database transaction processing, write business data to the main database, write to the local message table, commit the local transaction, call the del(business ID) method of TairManagerImpl, and release the Tair distributed lock.

[0011] If a local transaction commit fails, roll back the local database transaction, release the Tair distributed lock, and end the process; if a local transaction commits successfully, call the delete(business ID) method of TairManagerImpl to immediately release the Tair distributed lock, create a Message object, initialize the message to be sent, and call the send(message) method of DefaultMQProducer to asynchronously send the message to RockerMQ.

[0012] Upon successful RockerMQ message sending, a DefaultMQPushConsumer object is created, and the RockerMQ message consumer is initialized. By listening to the Broker, the consumeMessage() method is called, and the consumer pulls messages. If it is a duplicate message, the message is discarded and the process ends; if it is not a duplicate message, cross-database transaction processing is performed, cross-database business data is written, and the cross-database transaction is committed.

[0013] If the cross-database transaction commits successfully, the processing status is recorded, the local message table record is deleted, and the process ends. If the cross-database transaction commit fails, message retry is performed, with 10 retries. During the first 10 retries, the consumer pulls the message and checks for duplicates by storing the Message ID of the processed message. If the message is a duplicate, it is discarded, and the process ends. If the message is not a duplicate, cross-database transaction processing is performed, writing cross-database business data and committing the cross-database transaction. On the 11th retry, a new message instance MessageExt is created, the message is sent to the dead-letter queue, and manual processing is initiated, ending the process.

[0014] When retrying for the 11th time, RocketMQ automatically transfers it to the dead letter queue, releases Broker thread resources, ensures that other messages are processed normally, and avoids invalid messages blocking normal business processes; it fully retains business identifiers (such as business IDs), supports manual compensation and root cause analysis, and realizes fault isolation and manual fallback.

[0015] RockerMQ message sending failed, entering message recovery process;

[0016] The message recovery process begins, performing three quick memory retries with timeouts of 50ms, 100ms, and 200ms respectively. If one of the three retries succeeds, the methods described in

[0012] to

[0013] are executed. If all three retries fail, a ConcurrentLinkedQueue is created. <string>The three objects, File, and FileWriter, call the add(message) method of ConcurrentLinkedQueue and the write(message) method of FileWriter to write the message to the disk persistent queue for persistent storage in case of failure. A timed retry task is started, retrying 10 times. If the timed retry succeeds within 10 times, the delete() method of File is called to delete the disk record, and the methods described in

[0012] to

[0013] are executed. If the timed retry still fails after 10 times, the circuit breaker release mechanism is executed.

[0017] The circuit breaker release mechanism is executed. When the circuit breaker conditions are met, namely: the number of retries is greater than 10, the time consumption is greater than 80% of the lock's TTL, and the failure rate is greater than 50% within 1 minute, the delete(business ID) method of TairManagerImpl is called to forcibly release the Tair distributed lock, record the exception log, trigger the alarm notification, create a manual compensation task, switch to manual intervention, and end the process.

[0018] The technical solution provided by this invention has the following beneficial effects:

[0019] RocketMQ's two-phase commit mechanism eliminates synchronization failures between heterogeneous data sources such as MySQL, Oracle, SQL Server, MongoDB, and Elasticsearch, thus eradicating cross-database transaction failures. Tair distributed locks enable millisecond-level resource preemption (less than 5ms), avoiding concurrent operation conflicts. Persistent message delivery and dead-letter queues provide a safety net, eliminating data inconsistencies. Attached Figure Description

[0020] Appendix Figure 1 This is a business process diagram of an example of the main process of this invention.

[0021] Appendix Figure 2 This is a flowchart of the method in Embodiment 1 of the present invention.

[0022] Appendix Figure 3 This is a flowchart of the method in Embodiment 2 of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Main process example

[0025] As attached Figure 1 The diagram illustrates a method for achieving data consistency across multiple data sources using distributed locks and message queues. The specific steps are as follows:

[0026] Create TairManagerImpl and DefaultMQProducer objects, and initialize the Tair distributed lock and RocketMQ message producer;

[0027] Upon receiving a business request, the `setnx(business ID, "locked", 30)` method of `TairManagerImpl` is called to acquire a Tair distributed lock using the business ID as the key. The lock expires in 30 seconds, and the process is divided into two cases: success and failure, depending on the lock acquisition outcome.

[0028] If acquiring the lock fails, return a custom error message and end the process;

[0029] Once the lock is successfully acquired, execute local database transaction processing, write business data to the main database, write to the local message table, commit the local transaction, call the del(business ID) method of TairManagerImpl, and release the Tair distributed lock.

[0030] Create a Message object, initialize the message content to be sent, and call the send(message) method of DefaultMQProducer to send the message asynchronously to RockerMQ;

[0031] Upon successful RockerMQ message sending, a DefaultMQPushConsumer object is created, the RocketMQ message consumer is initialized, and the consumeMessage() method is called by listening to the Broker. The consumer pulls messages, performs cross-database transaction processing, writes cross-database business data, and commits the cross-database transaction.

[0032] If a RockerMQ message fails to send, exception handling is executed, the message recovery process is initiated, and then the process ends.

[0033] Example 1

[0034] As attached Figure 2 The diagram illustrates a method for consumers to fetch messages and perform cross-database transaction processing. The specific steps are as follows:

[0035] Create a DefaultMQPushConsumer object to initialize the RocketMQ message consumer. By listening to the Broker, call the consumeMessage() method to let the consumer pull messages.

[0036] Consumers retrieve messages and determine whether a message is a duplicate by storing the Message ID of processed messages. If a message is a duplicate, it is discarded and the process ends.

[0037] If the message is not a duplicate, then perform cross-database transaction processing, write cross-database business data, and commit the cross-database transaction;

[0038] If the cross-database transaction is successfully committed, the processing status is recorded and the local message table record is deleted, and the process ends. If the cross-database transaction fails to commit, a message retry is performed, with 10 retries. The first 10 retries are performed using the methods described in

[0035] to

[0037] .

[0039] When retrying for the 11th time, a new message instance MessageExt is created, the message is sent to the dead letter queue, and the process is switched to manual handling, ending the process.

[0040] Example 2

[0041] As attached Figure 3 The diagram illustrates a message recovery process. The specific steps are as follows:

[0042] Enter the message recovery process and perform 3 memory fast retries with timeouts of 50ms, 100ms, and 200ms respectively. If 1 out of 3 retries is successful.

[0043] The process involves creating a DefaultMQPushConsumer object, initializing the RocketMQ message consumer, and then calling the consumeMessage() method by listening to the Broker to retrieve messages.

[0044] When a consumer retrieves a message, if it is a duplicate message, the message is discarded and the process ends. If the message is not a duplicate message, a cross-database transaction is executed, which writes business data to the cross-database database, commits the cross-database transaction, records the processing status, and deletes the record from the local message table.

[0045] If all three retries fail, a ConcurrentLinkedQueue will be created. <string>The three objects, File, and FileWriter, call the add(message) method of ConcurrentLinkedQueue and the write(message) method of FileWriter to write the message to the disk persistent queue, which will be persistently stored in case of failure, and start a timed retry task to retry 10 times at a time.

[0046] If the timed retry succeeds within 10 attempts, the `delete()` method of `File` is called to delete the disk record, and the methods described in

[0042] to

[0043] are executed. If the timed retry still fails after 10 attempts, the circuit breaker release mechanism is executed;

[0047] The circuit breaker release mechanism will be implemented when the circuit breaker condition is met. The condition is as follows:

[0048] The number of retries exceeds 10, the time consumption exceeds 80% of the lock's TTL, and the failure rate exceeds 50% within 1 minute.

[0049] Call the delete(business ID) method of TairManagerImpl to forcibly release the Tair distributed lock;

[0050] Record the anomaly log, trigger the alarm notification, create a manual compensation task, switch to manual intervention, and end the process.< / string> < / string>

Claims

1. A method for implementing multi-data source data consistency by a distributed lock and a message middleware, characterized in that, Comprise: S1: receive service request, use service ID as key, through calling Tair distributed lock setnx method, and set a lock value and expiration time, to obtain Tair distributed lock, according to the lock acquisition situation is divided into success and failure two cases; S2: lock acquisition fails, return custom error message, end process; S3: lock acquisition succeeds, execute local database transaction processing, according to the transaction processing situation is divided into success and failure two cases; S4: transaction commit succeeds, release Tair distributed lock, asynchronously send message to RocketMq; S5: transaction commit fails, roll back transaction, release Tair lock, end process; S6: after releasing Tair distributed lock, asynchronously send message to RocketMq, according to the message sending situation is divided into success and failure two cases; S7: message sending succeeds, execute cross-database transaction processing, according to the cross-database transaction processing result is divided into success and failure two cases; S8: cross-database transaction processing succeeds, record processing state, delete local message record table record, and then end process; S9: cross-database transaction processing fails, perform 10 message retries, retry execute consumer pull message, judge whether the message is repeated, cross-database transaction processing, when the 11th retry is performed, move into the dead letter queue, end process; S10: message sending fails, execute exception handling, enter message recovery process; The message recovery process comprises: S101: perform 3 times of memory quick retries, wherein the 3 times of retries adopt an increasing timeout waiting strategy, and the timeout times are set to 50 milliseconds, 100 milliseconds and 200 milliseconds respectively; S102: if one of the 3 times of memory quick retries succeeds, execute the steps S7 to S9; S103: if all the 3 times of memory quick retries fail, write the message into a disk persistent queue, and start a timing retry task to perform timing 10 times of retries; S104: if success within 10 times of timing retries, delete the disk record, and execute the steps S7 to S9; S105: if still fail after the 10 times of timing retries, execute a fuse release mechanism; the fuse release mechanism is: when reaching a fuse condition, the system forcibly releases the Tair distributed lock, records an exception log, triggers an alarm notification, creates a manual compensation task, transfers to manual intervention, and ends the process; wherein the fuse condition comprises at least one of the following: retry number is greater than 10 times, total time consumption since service start is greater than 80% of lock TTL, failure rate is greater than 50% in 1 minute.

2. The method of claim 1, wherein the method is implemented by a distributed lock and a message middleware. In S3, lock acquisition succeeds, execute local database transaction processing, specifically comprising: in the local database transaction, write the business data into the master database, and write into the local message table, and submit the local database transaction; if the local database transaction commit succeeds, immediately release the Tair distributed lock, and after releasing the Tair distributed lock, execute the operation of asynchronously sending the message to RocketMq; if the local transaction commit fails, roll back the transaction and release the Tair distributed lock, end process.

3. The method of claim 1, wherein the method is implemented by a distributed lock and a message middleware. According to the result of RocketMq message sending, it is divided into two cases of success and failure; message sending is successful, the consumer pulls the message, and whether the message is repeated is judged by storing the message ID of the processed message; if it is a repeated message, the message is discarded and the process is ended; if it is not a repeated message, a cross-database database transaction processing is performed, cross-database database business data is written, and cross-database transaction submission is performed; if the cross-database transaction submission fails, a message retry mechanism is started: within the preset 10 times of retry, the step is repeatedly executed; when the retry number reaches the 11th time, the message is moved to the dead letter queue, the automatic retry is stopped and is changed to manual processing, and the process is ended.

4. The method of claim 3, wherein the method is implemented by a distributed lock and a message middleware. According to the result of cross-database transaction submission, it is divided into two cases of success and failure; cross-database transaction submission is successful, the processing state is recorded and the local message table record is deleted, and the process is ended; cross-database transaction submission fails, message retry mechanism is executed, 10 times of message retry is performed, the first 10 times of retry executes the S9 step, when the 11th time of retry, the message middleware automatically transfers the message of consumption failure to the dead letter queue, and immediately releases the Broker message processing thread resource occupied by it, so as to avoid that the message blocks the processing of subsequent normal messages; the message transferred to the dead letter queue will keep its business identification intact; a manual compensation task is created and an alarm notification is triggered, so as to carry out manual intervention, root cause analysis, and end the process.

Images