Method for realizing data consistency of multiple data sources through distributed lock and message-oriented middleware

Abstract

The invention relates to the technical field of data processing methods, and discloses a method for achieving multi-data-source data consistency through a distributed lock and message middleware, and the method comprises the steps: receiving a service request, using a service ID as a key to obtain a Tail distributed lock, and dividing the Tail distributed lock into a successful condition and a failed condition according to the condition of obtaining the lock; acquiring a locking failure condition, returning a self-defined error message, and ending the process; if the lock succeeds, executing a local database transaction, if the transaction submission fails, rolling back the transaction, releasing the Tair lock, and ending the process; if the transaction is submitted successfully, releasing the Tail lock, asynchronously sending a message to the Rocket Mq, and dividing the message into a successful condition and a failed condition according to the message sending condition; if the message is successfully sent, executing cross-database transaction processing, and ending the process; and if the message is failed to be sent, executing exception processing, such as performing a retry mechanism, fusing a release mechanism and ending the process. According to the method, the problems of cross-database transaction failure, concurrent operation conflict and data inconsistency of multiple databases are effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing methods, and in particular to a method for achieving data consistency among multiple data sources using a distributed lock and message middleware. Background Art

[0002] In recent years, with the rapid development of distributed system architecture, business data is often stored in multiple heterogeneous databases, such as MySQL, Oracle, Sqlserver, MongoDB, and Elasticsearch. In this case, ensuring the consistency of business operations across multiple data sources has become a key challenge.

[0003] Traditional solutions have the following flaws: an imbalance between performance and consistency. The transaction's two-phase commit (2PC) relies on a global lock, which reduces throughput, and single points of failure can cause system paralysis. Asynchronous communication reliability and complexity are inverted. Local message tables require business-coupled polling, and messages are easily lost under tens of thousands of concurrent connections. Traditional message queues cannot guarantee the atomic delivery of transactional messages, and consumption failures require manual compensation. Collaboration across heterogeneous data sources fails, and existing solutions cannot uniformly coordinate SQL and NoSQL operations. For example, MySQL succeeds, but Elasticsearch updates fail. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the prior art, the present invention provides a method for achieving data consistency among multiple data sources using distributed locks and message middleware, which effectively solves the problems of cross-database transaction failure, concurrent operation conflicts, and data inconsistency among multiple data sources.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution of the present invention is as follows:

[0006] A method for achieving data consistency among multiple data sources using a distributed lock and message middleware is provided, comprising the following steps:

[0007] Create TairManagerImpl and DefaultMQProducer objects, initialize Tair distributed lock and RocketMq message producer;

[0008] After receiving a business request, call the setnx(business ID, "locked", 30) method of TairManagerImpl. Use the business ID as the key to obtain the Tair distributed lock, which expires in 30 seconds. The lock acquisition is divided into two situations: success and failure.

[0009] If the lock acquisition fails, a custom error message is returned and the process ends;

[0010] After the lock is successfully acquired, the local database transaction is executed, the business data of the main database is written to the local message table, the local transaction is committed, and the del(business ID) method of TairManagerImpl is called to release the Tair distributed lock.

[0011] If the local transaction fails to commit, the local database transaction is rolled back, the Tair distributed lock is released, and the process ends. If the local transaction succeeds, the delete(business ID) method of TairManagerImpl is called to immediately release the Tair distributed lock, create a Message object, initialize the message to be sent, and call the send(message) method of DefaultMQProducer to asynchronously send the message to RockerMq.

[0012] The RockerMq message is sent successfully, creating a DefaultMQPushConsumer object and initializing the RocketMq message consumer. By listening to the Broker, the consumeMessage() method is called and the consumer pulls the message. If it is a duplicate message, the message is discarded and the process ends. If it is not a duplicate message, cross-database transaction processing is executed, cross-database business data is written, and cross-database transaction submission is completed.

[0013] If the cross-database transaction is successfully submitted, the processing status will be recorded, the local message table record will be deleted, and the process will end. If the cross-database transaction fails to be submitted, the message will be retried and 10 retries will be performed. During the first 10 retries, the consumer will pull the message and determine whether the message is a duplicate by storing the Message ID of the processed message. If it is a duplicate message, the message will be discarded and the process will end. If it is not a duplicate message, the cross-database transaction processing will be executed, the cross-database business data will be written to the cross-database database, and the cross-database transaction will be submitted. When the retry is for the 11th time, a new message instance MessageExt will be created, the message will be sent to the dead letter queue, and the message will be manually processed, ending the process.

[0014] When retrying for the 11th time, RocketMq automatically transfers it to the dead letter queue, freeing up Broker thread resources to ensure the normal processing of other messages and prevent invalid messages from blocking normal business processes; it fully retains business identifiers (such as business IDs), supports manual compensation and root cause analysis, and implements fault isolation and manual backup;

[0015] RockerMq message sending failed, entering the message recovery process;

[0016] Enter the message recovery process and perform three memory fast retries. The timeout periods for the three fast retries are 50ms, 100ms, and 200ms respectively. If one of the three retries succeeds, execute the methods described in

[0012] to

[0013] . If all three retries fail, create a ConcurrentLinkedQueue <string>, File, and FileWriter objects, calling the add(message) method of ConcurrentLinkedQueue and the write(message) method of FileWriter to write the message to the disk persistence queue. In case of failure, the message is stored persistently. The scheduled retry task is started and retried 10 times. If the scheduled retry succeeds within 10 times, the delete() method of File is called to delete the disk record and execute the methods described in

[0012] to

[0013] . If the scheduled retry still fails after 10 times, the fuse release mechanism is executed.

[0017] The circuit breaker release mechanism is executed. When the circuit breaker conditions are met (the number of retries is greater than 10, the time consumption is greater than 80% of the lock TTL, and the failure rate is greater than 50% in 1 minute), the delete(business ID) method of TairManagerImpl is called to forcibly release the Tair distributed lock, record the exception log, trigger an alarm notification, create a manual compensation task, switch to manual intervention, and end the process.

[0018] The technical solution provided by the present invention has the following beneficial effects:

[0019] The two-phase commit mechanism of the RocketMq message middleware eliminates synchronization failures between heterogeneous data sources such as MySQL, Oracle, Sqlserver, MongoDB, and Elasticsearch, and fundamentally solves cross-database transaction failures. Tair distributed locks achieve millisecond-level resource preemption (less than 5ms) to avoid concurrent operation conflicts. Persistent message delivery and dead-letter queues provide a safety net to eliminate data inconsistencies. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Attachment Figure 1 This is a business process diagram of the main process example of the present invention.

[0021] Attachment Figure 2 This is a flow chart of the method of Example 1 of the present invention.

[0022] Attachment Figure 3 This is a flow chart of the method of embodiment 2 of the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] Main process example:

[0025] Attachment Figure 1 As shown in the figure, a method for achieving data consistency across multiple data sources using distributed locks and message middleware is described. The specific steps are as follows:

[0026] Create TairManagerImpl and DefaultMQProducer objects, initialize Tair distributed lock and RocketMq message producer;

[0027] After receiving a business request, call the setnx(business ID, "locked", 30) method of TairManagerImpl. Use the business ID as the key to obtain the Tair distributed lock, which expires in 30 seconds. The lock acquisition is divided into two situations: success and failure.

[0028] If the lock acquisition fails, a custom error message is returned and the process ends;

[0029] After the lock is successfully acquired, the local database transaction is executed, the business data of the main database is written to the local message table, the local transaction is committed, and the del(business ID) method of TairManagerImpl is called to release the Tair distributed lock.

[0030] Create a message object, initialize the message content to be sent, call the send(message) method of DefaultMQProducer, and asynchronously send the message to RockerMq;

[0031] The RockerMq message is sent successfully, creating a DefaultMQPushConsumer object, initializing the RocketMq message consumer, calling the consumeMessage() method by listening to the Broker, and the consumer pulls the message, executes cross-database transaction processing, writes cross-database business data, and commits the cross-database transaction;

[0032] If RockerMq message sending fails, exception handling is performed, the message recovery process is entered, and then the process ends.

[0033] Example 1:

[0034] As attached Figure 2 The following is a method for consumers to pull messages and process cross-database transactions. The specific steps are as follows:

[0035] Create a DefaultMQPushConsumer object, initialize the RocketMq message consumer, and call the consumeMessage() method by listening to the Broker. The consumer pulls the message.

[0036] The consumer pulls the message and determines whether the message is a duplicate by storing the Message ID of the processed message. If it is a duplicate message, the message is discarded and the process ends;

[0037] If the message is not a duplicate, the cross-database transaction is processed, the cross-database business data is written to the cross-database, and the cross-database transaction is committed;

[0038] If the cross-database transaction is successfully submitted, the processing status is recorded and the local message table record is deleted, and the process ends. If the cross-database transaction fails to be submitted, the message is retried and 10 retries are performed. The first 10 retries are performed according to the method described in

[0035] to

[0037] .

[0039] When the retry is performed for the 11th time, a new message instance MessageExt is created, the message is sent to the dead letter queue, and manual processing is performed, ending the process.

[0040] Example 2:

[0041] As attached Figure 3 The following is a message recovery process method.

[0042] Enter the message recovery process and perform three memory fast retries. The timeout periods for the three fast retries are 50ms, 100ms, and 200ms respectively. If one of the three retries succeeds;

[0043] Create a DefaultMQPushConsumer object, initialize the RocketMq message consumer, listen to the Broker, call the consumeMessage() method, and the consumer pulls the message;

[0044] The consumer pulls the message. If it is a duplicate, it discards the message and ends the process. If it is not a duplicate, it executes cross-database transaction processing, writes cross-database business data, commits the cross-database transaction, records the processing status, and deletes the local message table record.

[0045] If all three retries fail, create a ConcurrentLinkedQueue <string>, File, and FileWriter objects, call the add(message) method of ConcurrentLinkedQueue and the write(message) method of FileWriter to write the message to the disk persistence queue, store it persistently in case of failure, start the scheduled retry task, and retry 10 times at a time;

[0046] If the scheduled retry succeeds within 10 times, the delete() method of File is called to delete the disk record and the methods described in

[0042] to

[0043] are executed. If the scheduled retry still fails after 10 times, the fuse release mechanism is executed;

[0047] The circuit breaker release mechanism is executed when the circuit breaker conditions are met. The conditions are:

[0048] The number of retries is greater than 10, the time consumed is greater than 80% of the lock TTL, and the failure rate is greater than 50% in 1 minute;

[0049] Call the delete(business ID) method of TairManagerImpl to forcibly release the Tair distributed lock;

[0050] Record exception logs, trigger alarm notifications, create manual compensation tasks, switch to manual intervention, and end the process.< / string> < / string>

Claims

1. A method for achieving data consistency among multiple data sources using a distributed lock and message middleware, characterized in that: include: S1: Receives a business request and uses the business ID as the key to obtain a Tair distributed lock. The lock acquisition is classified into two situations: success and failure. S2: Failed to acquire the lock, returned a custom error message, and ended the process; S3: The lock is acquired successfully and the local database transaction is executed. The transaction is either successful or failed depending on the transaction status. S4: The transaction is successfully committed, the Tair distributed lock is released, and a message is asynchronously sent to RocketMq. S5: Transaction commit fails, the transaction is rolled back, the Tair lock is released, and the process ends; S6: After releasing the Tair distributed lock, asynchronously send a message to RocketMq. The message sending status is divided into two cases: success and failure. S7: The message is sent successfully and the cross-database transaction is executed. The result of the cross-database transaction is divided into two cases: success and failure. S8: The cross-database transaction is successfully processed, the processing status is recorded, the local message record table record is deleted, and then the process ends; S9: If the cross-database transaction fails, the message is retried 10 times. The consumer is retried to pull the message, determine whether the message is a duplicate, and process the cross-database transaction. When the 11th retry is performed, the message is moved to the dead letter queue, and the process ends. S10: If the message fails to be sent, exception handling is performed and the message recovery process is entered, such as: a retry mechanism and a fuse release mechanism, and then the process ends.

2. A method for achieving data consistency among multiple data sources using a distributed lock and message middleware according to claim 1, characterized in that: The lock is successfully acquired in S3, and local database transaction processing is executed, including writing business data to the main database, writing to the local message table, and committing the local transaction.

3. The method for achieving data consistency among multiple data sources using a distributed lock and message middleware according to claim 1, characterized in that: According to the result of RocketMq message sending, there are two cases: success and failure. If the message is sent successfully, the consumer pulls the message and determines whether the message is a duplicate by storing the Message ID of the processed message. If it is a duplicate message, the message is discarded and the process ends. If it is not a duplicate message, cross-database transaction processing is executed, including: writing cross-database business data and committing cross-database transactions.

4. A method for achieving data consistency among multiple data sources using a distributed lock and message middleware according to claim 3, characterized in that: The cross-database transaction submission is divided into two cases: success and failure. If the cross-database transaction is successfully submitted, the processing status is recorded, the local message table record is deleted, and the process ends. If the cross-database transaction fails to be submitted, the message retry mechanism is executed, and 10 message retries are performed. The first 10 retries execute the above S9 step. When the retry is the 11th time, it is moved to the dead letter queue and switched to manual processing, ending the process.

5. The method for achieving data consistency among multiple data sources using a distributed lock and message middleware according to claim 1, characterized in that: According to the result of RocketMq message sending, there are two cases: success and failure. If the message sending fails, perform exception handling, enter the message recovery process, perform three memory fast retries, and if the retry is successful, execute the S8 step; If the retry fails, the circuit breaker release mechanism is executed.

6. The method for achieving data consistency among multiple data sources using a distributed lock and message middleware according to claim 5, characterized in that: Execute the message retry mechanism and perform three quick memory retries. If the retry fails, write to the disk persistence queue, start the scheduled retry task, and retry 10 times at a time. If the scheduled retry succeeds within 10 times, delete the disk record and execute the S8 step. If the scheduled retry still fails after 10 times, execute the fuse release mechanism.

7. A method for achieving data consistency among multiple data sources using a distributed lock and message middleware according to claim 5 or 6, characterized in that: Execute the circuit breaker release mechanism; when the circuit breaker condition is met, the system forcibly releases the Tair distributed lock, records the exception log, triggers an alarm notification, creates a manual compensation task, switches to manual intervention, and ends the process.

Images