Data synchronization method and system based on Rocket MQ

Abstract

The invention particularly relates to a data synchronization method and system based on Rocket MQ. According to the data synchronization method and system based on the Rocket MQ, the state of a Rocket MQ cluster is monitored, and monitored ground state information is recorded into a cluster state table; the method comprises the following steps: after a consumer program receives a main data change message, firstly writing the message into a cache table checkdatatable, and adding a checkstatus field for each message to identify whether the message is successfully processed or not, the initial value being "unprocessed"; under the control of the transaction, the consumer program updates the data in the business library; and after the business library is successfully updated, updating a checkstatus field of a corresponding message in the checkdatatable to be'processed ', submitting the offset of the Rocket MQ, and confirming that the message is processed. The data synchronization method and system based on the Rocket MQ are high in expansibility, capable of ensuring that data changes are reflected to all service systems in real time, capable of reducing delay, high in fault-tolerant capability, simpler and more efficient in fault recovery process and capable of remarkably improving the real-time performance and accuracy of synchronization of main data changes.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a data synchronization method and system based on RocketMQ. Background Art

[0002] In the information architecture of modern enterprises, real-time data synchronization between different business systems is crucial to ensure business consistency and continuity. However, traditional data synchronization technologies often rely on scheduled tasks or complex trigger mechanisms, which leads to certain delays in data synchronization and is prone to consistency issues when large-scale data changes occur. As a high-performance message queue product, RocketMQ can solve these problems well, but how to design a complete data change synchronization mechanism is still a challenge.

[0003] In order to overcome the delay and consistency problems in the prior art, the present invention proposes a data synchronization method and system based on RocketMQ. Summary of the invention

[0004] In order to make up for the defects of the prior art, the present invention provides a simple and efficient RocketMQ-based data synchronization method and system.

[0005] The present invention is achieved through the following technical solutions:

[0006] A data synchronization method based on RocketMQ, characterized in that it includes the following steps:

[0007] Step S1, monitor the status of the RocketMQ cluster, and record the monitored ground status information into the cluster status table;

[0008] Step S2: After receiving the master data change message, the consumer program first writes the message into the cache table check_data_table, and adds a check_status field to each message to indicate whether the message has been successfully processed. The initial value is "unprocessed".

[0009] Step S3: Under transaction control, the consumer program updates the data in the business library. After the business library is successfully updated, the check_status field of the corresponding message in check_data_table is updated to "processed", and the offset of RocketMQ is submitted to confirm that the message has been processed.

[0010] In step S2, the consumer program pulls 100 messages in batches each time, and saves the messages in memory after writing them into the cache table check_data_table.

[0011] In the step S3, under abnormal circumstances, if part of the message is updated successfully, only the check_status field of the successful part is updated, and the offset of the corresponding part of the message is submitted.

[0012] If an exception occurs in any of the steps S1 to S3, the consumer program will stop consuming subsequent messages and restart processing unfinished messages after the problem is resolved.

[0013] A data synchronization system based on RocketMQ, comprising:

[0014] The monitoring module is used to monitor the status of the RocketMQ cluster and record the monitored ground status information into the cluster status table;

[0015] The consumer module is used to receive master data change messages and write the messages to the cache table check_data_table. At the same time, a check_status field is added to each message to indicate whether the message has been successfully processed. The initial value is "unprocessed";

[0016] The consumer module updates the data in the business library under transaction control; after the business library is successfully updated, the check_status field of the corresponding message in the check_data_table is updated to "processed", and the offset of RocketMQ is submitted to confirm that the message has been processed.

[0017] The consumer module pulls 100 messages in batches each time and saves the messages in memory after writing them to the cache table check_data_table.

[0018] In an abnormal situation, if part of the message is updated successfully, the consumer module only updates the check_status field of the successful part and submits the offset of the corresponding part of the message.

[0019] If an exception occurs during message processing, the consumer module will stop subsequent message consumption and restart processing unfinished messages after the problem is resolved.

[0020] A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the above methods.

[0021] A computing device comprising:

[0022] One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0023] The beneficial effects of the present invention are as follows: the RocketMQ-based data synchronization method and system have strong scalability, can ensure that data changes are reflected in various business systems in real time, reduce delays, and have strong fault tolerance. The fault recovery process is simpler and more efficient, which significantly improves the real-time and accuracy of master data change synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Attached Figure 1 It is a schematic diagram of the data synchronization method based on RocketMQ of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] The data synchronization method based on RocketMQ includes the following steps:

[0028] Step S1, monitor the status of the RocketMQ cluster, and record the monitored ground status information into the cluster status table; so as to find problems in the cluster in the first time and take corresponding measures.

[0029] Step S2: After receiving the master data change message, the consumer program first writes the message into the cache table check_data_table, and adds a check_status field to each message to indicate whether the message has been successfully processed. The initial value is "unprocessed".

[0030] Step S3: Under transaction control, the consumer program updates the data in the business library. After the business library is successfully updated, the check_status field of the corresponding message in check_data_table is updated to "processed", and the offset of RocketMQ is submitted to confirm that the message has been processed.

[0031] Offset refers to the position of the message that the consumer group should consume next when consuming a queue. It is represented by a long integer value and is used to mark which message the consumer has consumed.

[0032] In step S2, the consumer program pulls 100 messages in batches each time, and saves the messages in memory after writing them into the cache table check_data_table.

[0033] In the step S3, under abnormal circumstances, if part of the message is updated successfully, only the check_status field of the successful part is updated, and the offset of the corresponding part of the message is submitted.

[0034] If any of the steps S1 to S3 is abnormal, such as failure to write to the cache table, failure to update the business database, etc., the consumer program will stop consuming subsequent messages and restart processing unfinished messages after the problem is resolved. This mechanism ensures data consistency even when a network outage or other failure occurs.

[0035] The RocketMQ-based data synchronization system includes:

[0036] The monitoring module is used to monitor the status of the RocketMQ cluster and record the monitored ground status information into the cluster status table;

[0037] The consumer module is used to receive master data change messages and write the messages to the cache table check_data_table. At the same time, a check_status field is added to each message to indicate whether the message has been successfully processed. The initial value is "unprocessed";

[0038] The consumer module updates the data in the business library under transaction control; after the business library is successfully updated, the check_status field of the corresponding message in the check_data_table is updated to "processed", and the offset of RocketMQ is submitted to confirm that the message has been processed.

[0039] The consumer module pulls 100 messages in batches each time and saves the messages in memory after writing them to the cache table check_data_table.

[0040] In an abnormal situation, if part of the message is updated successfully, the consumer module only updates the check_status field of the successful part and submits the offset of the corresponding part of the message.

[0041] If an exception occurs during message processing, the consumer module will stop subsequent message consumption and restart processing unfinished messages after the problem is resolved.

[0042] The computer-readable storage medium stores one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device performs any one of the above methods.

[0043] The computing device comprises:

[0044] One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0045] In summary, the data synchronization method and system based on RocketMQ has the following characteristics:

[0046] 1) Enhanced data consistency:

[0047] By introducing a cache table mechanism, each master data change message is accurately processed in the business system, ensuring data consistency even in the event of network or system anomalies. The check_status field in the cache table is used to track the status of message processing to ensure that only successfully processed messages are submitted.

[0048] 2) Improved system stability:

[0049] The exception handling mechanism ensures that if an error occurs in any step, the system can immediately stop subsequent message consumption and restart processing unfinished messages after the problem is resolved. This mechanism greatly enhances the stability and reliability of the system.

[0050] 3) Improved real-time response capabilities:

[0051] After receiving the message, the consumer program updates the business database through transaction control to ensure that data changes can be reflected in various business systems in real time, reducing delays and improving the system's real-time response capabilities.

[0052] 4) Simplified fault recovery process:

[0053] In abnormal situations, such as when some messages are updated successfully, the system only updates the check_status field of the successful part and commits the offset of this part of the message. This makes the fault recovery process simpler and more efficient, and avoids unnecessary repeated processing.

[0054] 5) Enhanced fault tolerance:

[0055] Through transaction control and cache table mechanisms, the system can automatically suspend message consumption when a network outage or other failure occurs, and continue to process unfinished messages after recovery, greatly enhancing the system's fault tolerance.

[0056] 6) Enhanced scalability:

[0057] The solution is designed with good scalability and can adapt to the growth of business systems of different sizes and data volumes. By flexibly configuring RocketMQ clusters and consumer programs, the system can be dynamically adjusted according to actual needs to support future business expansion.

[0058] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A data synchronization method based on RocketMQ, characterized by: The following steps are involved: Step S1, monitor the status of the RocketMQ cluster, and record the monitored ground status information into the cluster status table; Step S2: After receiving the master data change message, the consumer program first writes the message to the cache table check_data_table, and adds a check_status field to each message to indicate whether the message has been successfully processed. The initial value is "unprocessed". Step S3, under transaction control, the consumer program updates the data in the business library; after the business library is successfully updated, the check_status field of the corresponding message in check_data_table is updated to "processed", and the offset of RocketMQ is submitted to confirm that the message has been processed.

2. The RocketMQ-based data synchronization method according to claim 1, characterized in that: In step S2, the consumer program pulls 100 messages in batches each time, and saves the messages in memory after writing them into the cache table check_data_table.

3. The RocketMQ-based data synchronization method according to claim 1 or 2, characterized in that: In the step S3, under abnormal circumstances, if part of the message is updated successfully, only the check_status field of the successful part is updated, and the offset of the corresponding part of the message is submitted.

4. The data synchronization method based on RocketMQ according to claim 1, characterized in that: If an exception occurs in any of the steps S1 to S3, the consumer program will stop consuming subsequent messages and restart processing unfinished messages after the problem is resolved.

5. A data synchronization system based on RocketMQ, characterized by: include: The monitoring module is used to monitor the status of the RocketMQ cluster and record the monitored ground status information into the cluster status table; The consumer module is used to receive master data change messages and write them to the cache table check_data_table. At the same time, a check_status field is added to each message to indicate whether the message has been successfully processed. The initial value is "unprocessed"; The consumer module updates the data in the business library under transaction control; after the business library is successfully updated, the check_status field of the corresponding message in the check_data_table is updated to "processed", and the offset of RocketMQ is submitted to confirm that the message has been processed.

6. The RocketMQ-based data synchronization system according to claim 5, characterized in that: The consumer module pulls 100 messages in batches each time and saves the messages in memory after writing them to the cache table check_data_table.

7. The RocketMQ-based data synchronization system according to claim 5 or 6, characterized in that: In an abnormal situation, if part of the message is updated successfully, the consumer module only updates the check_status field of the successful part and submits the offset of the corresponding part of the message.

8. The RocketMQ-based data synchronization system according to claim 5, characterized in that: If an exception occurs during message processing, the consumer module will stop subsequent message consumption and restart processing unfinished messages after the problem is resolved.

9. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions which, when executed by a computing device, cause the computing device to perform any one of the methods according to claims 1 to 4.

10. A computing device, characterized in that: include: One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods according to claims 1 to 4.

Images