Message synchronization method and apparatus, storage medium, and electronic device

Abstract

The application provides a message synchronization method and device, a storage medium and an electronic device, which are applied to a remote disaster recovery system. The disaster recovery system comprises a first MQ cluster, a processing component and a second MQ cluster. The method comprises the following steps: the processing component acquires a message synchronization task and at least one to-be-synchronized message corresponding to the message synchronization task in the first MQ cluster; the processing component determines routing information of a message topic corresponding to the task based on the message synchronization task; the routing information comprises a correspondence between the message topic and at least one message server and a correspondence between the message server and a message queue; and the processing component synchronizes each to-be-synchronized message to the second MQ cluster based on the routing information, thereby completing full-amount synchronization of the message and ensuring that the order of the synchronized message is consistent with that in the first MQ cluster. The two clusters do not affect each other, and when a user sends a message, the message only flows in the same region, thereby guaranteeing the performance of RocketMQ and the redundancy of the message.

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, and in particular to a message synchronization method, apparatus, storage medium, and electronic device. Background Technology

[0002] Off-site disaster recovery refers to the process of replicating the entire system or a portion of its data set to an off-site site to prevent data loss due to operational errors or system failures. Currently, the mainstream message queue (MQ) middleware RocketMQ is widely used in various internet system platforms, and how to perform off-site disaster recovery based on RocketMQ is an important issue that urgently needs to be addressed.

[0003] In existing technologies, most deployments are based on a master-slave architecture, but in reality, it is still the same MQ cluster with Broker message servers deployed in two different regions. Message synchronization is achieved by establishing a connection channel between the local MQ cluster's message server Broker and the remote Broker. Cross-network transmission between different regions introduces latency, which seriously affects the performance of RocketMQ and increases the time it takes for RocketMQ users to send messages. Summary of the Invention

[0004] This invention provides a message synchronization method to address the performance defects in existing technologies for remote disaster recovery, where message synchronization is achieved by establishing a connection channel between a local RocketMQ cluster broker and a remote broker. These issues arise from latency in cross-regional network transmission during message sending. The new method achieves full synchronization of message topics, ensuring that the synchronized message order matches the message order in the first RocketMQ cluster. Furthermore, the two RocketMQ clusters are completely isolated and do not interfere with each other. When a RocketMQ user sends a message, the message only circulates within the same region, minimizing network latency. This ensures both the overall performance of RocketMQ and message redundancy.

[0005] On one hand, this invention provides a message synchronization method applied to a remote disaster recovery system, the remote disaster recovery system including a first message queue (MQ) cluster, a processing component, and a second MQ cluster, the method including:

[0006] The processing component obtains the message synchronization task and retrieves at least one message to be synchronized corresponding to the message synchronization task in the first MQ cluster;

[0007] The processing component determines the routing information of the message topic corresponding to the message synchronization task based on the message synchronization task. The routing information includes: the correspondence between the message topic and at least one message server used to store the messages of the message topic, and the correspondence between the message topic and the message queue in the message server.

[0008] The processing component synchronizes each message to be synchronized to the second MQ cluster based on the routing information of the message topic.

[0009] Optionally, the message synchronization method provided by the present invention, wherein the messages to be synchronized are ordinary messages, synchronizes each message to be synchronized to a second MQ cluster based on the routing information of the message topic, including:

[0010] For any ordinary message, based on the correspondence between the ordinary message and the message server, and the correspondence between the ordinary message and the message queue in the message server, the ordinary message is synchronized to the message queue of the message server corresponding to the ordinary message.

[0011] Optionally, the message synchronization method provided by the present invention, where the synchronized message is a dead-letter message, further includes the following before synchronizing each message to be synchronized to the second MQ cluster based on the routing information of the message topic:

[0012] Obtain operation permissions for dead-letter messages in the second MQ cluster;

[0013] Change the operation permissions to read and write permissions.

[0014] Optionally, in the message synchronization method provided by the present invention, when the message to be synchronized is a retry message, the processing component determines the routing information of the message topic corresponding to the message synchronization task based on the message synchronization task, including:

[0015] The latency level of each retry message is obtained based on the number of retries for each retry message.

[0016] Based on the delay level of each retry message and the message synchronization task, determine the routing information;

[0017] The method also includes:

[0018] Send the delay level and routing information of each retry message to the second MQ cluster;

[0019] Based on the routing information of the message topic, each message to be synchronized is synchronized to the second MQ cluster, including:

[0020] Each retry message is synchronized to the second MQ cluster based on its latency level and routing information.

[0021] Optionally, the message synchronization method provided by the present invention, when the message to be synchronized is a retry message, synchronizes each retry message to the second MQ cluster based on the delay level and routing information of each retry message, including:

[0022] For any retry message, the retry message is placed into the delay queue of the message server corresponding to the retry message in the second MQ cluster according to the delay level and routing information of the retry message;

[0023] If the delay time corresponding to the delay queue is reached, a retry message will be sent to the retry queue of the message server corresponding to the retry message.

[0024] Optionally, the message synchronization method provided by the present invention, when the message to be synchronized is a delayed message, further includes the following before synchronizing each message to be synchronized to the second MQ cluster based on the routing information of the message topic:

[0025] Based on the storage timestamp of each delayed message and the delay level in the first MQ cluster, the remaining delivery time of each delayed message is determined.

[0026] Based on the remaining delivery time of each delayed message, determine the delay level of each delayed message in the second MQ cluster;

[0027] Based on the routing information of the message topic, each message to be synchronized is synchronized to the second MQ cluster, including:

[0028] For any delayed message, the message server where the delayed message is located is determined based on the correspondence between the delayed message and the message server in the second MQ cluster;

[0029] Based on the delay level of the delayed message and the message server where the delayed message is located, determine the message queue of the delayed message on the message server, and synchronize the delayed message to the message queue corresponding to the message server.

[0030] Optionally, the message synchronization method provided by the present invention further includes, when the message to be synchronized is a delayed message:

[0031] The delay duration of the delayed message is determined based on the delay level corresponding to the delayed message.

[0032] The original delivery timestamp is determined based on the delay duration and the storage timestamp of the delayed message.

[0033] Determine the remaining delivery time based on the original delivery timestamp and the current time.

[0034] Optionally, the message synchronization method provided by the present invention, when the message to be synchronized is a delayed message, further includes, before determining the delay level of each delayed message in the second MQ cluster based on the remaining delivery time of each delayed message:

[0035] Based on the remaining delivery time and a preset threshold, determine whether to synchronize delayed messages;

[0036] If so, then perform the step of determining the delay level of each delayed message in the second MQ cluster based on the remaining delivery time of each delayed message;

[0037] If not, then there is no need to synchronize delayed messages.

[0038] Optionally, the message synchronization method provided by the present invention further includes:

[0039] Get the consumption progress of at least one message from the message topic subscribed to by the subscription group in the message synchronization task;

[0040] Determine the location of each message in the message queue of the second MQ cluster;

[0041] The consumption progress of each message is updated based on its position in the message queue of the second MQ cluster.

[0042] Optionally, the message synchronization method provided by the present invention further includes:

[0043] Synchronize the routing information to the name server corresponding to the message topic in the second MQ cluster.

[0044] On the other hand, the present invention also provides a message synchronization device for use in a remote disaster recovery system. The remote disaster recovery system includes a first message queue (MQ) cluster, a processing component, and a second MQ cluster. The device includes:

[0045] The acquisition module is used to acquire message synchronization tasks through the processing component and to acquire at least one message to be synchronized corresponding to the message synchronization task in the first MQ cluster.

[0046] The processing module is used to determine the routing information of the message topic corresponding to the message synchronization task based on the message synchronization task through the processing component; the routing information includes: the correspondence between the message topic and at least one message server used to store the message topic, and the correspondence between the message topic and the message queue in the message server;

[0047] The processing module is also used to synchronize each message to be synchronized to the second MQ cluster by using the routing information of the message topic through the processing component.

[0048] On the other hand, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any message synchronization method when executing the program.

[0049] On the other hand, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any message synchronization method.

[0050] On the other hand, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements any message synchronization method.

[0051] This invention provides a message synchronization method applied to a remote disaster recovery system. The remote disaster recovery system includes a first message queue (MQ) cluster, a processing component, and a second MQ cluster. In this invention, the processing component determines all messages to be synchronized in a message synchronization task, and determines the routing information of the message topic for each message to be synchronized. Since the routing information of each message topic includes the correspondence between the message topic and the message server storing the message topic, as well as the correspondence between the message topic and the message queue in the proxy server storing the message topic, based on the above correspondence between the message topic, the message server, and the message queue in the message server, it is determined that each message to be synchronized needs to be synchronized to the remote second MQ cluster. The specific queue position and message synchronization flow are determined to complete real-time message synchronization and backup. Because the above message synchronization process uses a component independent of the first MQ cluster, the synchronization flow of the message to be synchronized is determined by combining the routing information of the message topic to perform full message synchronization, rather than just transmitting individual messages through the message server. This achieves full synchronization of the message topic, and the message order after synchronization is consistent with the message order in the first MQ cluster. Furthermore, the two RocketMQ clusters are completely isolated and do not affect each other. When RocketMQ users send messages, the messages only circulate within the same region, resulting in low network latency. This ensures both the overall performance of RocketMQ and message redundancy. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating the message synchronization method provided by the present invention;

[0054] Figure 2 This is a schematic diagram of the off-site disaster recovery system provided by the present invention;

[0055] Figure 3 This is one of the schematic diagrams illustrating the principle of the ordinary message synchronization method provided by the present invention;

[0056] Figure 4 This is the second schematic diagram of the principle of the ordinary message synchronization method provided by the present invention;

[0057] Figure 5 This is a schematic diagram illustrating the principle of the delayed message synchronization method provided by the present invention;

[0058] Figure 6 This is a schematic diagram of the message synchronization device provided by the present invention;

[0059] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0061] First, some of the terms and application scenarios involved in the embodiments of the present invention will be introduced.

[0062] In this invention embodiment, the so-called message queue (MQ) middleware refers to a container or service that stores messages during message transmission. It is an important component for distributed systems to achieve advanced effects such as high performance, high availability, and high scalability, and is suitable for various scenarios, such as message communication, asynchronous processing, application decoupling, and traffic shaping. Currently, the mainstream message queue, RocketMQ (Rocket Message Queue), is an open-source message queue from Alibaba, developed in Java. It features high throughput (100,000-level), very high availability, and suitability for large-scale distributed system applications, and has been widely used in various Internet system platforms. The following explains the relevant terms in RocketMQ used in this invention:

[0063] Message Topic: A first-level message type to which the message publisher sends messages.

[0064] Message: A data message that is sent by a message publisher based on a message topic and ultimately delivered to a consumer.

[0065] Queues: Each message topic will have one or more queues to store messages.

[0066] Message Producer: Responsible for producing and sending messages to the message topic.

[0067] A message consumer is responsible for receiving and consuming messages from a message topic.

[0068] SubscriptionGroup is a logical group that records the consumption progress of subscribed topics.

[0069] Delayed message: A message publisher sends a message to a message queue server, but does not expect the message to be delivered immediately. Instead, it is delivered to the message subscriber for consumption after a certain delay. This message is called a delayed message.

[0070] Retry messages: When a consumer fails to consume a message, the message queue will automatically and continuously retry the message. By default, the message queue RocketMQ allows a maximum of 16 retries per message.

[0071] Dead-letter messages: Messages that cannot be consumed under normal circumstances are called dead-letter messages, and the special queues that store dead-letter messages are called dead-letter queues.

[0072] Consumption progress QueueOffset: Used to record the consumption progress of each subscribed topic in each message queue, that is, the consumption point of the subscribed topic message in each queue.

[0073] The following is combined Figures 1-7 The technical solution of the present invention will be described in detail with reference to specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0074] This invention provides a message synchronization method applied to a remote disaster recovery system. The remote disaster recovery system includes a first message queue (MQ) cluster, a processing component, and a second MQ cluster, such as... Figure 1 As shown, the method includes:

[0075] Step 101: The processing component obtains the message synchronization task and obtains at least one message to be synchronized corresponding to the message synchronization task in the first MQ cluster;

[0076] Specifically, the off-site disaster recovery system of this invention includes a first MQ cluster, a processing component, and a second MQ cluster, with the processing component operating independently of the first MQ cluster. For example, when the processing component starts, it acquires a message synchronization task, such as synchronizing a message from the first MQ cluster to the second MQ cluster; or, the processing component may trigger the acquisition of the message synchronization task under other conditions, such as receiving a user instruction message.

[0077] It is understandable that the message synchronization task in this invention generally needs to be synchronized in different locations. For example, the message synchronization task is to synchronize the topic message from Shanghai to Wuhan. Therefore, the second MQ cluster and the first MQ cluster usually need to be deployed in different regions, and the processing components are independent of the first MQ cluster. Therefore, the acquisition operation of the message synchronization task will not affect the message transmission efficiency of the first MQ cluster.

[0078] Step 102: The processing component determines the routing information of the message topic corresponding to the message synchronization task based on the message synchronization task. The routing information includes: the correspondence between the message topic and at least one message server used to store the messages of the message topic, and the correspondence between the message topic and the message queue in the message server.

[0079] Specifically, the processing component parses the message synchronization task to obtain the message topic corresponding to each message in the first MQ cluster. Further, based on the message topic, it obtains the routing information for each message topic. The routing information includes the correspondence between the message topic and the message server (Broker) in the first MQ cluster, and the correspondence between the message topic and the message queue. That is, based on a given message topic, the processing component needs to determine which Brokers the message topic in the first MQ cluster is stored on, and also know which queues within those Brokers the message topic is specifically stored in, thereby making full use of the relevant topic routing information in the message synchronization task.

[0080] Step 103: The processing component synchronizes each message to be synchronized to the second MQ cluster based on the routing information of the message topic.

[0081] Specifically, after determining the routing information of the message topic corresponding to each message to be synchronized, the specific flow direction of each message to be synchronized is determined, and the message synchronization process is completed based on the determined message flow direction.

[0082] This invention provides a message synchronization method applied to a remote disaster recovery system. The remote disaster recovery system includes a first message queue (MQ) cluster, a processing component, and a second MQ cluster. In this invention, the processing component determines all messages to be synchronized in a message synchronization task. For each message to be synchronized, it determines the routing information of the message topic. Since the routing information of each message topic includes the correspondence between the message topic and the message server storing the message topic, as well as the correspondence between the message topic and the message queue in the proxy server storing the message topic, the specific queue position and specific message synchronization flow of each message to be synchronized to the remote second MQ cluster are determined based on the above correspondence between the message topic, the message server, and the message queue in the message server. This completes the real-time synchronization and backup of messages. Because the above message synchronization process uses a component independent of the first MQ cluster, it combines the routing information of the message topic to determine the synchronization flow of the message to be synchronized for full message synchronization, rather than just transmitting individual messages through the message server, it achieves full synchronization of message topics and ensures that the message order after synchronization is consistent with the message order in the first MQ cluster, ensuring that message synchronization will not fail due to the failure of the first MQ cluster node.

[0083] like Figure 2As shown, the above-mentioned off-site disaster recovery system includes a first message queue (MQ) cluster, a processing component, and a second MQ cluster.

[0084] Specifically, the first MQ cluster contains a Nameserver and multiple message brokers. The processing component connects to the first MQ cluster for message synchronization. The second MQ cluster contains a Nameserver and multiple brokers, and the processing component also connects to the second MQ cluster. It can be understood that the processing component is the main execution entity in the message synchronization task. By obtaining the message synchronization task, it synchronizes the topic messages and key data, such as subscription groups and subscription group consumption progress, from the first MQ cluster to the second MQ cluster in real time.

[0085] Furthermore, based on actual deployment needs, there can be multiple processing components in this off-site disaster recovery system. For example, the first MQ cluster is in Shanghai, and processing components 1, 2, 3 and the second MQ cluster are deployed in Wuhan. When a processing component starts up, for example, it obtains a message topic synchronization task. The off-site disaster recovery system can use a load balancing algorithm to allocate the topics in the message synchronization task according to the number of queues based on the number of processing components in the system. Each processing component performs message synchronization according to the sub-tasks allocated to it by the load balancing algorithm.

[0086] Because the processing component is independent of the first MQ cluster, stateless and highly scalable, it will not affect the message transmission efficiency of the first MQ cluster itself, thus achieving non-intrusive synchronous backup of an independent cluster.

[0087] Optionally, the message synchronization method provided in this embodiment of the invention, when the message to be synchronized is a normal message, is illustrated in the following schematic diagram. Figure 3 , Figure 4 As shown in the diagram, "qId" refers to the message queue sequence number (QueueId, qId). The method includes:

[0088] For any ordinary message, based on the correspondence between the ordinary message and the message server, and the correspondence between the ordinary message and the message queue in the message server, the ordinary message is synchronized to the message queue of the message server corresponding to the ordinary message.

[0089] Specifically, for the synchronization of ordinary messages, after determining the Broker corresponding to the topic of each message to be synchronized and the queue of the message to be synchronized in the corresponding Broker, the specific flow of each message to be synchronized in the second MQ cluster can be determined. Based on the determined flow, the messages are stored in the designated locations, thus completing the synchronization of ordinary messages. The synchronization of ordinary messages follows a simple "consume-send" pattern. After determining the synchronization flow of each message, the message publisher in the first MQ cluster sends each message in the topic in sequence, and the consumers in the second MQ cluster consume the topic messages in sequence. The message order sent to the second MQ cluster is strictly consistent with that of the first MQ cluster, and the message order of each queue on each Broker is consistent before and after synchronization.

[0090] It is understandable that processing components may experience failures, restarts, or failover. Based on the message synchronization method described above, this will only lead to inaccurate message synchronization progress and a small amount of duplicate information. However, no information will be lost in this case, which conforms to RocketMQ's "at least once" strategy. A schematic diagram illustrating the principle behind duplicate information during message synchronization is shown below. Figure 4 As shown.

[0091] The method provided in this embodiment of the invention can ensure that the message order synchronized to the second MQ cluster is strictly consistent with that in the first MQ cluster, without causing message order disorder.

[0092] Optionally, the message synchronization method provided in this embodiment of the invention, when the message to be synchronized is a dead letter message, before synchronizing each message to be synchronized to the second MQ cluster based on the routing information of the message topic, further includes:

[0093] Obtain operation permissions for dead-letter messages in the second MQ cluster;

[0094] Change the operation permissions to read and write permissions.

[0095] Specifically, dead-letter messages refer to messages that cannot be consumed under normal circumstances. The special queue for storing dead-letter messages is called a dead-letter queue (DLQ). Dead-letter messages differ from synchronous messages. The perm permission of a dead-letter queue is set to "2: read-only" (there are three perm permissions: 2: read-only, 4: write-only, 6: read-write). Under normal circumstances, they cannot be consumed. Therefore, it is necessary to detect and modify the dead-letter queue permissions through a component before synchronizing the messages in the same way as normal messages.

[0096] Specifically, when the message to be synchronized is a dead letter message, before synchronizing the message, it is necessary to first obtain the operation permission for the dead letter queue in the first MQ cluster and modify the operation permission of the dead letter queue in the first MQ cluster to "6: Read and Write". Then, synchronization is performed in the same way as normal message synchronization, which will not be elaborated here.

[0097] The method provided in this embodiment of the invention can fully synchronize dead-letter messages in the first MQ cluster to the second MQ cluster by modifying the operation permissions of the dead-letter queue in the first MQ cluster, thus avoiding message synchronization failure during the synchronization process due to the dead-letter messages not being consumed normally.

[0098] Optionally, in the message synchronization method provided in this embodiment of the invention, when the message to be synchronized is a retry message, step 102 specifically includes:

[0099] The latency level of each retry message is obtained based on the number of retries for each retry message.

[0100] Based on the delay level of each retry message and the message synchronization task, determine the routing information;

[0101] The method also includes:

[0102] Send the delay level and routing information of each retry message to the second MQ cluster;

[0103] Each retry message is synchronized to the second MQ cluster based on its latency level and routing information.

[0104] Specifically, message retry refers to the automatic and continuous message retry mechanism in RocketMQ message queue when a consumer fails to consume a message. For example, the message queue allows each message to be retried up to 16 times by default. If a message has not been consumed normally after reaching the maximum number of retries, it will enter the dead letter queue.

[0105] Furthermore, if the message to be synchronized is a retry message, the processing component needs to first parse the original topic name of the retry topic based on each retry message, that is, the topic name in the first MQ cluster, obtain the number of retries of the retry message in the first MQ cluster based on the topic name, and obtain the latency level of the retry message.

[0106] Furthermore, the message latency level during message synchronization is calculated based on the latency level of the retry message. The calculation method is as follows:

[0107] Message latency level during synchronization = original message latency level - 1

[0108] Furthermore, after obtaining the message delay level for the retry message synchronization, the delay level and topic routing information are set for the current message to be synchronized, and the processing component sends the delay level and topic routing information of the retry message to the second MQ cluster.

[0109] Furthermore, RocketMQ's internal native logic will eventually convert this message into a retry message.

[0110] The method provided in this embodiment of the invention decrements the latency level of all messages by one during retry message synchronization to obtain the latency level of the delayed message. Further, the latency level and topic routing information of the retry message are sent to a second MQ cluster, thereby completing the retry message synchronization. This fully synchronizes the current retry message and its topic routing information settings to the second MQ cluster, preventing message synchronization failure due to the retry count reaching the maximum retry limit.

[0111] Optionally, during message synchronization retry, step 102 further includes:

[0112] For any retry message, the retry message is placed into the delay queue of the message server corresponding to the retry message in the second MQ cluster according to the delay level and routing information of the retry message;

[0113] If the delay time corresponding to the delay queue is reached, a retry message will be sent to the retry queue of the message server corresponding to the retry message.

[0114] Specifically, because the message has a set delay level, RocketMQ's internal mechanism will first put it into the delay queue according to the message's delay level and topic routing information. After the delay time corresponding to the delay queue is reached, the message will be automatically sent to the retry queue, thereby completing the synchronization of retry messages.

[0115] Understandably, since the delay levels of delayed messages synchronized to the second MQ cluster have been reduced by one, direct consumption would result in a large number of duplicate messages. The message synchronization method provided by this invention can update the consumption progress in real time, thus avoiding the problem of a large number of duplicate messages.

[0116] The method provided in this embodiment of the invention first places the retry message into a delay queue based on the delay level and topic routing information set during the message synchronization. After the delay duration corresponding to the delay queue is reached, the internal mechanism of RocketMQ automatically sends the message to the retry queue, thereby completing the synchronization of the retry message and avoiding message loss during the synchronization process due to the characteristics of retry messages.

[0117] Optionally, in the message synchronization method provided by the embodiments of the present invention, when the message to be synchronized is a delayed message, the following embodiments are combined with Figure 5 The content is explained below. Before synchronizing each message to be synchronized to the second MQ cluster based on the routing information of the message topic, the method also includes:

[0118] (1) Determine the remaining delivery time of each delayed message based on the storage timestamp of each delayed message and the delay level in the first MQ cluster;

[0119] (2) Determine the delay level of each delayed message in the second MQ cluster based on the remaining delivery time of each delayed message;

[0120] The method of synchronizing each message to be synchronized to the second MQ cluster based on the routing information of the message topic includes:

[0121] (3) For any delayed message, determine the message server where the delayed message is located based on the correspondence between the delayed message and the message server in the second MQ cluster;

[0122] (4) Based on the delay level of the delayed message and the message server where the delayed message is located, determine the message queue of the delayed message on the message server and synchronize the delayed message to the message queue corresponding to the message server.

[0123] Specifically, delayed messages differ from regular messages because they have only one topic, "SCHEDULE_TOPIC_XXXX", which carries all delayed messages in the system. In the message synchronization method provided by this invention, when the message to be synchronized is a delayed message, before synchronizing the message based on the routing information of the message topic, the method first parses each delayed message and calculates the remaining delivery time of the delayed message based on the storage timestamp of the delayed message and the delay level of the delayed message in the first MQ cluster.

[0124] Furthermore, based on the calculated remaining delivery time of the delayed message, a mapping calculation is performed between the delay level configuration of the MQ cluster and the delay level of the second MQ cluster to determine and set the delay level for this delayed information. The delay level configuration of the MQ cluster is shown in Table 1 below:

[0125] Table 1

[0126]

[0127]

[0128] For example, if the remaining delivery time of a delayed message is 5 minutes, and these 5 minutes are mapped to the "RMQ default delay time" row in Table 1, the delay level of the delayed message is determined to be 9.

[0129] Furthermore, based on the delay level of the delayed message and referring to Table 1: Delay Level Configuration of the MQ Cluster, the Broker in the second MQ cluster where the delayed message resides and the corresponding message queue within that Broker are determined. For example, if the delay level of the delayed message is known to be 9, matching it in Table 1 reveals that the queue number storing the delayed message is 8. Combining the queue where the delayed message resides and the topic routing information of the delayed message, the specific synchronization flow of the delayed message is further determined. Then, message synchronization is performed in the same manner as normal message synchronization, updating the delayed message to the corresponding message queue.

[0130] Understandably, the processing components within the off-site disaster recovery system can start consuming messages from all queues of the delayed message topic in the first MQ cluster based on the consumption progress of that topic.

[0131] For example, such as Figure 5 As shown, the processing component can determine the consumption progress of delayed messages in the first MQ cluster, which is the specific position of the currently consumed information in the message queue of the first MQ cluster. See details... Figure 5 The consumption progress of each queue in the delayed message topic of the first MQ cluster is "the consumption progress of qId=0, qId=1, qId=2, and qId=3 is all 1". During synchronization, the processing component will synchronize this consumption progress and consume the delayed messages based on this consumption progress. After consumption, some duplicate messages in the delayed message queue have been consumed and will not be synchronized to the messages in the second MQ cluster. The messages synchronized to the second MQ cluster will not contain consumed messages. In this way, a large number of duplicate message deliveries can be reduced during message synchronization.

[0132] The method provided in this invention calculates the remaining delivery time of each delayed message, determines the delay level of each delayed message in the second MQ cluster, and further synchronizes the delayed messages based on the delay level and the routing information of the corresponding topic in the second MQ cluster. This method fully considers the characteristics of delayed messages and combines the calculation of delay level for message synchronization, resulting in good message synchronization effect and reducing a large number of duplicate message deliveries.

[0133] Optionally, in the message synchronization method provided in this embodiment of the invention, when the message to be synchronized is a delayed message, the method further determines the remaining delivery time of each delayed message, the method including:

[0134] The delay duration of the delayed message is determined based on the delay level corresponding to the delayed message.

[0135] The original delivery timestamp is determined based on the delay duration and the storage timestamp of the delayed message.

[0136] Determine the remaining delivery time based on the original delivery timestamp and the current time.

[0137] Specifically, when the message to be synchronized is a delayed message, based on the delay level configuration information of the first MQ cluster, the corresponding delay level, i.e., the message storage queue QueueId, is first obtained from the delayed message to determine the delay duration. Further, based on the delay duration and the storage timestamp of the delayed message, the original delivery timestamp is determined, using the following formula:

[0138] Original delivery timestamp = Delayed message storage timestamp + Delay duration;

[0139] Furthermore, based on the original delivery timestamp and the current time, the remaining delivery time of the delayed message is determined using the following formula:

[0140] Remaining delivery time = Original delivery timestamp - Current time.

[0141] The method provided in this embodiment of the invention determines the remaining delivery time of each delayed message, which facilitates subsequent judgment on whether the delayed message needs to be synchronized, thereby reducing the duplication of some messages.

[0142] Optionally, the message synchronization method provided in this embodiment of the invention, when the message to be synchronized is a delayed message, determines the delay level of each delayed message in the second MQ cluster based on the remaining delivery time of each delayed message. The method further includes:

[0143] Based on the remaining delivery time and a preset threshold, determine whether to synchronize delayed messages;

[0144] If so, then perform the step of determining the delay level of each delayed message in the second MQ cluster based on the remaining delivery time of each delayed message;

[0145] If not, then there is no need to synchronize delayed messages.

[0146] Specifically, after calculating the remaining delivery time (timeRemaining) of the delayed message, the remaining delivery time is compared with a preset threshold (timeDiff) to determine whether message synchronization is needed, as follows:

[0147] a. If timeRemaining<(timeDiff×-1),

[0148] This means that the delayed message has already been delivered in the first MQ cluster and does not need to be synchronized.

[0149] b. If timeRemaining<0&&timeRemaining>=(timeDiff×-1),

[0150] If the remaining delivery time is within the time difference allowable range, message synchronization will be performed, and the delay level of the delayed message will be set to the minimum level 1.

[0151] c. If timeRemaining>0&&timeRemaining>=(timeDiff×-1),

[0152] Then, based on the timeRemaining configuration, the corresponding latency level is mapped to the latency level in the second MQ cluster.

[0153] For example, the delayed message synchronization method refers to the following process:

[0154] Assuming the current time is 11:00:00 on February 1, 2023, the storage timestamp of the delayed message to be synchronized is 10:00:00 on February 1, 2023, and the queue number of the delayed message in the first MQ cluster is 15, calculate the remaining delivery time of this delayed message as follows:

[0155] The delayed message has a queue number of 15 in the first MQ cluster, which corresponds to Table 1, indicating a delay duration of 30 minutes. Therefore:

[0156] Original delivery timestamp: The storage timestamp of the delayed message plus the delay duration, i.e., February 1, 2023, 10:00:00, plus 30 minutes, gives the original delivery timestamp as February 1, 2023, 10:30:00.

[0157] Remaining delivery time: Subtract the current time from the original delivery timestamp, i.e., 11:00:00 on February 1, 2023 minus 10:30:00 on February 1, 2023, to get the remaining delivery time of 30 minutes.

[0158] Furthermore, the remaining delivery time is compared with a preset time threshold, timeDiff. For example, if the preset timeDiff is 1 minute, the remaining delivery time of the delayed message (30 minutes) is compared with "1 minute multiplied by -1". If the following judgment condition is met:

[0159] timeRemaining>0&&timeRemaining>=(timeDiff×-1)

[0160] Therefore, based on the timeRemaining configuration in the second MQ cluster, the corresponding delay level is mapped to, and the delayed message is synchronized. The specific process is as follows:

[0161] The remaining delivery time of 30 minutes is mapped to the latency level configuration of the MQ cluster in Table 1, resulting in a latency level of 16 and a queue number of 15.

[0162] Furthermore, by combining the topic routing information of the delayed message in the second MQ cluster with the specific queue number in the Broker where the message is stored, the delayed message is synchronized in the same way as a normal message, thus completing the delayed message synchronization.

[0163] The method provided in this invention determines whether to synchronize delayed messages based on the remaining delivery time of the delayed message and a preset threshold, which is more suitable for the characteristics of delayed messages and avoids the delivery of a large amount of duplicate information.

[0164] Optionally, the message synchronization method provided in this embodiment of the invention further includes:

[0165] Get the consumption progress of at least one message from the message topic subscribed to by the subscription group in the message synchronization task;

[0166] Determine the location of each message in the message queue of the second MQ cluster;

[0167] The consumption progress of each message is updated based on its position in the message queue of the second MQ cluster.

[0168] Specifically, based on the consumption progress of the subscription group in the first MQ cluster, that is, the consumption progress of each message topic in its respective queue, the message at that point is obtained according to this consumption progress. Then, the message is searched under the same topic route in the second MQ cluster to obtain the message's position in the message queue in the second MQ cluster. The consumption progress of each message is updated, that is, the consumption progress QueueOffset of the queue is updated.

[0169] Understandably, there is a crucial prerequisite for directly updating the consumption progress of each message here: the synchronization process of all message types, except for delayed messages, must be strictly ordered. Otherwise, directly updating the consumption progress here may result in message loss.

[0170] Specifically, a Queue is an infinitely long array. When a message arrives, its index increases by 1. The index is called QueueOffset, which indicates the position of the message within a Queue. The value of QueueOffset can be used to locate the message or instruct the consumer to start processing from this message.

[0171] For example, a subscription group subscribes to message topic B, and one of the brokers corresponding to topic B is Broker1, which belongs to queue 1.

[0172] The message being consumed by the subscription group in message queue 1 of Broker1 in the first MQ cluster is the 5th message, that is, the consumption progress QueueOffset of this queue is 5. The processing component obtains the specific message stored at this consumption progress, parses it based on the message, obtains the Broker corresponding to the message in the second MQ cluster and the queue information corresponding to the Broker, and updates the consumption progress QueueOffset of the queue.

[0173] Furthermore, based on the example method described above, the queue position of each message in the message topic subscribed to by the subscription group in the second MQ cluster can be updated.

[0174] The method provided in this embodiment of the invention can correct the phenomenon of duplicate message synchronization if the processing component restarts for other reasons or fails between clusters during the message synchronization process, resulting in duplicate message synchronization. This invention will retrieve the earlier message when searching for messages in the remote MQ cluster according to the message attributes. Therefore, the above operation will correct the phenomenon of duplicate message synchronization, minimize the existence of duplicate messages, and reduce the impact on the business system.

[0175] Optionally, the message synchronization method provided in this embodiment of the invention further includes:

[0176] Synchronize the routing information to the name server corresponding to the message topic in the second MQ cluster.

[0177] Specifically, in the message synchronization method provided by the present invention, the processing component also synchronizes the routing information of the message topic in the message synchronization task to the second MQ cluster. That is, the message synchronization task is parsed to obtain the correspondence between the message topic and the Broker used to store the message of the message topic, as well as the correspondence between the message topic and the message queue in the Broker. At the same time as message synchronization, the above correspondence is also synchronized to the Nameserver in the second MQ cluster.

[0178] The message synchronization method provided in this invention provides real-time backup of key data such as topic messages, subscription groups, subscription group consumption progress, and routing information. It performs real-time backup of the entire cluster without affecting the first MQ cluster, resulting in strong system scalability and high message transmission efficiency.

[0179] Figure 6 This invention provides a message synchronization device applied to a remote disaster recovery system. The remote disaster recovery system includes a first message queue (MQ) cluster, a processing component, and a second MQ cluster, such as... Figure 6 As shown, the device includes:

[0180] The acquisition module is used to acquire a message synchronization task through the processing component and acquire at least one message to be synchronized corresponding to the message synchronization task in the first MQ cluster.

[0181] The processing module is used to determine the routing information of the message topic corresponding to the message synchronization task based on the message synchronization task through the processing component; the routing information includes: the correspondence between the message topic and at least one message server used to store messages of the message topic, and the correspondence between the message topic and the message queue in the message server;

[0182] The processing module is also used to synchronize each of the messages to be synchronized to the second MQ cluster based on the routing information of the message topic through the processing component.

[0183] It is understood that the device may be located in the processing component, or in the system, and may communicate with the processing component to further implement the above-mentioned message synchronization method.

[0184] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a message synchronization method, which includes:

[0185] The processing component obtains the message synchronization task and retrieves at least one message to be synchronized corresponding to the message synchronization task in the first MQ cluster;

[0186] The processing component determines the routing information of the message topic corresponding to the message synchronization task based on the message synchronization task. The routing information includes: the correspondence between the message topic and at least one message server used to store the messages of the message topic, and the correspondence between the message topic and the message queue in the message server.

[0187] The processing component synchronizes each message to be synchronized to the second MQ cluster based on the routing information of the message topic.

[0188] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the message synchronization method provided by the above methods, the method comprising:

[0190] The processing component obtains the message synchronization task and retrieves at least one message to be synchronized corresponding to the message synchronization task in the first MQ cluster;

[0191] The processing component determines the routing information of the message topic corresponding to the message synchronization task based on the message synchronization task. The routing information includes: the correspondence between the message topic and at least one message server used to store the messages of the message topic, and the correspondence between the message topic and the message queue in the message server.

[0192] The processing component synchronizes each message to be synchronized to the second MQ cluster based on the routing information of the message topic.

[0193] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the message synchronization methods provided by the methods described above, the method comprising:

[0194] The processing component obtains the message synchronization task and retrieves at least one message to be synchronized corresponding to the message synchronization task in the first MQ cluster;

[0195] The processing component determines the routing information of the message topic corresponding to the message synchronization task based on the message synchronization task. The routing information includes: the correspondence between the message topic and at least one message server used to store the messages of the message topic, and the correspondence between the message topic and the message queue in the message server.

[0196] The processing component synchronizes each message to be synchronized to the second MQ cluster based on the routing information of the message topic.

[0197] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0198] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A message synchronization method, characterized by, Applied to a remote disaster recovery system, the remote disaster recovery system includes a first message queue (MQ) cluster, a processing component, and a second MQ cluster, the method includes: The processing component obtains a message synchronization task and obtains at least one message to be synchronized corresponding to the message synchronization task in the first MQ cluster. The processing component determines the routing information of the message topic corresponding to the message synchronization task based on the message synchronization task; the routing information includes: the correspondence between the message topic and at least one message server used to store messages of the message topic, and the correspondence between the message topic and the message queue in the message server; The processing component synchronizes each of the messages to be synchronized to the second MQ cluster based on the routing information of the message topic; In the case that the message to be synchronized is a delayed message, before synchronizing each of the messages to be synchronized to the second MQ cluster based on the routing information of the message topic, the method further includes: Based on the storage timestamp of each delayed message and the delay level in the first MQ cluster, the remaining delivery time of each delayed message is determined. Based on the remaining delivery time of each delayed message, the delay level of each delayed message in the second MQ cluster is determined; The process of synchronizing each message to be synchronized to the second MQ cluster based on the routing information of the message topic includes: For any of the delayed messages, the message server where the delayed message is located is determined based on the correspondence between the delayed message and the message server of the delayed message in the second MQ cluster; Based on the delay level of the delayed message and the message server where the delayed message is located, the message queue of the delayed message on the message server is determined, and the delayed message is synchronized to the message queue corresponding to the message server.

2. The message synchronization method of claim 1, wherein, The messages to be synchronized are ordinary messages. The process of synchronizing each message to the second MQ cluster based on the routing information of the message topic includes: For any of the aforementioned ordinary messages, based on the correspondence between the ordinary message and the message server, and the correspondence between the ordinary message and the message queue in the message server, the ordinary message is synchronized to the message queue of the message server corresponding to the ordinary message.

3. The message synchronization method of claim 2, wherein, The synchronization message is a dead-letter message. Before synchronizing each of the messages to be synchronized to the second MQ cluster based on the routing information of the message topic, the process further includes: Obtain operation permissions for the dead letter message in the second MQ cluster; Change the operation permission to read and write permission.

4. The message synchronization method of claim 1, wherein, When the message to be synchronized is a retry message, the processing component determines the routing information of the message topic corresponding to the message synchronization task based on the message synchronization task, including: Based on the number of retries for each retry message, the delay level of each retry message is obtained; The routing information is determined based on the delay level of each retry message and the message synchronization task; The method further includes: The delay level and routing information of each of the retry messages are sent to the second MQ cluster; synchronizing each of the to-be-synchronized messages to the second MQ cluster based on the routing information of the message topic, comprising: synchronizing each of the retry messages to the second MQ cluster based on the delay level of each of the retry messages and the routing information.

5. The message synchronization method of claim 4, wherein, synchronizing each of the retry messages to the second MQ cluster based on the delay level of each of the retry messages and the routing information, comprising: for any of the retry messages, putting the retry message into a delay queue of a message server corresponding to the retry message in the second MQ cluster according to the delay level of the retry message and the routing information; when a delay duration corresponding to the delay queue is reached, sending the retry message to a retry queue of the message server corresponding to the retry message.

6. The message synchronization method of claim 1, wherein, the determining of the remaining delivery duration of each of the delay messages, comprising: determining a delay duration of the delay message based on the delay level corresponding to the delay message; determining an original delivery timestamp based on the delay duration and a storage timestamp of the delay message; determining the remaining delivery duration based on the original delivery timestamp and a current time.

7. The message synchronization method of claim 1, wherein, the determining of each of the delay messages in the delay level of the second MQ cluster based on the remaining delivery duration of each of the delay messages, further comprising: determining whether to synchronize the delay message based on the remaining delivery duration and a preset threshold; if yes, performing the step of determining each of the delay messages in the delay level of the second MQ cluster based on the remaining delivery duration of each of the delay messages; if no, the delay message does not need to be synchronized.

8. The message synchronization method according to any one of claims 1-7, characterized by, the method further comprising: obtaining a consumption progress of at least one message of the message topic subscribed by a subscription group in the message synchronization task; determining a position of each of the messages in a message queue in the second MQ cluster; updating the consumption progress of each of the messages based on the position of each of the messages in the message queue in the second MQ cluster.

9. The message synchronization method according to any one of claims 1-7, characterized by, the method further comprising: synchronizing the routing information to a name server corresponding to the message topic in the second MQ cluster.

10. A message synchronization apparatus, characterized by comprising: application to a disaster recovery system in different places, the disaster recovery system in different places comprising a first message queue (MQ) cluster, a processing component, and a second MQ cluster, the apparatus comprising: an obtaining module configured to obtain a message synchronization task through the processing component and obtain at least one to-be-synchronized message corresponding to the message synchronization task in the first MQ cluster; a processing module configured to determine, through the processing component and based on the message synchronization task, routing information of a message topic corresponding to the message synchronization task; the routing information comprising a correspondence between the message topic and at least one message server storing messages of the message topic, and a correspondence between the message server and a message queue; the processing module is further configured to synchronize, through the processing component and based on the routing information of the message topic, each of the to-be-synchronized messages to the second MQ cluster; in a case where the to-be-synchronized message is a delay message, the processing module is specifically configured to: determine a remaining delivery time length of each of the delay messages based on a storage time stamp of each of the delay messages and a delay level of the first MQ cluster; determine a delay level of each of the delay messages in the second MQ cluster based on the remaining delivery time length of each of the delay messages; for any of the delay messages, determine a message server where the delay message is located based on a correspondence between the delay message and the message server in the second MQ cluster; determine a message queue of the delay message in the message server based on the delay level of the delay message and the message server where the delay message is located, and synchronize the delay message into the message queue corresponding to the message server.

11. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the message synchronization method according to any one of claims 1 to 9 when executing the program.

12. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the message synchronization method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the message synchronization method according to any one of claims 1 to 9.

Images