Message processing method based on message queue system and related equipment thereof

Abstract

The invention discloses a message processing method based on a message queue system and related equipment thereof, and the method comprises the steps: firstly receiving a first message generated in business service, and writing a first storage offset (such as offset) configured for the first message in the first message; the first message carrying the first storage offset is stored in a first message queue cluster, and the first message in the first message queue cluster is synchronously stored in a second message queue cluster through a special data synchronization service, so that the second message queue cluster is used for storing the first message according to the first storage offset; and enabling the second message queue cluster to support message backup of the first message queue cluster, so that backup can be realized through a specified Offset write-in mode to ensure that the storage positions of the message in different clusters are kept consistent. Therefore, the defect caused by inconsistency of offsets corresponding to the same message in different clusters can be effectively overcome, and the remote disaster recovery effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a message processing method based on a message queue system and related equipment. Background Art

[0002] The message queue system (such as Kafka or RocketMQ) is used to perform message-related processing, such as storing messages, forwarding messages, etc., and after the system has been configured with a topic, the system works as follows: after the Topic receives the message provided by the producer, the Topic saves the message and delivers the message to the consumer who has subscribed to the Topic.

[0003] In addition, in order to better improve service effects, the message queue system has a need for off-site disaster recovery. Off-site disaster recovery refers to building one or more sets of the same applications or services (such as message queue systems) in computer rooms deployed in different regions to take over immediately after a disaster, thereby reducing the impact of a single computer room failure (such as network interruption, computer room fire, computer room power outage, etc.) on the overall service. Summary of the invention

[0004] In order to meet the above requirements, the present application provides a message processing method based on a message queue system and related devices thereof.

[0005] In order to achieve the above objectives, the technical solutions provided by this application are as follows:

[0006] The present application provides a message processing method based on a message queue system, the method comprising: receiving a first message generated in a business service, writing a first storage offset configured for the first message in the first message, wherein the storage offset in the first message queue increases with message storage; storing the first message carrying the first storage offset in the first message queue cluster and synchronizing and storing the first message in the first message queue cluster to the second message queue cluster through a dedicated data synchronization service, the second message queue cluster being used to store the first message according to the first storage offset, and the second message queue cluster supporting message backup of the first message queue cluster.

[0007] In one possible implementation, the first message queue cluster and the second message queue cluster are deployed across regions, and the first message queue cluster and the second message queue cluster are pre-configured with the dedicated data synchronization service, and the dedicated data synchronization service is used to send the first message in the first message queue cluster unidirectionally to the second message queue cluster.

[0008] In one possible implementation, the first message queue cluster and the second message queue cluster both include multiple containers created by configuring the same topic, the multiple containers include a first container and a second container, and the topic is used to indicate the message categories stored in the multiple containers; the dedicated data synchronization service configured between the first container in the first message queue cluster and the first container in the second message queue cluster is used to unidirectionally synchronize messages received and stored by the first message queue from the business service to the second message queue cluster; the dedicated data synchronization service configured between the second container in the first message queue cluster and the second container in the second message queue cluster is used to unidirectionally synchronize messages received and stored by the second message queue from the business service to the first message queue cluster.

[0009] In one possible implementation, writing the first storage offset configured for the first message in the first message includes: writing the first storage offset configured for the first message in a message header of the first message; or writing the first message offset configured for the first message in an attribute feature of the first message.

[0010] In one possible implementation, the method also includes: after obtaining the first storage offset of the first message through the second message queue cluster, determining whether the first storage offset is greater than the maximum storage offset in the second message queue cluster; if so, storing the first message in the second message queue cluster according to the first storage offset; if not, discarding the first message.

[0011] In one possible implementation, the receiving of a first message generated in a business service includes: receiving a first message generated in a business service corresponding to the first message queue cluster; the method also includes: receiving a second message generated in a business service corresponding to the second message queue cluster, and writing a second storage offset configured for the second message in the second message, wherein the storage offset in the second message queue increases with message storage; storing the second message carrying the second storage offset in the second message queue cluster and updating the second storage offset of the second message to metadata, wherein the metadata is configured to support the first message queue cluster to read the second storage offset of the second message and synchronously store the second message in the second message queue cluster to the first message queue cluster through a dedicated data synchronization service.

[0012] In one possible implementation, the method also includes: obtaining first consumption information generated via the first message queue cluster, the first consumption information being used to indicate messages consumed via the first message queue cluster; wherein a third container and a fourth container are configured in the first message queue cluster, and the fourth container is configured as a shadow container of the third container, for caching part of the updated messages in the third container; storing the first consumption information in the third container in the first message queue cluster; if the first consumption information meets a preset synchronization condition, storing the first consumption information in the fourth container in the first message queue cluster, and synchronously storing the first consumption information in the fourth container in the second message queue cluster through a dedicated data synchronization service.

[0013] In a possible implementation manner, the method further includes: responding to a message consumption request by a first message queue cluster deployed in the first area, wherein the first message queue cluster supports consumption of messages generated by business services corresponding to each area.

[0014] In one possible implementation, the method also includes: receiving a message consumption request for a specified topic, the message consumption request being used to request consumption of messages stored in a message queue cluster deployed in any one of multiple regions, responding to the message consumption request through a specified message queue cluster, and topics with the same name in different regions supporting message consumption from a message queue cluster deployed in any region, and the topic being used to indicate a message category stored in the message queue cluster.

[0015] The present application provides a message processing device based on a message queue system, including: a first receiving unit, used to receive a first message generated in a business service, and write a first storage offset configured for the first message in the first message, wherein the storage offset in the first message queue increases with the message storage; a first storage unit, used to store the first message carrying the first storage offset in the first message queue cluster and synchronize the first message in the first message queue cluster to the second message queue cluster through a dedicated data synchronization service, the second message queue cluster being used to store the first message according to the first storage offset, and the second message queue cluster supporting message backup of the first message queue cluster.

[0016] The present application provides an electronic device, which includes: a processor and a memory; the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes the message processing method based on the message queue system provided by the present application.

[0017] The present application provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes the message processing method based on the message queue system provided by the present application.

[0018] The present application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program codes for executing the message processing method based on the message queue system provided by the present application.

[0019] Compared with the related art, this application has at least the following advantages:

[0020] In the message processing method based on the message queue system provided in the present application, a first message generated in a business service is first received, and a first storage offset (such as offset) configured for the first message is written in the first message; then the first message carrying the first storage offset is stored in the first message queue cluster and the first message in the first message queue cluster is synchronously stored in the second message queue cluster through a dedicated data synchronization service, so that the second message queue cluster is used to store the first message according to the first storage offset, and the second message queue cluster supports the message backup of the first message queue cluster, so that backup can be achieved by specifying the Offset writing method to ensure that the storage location of the message in different clusters remains consistent, thereby effectively overcoming the defects caused by the inconsistent offsets corresponding to the same message in different clusters, which is conducive to improving the disaster recovery effect in different locations. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A flowchart of a message processing method based on a message queue system provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of configuration results of the same Topic configured for multiple regions provided in an embodiment of the present application;

[0024] Figure 3 Another schematic diagram of configuration results of the same Topic configured in multiple regions provided in an embodiment of the present application;

[0025] Figure 4A schematic diagram of a message synchronization process between multiple regions provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of a synchronization process of consumption information between multiple regions provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the structure of a message processing device based on a message queue system provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to better understand the technical solution provided by this application, some technical terms are introduced below.

[0030] Kafka is a high-throughput, distributed, and scalable message middleware system. Kafka can process a large amount of real-time data streams and is widely used in scenarios such as log collection, event processing, stream processing, and message queues. In addition, Kafka's architecture includes components such as Producer, Consumer, and Broker. Producer can send messages to the Kafka cluster, Consumer can subscribe to messages from the Kafka cluster and process them, while Broker is the message transit server responsible for storing and forwarding messages.

[0031] A Topic is a logical container for messages, also known as a message queue collection, which is used to classify and store messages. In Kafka, messages are published to a specified Topic and can be subscribed to by one or more Consumers. Topic is one of the core concepts of Kafka and is the basis for message delivery. It can be seen that a Topic is used to store the same type of messages. For example, TopicA is used to store order messages from a restaurant, and TopicB is used to store transaction messages from a store.

[0032] Partition is a mechanism for implementing data sharding in Kafka. A Topic can be divided into multiple Partitions, and each Partition is an ordered message queue. When a message is sent to a Topic, it will be hashed according to the specified key and then assigned to the corresponding Partition.

[0033] Producer refers to the producer of messages, which is used to send messages to the specified Topic. Producer is responsible for sending messages to the Broker node in the Kafka cluster, and can specify the key of the message when sending the message so that Kafka can assign the message to the specified Partition.

[0034] Consumer is a message consumer, which is used to receive messages from a specified Topic. Consumer is responsible for getting messages from the Broker node in the Kafka cluster, and can specify which Partition to get messages from. Consumers can consume messages in different ways, such as batch consumption, polling consumption, etc.

[0035] Broker is a node in the Kafka cluster that is used to store and manage messages. Broker is one of the core components of Kafka and is responsible for receiving and processing messages sent by producers and storing them on disk. It is also responsible for forwarding messages to consumers.

[0036] Offset is an identifier that identifies the position of each message in a Partition. Each Partition has a unique offset value, and consumers can obtain messages at a specified position based on the offset. Kafka also provides a special Topic called __consumer_offset, which is used to store consumer consumption information.

[0037] After research, it was found that the above-mentioned off-site disaster recovery requirements can be achieved with the help of synchronous replication, and the implementation process is: while writing a message to the local message queue, the message is also synchronously written to the message queues in various remote locations (or computer rooms), so that these remote message queues store copies of the message, so that after confirming that all remote copies have been written successfully, it can be determined that the current message has been written successfully.

[0038] The study also found that the above solution has the following defects: due to the uncontrollable network delay of the remote computer room, the message writing delay is high, which affects the remote disaster recovery effect. In addition, due to the characteristics of the solution itself, the solution only allows messages to be written to the message queue of one area at the same time. For example, at a certain moment, if the producer sends a message to the message queue of area 1, the producers in other areas cannot send messages to the message queue of the corresponding area (such as the producer in area 2 cannot send messages to the message queue of area 2, etc.), so that the producers in other areas can only wait until the aforementioned message has been written successfully before sending the message, which affects the message writing experience.

[0039] After research, it was found that in order to improve writing efficiency, the above-mentioned disaster recovery requirements can be achieved with the help of asynchronous replication, and the implementation process is: after writing a message to the local message queue, the current message can be regarded as written successfully, so that after a certain delay, the message can be asynchronously copied to the message queues (or computer rooms) in various locations for data redundancy.

[0040] After further research, it was found that the above solution has the following defects: because each message in a cluster corresponds to a unique Offset, and each write Offset will be incremented, so the Offset corresponding to the same message in different clusters may be different (for example, the Offset may change in scenarios such as sending failure, retry, cluster failure, etc.), resulting in the defect of Offset misalignment in clusters deployed in different regions.

[0041] Further research has found that one solution to the above defect (such as Uber's Kafka replica synchronization solution) is: after copying the message from cluster 1 to cluster 2, map the offset in cluster 1 with the offset in cluster 2, so that remote disaster recovery can be achieved based on the mapping. In addition, another solution to the above defect (such as confluent's remote data center replication mechanism solution and RocketMQ's remote replication solution) is: in the message sent from cluster 1 to cluster 2, the header information of the message is carried, and the header information records the timestamp (Time stamp) generated by the message, so that the corresponding relationship between the offsets in different clusters can be determined based on the timestamp. It can be seen that under this solution, although it cannot guarantee that the offsets of the same message in the two clusters are equal, it can guarantee that the timestamp information it carries is equal. Therefore, when a failure occurs and the cluster is switched, the corresponding offset in the new cluster can be queried by the timestamp to achieve remote disaster recovery.

[0042] After research, it was also found that both solutions shown in the previous paragraph require relatively complex mechanisms to ensure that the same message in two remote clusters can be mapped to each other, so both solutions have their own defects. For example, the solution implemented by offset mapping requires an additional third-party application to store the offset mapping relationship, which affects the disaster recovery effect of different locations. For another example, in the solution implemented by time stamp mapping, when there are multiple data with the same timestamp, it is impossible to accurately find the consumption location, there will be duplicate data, and the switching solution requires querying, which is relatively complicated, affecting the disaster recovery effect of different locations.

[0043] Based on the above research, in order to better overcome the defects shown in the previous paragraph, the present application provides a multi-regional disaster recovery solution implemented by means of a specified offset writing method, which includes: after writing a message to cluster 1, the message and the offset corresponding to the message in cluster 1 are sent to cluster 2, so that cluster 2 can write and process the message according to the offset, so that the offset corresponding to the message in cluster 2 is the same as the offset corresponding to the message in cluster 1, and then the offsets corresponding to the same message in different clusters are equal, which can effectively overcome the defects caused by the inconsistent offsets corresponding to the same message in different clusters, thereby helping to improve the effect of remote disaster recovery.

[0044] As an example, when the off-site disaster recovery solution provided by the present application involves N clusters, these clusters can meet the following constraints: each cluster is configured with the same Topic so that these clusters can all receive messages sent for the Topic; different clusters are deployed in different areas so that these clusters can back up each other; for the i-th cluster, after the i-th cluster receives the message sent for the Topic, it is not only necessary to store the message in the i-th cluster, but also to synchronize the message and the offset corresponding to the message in the i-th cluster to the other clusters in the N clusters except the i-th cluster, so that the other clusters can store the message according to the offset, so that the same message is in an offset-aligned state in different clusters, which is conducive to improving the off-site disaster recovery effect. Wherein, i is a positive integer, i≤N, and N is a positive integer.

[0045] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] In order to better understand the technical solution provided by the present application, the message processing method based on the message queue system provided by the present application is described below with reference to some drawings. Figure 1 As shown, the message processing method based on the message queue system provided in the embodiment of the present application includes the following S1-S2.

[0047] S1: receiving a first message generated in a business service, and writing a first storage offset configured for the first message into the first message, wherein the storage offset in the first message queue increases as the message is stored.

[0048] Among them, business service refers to the service used to generate messages, such as Producer.

[0049] The first message refers to a message generated by a business service for a first message queue cluster, in the expectation that the first message queue cluster will store the message and synchronize it to other message queue clusters.

[0050] The first message queue cluster refers to the message queue cluster deployed in the first region, such as the i-th cluster or Figure 2 A logical cluster is deployed in the region 1 shown, so that the cluster can at least be used to store messages generated by the business service corresponding to the region.

[0051] The first storage offset is used to describe the storage location of the first message, and the present application does not limit the implementation method of the first storage offset. For example, it can be implemented using offset.

[0052] S2: Store the first message carrying the first storage offset in the first message queue cluster and synchronously store the first message in the first message queue cluster in the second message queue cluster through a dedicated data synchronization service, the second message queue cluster is used to store the first message according to the first storage offset, and the second message queue cluster supports message backup of the first message queue cluster.

[0053] Among them, the dedicated data synchronization service is used to achieve data synchronization between different message queue clusters.

[0054] The second message queue cluster refers to a cluster different from the first message queue cluster, and the other clusters except the i-th cluster among the above N clusters, or Figure 2 The logical cluster deployed in zone 2 as shown, or Figure 2 The logical cluster deployed in the area 3 shown is such that the second message queue cluster is at least used to back up the messages stored in the first message queue cluster.

[0055] Based on the above S1 to S2, it can be known that the present application provides a multi-regional disaster recovery solution implemented by means of a specified offset writing method, which includes: after writing a message to cluster 1, synchronizing the message and the offset corresponding to the message in cluster 1 to cluster 2, so that cluster 2 can write and process the message according to the offset, so that the offset corresponding to the message in cluster 2 is the same as the offset corresponding to the message in cluster 1, and then the offsets corresponding to the same message in different clusters are equal, which can effectively overcome the defects caused by the inconsistent offsets corresponding to the same message in different clusters, thereby helping to improve the effect of remote disaster recovery.

[0056] After research, it is found that in the deployment topology of message queues, there are two concepts: multi-region (Region) and multi-data center (Internet data center, Idc). Among them, multi-data center refers to a physical area (such as Figure 2 In any region or country shown in the figure, there are multiple physical clusters with close locations, and these physical clusters can be connected through relatively low-cost dedicated network lines, so that these physical clusters can form a logical cluster and be used as a cluster deployed in the physical region. In addition, the present application is not limited to multiple regions. For example, multiple regions can refer to multiple countries or regions, so that the multiple regions have the characteristics of relatively long physical distances, so that the deployment topology of multiple regions is more suitable for the remote disaster recovery solution provided by the present application to save costs.

[0057] It can be seen that in one possible implementation, the first message queue cluster and the second message queue cluster are deployed across regions, and the first message queue cluster and the second message queue cluster are pre-configured with a dedicated data synchronization service, so that data synchronization between the two clusters can be achieved with the help of the dedicated data synchronization service in the future, thereby achieving a multi-region deployment topology at a lower cost.

[0058] In addition, in some scenarios, message queue clusters deployed in different regions can be configured with corresponding business services so that the message queue cluster deployed in the region can store messages generated by the corresponding business services and synchronize the messages to message queue clusters deployed in other regions for backup.

[0059] It can be seen that, in a possible implementation manner, the message processing method based on the message queue system may include the following steps 11 to 14.

[0060] Step 11: Receive a first message generated in a business service corresponding to a first message queue cluster, and write a first storage offset configured for the first message into the first message, wherein the storage offset in the first message queue increases as the message is stored.

[0061] The business service corresponding to the first message queue cluster is used to produce messages to provide to the cluster for storage and forwarding.

[0062] Step 12: Store the first message carrying the first storage offset in the first message queue cluster and synchronize the first message in the first message queue cluster to the second message queue cluster through a dedicated data synchronization service. The second message queue cluster is used to store the first message according to the first storage offset. The second message queue cluster supports message backup of the first message queue cluster.

[0063] Step 13: Receive a second message generated in the business service corresponding to the second message queue cluster, and write a second storage offset configured for the second message into the second message, wherein the storage offset in the second message queue increases as the message is stored.

[0064] The business service corresponding to the second message queue cluster is used to produce messages to provide to the cluster for storage and forwarding.

[0065] The second message refers to a message generated by the business service for the second message queue cluster, in the expectation that the second message queue cluster will store the message and synchronize it to other message queue clusters.

[0066] The second storage offset is used to describe the storage location of the second message, and the present application does not limit the implementation method of the second storage offset. For example, it can be implemented using offset.

[0067] Step 14: Store the second message carrying the second storage offset in the second message queue cluster and synchronously store the second message in the second message queue cluster to the first message queue cluster through a dedicated data synchronization service, wherein the first message queue cluster supports message backup of the second message queue cluster.

[0068] Based on the above steps 11 to 14, it can be known that in the multi-regional disaster recovery solution based on asynchronous replication provided by the present application, when message queue clusters are deployed in multiple regions (such as transoceanic regions), for the message queue cluster deployed in any region, after the cluster receives the message generated by the Producer corresponding to the cluster, the message is first stored in the cluster according to the offset configured for the message, and then the offset and the message are synchronized to the message queue clusters deployed in other regions, so that the message queue clusters deployed in other regions can store the message according to the offset, thereby ensuring that the storage location of the message in different clusters is consistent by specifying the Offset write method, so that the message queue clusters deployed in different regions can back up each other to ensure that the producers in each region can write messages to the same Topic at the same time or at different times, thereby realizing masterless multi-write.

[0069] After research, it was found that for cluster 1 and cluster 2 that back up each other, after cluster 2 receives the message synchronized by cluster 1, cluster 2 may synchronize the message back to cluster 1, causing a loop problem. Therefore, in order to overcome this problem, the present application provides a solution. In this solution, when the first message queue cluster and the second message queue cluster are deployed across regions, and the first message queue cluster and the second message queue cluster are pre-configured with multiple dedicated data synchronization services, one dedicated data synchronization service is used to send the first message in the first message queue cluster unidirectionally to the second message queue cluster, and another dedicated data synchronization service is used to send the second message in the second message queue cluster unidirectionally to the first message queue cluster. In this way, the loop problem can be overcome through one-way data synchronization.

[0070] In addition, in order to better implement remote disaster recovery with the help of a specified offset writing method, the present application provides a deployment method for multiple message queue clusters involved in the disaster recovery solution (such as Figure 3 The deployment method shown in FIG. 1 includes: for a group of groups configured with the same Topic (such as Figure 3 For the multiple message queue clusters of Topic 1) shown in FIG. 1 , each message queue cluster includes multiple containers created by configuring the Topic, so that the messages stored in the multiple containers belong to the category specified by the Topic, and a dedicated data synchronization service (Mirror Service) is configured between the same type of containers (such as containers with the same name) in different message queue clusters, so that data synchronization between different message queue clusters can be completed with the help of the Mirror Service later.

[0071] It should be noted that the Mirror Service is used to implement data synchronization based on asynchronous replication, and the Mirror Service can pass the message stored in the upstream cluster and its corresponding offset to the downstream cluster, so that the downstream cluster stores the message according to the offset, ensuring that the offset of the same message in the source cluster and the target cluster is consistent. In addition, since the delay between regions is relatively high, an independently deployed Mirror Service can be used for data synchronization. In addition, the number of containers in the above-mentioned "multiple containers" is determined based on the number of clusters in the above-mentioned "multiple clusters", so that the "multiple containers" at least include the containers corresponding to the regions to which each cluster belongs under the same Topic. In addition, the multiple clusters meet the following constraints: ① The physical names of clusters deployed in different regions are different, so that different clusters can be distinguished by means of the physical names in the future, and the containers corresponding to the same Topic in each region can be distinguished by using the physical names as suffixes. ② The logical names (such as the names that users can view) of clusters deployed in different regions are the same, so that the logical names can be used to know that these clusters are data copies of each other in the future. ③Topics with the same name can be created between multiple clusters, and when creating a Topic, you can choose to specify a primary region for the producer to ensure that messages written to the Topic can only be executed through the cluster deployed in the primary region. When registering a consumer, you can also choose to specify a primary region for the consumer to ensure that messages written to the Topic can only be consumed through the cluster deployed in the primary region. ④When creating a container for a Topic, create containers corresponding to each region under the Topic so that the suffixes of the names of these containers are determined according to the physical names of the clusters deployed in these regions, and the names of these containers also include the names specified for the Topic (such as Topic1). ⑤When creating a container for a Topic, a one-way Mirror Service will also be created between containers of the same type (such as containers with the same name) to synchronize data between regions, such as messages, consumption information (such as consumer offset), etc., without having to deal with data loop issues.

[0072] Based on the above two paragraphs, it can be known that in one possible implementation, the first message queue cluster and the second message queue cluster both include multiple containers created by configuring the same Topic, the multiple containers include a first container and a second container, and the Topic is used to indicate the message categories stored in the multiple containers; the dedicated data synchronization service configured between the first container in the first message queue cluster and the first container in the second message queue cluster is used to unidirectionally synchronize messages received and stored by the first message queue from the business service to the second message queue cluster; the dedicated data synchronization service configured between the second container in the first message queue cluster and the second container in the second message queue cluster is used to unidirectionally synchronize messages received and stored by the second message queue from the business service to the first message queue cluster.

[0073] It can be seen that for cluster 1, cluster 2, and cluster 3 (such as Figure 3 For example, cluster 1, cluster 2, and cluster 3 are all configured with the same Topic (such as Figure 3 , and clusters 1, 2, and 3 all include multiple containers created for the Topic. If the multiple containers include container 1 (such as Figure 3 Topic1_global_region1), container 2 (such as Figure 3 Topic1_global_region2) and container 3 (as shown Figure 3 As shown in Topic1_global_region3), a one-way dedicated data synchronization service is configured between container 1 in cluster 1 and container 1 in cluster 2, a one-way dedicated data synchronization service is configured between container 2 in cluster 1 and container 2 in cluster 2, and a one-way dedicated data synchronization service is configured between container 3 in cluster 1 and container 3 in cluster 2. The solution may include: container 1 in cluster 1 stores message 1 generated by the business service corresponding to cluster 1, and container 1 in cluster 1 synchronizes message 1 and offset 1 corresponding to message 1 to cluster 1. 2 and container 1 in cluster 3 for backup; similarly, container 2 in cluster 2 stores message 2 generated by the business service corresponding to cluster 2, and container 2 in cluster 2 synchronizes message 2 and offset 2 corresponding to message 2 to container 2 in cluster 1 and container 2 in cluster 3 for backup; similarly, container 3 in cluster 3 stores message 3 generated by the business service corresponding to cluster 3, and container 3 in cluster 3 synchronizes message 3 and offset 3 corresponding to message 3 to container 3 in cluster 1 and container 3 in cluster 2 for backup.

[0074] Based on the above four paragraphs, it can be seen that the remote disaster recovery solution provided by this application that can better meet the specified Offset writing requirements has the following characteristics: the solution involves logical clusters deployed in N regions; the message queue clusters deployed in the N regions are all configured with the same Topic (such as Figure 3 Topic1 shown in Figure 3 For example, in the region 1 shown in the figure, the message queue cluster deployed in the i-th region contains not only the container corresponding to the i-th region under the Topic (such as Figure 3 Topic1_global_region1) shown in the figure, there are also containers corresponding to other regions under this Topic (such as Figure 3 Topic1_global_region2 and Topic1_global_region3) are shown, so that the message queue cluster deployed in the i-th region will have containers corresponding to N regions under the Topic, so that the message queue cluster deployed in the i-th region can not only record the messages generated by the business services corresponding to the region, but also record the messages generated by the business services corresponding to other regions, thereby enabling the message queue cluster deployed in the i-th region to fully reproduce the data view in the message queue clusters deployed in each region.

[0075] It can be seen that the off-site disaster recovery solution shown in the previous paragraph allows message production services to be deployed separately in different regions, and data is written to the Topics of their respective regions at the same time. In addition, the data in this region will be asynchronously copied to the clusters deployed in other regions through the mirror service, while ensuring that the offset of the data is consistent in different regions. In this way, each region not only has the Topic of this region, but also has backup data from Topics in other regions. In addition, for producers in each region, the messages they send can only be written to the Topics of this region (such as containers with the physical name of the cluster deployed in this region as the suffix), and cannot be written to Topics in other regions; however, for consumers in each region, they can not only consume (such as subscribe to) messages stored in the Topics of this region, but also consume messages stored in Topics in other regions. In addition, containers with the same suffix (such as Figure 3 ) in each region must present the same Partition to ensure that the source and target Topic data are aligned when data is synchronized. In addition, you need to create a one-way Mirror Service for the same Topic deployed in different regions (such as Figure 4 and Figure 5It should be noted that for multiple regions configured with the same Topic, as the number of regions increases, the number of containers that need to be created in the cluster deployed in each region when configuring the Topic will also increase.

[0076] In addition, in order to better implement the specified Offset writing, the present application also provides a specified Offset writing scheme, and the scheme satisfies the constraints shown in (1)-(2) below.

[0077] (1) For the sender of the synchronous message, according to the type of message queue storing the message in the sender, the Offset corresponding to the message is written into the attribute of the message so that the Offset can be provided to other clusters as an attribute. For example, since Kafka provides a mechanism for transmitting metadata between producers and consumers, called Kafkaheader, the Offset can be written into the attribute by setting a certain metadata in the Kafka header (such as GLOBAL_MQ_WRITE_OFFSET=10000). For another example, in RocketMQ, the Offset can be written into the attribute by setting the same content in the properties attribute.

[0078] Based on the content of the above paragraph, it can be known that in one possible implementation, the above step of "writing the first storage offset configured for the first message in the first message" may include: writing the first storage offset configured for the first message in the message header of the first message (such as Kafka header); or, writing the first message offset configured for the first message in the attribute characteristics of the first message (such as properties), so as to better adapt to the specified Offset writing scheme in different scenarios.

[0079] (2) For the receiver of the synchronous message, the receiver needs to read the attribute carrying the Offset, and when writing the message to the disk data, set the Offset of the message to the attribute value of the attribute to ensure that the Offset corresponding to the message in the receiver is consistent with the Offset corresponding to the message in the sender. For example, when the receiver is implemented by Kafka, the incremental Offset generated by the receiver itself is no longer used when writing the message, and the Offset in the write request metadata received by the receiver is used first, so that when it is determined that the Offset of the current storage system is less than the Offset in the write request, the write is supported. For another example, when the receiver is implemented by RocketMQ, the Queue Offset attribute of the message is set to the Offset in the write request when writing CommitLog, and when generating the ConsumeQueue index, the blank Offset between the last Offset and this Offset is filled with 0.

[0080] It should be noted that in any message queue system, the Offset is continuous and increasing. However, in the solution implemented by writing with a specified Offset, it is only necessary to ensure that the written Offset is increasing, and there is no need to ensure a certain continuity to prevent some unexpected failure scenarios. In addition, if the Offset specified by the sender is less than or equal to the current maximum Offset of the receiver, it can be determined that the current write request is a request that is repeatedly sent due to problems such as restart or failure of the mirror service, so it can be determined that the write request does not need to be processed, so the request can be directly rejected.

[0081] Based on the above two paragraphs, it can be known that the process of storing the first message in the second message queue cluster includes: after obtaining the first storage offset of the first message through the second message queue cluster, determining whether the first storage offset is greater than the maximum storage offset in the second message queue cluster; if so, storing the first message in the second message queue cluster according to the first storage offset; if not, discarding the first message, which can effectively overcome the impact caused by problems with the dedicated data synchronization service itself.

[0082] After research, it was found that for a message queue cluster (such as Figure 3 For example, for a logical cluster deployed in region 1 as shown in FIG. 1 , consumers can not only subscribe to messages recorded by themselves through the cluster (such as those recorded by Figure 3 As shown in Topic1_global_region1), you can also subscribe to messages synchronized in other regions (such as those stored in Figure 3As shown in Topic1_global_region2, consumers can consume messages generated by the cluster for different regions without increasing the cross-region bandwidth cost. It can be seen that in one possible implementation, a message consumption request is responded to by a first message queue cluster deployed in the first region, and the first message queue cluster supports consumption of messages generated by business services corresponding to each region.

[0083] As an example, for the i-th cluster among multiple message queue clusters configured with the same Topic, the i-th cluster may include multiple containers created for the Topic, the multiple containers include containers corresponding to the regions to which each cluster belongs under the Topic, and the working principle of the i-th cluster may include: first, receiving a message subscription request triggered by a consumer for the Topic, so that the request is used to indicate which container under the Topic the consumer wants to consume messages stored in; then, parsing the container identifier (such as Figure 3 The container name "Topic1_global_region2" shown in the figure is used to indicate the container specified by the consumer for consumption; the message in the container indicated by the container identifier is then pushed to the consumer, so that the consumers in this region can not only consume the messages generated by the business service corresponding to the i-th cluster through the i-th cluster, but also consume the messages backed up by the i-th cluster through the i-th cluster, that is, the messages generated by the business services corresponding to the message queue clusters deployed in other regions, so that consumers can consume the messages generated by the business services corresponding to different regions with the same cluster, and then when the cluster deployed in a single region fails, it can directly switch to the cluster deployed in other regions to provide services without performing other additional operations (such as Offset mapping), which is conducive to improving efficiency.

[0084] In addition, in order to better improve the message experience, for Topics with the same name in different regions, you can consume the messages stored in the Topic from a message queue cluster deployed in any region. Based on this, it can be seen that in one possible implementation, the message processing method based on the message queue system provided by this application may include: receiving a message consumption request for a specified Topic, the message consumption request is used to request the consumption of messages stored in a message queue cluster deployed in any of multiple regions, responding to the message consumption request through a specified message queue cluster, and Topics with the same name in different regions support message consumption from a message queue cluster deployed in any region. The Topic is used to indicate the message category stored in the message queue cluster.

[0085] Research has found that in order to better improve the disaster recovery effect of different locations, the consumption progress of each cluster can be synchronized with other clusters to ensure that each cluster can accurately understand the corresponding consumption situation in each region.

[0086] Based on the above research, the present application also provides an implementation of a message processing method based on a message queue system, which may include at least some or all of the steps 21 to 23 below.

[0087] Step 21: Acquire first consumption information generated via the first message queue cluster, where the first consumption information is used to indicate messages consumed via the first message queue cluster.

[0088] The first consumption information is generated by the consumer consuming messages through the first message queue cluster, so that the first consumption information can indicate what message the consumer has consumed through the first message queue cluster (eg, the message corresponding to which Offset has been consumed).

[0089] In addition, the present application does not limit the implementation method of the first consumption information. For example, it may include the Offset corresponding to the consumed message, so that the first consumption information can indicate the position of the message consumed by the consumer via the first message queue cluster.

[0090] Step 22: Synchronize the first consumption information to the second message queue cluster, where the second message queue cluster is used to store the first consumption information.

[0091] Step 23: Store the second consumption information synchronized via the second message queue cluster into the first message queue cluster, where the second consumption information is used to indicate the messages consumed via the second message queue cluster.

[0092] The second consumption information is generated by the consumer consuming messages through the second message queue cluster, so that the second consumption information can indicate what messages the consumer has consumed through the second message queue cluster (eg, the message corresponding to which Offset has been consumed).

[0093] In addition, the present application does not limit the implementation method of the second consumption information. For example, it may include the Offset corresponding to the consumed message, so that the second consumption information can indicate the position of the message consumed by the consumer via the second message queue cluster.

[0094] In addition, the present application does not limit the execution time of step 23.

[0095] Based on the relevant contents of steps 21 to 23 above, it can be known that for the i-th cluster (such as the first message queue cluster or the second message queue cluster) among multiple message queue clusters configured with the same Topic, the working principle of the i-th cluster may include: after sending the message stored in any container under the Topic to the consumer who has subscribed to the container, the Offset corresponding to the message is stored as consumption information, and the consumption information is synchronized to other clusters, so that each cluster stores the consumption information, so that each cluster can obtain the consumption progress from the consumption information.

[0096] In addition, in some scenarios, in order to better improve the disaster recovery effect in different regions, for the i-th cluster (such as the first message queue cluster or the second message queue cluster) among multiple clusters configured with the same Topic, the i-th cluster may also include a container (such as Figure 3 The _consumer_offset container shown in the figure is used to store the consumption progress of the messages in the container corresponding to the Topic in the area to which each cluster belongs, so that the working principle of the i-th cluster may include: after sending the message stored in any container under the Topic to the consumer who has subscribed to the container, the Offset corresponding to the message is stored in the _consumer_offset container as consumption information, and the consumption information is synchronized to the _consumer_offset containers in other clusters through the _consumer_offset container, so that each cluster stores the consumption information, so that each cluster can learn the consumption progress from the consumption information.

[0097] The research also found that in some scenarios, some consumers have the following consumption needs: their consumption progress is only recorded in the local cluster and their consumption progress cannot be synchronized to other areas.

[0098] Based on the above research, the present application also provides an implementation of a message processing method based on a message queue system, which may include at least some or all of the steps 31 to 33 below.

[0099] Step 31: Acquire and store first consumption information generated via the first message queue cluster, where the first consumption information is used to indicate messages consumed via the first message queue cluster.

[0100] Step 32: If the first consumption information meets a preset synchronization condition, the first consumption information is synchronized to a second message queue cluster, and the second message queue cluster is used to store the first consumption information.

[0101] The preset synchronization condition refers to the condition that needs to be met by the consumption information that needs to be synchronized to other clusters. For example, it can be specifically: the consumption information carries a preset synchronization identifier. It should be noted that for a consumption information, whether the information carries the synchronization identifier is determined based on the consumption configuration information set when the consumer subscribes to the Topic. The consumption configuration information is used to characterize its consumption needs, such as whether to synchronize.

[0102] Step 33: If the second consumption information meets the preset synchronization condition, the second consumption information synchronized via the second message queue cluster is stored in the first message queue cluster, where the second consumption information is used to indicate the message consumed via the second message queue cluster.

[0103] Based on the relevant contents of steps 31 to 33 above, it can be known that for the i-th cluster (such as the first message queue cluster or the second message queue cluster) among multiple clusters configured with the same Topic, the working principle of the i-th cluster may include: after sending the message stored in any container under the Topic to the consumer who has subscribed to the container, first store the Offset corresponding to the message as consumption information, and then determine whether the consumption information meets the preset synchronization conditions. If not, do not synchronize. If satisfied, synchronize the consumption information to other clusters so that each cluster stores the consumption information, so that each cluster can obtain the consumption progress from the consumption information.

[0104] In addition, in order to better implement the solution shown in the previous paragraph, the present application provides a possible implementation method of the above-mentioned multiple containers. In this method, for the i-th cluster (such as the first message queue cluster or the second message queue cluster) among the multiple clusters configured with the same Topic, the i-th cluster may include multiple containers created by configuring the Topic (target topic), and the multiple containers include not only the containers corresponding to the areas to which each cluster belongs under the Topic (such as the first container and the second container), but also the third container (such as Figure 3 _consumer_offset container shown) and the fourth container (as shown Figure 3 _consumer_offset_shadow container shown). The third container is used not only to record the consumption information generated by the i-th cluster, but also to record the consumption information synchronized by other clusters. The fourth container is used to record the consumption information generated by the i-th cluster and needs to be synchronized to other clusters. For ease of understanding, two examples are used for explanation below.

[0105] Example 1, when the above-mentioned remote disaster recovery solution involves a first message queue cluster and a second message queue cluster, the first message queue cluster and the second message queue cluster are both configured with the same Topic, the first message queue cluster and the second message queue cluster both include multiple containers created by configuring the Topic (target topic), the multiple containers include at least a third container and a fourth container, and a dedicated data synchronization service is configured between the fourth container in the first message queue cluster and the third container in the second message queue cluster, the working principle of the system includes: first, the first message queue cluster obtains the first consumption information generated by the first message queue cluster through the third container in the first message queue cluster, so that the first consumption information is used to describe the consumption information generated by the first message queue cluster. The first message queue cluster consumes messages; then, determines whether the first consumption information meets the preset synchronization condition (such as whether it carries a synchronization identifier), so that when it is determined that the first consumption information meets the preset synchronization condition, the first message queue cluster stores the first consumption information through the third container in the first message queue cluster and the fourth container in the first message queue cluster, and the first message queue cluster synchronizes the first consumption information to the third container in the second message queue cluster through the fourth container in the first message queue cluster, so that the third container in the second message queue cluster is used to store the first consumption information, so as to synchronize part of the consumption information generated in the first message queue cluster to other clusters, which is conducive to better meeting some consumption needs.

[0106] It can be seen that, under a possible implementation mode, the message processing method based on the message queue system provided by the present application may also include: first obtaining first consumption information generated by the first message queue cluster, the first consumption information is used to indicate the message consumed by the first message queue cluster; wherein the first message queue cluster is configured with a third container and a fourth container, and the fourth container is configured as a shadow container of the third container, and is used to cache part of the updated messages in the third container; then storing the first consumption information in the third container in the first message queue cluster; if the first consumption information meets the preset synchronization condition, storing the first consumption information in the fourth container in the first message queue cluster, and synchronously storing the first consumption information in the fourth container to the second message queue cluster through a dedicated data synchronization service.

[0107] Example 2, when the above-mentioned remote disaster recovery solution involves a first message queue cluster and a second message queue cluster, the first message queue cluster and the second message queue cluster are both configured with the same Topic, the first message queue cluster and the second message queue cluster both include multiple containers created by configuring the Topic (target topic), the multiple containers include at least a third container and a fourth container, and a dedicated data synchronization service is configured between the third container in the first message queue cluster and the fourth container in the second message queue cluster, the working principle of the system includes: first, the second message queue cluster obtains the second consumption information generated by the second message queue cluster through the third container in the second message queue cluster, so that the second consumption information is used to describe the consumption information generated by the first message queue cluster; The method further comprises the following steps: first, determining a message consumed by the second message queue cluster; second, determining whether the second consumption information meets a preset synchronization condition (such as whether it carries a synchronization identifier); so that when it is determined that the second consumption information meets the preset synchronization condition, the second message queue cluster stores the second consumption information through a third container in the second message queue cluster and a fourth container in the second message queue cluster, and the second message queue cluster synchronizes the second consumption information to the third container in the first message queue cluster through the fourth container in the second message queue cluster, so that the third container in the first message queue cluster stores the second consumption information, so as to synchronize part of the consumption information generated in the second message queue cluster to other clusters, thereby helping to better meet some consumption needs.

[0108] It can be seen that, under a possible implementation mode, the message processing method based on the message queue system provided by the present application may also include: first obtaining the second consumption information generated by the second message queue cluster, the second consumption information is used to indicate the message consumed by the second message queue cluster; wherein the second message queue cluster is configured with a third container and a fourth container, and the fourth container is configured as a shadow container of the third container, and is used to cache part of the updated messages in the third container; then the second consumption information is stored in the third container in the second message queue cluster; if the second consumption information meets the preset synchronization condition, the second consumption information is stored in the fourth container in the second message queue cluster, and the second consumption information in the fourth container is synchronously stored in the first message queue cluster through a dedicated data synchronization service.

[0109] It should be noted that for a cluster deployed in any region, it is necessary not only to create corresponding containers under the same Topic in each region in the cluster to achieve data backup between different regions by specifying the Offset, but also to create a __consumer_offset container and a __consumer_offset_shadow topic container in the cluster, and to ensure that the number of partitions in these two containers remains consistent.

[0110] Based on the above-mentioned relevant contents of the remote disaster recovery system, it can be known that the system has the advantages shown in (i) to (vi) below.

[0111] (I) The present application provides a solution for writing by specifying an Offset in a message queue, and at the same time, adapts and modifies message queues such as Rocketmq and Kafka involved in actual application scenarios to better meet the needs of writing by specifying an Offset, so that the same message in different clusters can be guaranteed to be completely consistent in information such as partitions and Offsets.

[0112] (II) For the Offset-specified write solution provided by this application, the solution also adopts cross-region asynchronous replication to provide a unified and consistent perspective of data off-site copies, thereby ensuring that clusters deployed in each region can see a complete and consistent data view with consistent Offsets. In addition, when multiple regions are configured with the same Topic (such as Figure 3 Topic 1) shown in the figure, you can use multiple containers created in each region for the Topic to implement masterless multi-write, so as to ensure that the production services in each region can write to the Topic at the same time without special processing.

[0113] (III) Based on the offset synchronization mechanism provided by this application, the consumption status of each cluster can be synchronized to other clusters, which facilitates lossless switching between multiple clusters. In this way, when a cluster deployed in a single area fails, it can be quickly switched to clusters deployed in other areas to continue providing services, thereby ensuring that the data viewed with the help of these clusters is consistent.

[0114] (iv) Because the data stored in containers with the same name in different regions is exactly the same, consumers can use the cluster deployed in any region to specify the consumption of messages stored in any container, which helps improve the consumer experience.

[0115] (V) Because the cluster deployed in the same region includes multiple containers created for the same Topic and used to store messages generated in different regions, the cluster can provide consumers with messages generated in each region, thereby enabling multiple different consumption tasks to be started through the cluster without increasing the cross-region bandwidth cost.

[0116] (VI) The remote disaster recovery system provided by this application supports simultaneous writing and consumption of data in multiple regions. It can be seen that the clusters deployed in each region can simultaneously write messages sent to the same Topic, and consumers can consume the full amount of data with the help of the clusters deployed in each region.

[0117] Based on the relevant contents of the message processing method based on the message queue system provided in the embodiment of the present application, the embodiment of the present application also provides a message processing device based on the message queue system. Figure 6 Explain and illustrate. Figure 6 This is a schematic diagram of a message processing device based on a message queue system provided in an embodiment of the present application. It should be noted that for the technical details of the message processing device based on a message queue system provided in an embodiment of the present application, please refer to the relevant content of the message processing method based on a message queue system above.

[0118] like Figure 6 As shown, the message processing device 600 based on the message queue system provided in the embodiment of the present application includes:

[0119] The first receiving unit 601 is used to receive a first message generated in a business service, and write a first storage offset configured for the first message into the first message, wherein the storage offset in the first message queue increases as the message is stored;

[0120] The first storage unit 602 is used to store the first message carrying the first storage offset in the first message queue cluster and synchronize the first message in the first message queue cluster to the second message queue cluster through a dedicated data synchronization service, the second message queue cluster is used to store the first message according to the first storage offset, and the second message queue cluster supports message backup of the first message queue cluster.

[0121] In one possible implementation, the first message queue cluster and the second message queue cluster are deployed across regions, and the first message queue cluster and the second message queue cluster are pre-configured with the dedicated data synchronization service, and the dedicated data synchronization service is used to send the first message in the first message queue cluster unidirectionally to the second message queue cluster.

[0122] In one possible implementation, the first message queue cluster and the second message queue cluster both include multiple containers created by configuring the same topic, the multiple containers include a first container and a second container, and the topic is used to indicate the message categories stored in the multiple containers; the dedicated data synchronization service configured between the first container in the first message queue cluster and the first container in the second message queue cluster is used to unidirectionally synchronize messages received and stored by the first message queue from the business service to the second message queue cluster; the dedicated data synchronization service configured between the second container in the first message queue cluster and the second container in the second message queue cluster is used to unidirectionally synchronize messages received and stored by the second message queue from the business service to the first message queue cluster.

[0123] In a possible implementation manner, the first receiving unit 601 is specifically used to: write a first storage offset configured for the first message in a message header of the first message; or write a first message offset configured for the first message in an attribute feature of the first message.

[0124] In one possible implementation, the device 600 also includes: a first processing unit, used to determine whether the first storage offset of the first message is greater than the maximum storage offset in the second message queue cluster after obtaining the first storage offset of the first message through the second message queue cluster; if so, storing the first message in the second message queue cluster according to the first storage offset; if not, discarding the first message.

[0125] In a possible implementation manner, the first receiving unit 601 is specifically configured to: receive a first message generated in a business service corresponding to the first message queue cluster;

[0126] The device 600 further includes:

[0127] A second receiving unit is used to receive a second message generated in the business service corresponding to the second message queue cluster, and write a second storage offset configured for the second message into the second message, wherein the storage offset in the second message queue increases with the message storage;

[0128] The second storage unit is used to store the second message carrying the second storage offset in the second message queue cluster and synchronously store the second message in the second message queue cluster to the first message queue cluster through a dedicated data synchronization service, and the first message queue cluster supports message backup of the second message queue cluster.

[0129] In a possible implementation manner, the device 600 further includes:

[0130] A first acquisition unit is configured to acquire first consumption information generated by the first message queue cluster, where the first consumption information is used to indicate messages consumed by the first message queue cluster; wherein a third container and a fourth container are configured in the first message queue cluster, and the fourth container is configured as a shadow container of the third container, and is used to cache part of the messages updated in the third container;

[0131] A third storage unit, used to store the first consumption information in a third container in the first message queue cluster;

[0132] The fourth storage unit is used to store the first consumption information in a fourth container in the first message queue cluster if the first consumption information meets a preset synchronization condition, and to synchronize and store the first consumption information in the fourth container in the second message queue cluster through a dedicated data synchronization service.

[0133] In a possible implementation manner, the device 600 further includes:

[0134] The second processing unit is used to respond to the message consumption request through a first message queue cluster deployed in the first area, where the first message queue cluster supports consuming messages generated by business services corresponding to each area.

[0135] In a possible implementation manner, the device 600 further includes:

[0136] The third processing unit is used to receive a message consumption request for a specified topic, where the message consumption request is used to request consumption of messages stored in a message queue cluster deployed in any of the multiple regions, and respond to the message consumption request through a specified message queue cluster. Topics with the same name in different regions support message consumption from a message queue cluster deployed in any region, and the topic is used to indicate a message category stored in the message queue cluster.

[0137] Based on the relevant content of the message processing device 600 based on the above-mentioned message queue system, it can be known that the working principle of the device 600 includes: first receiving the first message generated in the business service, and writing the first storage offset (such as offset) configured for the first message in the first message; then storing the first message carrying the first storage offset to the first message queue cluster and synchronously storing the first message in the first message queue cluster to the second message queue cluster through a dedicated data synchronization service, so that the second message queue cluster is used to store the first message according to the first storage offset, and the second message queue cluster supports the message backup of the first message queue cluster, so that backup can be achieved by specifying the Offset writing method to ensure that the storage location of the message in different clusters remains consistent, thereby effectively overcoming the defects caused by the inconsistent offsets corresponding to the same message in different clusters, which is conducive to improving the disaster recovery effect in different locations.

[0138] In addition, an embodiment of the present application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the message processing method based on the message queue system provided in the embodiment of the present application.

[0139] See also Figure 7 , which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0140] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0141] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0142] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0143] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept, and the technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0144] An embodiment of the present application also provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the message processing method based on the message queue system provided in the embodiment of the present application.

[0145] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0146] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0147] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0148] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can execute the method.

[0149] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0150] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0151] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0152] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0153] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0154] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0155] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0156] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0157] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0158] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A message processing method based on a message queue system, characterized in that: The method comprises: receiving a first message generated in the business service, and writing a first storage offset configured for the first message into the first message, wherein the storage offset in the first message queue increases as the message is stored; A first message carrying a first storage offset is stored in the first message queue cluster, and the first message in the first message queue cluster is synchronously stored in the second message queue cluster through a dedicated data synchronization service. The second message queue cluster is used to store the first message according to the first storage offset. The second message queue cluster supports message backup of the first message queue cluster.

2. The method according to claim 1, characterized in that The first message queue cluster and the second message queue cluster are deployed across regions, and the first message queue cluster and the second message queue cluster are pre-configured with the dedicated data synchronization service, and the dedicated data synchronization service is used to unidirectionally send the first message in the first message queue cluster to the second message queue cluster.

3. The method according to claim 1, characterized in that The first message queue cluster and the second message queue cluster both include multiple containers created by configuring the same topic, the multiple containers include a first container and a second container, and the topic is used to indicate the message categories stored in the multiple containers; The dedicated data synchronization service configured between the first container in the first message queue cluster and the first container in the second message queue cluster is used to unidirectionally synchronize the messages received and stored by the first message queue from the business service to the second message queue cluster; The dedicated data synchronization service configured between the second container in the first message queue cluster and the second container in the second message queue cluster is used to unidirectionally synchronize the messages received and stored by the second message queue from the business service to the first message queue cluster.

4. The method according to claim 1, characterized in that The step of writing a first storage offset configured for the first message into the first message includes: Writing a first storage offset configured for the first message into a message header of the first message; or, A first message offset configured for the first message is written into an attribute feature of the first message.

5. The method according to claim 1, characterized in that The method further comprises: After obtaining the first storage offset of the first message through the second message queue cluster, it is determined whether the first storage offset is greater than the maximum storage offset in the second message queue cluster. If so, the first message is stored in the second message queue cluster according to the first storage offset; if not, the first message is discarded.

6. The method according to claim 1, characterized in that The receiving a first message generated in the business service includes: Receiving a first message generated in a business service corresponding to the first message queue cluster; The method further comprises: receiving a second message generated in the business service corresponding to the second message queue cluster, and writing a second storage offset configured for the second message into the second message, wherein the storage offset in the second message queue increases as the message is stored; A second message carrying a second storage offset is stored in the second message queue cluster, and the second message in the second message queue cluster is synchronously stored in the first message queue cluster through a dedicated data synchronization service. The first message queue cluster supports message backup of the second message queue cluster.

7. The method according to claim 1, characterized in that The method further comprises: Acquire first consumption information generated by the first message queue cluster, where the first consumption information is used to indicate messages consumed by the first message queue cluster; wherein a third container and a fourth container are configured in the first message queue cluster, and the fourth container is configured as a shadow container of the third container, and is used to cache part of the messages updated in the third container; Storing the first consumption information in a third container in the first message queue cluster; If the first consumption information meets the preset synchronization condition, the first consumption information is stored in the fourth container in the first message queue cluster, and the first consumption information in the fourth container is synchronously stored in the second message queue cluster through a dedicated data synchronization service.

8. The method according to claim 1, characterized in that The method further comprises: The message consumption request is responded to by a first message queue cluster deployed in the first area, where the first message queue cluster supports consumption of messages generated by business services corresponding to each area.

9. The method according to claim 1, characterized in that: The method further comprises: Receive a message consumption request for a specified topic, where the message consumption request is used to request consumption of messages stored in a message queue cluster deployed in any of multiple regions. The message consumption request is responded to by a specified message queue cluster. Topics with the same name in different regions support message consumption from a message queue cluster deployed in any region. The topic is used to indicate the message category stored in the message queue cluster.

10. A message processing device based on a message queue system, characterized in that: include: A first receiving unit, configured to receive a first message generated in a business service, and write a first storage offset configured for the first message into the first message, wherein the storage offset in the first message queue increases as the message is stored; The first storage unit is used to store a first message carrying a first storage offset in the first message queue cluster and synchronously store the first message in the first message queue cluster to the second message queue cluster through a dedicated data synchronization service, the second message queue cluster is used to store the first message according to the first storage offset, and the second message queue cluster supports message backup of the first message queue cluster.

11. An electronic device, characterized in that: The device comprises: a processor and a memory; The memory is used to store instructions or computer programs; The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the method according to any one of claims 1 to 9.

12. A computer-readable medium, characterized in that The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that It comprises a computer program carried on a non-transitory computer-readable medium, the computer program comprising a program code for executing the method according to any one of claims 1 to 9.

Images