Message processing method and device and message system
By switching to the Redis cluster to process messages when the MQ cluster fails and resend them to the MQ cluster after recovery, the reliability and fault tolerance of message processing in the event of MQ cluster failure is solved, ensuring high availability of the business system.
Patent Information
- Application Number
- CN202410996700.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-29
AI Technical Summary
In distributed systems, when a message queue (MQ) cluster fails, how to quickly and reliably process messages to ensure high availability and high fault tolerance of the business system and avoid business interruptions and data loss.
When the MQ cluster is unavailable, disconnect and establish a connection to the Redis cluster to publish a message to the target message channel in the Redis cluster; when the MQ cluster is restored to be available, obtain the message from the Redis cluster and resend it to the MQ cluster when the MQ cluster is re-issued. , disconnect Redis connection and re-establish the MQ cluster connection.
During the failure of the MQ cluster, the release and consumption of messages is realized through the Redis cluster, ensuring that messages are resent to the MQ cluster after failure recovery, avoiding the inability to obtain messages during the failure, and improving the reliability and fault tolerance of the system .
Smart Images

Figure CN119046027A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a message processing method, device and message system. Background Art
[0002] At present, in distributed systems, MQ (Message Queue) clusters play a vital role. They can not only improve the scalability and flexibility of distributed systems, but also significantly reduce the coupling between distributed system components, making distributed systems more robust and easier to maintain. However, the risk of failure of MQ clusters also comes with it, which poses a major challenge to business systems that are highly dependent on message transmission.
[0003] In key business systems such as financial transactions, order processing, and real-time data analysis, the reliability and real-time performance of messages are crucial. These business systems usually require MQ clusters to ensure high availability and high fault tolerance to ensure that services can be quickly restored when MQ cluster failures occur, avoiding business interruptions and data loss. Therefore, a fast and reliable message processing solution is urgently needed during MQ cluster failures. Summary of the invention
[0004] Based on this, it is necessary to provide a message processing method, device and message system to address the above technical problems.
[0005] In a first aspect, a method for processing a message is provided, the method being applied to a message producer, the method comprising:
[0006] When it is detected that the MQ cluster is unavailable, disconnect the connection with the MQ cluster, establish a connection with the Redis cluster, and publish a message to a target message channel in the Redis cluster; the target message channel is a message channel subscribed by a message consumer;
[0007] When it is detected that the MQ cluster is restored to an available state, the message published to the target message channel is obtained from the Redis cluster, the message published to the target message channel is sent to the MQ cluster, the connection with the Redis cluster is disconnected, and the connection with the MQ cluster is re-established.
[0008] As an optional implementation manner, obtaining the message published to the target message channel from the Redis cluster, and sending the message published to the target message channel to the MQ cluster includes:
[0009] Create a recovery message consumer subscribing to the target message channel;
[0010] Obtaining the message published to the target message channel from the Redis cluster through the recovery message consumer;
[0011] The message published to the target message channel is sent to the MQ cluster through the recovery message consumer.
[0012] As an optional implementation, before publishing a message to a target message channel in the Redis cluster, the method further includes:
[0013] Determine whether the target message channel exists in the Redis cluster;
[0014] If the target message channel does not exist in the Redis cluster, the target message channel is created in the Redis cluster; otherwise, the step of publishing a message to the target message channel in the Redis cluster is performed.
[0015] In a second aspect, a message processing method is provided, the method being applied to a message consumer, the method comprising:
[0016] When it is detected that the MQ cluster is unavailable, disconnect from the MQ cluster, establish a connection with the Redis cluster, subscribe to the target message channel in the Redis cluster, and receive messages published by the message producer to the target message channel;
[0017] When it is detected that the MQ cluster is restored to an available state and all messages under the target message channel are received, the connection with the Redis cluster is disconnected, the connection with the MQ cluster is re-established, and the messages sent by the MQ cluster are received.
[0018] As an optional implementation manner, after receiving the message sent by the MQ cluster, the method further includes:
[0019] Determine whether the message identifier list in the Redis cluster contains the message identifier corresponding to the message sent by the MQ cluster; wherein the message identifier list stores the message identifier corresponding to the message published by the message producer to the target message channel when the MQ cluster is unavailable;
[0020] If the message identifier list does not contain the message identifier corresponding to the message sent by the MQ cluster, the message sent by the MQ cluster is processed; otherwise, the message sent by the MQ cluster is discarded.
[0021] In a third aspect, a message processing device is provided, the device is applied to a message producer, and the device includes:
[0022] A publishing module is used to disconnect from the MQ cluster, establish a connection with the Redis cluster, and publish a message to a target message channel in the Redis cluster when detecting that the MQ cluster is unavailable; the target message channel is a message channel subscribed by a message consumer;
[0023] A connection module is used to obtain the message published to the target message channel from the Redis cluster when it is detected that the MQ cluster has recovered to an available state, send the message published to the target message channel to the MQ cluster, disconnect from the Redis cluster, and re-establish the connection with the MQ cluster.
[0024] As an optional implementation manner, the connection module is specifically used to:
[0025] Create a recovery message consumer subscribing to the target message channel;
[0026] Obtaining the message published to the target message channel from the Redis cluster through the recovery message consumer;
[0027] The message published to the target message channel is sent to the MQ cluster through the recovery message consumer.
[0028] As an optional implementation, the device further includes:
[0029] A judgment module is used to judge whether the target message channel exists in the Redis cluster;
[0030] The determination module is used to create the target message channel in the Redis cluster if the target message channel does not exist in the Redis cluster; otherwise, execute the step of publishing a message to the target message channel in the Redis cluster.
[0031] In a fourth aspect, a message processing device is provided, the device is applied to a message consumer, and the device includes:
[0032] A first receiving module is used to disconnect from the MQ cluster, establish a connection with the Redis cluster, subscribe to a target message channel in the Redis cluster, and receive messages published by a message producer to the target message channel when detecting that the MQ cluster is unavailable;
[0033] The second receiving module is used to disconnect from the Redis cluster, re-establish the connection with the MQ cluster, and receive messages sent by the MQ cluster when it is detected that the MQ cluster is restored to an available state and all messages under the target message channel are received.
[0034] As an optional implementation, the device further includes:
[0035] A judgment module, used to judge whether the message identifier list in the Redis cluster contains the message identifier corresponding to the message sent by the MQ cluster; wherein the message identifier list stores the message identifier corresponding to the message published by the message producer to the target message channel when the MQ cluster is unavailable;
[0036] The determination module is used to process the message sent by the MQ cluster if the message identifier list does not contain the message identifier corresponding to the message sent by the MQ cluster; otherwise, discard the message sent by the MQ cluster.
[0037] In a fifth aspect, a message system is provided, the message system comprising a message queue MQ cluster, a Redis cluster, a message producer and a message consumer;
[0038] The message producer executes any one of the methods described in the first aspect; and the message consumer executes any one of the methods described in the second aspect.
[0039] In a sixth aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement the method described in any one of the first aspect or the second aspect when executing the program stored in the memory.
[0040] In a seventh aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method described in any one of the first aspect or the second aspect is implemented.
[0041] The present application provides a message processing method, device and message system. The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:
[0042] When the message producer detects that the MQ cluster is unavailable, it disconnects from the MQ cluster, establishes a connection with the Redis cluster, and publishes messages to the target message channel in the Redis cluster. The target message channel is the message channel subscribed by the message consumer. When the message producer detects that the MQ cluster has recovered to an available state, it obtains the message published to the target message channel from the Redis cluster, sends the message published to the target message channel to the MQ cluster, disconnects from the Redis cluster, and re-establishes the connection with the MQ cluster. In this way, during the MQ cluster failure, message publishing and consumption are realized based on the Redis cluster, and after the MQ cluster failure is restored, the messages published to the Redis cluster are resent to the MQ cluster, so as to avoid the new message consumers being unable to obtain the messages published by the message producer during the MQ cluster failure.
[0043] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 An architectural diagram of a message system provided in an embodiment of the present application;
[0046] Figure 2 A flowchart of a message processing method provided in an embodiment of the present application;
[0047] Figure 3 A flowchart of a message processing method provided in an embodiment of the present application;
[0048] Figure 4 A flowchart of an example of a message processing method provided in an embodiment of the present application;
[0049] Figure 5 A schematic diagram of the structure of a message processing device provided in an embodiment of the present application;
[0050] Figure 6 A schematic diagram of the structure of a message processing device provided in an embodiment of the present application;
[0051] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0053] In order to facilitate the understanding of the embodiments of the present application, the Redis in the embodiments of the present application is first introduced. (1) Redis's Publish / Subscribe (Pub / Sub) mechanism: Redis's publish / subscribe function is based on the concept of channels. Any number of clients can subscribe to one or more channels to receive messages. Each channel corresponds to a topic. When a client publishes a message to a channel, all clients subscribed to the channel will receive the message. This mechanism is naturally suitable for point-to-point or point-to-multipoint message transmission scenarios. (2) Memory priority processing: Redis stores data in memory, providing extremely low latency and high throughput, which is very beneficial for the real-time requirements of message delivery. Although it is not designed for persistent messages, through AOF (Append Only File) or RDB (Redis Data Base) persistence, it can ensure that messages are not lost to a certain extent. (3) Temporary architecture design principles: When designing a temporary solution, follow the principle of "minimum change, maximum benefit", minimize the intrusion into the existing system architecture, ensure smooth cutback after the MQ system is restored, and reduce business and technical debt.
[0054] A message processing method provided in an embodiment of the present application can be applied to a message system, and specifically, can be applied to message producers and message consumers in a message system. Figure 1 This is an architecture diagram of a message system provided in an embodiment of the present application. Figure 1As shown, the message system includes an MQ cluster 110, a Redis cluster 120, a message producer 130, and a message consumer 140. Among them, the Redis cluster version can use 4.0 and above, supports the publish / subscribe mode, selects AOF+RDB as the persistence method, and the cache strategy setting in the cluster never expires. When it is detected that the MQ cluster 110 is unavailable, the message producer 130 disconnects from the MQ cluster 110, establishes a connection with the Redis cluster 120, and publishes a message to the target message channel in the Redis cluster 120. When it is detected that the MQ cluster 110 is unavailable, the message consumer 140 disconnects from the MQ cluster 110, establishes a connection with the Redis cluster 120, subscribes to the target message channel in the Redis cluster 120, and receives the message published by the message producer 130 to the target message channel. When it is detected that the MQ cluster 110 has recovered to an available state, the message producer 130 obtains the message published to the target message channel from the Redis cluster 120, sends the message published to the target message channel to the MQ cluster 110, disconnects from the Redis cluster 120, and re-establishes the connection with the MQ cluster 110. When it is detected that the MQ cluster 110 has recovered to an available state and all messages under the target message channel have been received, the message consumer 140 disconnects from the Redis cluster 120, re-establishes the connection with the MQ cluster 110, and receives the message sent by the MQ cluster 110.
[0055] It should be noted that the MQ cluster is a system for message publishing and subscription. The MQ system can also be generally referred to as a message middleware or a message queue system. Common MQ clusters may include Apache Kafka, ActiveMQ, RocketMQ, Amazon SQS, and the like.
[0056] The following will describe in detail a message processing method provided by an embodiment of the present application in conjunction with a specific implementation method. Figure 2 A flowchart of a message processing method provided in an embodiment of the present application, the method is applied to a message producer, such as Figure 2 As shown, the specific steps are as follows:
[0057] Step 201, when it is detected that the MQ cluster is unavailable, disconnect from the MQ cluster, establish a connection with the Redis cluster, and publish a message to a target message channel in the Redis cluster, where the target message channel is a message channel subscribed by a message consumer.
[0058] In implementation, the message producer can monitor the status of the MQ cluster in real time. Specifically, the message producer can monitor the status of the message storage subcluster (i.e., broker subcluster) in the MQ cluster in real time. When the message producer detects that the MQ cluster is unavailable, it means that the MQ cluster has failed. Accordingly, the message producer can disconnect from the MQ cluster and establish a connection with the backup cluster of the MQ cluster (i.e., Redis cluster). Then, the message producer can publish messages to the target message channel subscribed by the message consumer in the Redis cluster. In this way, in the event of a MQ cluster failure, the message consumer can receive the message published by the message producer through the Redis cluster.
[0059] It should be noted that when the message producer detects that the MQ cluster is unavailable, the message producer can first mark the MQ cluster as unavailable and then reconnect to the MQ cluster. If the preset number of retries is reached and the MQ cluster still cannot be reconnected, the message producer determines that the MQ cluster is unavailable.
[0060] As an optional implementation, before the message producer publishes a message to the target message channel in the Redis cluster, the message producer further performs the following steps:
[0061] Step 1: Determine whether the target message channel exists in the Redis cluster.
[0062] In implementation, before a message producer publishes a message to a target message channel in a Redis cluster, the message producer may determine whether the target message channel exists in the Redis cluster.
[0063] Step 2: If the target message channel does not exist in the Redis cluster, create the target message channel in the Redis cluster. Otherwise, execute the step of publishing a message to the target message channel in the Redis cluster.
[0064] In implementation, if the target message channel does not exist in the Redis cluster, the message producer can create the target message channel in the Redis cluster. If the target message channel exists in the Redis cluster, the message producer can directly publish messages to the target message channel in the Redis cluster.
[0065] As an optional implementation, after the message producer publishes a message to a target message channel in a Redis cluster, the message producer further performs the following steps: storing a message identifier corresponding to the message published to the target message channel in a message identifier list in the Redis cluster.
[0066] In implementation, after the MQ cluster is restored to an available state, the message producer needs to synchronize the messages published to the Redis cluster during the MQ cluster failure to the MQ cluster. Subsequently, the MQ cluster will send the synchronized messages to the message consumers. For message consumers that have received messages published by the message producer to the Redis cluster during the MQ cluster failure, they will receive the same message repeatedly. Therefore, in order to avoid the above problems, after the message producer publishes a message to the target message channel in the Redis cluster, the message identifier (message ID) corresponding to the message published to the target message channel can be further stored in the message identifier list (unque_list) in the Redis cluster. Subsequently, after the message consumer receives the message sent by the MQ cluster, it can determine whether the message sent by the MQ cluster is repeated based on the message identifier list.
[0067] Step 202, when it is detected that the MQ cluster is restored to an available state, the message published to the target message channel is obtained from the Redis cluster, the message published to the target message channel is sent to the MQ cluster, and the connection with the Redis cluster is disconnected, and the connection with the MQ cluster is re-established.
[0068] In implementation, when the message producer detects that the MQ cluster has recovered to an available state, it indicates that the MQ cluster failure has been recovered. Accordingly, the message producer can obtain the message published to the target message channel from the Redis cluster, send the message published to the target message channel to the MQ cluster, disconnect from the Redis cluster, and re-establish the connection with the MQ cluster. After the message producer sends all the messages published to the target message channel to the MQ cluster, the message producer can re-establish the connection with the MQ cluster and can send messages to the MQ cluster. In this way, the message producer restores the messages published to the target message channel to the MQ cluster, which can ensure that the message consumers who have not received the messages published by the message producer to the Redis cluster during the MQ cluster failure can receive the messages published by the message producer during the MQ cluster failure.
[0069] It should be noted that before the message producer sends all the messages published to the target message channel to the MQ cluster, the message producer still publishes messages to the target message channel in the Redis cluster.
[0070] As an optional implementation, the message producer obtains the message published to the target message channel from the Redis cluster, and sends the message published to the target message channel to the MQ cluster. The processing process is as follows:
[0071] Step 1: Create a recovery message consumer that subscribes to the target message channel.
[0072] During implementation, when the message producer detects that the MQ cluster is restored to an available state, the message producer may create a recovery message consumer subscribed to a target message channel.
[0073] Step 2: Recover the message consumer and obtain the message published to the target message channel from the Redis cluster.
[0074] In implementation, after the message producer creates a recovery message consumer that subscribes to the target message channel, the message producer can obtain the message published to the target message channel from the Redis cluster through the recovery message consumer. Specifically, the recovery message consumer pulls the message published by the message producer to the target message channel from the Redis cluster.
[0075] Step 3: Send the message published to the target message channel to the MQ cluster by restoring the message consumer.
[0076] In implementation, the recovered message consumer may send the received message to the message producer, which then sends the message to the MQ cluster.
[0077] The following will describe in detail a message processing method provided by an embodiment of the present application in conjunction with a specific implementation method. Figure 3 A flowchart of a message processing method provided in an embodiment of the present application, the method is applied to a message consumer, such as Figure 3 As shown, the specific steps are as follows:
[0078] Step 301, when it is detected that the MQ cluster is unavailable, disconnect from the MQ cluster, establish a connection with the Redis cluster, subscribe to a target message channel in the Redis cluster, and receive messages published by a message producer to the target message channel.
[0079] In implementation, message consumers can monitor the status of the MQ cluster in real time. Specifically, message consumers can monitor the status of the message storage subcluster (i.e., broker subcluster) in the MQ cluster in real time. When the message consumer detects that the MQ cluster is unavailable, it means that the MQ cluster has failed. Accordingly, the message consumer can disconnect from the MQ cluster and establish a connection with the Redis cluster to switch to the Redis cluster. The message consumer can subscribe to the target message channel in the Redis cluster and receive messages published by the message producer to the target message channel. In this way, in the event of a MQ cluster failure, the message consumer can receive messages published by the message producer through the Redis cluster.
[0080] It should be noted that the message consumer can continuously reconnect to the MQ cluster while receiving the messages published by the message producer to the target message channel.
[0081] Step 302: When it is detected that the MQ cluster is restored to an available state and all messages under the target message channel are received, the connection with the Redis cluster is disconnected, the connection with the MQ cluster is re-established, and the messages sent by the MQ cluster are received.
[0082] In implementation, when the message consumer detects that the MQ cluster has recovered to an available state, it indicates that the MQ cluster has recovered from the failure. Accordingly, after receiving all messages under the target message channel, the message consumer can disconnect from the Redis cluster and re-establish the connection with the MQ cluster to switch back to the MQ cluster. The message consumer can receive messages sent by the MQ cluster.
[0083] As an optional implementation, after the message consumer receives the message sent by the MQ cluster, the message consumer further performs the following steps:
[0084] Step 1: Determine whether the message identifier list in the Redis cluster contains the message identifier corresponding to the message sent by the MQ cluster. The message identifier list stores the message identifier corresponding to the message published by the message producer to the target message channel when the MQ cluster is unavailable.
[0085] In implementation, after the MQ cluster is restored to an available state, the message producer needs to synchronize the messages published to the Redis cluster during the MQ cluster failure to the MQ cluster. Subsequently, the MQ cluster will send the synchronized messages to the message consumers. For message consumers that have received messages published by message producers to the Redis cluster during the MQ cluster failure, they will receive the same message repeatedly. Therefore, in order to avoid the above problems, after the message consumer receives the message sent by the MQ cluster, it can determine whether the message identifier list in the Redis cluster contains the message identifier corresponding to the message sent by the MQ cluster. Among them, the message identifier list stores the message identifiers corresponding to the messages published by the message producer to the target message channel when the MQ cluster is unavailable.
[0086] Step 2: If the message identifier list does not contain the message identifier corresponding to the message sent by the MQ cluster, the message sent by the MQ cluster is processed. Otherwise, the message sent by the MQ cluster is discarded.
[0087] In implementation, if the message identifier list does not contain the message identifier corresponding to the message sent by the MQ cluster, it means that the message published by the MQ cluster is not a duplicate message. At this time, the message consumer can process the message sent by the MQ cluster. If the message identifier list contains the message identifier corresponding to the message sent by the MQ cluster, it means that the message published by the MQ cluster is a duplicate message. At this time, the message consumer can discard the message sent by the MQ cluster and return a consumption success indication to the MQ cluster.
[0088] Figure 4 This is a flowchart of an example of a message processing method provided in an embodiment of the present application. Figure 4 As shown, the specific processing process is as follows:
[0089] Step 401: When it is detected that the MQ cluster is unavailable, the message producer disconnects from the MQ cluster and establishes a connection with the Redis cluster.
[0090] Step 402: The message producer creates a target message channel in the Redis cluster and publishes a message to the target message channel in the Redis cluster, wherein the target message channel is a message channel subscribed by the message consumer.
[0091] Step 403: The Redis cluster stores the message identifier corresponding to the message published by the message producer to the target message channel in a message identifier list.
[0092] Step 404, when it is detected that the MQ cluster is unavailable, the message consumer disconnects from the MQ cluster, establishes a connection with the Redis cluster, subscribes to the target message channel in the Redis cluster, and receives the message published by the message producer to the target message channel.
[0093] Step 405: When it is detected that the MQ cluster is restored to an available state, the message producer creates a recovery message consumer subscribing to the target message channel.
[0094] Step 406: The message producer obtains the message published to the target message channel from the Redis cluster by restoring the message consumer, and sends the message published to the target message channel to the MQ cluster.
[0095] Step 407: The message producer disconnects from the Redis cluster, re-establishes the connection with the MQ cluster, and sends a message to the MQ cluster.
[0096] Step 408, when the message consumer detects that the MQ cluster has recovered to an available state and has received all messages under the target message channel, it disconnects from the Redis cluster, re-establishes the connection with the MQ cluster, and receives the messages sent by the MQ cluster.
[0097] Step 409, the message consumer determines whether the message identifier list in the Redis cluster contains the message identifier corresponding to the message sent by the MQ cluster. The message identifier list stores the message identifier corresponding to the message published by the message producer to the target message channel when the MQ cluster is unavailable. If it is included, the message consumer executes step 410. If not, the message consumer executes step 411.
[0098] Step 410: The message consumer discards the message sent by the MQ cluster.
[0099] Step 411: The message consumer processes the message sent by the MQ cluster.
[0100] The present application provides a method for processing messages. When the message producer detects that the MQ cluster is in an unavailable state, it disconnects from the MQ cluster, establishes a connection with the Redis cluster, and publishes a message to a target message channel in the Redis cluster. The target message channel is a message channel subscribed to by a message consumer. When the message consumer detects that the MQ cluster is in an unavailable state, it disconnects from the MQ cluster, establishes a connection with the Redis cluster, subscribes to the target message channel in the Redis cluster, and receives messages published by the message producer to the target message channel. Afterwards, when the message producer detects that the MQ cluster has recovered to an available state, it obtains the message published to the target message channel from the Redis cluster, sends the message published to the target message channel to the MQ cluster, disconnects from the Redis cluster, and re-establishes the connection with the MQ cluster. When the message consumer detects that the MQ cluster has recovered to an available state and has received all the messages under the target message channel, it disconnects from the Redis cluster, re-establishes the connection with the MQ cluster, and receives the message sent by the MQ cluster. In this way, during the MQ cluster failure process, the publishing and consumption of messages are realized based on the Redis cluster. At the same time, after the MQ cluster failure is restored, the message producer will resend the messages published to the Redis cluster to the MQ cluster to prevent the new message consumers from being unable to obtain the messages published by the message producer during the MQ cluster failure.
[0101] It should be understood that although Figure 2 , Figure 3 and Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , Figure 3 and Figure 4 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.
[0102] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.
[0103] The embodiment of the present application also provides a message processing device, which is applied to a message producer, such as Figure 5 As shown, the device comprises:
[0104] The publishing module 510 is used to disconnect from the MQ cluster, establish a connection with the Redis cluster, and publish a message to a target message channel in the Redis cluster when detecting that the MQ cluster is unavailable; the target message channel is a message channel subscribed by the message consumer;
[0105] The connection module 520 is used to obtain the message published to the target message channel from the Redis cluster when it is detected that the MQ cluster has recovered to an available state, send the message published to the target message channel to the MQ cluster, disconnect from the Redis cluster, and re-establish the connection with the MQ cluster.
[0106] As an optional implementation, the connection module 520 is specifically configured to:
[0107] Create a recovery message consumer subscribed to the target message channel;
[0108] By restoring the message consumer, obtain the message published to the target message channel from the Redis cluster;
[0109] By restoring the message consumer, the messages published to the target message channel are sent to the MQ cluster.
[0110] As an optional implementation, the device further includes:
[0111] The judgment module is used to judge whether the target message channel exists in the Redis cluster;
[0112] The determination module is used to create a target message channel in the Redis cluster if the target message channel does not exist in the Redis cluster; otherwise, execute the step of publishing a message to the target message channel in the Redis cluster.
[0113] The embodiment of the present application also provides a message processing device, which is applied to a message consumer, such as Figure 6 As shown, the device comprises:
[0114] The first receiving module 610 is used to disconnect from the MQ cluster, establish a connection with the Redis cluster, subscribe to a target message channel in the Redis cluster, and receive a message published by a message producer to the target message channel when detecting that the MQ cluster is unavailable;
[0115] The second receiving module 620 is used to disconnect from the Redis cluster, re-establish the connection with the MQ cluster, and receive messages sent by the MQ cluster when it is detected that the MQ cluster is restored to an available state and all messages under the target message channel are received.
[0116] As an optional implementation, the device further includes:
[0117] A judgment module is used to judge whether the message identifier list in the Redis cluster contains the message identifier corresponding to the message sent by the MQ cluster; wherein the message identifier list stores the message identifier corresponding to the message published by the message producer to the target message channel when the MQ cluster is unavailable;
[0118] The determination module is used to process the message sent by the MQ cluster if the message identifier list does not contain the message identifier corresponding to the message sent by the MQ cluster; otherwise, the message sent by the MQ cluster is discarded.
[0119] For the specific definition of the message processing device, please refer to the definition of the message processing method above, which will not be repeated here. Each module in the above message processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0120] In one embodiment, a message system is provided, the message system comprising an MQ cluster, a Redis cluster, a message producer, and a message consumer;
[0121] Among them, the message producer executes the method steps of message processing implemented by the above message producer; the message consumer executes the method steps of message processing implemented by the above message consumer.
[0122] In one embodiment, an electronic device is provided, such as Figure 7As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704; the memory 703 is used to store computer programs; the processor 701 is used to implement the method steps for processing the above-mentioned message when executing the program stored in the memory 703.
[0123] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned message processing method are implemented.
[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0125] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0126] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0127] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0129] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A message processing method, characterized in that: The method is applied to a message producer, and the method comprises: When it is detected that the MQ cluster is unavailable, disconnect the connection with the MQ cluster, establish a connection with the Redis cluster, and publish a message to a target message channel in the Redis cluster; the target message channel is a message channel subscribed by a message consumer; When it is detected that the MQ cluster is restored to an available state, the message published to the target message channel is obtained from the Redis cluster, the message published to the target message channel is sent to the MQ cluster, the connection with the Redis cluster is disconnected, and the connection with the MQ cluster is re-established.
2. The method according to claim 1, characterized in that: The acquiring the message published to the target message channel from the Redis cluster, and sending the message published to the target message channel to the MQ cluster, includes: Create a recovery message consumer subscribing to the target message channel; Obtaining the message published to the target message channel from the Redis cluster through the recovery message consumer; The message published to the target message channel is sent to the MQ cluster through the recovery message consumer.
3. The method according to claim 1, characterized in that Before publishing a message to a target message channel in the Redis cluster, the method further includes: Determine whether the target message channel exists in the Redis cluster; If the target message channel does not exist in the Redis cluster, the target message channel is created in the Redis cluster; otherwise, the step of publishing a message to the target message channel in the Redis cluster is performed.
4. A message processing method, characterized in that: The method is applied to a message consumer, and the method comprises: When it is detected that the MQ cluster is unavailable, disconnect from the MQ cluster, establish a connection with the Redis cluster, subscribe to the target message channel in the Redis cluster, and receive messages published by the message producer to the target message channel; When it is detected that the MQ cluster is restored to an available state and all messages under the target message channel are received, the connection with the Redis cluster is disconnected, the connection with the MQ cluster is re-established, and the messages sent by the MQ cluster are received.
5. The method according to claim 4, characterized in that After receiving the message sent by the MQ cluster, the method further includes: Determine whether the message identifier list in the Redis cluster contains the message identifier corresponding to the message sent by the MQ cluster; wherein the message identifier list stores the message identifier corresponding to the message published by the message producer to the target message channel when the MQ cluster is unavailable; If the message identifier list does not contain the message identifier corresponding to the message sent by the MQ cluster, the message sent by the MQ cluster is processed; otherwise, the message sent by the MQ cluster is discarded.
6. A message processing device, characterized in that: The device is applied to a message producer, and comprises: A publishing module is used to disconnect from the MQ cluster, establish a connection with the Redis cluster, and publish a message to a target message channel in the Redis cluster when detecting that the MQ cluster is unavailable; the target message channel is a message channel subscribed by a message consumer; The sending module is used to obtain the message published to the target message channel from the Redis cluster when it is detected that the MQ cluster has recovered to an available state, send the message published to the target message channel to the MQ cluster, disconnect from the Redis cluster, and re-establish the connection with the MQ cluster.
7. A message processing device, characterized in that: The device is applied to a message consumer, and the device comprises: A first receiving module is used to disconnect from the MQ cluster, establish a connection with the Redis cluster, subscribe to a target message channel in the Redis cluster, and receive messages published by a message producer to the target message channel when detecting that the MQ cluster is unavailable; The second receiving module is used to disconnect from the Redis cluster, re-establish the connection with the MQ cluster, and receive messages sent by the MQ cluster when it is detected that the MQ cluster is restored to an available state and all messages under the target message channel are received.
8. A message system, characterized in that: The message system includes a message queue MQ cluster, a Redis cluster, a message producer and a message consumer; The message producer executes the method described in any one of claims 1 to 3; the message consumer executes the method described in any one of claims 4 to 5.
9. An electronic device, characterized in that: The invention comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the method described in any one of claims 1-3 or 4-5 when executing the program stored in the memory.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 or 4 to 5 is implemented.
