Repeated operation prevention method based on Redis when RabbitMQ multiple steps are used

Abstract

The invention relates to the field of RabbitMq message queues, and discloses a method for preventing repeated operation based on Redis when multiple steps of RabbitMQ are used, and the method comprises the following steps: firstly, generating a unique identifier mqmsgid for each message monitored by an mq consumer; converting a hook file which can be operated by the resident queue task into key-gt; a value array is used; the key-gt of the hook file conversion is circularly processed; a value array is used; after the operation of the current key is completed, the redi cache updates the mqmsgid processed key; when the whole message consumption is finished, clearing the redi cache corresponding to the mqmsgid; according to the method and the device, the problems that the unprocessed message cannot repeatedly process the processed logic when the task is restarted, and the effects of preventing repeated data operation, repeated message sending, repeated statistical calculation and the like are achieved.

Description

Technical Field

[0001] The present invention relates to the field of RabbitMQ message queues, and in particular to a Redis-based anti-duplicate operation method when using RabbitMQ multi-step. Background Art

[0002] Key-value data storage, as a simple and efficient data storage method, is widely used in mobile applications, embedded systems, and distributed environments. It is suitable for storing lightweight, unstructured data such as application configuration, user settings, and cached data. On the Android platform, SharedPreferences is the most commonly used key-value storage tool, which can save application configuration information and user preferences in the form of key-value pairs.

[0003] Currently, RocketMQ queues are often used for message push. RocketMQ queues are a message middleware with a queue model. They feature high performance, high reliability, high real-time performance, and distribution. However, they are not open source and cannot achieve business decoupling. Message sending failures may lead to an infinite loop.

[0004] Although there are some improvement solutions in the existing technology, such as using SQLite database and encrypted storage, these methods are often independent solutions to specific problems and lack flexibility and uniformity. Since the RabbitMq source code does not have logic for whether to repeat the processing of multi-step hooks, when a message is restarted during multi-step processing, the processed step logic will be re-run, resulting in repeated data storage, repeated data modification, overwriting other data, repeated message sending, repeated statistical calculations and other problems. Summary of the Invention

[0005] The present invention aims to provide a Redis-based anti-duplicate operation method when using RabbitMQ multi-step, which solves the problem that unprocessed completed messages will not repeat the processed logic when the task is restarted, and has the effects of preventing repeated operation of data, repeated sending of messages, and repeated statistical calculations.

[0006] In order to achieve the above object, the present invention provides the following method: The present invention provides a Redis-based anti-duplicate operation method when using RabbitMQ multi-step: S1: Generate a unique identifier mq_msg_id for each message monitored by the mq consumer; S2: Convert the hook file that can be run by the resident queue task into a key->value array; S3: looping through the key->value array of the hook file conversion; S4: After the current key is completed, the redi cache updates the mq_msg_id processed key; S5: When the entire message consumption is completed, clear the redi cache corresponding to the mq_msg_id.

[0007] Preferably, the unique identifier mq_msg_id is generated by converting the request parameters into a json string, concatenating the current timestamp, and then performing md5 processing on the resulting string.

[0008] Preferably, the step of looping through the key->value array of the hook file conversion includes: determining in each loop step whether there is a key cache for the current mq_msg_id of redis; if the key cache exists, the hook file logic program that is less than or equal to the key of the cache itself is skipped and not run; if the key cache does not exist, the redis cache has processed the key when the current hook logic is run.

[0009] Preferably, the method further includes deleting a message queue, which specifically includes: in the online environment where RabbitMq is served, detecting whether there are consumers and backlog messages in the message queue, and if neither exists, deleting the message queue and deleting the queue information corresponding to the database entity.

[0010] Preferably, it also includes taking the production message queue offline. The production message queue offline specifically includes detecting whether there are consumers and backlog messages in the production message queue. If neither exists, it is determined whether the production message queue is in an online environment. If so, the production message queue is taken offline and the queue information corresponding to the database entity is taken offline.

[0011] Preferably, while performing step S3, a preset cluster configuration file is obtained, and all cluster nodes in the preset cluster configuration file are parsed to obtain; wherein, all cluster nodes include preset master cluster nodes and slave cluster nodes; the RabbitMQ service and its installation package are pushed to all cluster nodes, and the RabbitMQ service is installed for each cluster node; the RabbitMQ service is started for the preset master cluster node, and a corresponding communication file is generated; the communication file is sent to all slave cluster nodes, and the RabbitMQ service of each slave cluster node is started; the join cluster operation is performed on the slave cluster node to complete the cluster deployment.

[0012] Preferably, before installing the RabbitMQ service for each cluster node, the method further includes: detecting whether the cluster node has installed the RabbitMQ service, and determining whether to start the step of installing the RabbitMQ service for each cluster node based on the detection result.

[0013] Preferably, the communication file is a cookie file.

[0014] Preferably, the performing of the join-cluster operation on the slave cluster node is to perform the join-cluster operation on the slave cluster node using a join_cluster command.

[0015] Preferably, the preset cluster configuration file is parsed to obtain the attribute information of the cluster node in the preset cluster configuration file; it is determined whether the attribute information is a preset default attribute; if not, the attributes of the cluster node are configured according to the attribute information using a preset attribute change command.

[0016] The beneficial effects of the present invention are reflected in that the present invention caches the keys of processed hooks in multiple hooks when the RabbitMq task is restarted by using redis cache, which solves the problem that unprocessed messages will not be repeatedly processed during restart, and has the effects of preventing repeated operation of data, repeated sending of messages, and repeated statistical calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly describes the drawings required for the specific embodiments or the description of the prior art. Similar elements or parts are generally identified by similar reference numerals throughout the drawings. Elements or parts in the drawings are not necessarily drawn to scale.

[0018] Figure 1 A flowchart of a Redis-based anti-duplicate operation method when using RabbitMQ multi-step is provided in an embodiment of the present invention; Figure 2 A schematic diagram of a message sending process for a Redis-based anti-duplicate operation method when using RabbitMQ multi-step, provided by an embodiment of the present invention; Figure 3 The present invention provides a flowchart of a method for sending, receiving and processing messages based on Redis to prevent repeated operations when using RabbitMQ multiple steps. DETAILED DESCRIPTION

[0019] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0020] The terms "first," "second," and so on, in the description and claims of the present invention and the accompanying drawings are used to distinguish between different items, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or end comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed therein, or may optionally include other steps or elements inherent to such process, method, product, or end.

[0021] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0022] Currently, RocketMQ queues are often used for message push. RocketMQ queues are a message middleware with a queue model. They feature high performance, high reliability, high real-time performance, and distribution. However, they are not open source and cannot achieve business decoupling. Message sending failures may lead to an infinite loop.

[0023] Although there are some improvement solutions in the existing technology, such as using SQLite database and encrypted storage, these methods are often independent solutions to specific problems and lack flexibility and uniformity. Since the RabbitMq source code does not have logic for whether to repeat the processing of multi-step hooks, when a message is restarted during multi-step processing, the processed step logic will be re-run, resulting in repeated data storage, repeated data modification, overwriting other data, repeated message sending, repeated statistical calculations and other problems.

[0024] The present invention aims to provide a Redis-based anti-duplicate operation method when using RabbitMQ multi-step, which solves the problem that unprocessed completed messages will not repeat the processed logic when the task is restarted, and has the effects of preventing repeated operation of data, repeated sending of messages, and repeated statistical calculations.

[0025] like Figure 1-3 As shown, a specific embodiment of the present invention provides a Redis-based anti-duplicate operation method when using RabbitMQ multi-step, including the following steps: S1: Generate a unique identifier mq_msg_id for each message monitored by the mq consumer.

[0026] In an embodiment of the present invention, the unique identifier mq_msg_id is generated by converting the request parameter into a JSON string, adding the current timestamp, and then performing MD5 processing on the resulting string.

[0027] S2: Convert the hook file that can be run by the resident queue task into a key->value array.

[0028] In an embodiment of the present invention, the step of looping the key->value array converted by the hook file includes: judging whether there is a key cache for the current mq_msg_id in redis in each loop; if there is a key cache, the hook file logic program with a key less than or equal to the cache itself is skipped and not run; if there is no key cache, when the current hook logic is finished running, the redis cache has processed the key S3: The key->value array for loop processing hook file conversion.

[0029] In an embodiment of the present invention, a preset cluster configuration file is obtained, and all cluster nodes in the preset cluster configuration file are parsed to obtain the preset master cluster node and slave cluster nodes; the RabbitMQ service and its installation package are pushed to all cluster nodes, and the RabbitMQ service is installed for each cluster node; the RabbitMQ service is started for the preset master cluster node, and a corresponding communication file is generated; the communication file is sent to all slave cluster nodes, and the RabbitMQ service of each slave cluster node is started; a join cluster operation is performed on the slave cluster node to complete the cluster deployment; before installing the RabbitMQ service for each cluster node, it also includes: detecting whether the cluster node has installed the RabbitMQ service, and determining whether to start the step of installing the RabbitMQ service for each cluster node based on the detection result; the communication file is a cookie file; performing a join cluster operation on the slave cluster node is to perform a join cluster operation on the slave cluster node using the join_cluster command; parsing the preset cluster configuration file to obtain the attribute information of the cluster node in the preset cluster configuration file; judging whether the attribute information is the preset default attribute; if not, configuring the attributes of the cluster node based on the attribute information using the preset attribute change command.

[0030] S4: After the current key is completed, the redi cache updates the mq_msg_id processed key.

[0031] S5: When the entire message consumption is completed, clear the redi cache corresponding to mq_msg_id.

[0032] In an embodiment of the present invention, it also includes deleting a message queue. Deleting a message queue specifically includes: in the online environment of the service where RabbitMq is located, detecting whether there are consumers and backlogged messages in the message queue. If neither exists, deleting the message queue and deleting the queue information corresponding to the database entity; it also includes taking the production message queue offline. Taking the production message queue offline specifically includes: detecting whether there are consumers and backlogged messages in the production message queue. If neither exists, determining whether the production message queue is in the online environment. If so, taking the production message queue offline and taking the queue information corresponding to the database entity offline.

[0033] The beneficial effects of the present invention are reflected in that the present invention caches the keys of processed hooks in multiple hooks when the RabbitMq task is restarted by using redis cache, which solves the problem that unprocessed messages will not be repeatedly processed during restart, and has the effects of preventing repeated operation of data, repeated sending of messages, and repeated statistical calculations.

[0034] The above description is merely an embodiment of the present invention. Common knowledge such as the specific technical solutions or features of the solutions is not described in detail here. It should be noted that those skilled in the art may make several modifications and improvements without departing from the solution of the present invention, and these modifications and improvements should also be considered as the scope of protection of the present invention. These modifications and improvements will not affect the effects of the present invention and the practicality of the patent. The scope of protection claimed in this application shall be based on the content of the claims, and the specific embodiments and other descriptions in the specification may be used to interpret the content of the claims.

Claims

1. A method for preventing repeated operations based on Redis when using RabbitMQ multiple steps, characterized in that: The method comprises: S1: Generate a unique identifier mq_msg_id for each message monitored by the mq consumer; S2: Convert the hook file that can be run by the resident queue task into a key->value array; S3: looping through the key->value array of the hook file conversion; S4: After the current key is completed, the redi cache updates the mq_msg_id processed key; S5: When the entire message consumption is completed, clear the redi cache corresponding to the mq_msg_id.

2. The method for preventing repeated operations based on Redis when using RabbitMQ multiple steps according to claim 1, characterized in that: The unique identifier mq_msg_id is generated by converting the request parameters into a json string, adding the current timestamp, and then performing md5 processing on the resulting string.

3. The method for preventing repeated operations based on Redis when using RabbitMQ multiple steps according to claim 1, characterized in that: The step of looping through the key->value array of the hook file conversion includes: Determine whether the key cache exists for the current mq_msg_id in redis in each step of the loop; If the key cache exists, the hook file logic program with a key less than or equal to the cache itself will be skipped and not executed; If the key cache does not exist, the redis cache has processed the key when the current hook logic is run.

4. The method for preventing repeated operations based on Redis when using RabbitMQ multiple steps according to claim 1, characterized in that: It also includes deleting the message queue, which specifically includes: in the online environment where RabbitMq is served, detecting whether the message queue has consumers and backlog messages. If neither exists, deleting the message queue and deleting the queue information corresponding to the database entity.

5. The method for preventing repeated operations based on Redis when using RabbitMQ multiple steps according to claim 1, characterized in that: It also includes taking the production message queue offline. The production message queue offline specifically includes detecting whether there are consumers and backlog messages in the production message queue. If neither exists, determining whether the production message queue is in an online environment. If so, taking the production message queue offline and the queue information corresponding to the database entity offline.

6. A Redis-based method for preventing repeated operations when using RabbitMQ multiple steps according to any one of claims 1 to 5, characterized in that: While performing step S3, a preset cluster configuration file is obtained, and all cluster nodes in the preset cluster configuration file are parsed; wherein all cluster nodes include a preset master cluster node and a slave cluster node; Push the RabbitMQ service and its installation package to all cluster nodes, and install the RabbitMQ service for each cluster node; Start the RabbitMQ service for the preset master cluster node and generate corresponding communication files; Sending the communication file to all the slave cluster nodes and starting the RabbitMQ service of each of the slave cluster nodes; The slave cluster node is added to the cluster to complete the cluster deployment.

7. The method for preventing repeated operations based on Redis when using RabbitMQ multiple steps according to claim 6, characterized in that Before installing the RabbitMQ service for each cluster node, the method further includes: Detect whether the cluster node has installed the RabbitMQ service, and determine whether to start the step of installing the RabbitMQ service for each cluster node according to the detection result.

8. The method for preventing repeated operations based on Redis when using RabbitMQ multiple steps according to claim 7, characterized in that: The communication file is a cookie file.

9. The method for preventing repeated operations based on Redis when using RabbitMQ multiple steps according to claim 8, characterized in that: The performing the joining-cluster operation on the slave cluster node is performing the joining-cluster operation on the slave cluster node using a join_cluster command.

10. The method for preventing repeated operations based on Redis when using RabbitMQ multiple steps according to claim 9, characterized in that: Parsing the preset cluster configuration file to obtain attribute information of the cluster node in the preset cluster configuration file; Determining whether the attribute information is a preset default attribute; If not, configure the attributes of the cluster node using a preset attribute change command according to the attribute information.

Images