A distributed blocking queue implementation method

Abstract

The present invention belongs to the field of message processing technology and discloses a method for implementing a distributed blocking queue, comprising the following steps: Step 1: When a configuration item changes, each application instance updates its local configuration cache, then competes for a distributed lock and submits a global "data refresh" event to a message queue. The purpose of the distributed lock is to avoid duplicate events; Step 2: When an attribute of an advertising contract changes, the instance that executes the contract modification submits a contract "data refresh" event to the message queue; Step 3: The event data must include the event type. The solution adopted herein utilizes MySQL data tables and Redis distributed locks to implement a message queue, and updates the consumer's survival status at a fixed period through ScheduledExecutorService. A worker distributed lock is introduced for the consumer, meeting the business scenario's requirements for non-repeatable messages and sequential processing one by one. The implementation is simple and easy, and relatively few middleware are introduced, reducing the complexity of the system architecture.

Description

Technical Field

[0001] The invention belongs to the technical field of message processing, and in particular relates to an implementation method of a distributed blocking queue. Background Art

[0002] As the functions of arbitrary applications become increasingly complex, the number of configuration items in the applications also increases. There is an increasing need for configuration management permissions and review mechanisms, as well as the effective time, release method and effective scope granularity of the changes. Therefore, more and more applications or systems have introduced configuration management centers to meet the corresponding configuration management requirements.

[0003] Existing message queues are usually implemented using Kafka or Redis lists, and consumers use the corresponding Kafka clients and Redis clients. Although this can meet the "first in, first out" consumption strategy and avoid blocking message reading, for multi-instance scenarios, it is not easy for clients to block each other, and it is not easy to ensure that the next task is processed after the previous task is completed. In addition, if the client is accidentally interrupted or offline during the processing process, the unfinished task is not easy to retry or recover. In the process of using Kafka to implement message queues, due to the uncertainty of task time consumption, client rebalance is very likely to occur, which can easily cause repeated consumption of messages and waste of client computing resources. Therefore, it needs to be improved. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for implementing a distributed blocking queue to solve the problems raised in the above background technology.

[0005] In order to achieve the above object, the present invention provides the following technical solution: a method for implementing a distributed blocking queue, comprising the following steps:

[0006] Step 1: When a configuration item changes, each application instance updates its local configuration cache, competes for a distributed lock, and submits a global "data refresh" event to the message queue. The purpose of the distributed lock is to avoid duplicate events.

[0007] Step 2: After the properties of the advertising contract are changed, the instance that executes the contract modification submits the contract "data refresh" event to the message queue;

[0008] Step 3: Event data must include event type, trigger / creation time, data before change (configuration / contract attributes), event status (default is "pending"), and event unique key (optional UUID);

[0009] Step 4: Each consumer instance updates its survival flag through an independent background thread, such as ScheduledExecutorService in Java, at the same fixed period, such as every 5 seconds. The solution adopted here is to write a key-value pair to Redis and set an expiration time (30 seconds by default). The key structure is "instance IP address@instance MAC address" to uniquely identify the instance in the distributed architecture; the value can be any value, such as the current timestamp at the time of writing. If the instance goes offline unexpectedly, its survival flag will disappear after a maximum of 30 seconds.

[0010] Step 5: The task scheduling platform periodically polls each instance. The awakened instance first attempts to obtain a distributed lock (worker). If the lock is not obtained, that is, there is no consumer instance currently executing the task, the instance must immediately set a lock and use "instance IP address@instance MAC address" as the lock value. If the lock is obtained, the lock value must be extracted to check whether the corresponding instance survival mark exists to determine the other party's survival status. If the survival mark has disappeared, that is, the corresponding instance has gone offline, you can try to obtain the worker lock. If the survival mark exists, the corresponding instance is considered online, and this scheduling is completed. The open source scheduling platform XXL-JOB is used here, and other platforms with similar functions are also applicable. The distributed lock is implemented through Redis and does not specify an expiration time. Based on the above mechanism, it can be ensured that only one consumer instance is active.

[0011] Step 6: The consumer instance that successfully acquires the worker lock reads the earliest pending event from the message queue. If there are no pending events, the scheduling ends. Otherwise, the event status is updated to "processing" and the corresponding processing logic is completed according to the event type.

[0012] Step 7: After the consumer completes the processing, it exits successfully or throws an exception, updates the event status, and releases the worker lock.

[0013] Preferably, the distributed lock described in step 1 is mutually exclusive and has an anti-deadlock function.

[0014] Preferably, the content of the advertising contract described in step 2 is legal, and the spokesperson of the advertising contract is legal and compliant.

[0015] Preferably, the unique key described in step three is secure and orderly.

[0016] Preferably, the distributed architecture described in step 4 has high availability and strong fault tolerance.

[0017] Preferably, the task scheduling platform described in step five includes a scheduled scheduling time, and the task scheduling platform has a scheduling dependency.

[0018] Preferably, the instance survival flag described in step five has a length limit, and the instance survival flag has a special character limit.

[0019] Preferably, the XXL-JOB in step five includes a port number, and the XXL-JOB includes a log path.

[0020] Preferably, the Redis described in step 4 has a data persistence function, and the Redis does not support custom identity authentication.

[0021] The beneficial effects of the present invention are as follows:

[0022] The solution adopted here uses MySQL data tables and Redis distributed locks to implement message queues, and updates the consumer's survival status at a fixed period through ScheduledExecutorService. It introduces worker distributed locks for consumers, meeting the business scenario's requirements for non-repeatable messages and sequential processing one by one. It is simple and easy to implement, and relatively few middleware are introduced, reducing the complexity of the system architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a functional architecture diagram of the present invention. DETAILED DESCRIPTION

[0024] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0025] like Figure 1 As shown, an embodiment of the present invention provides a method for implementing a distributed blocking queue, comprising the following steps:

[0026] Step 1: When a configuration item changes, each application instance updates its local configuration cache, competes for a distributed lock, and submits a global "data refresh" event to the message queue. The purpose of the distributed lock is to avoid duplicate events.

[0027] Step 2: After the properties of the advertising contract are changed, the instance that executes the contract modification submits the contract "data refresh" event to the message queue;

[0028] Step 3: Event data must include event type, trigger / creation time, data before change (configuration / contract attributes), event status (default is "pending"), and event unique key (optional UUID);

[0029] Step 4: Each consumer instance updates its survival flag through an independent background thread, such as ScheduledExecutorService in Java, at the same fixed period, such as every 5 seconds. The solution adopted here is to write a key-value pair to Redis and set an expiration time (30 seconds by default). The key structure is "instance IP address@instance MAC address" to uniquely identify the instance in the distributed architecture; the value can be any value, such as the current timestamp at the time of writing. If the instance goes offline unexpectedly, its survival flag will disappear after a maximum of 30 seconds.

[0030] Step 5: The task scheduling platform periodically polls each instance. The awakened instance first attempts to obtain a distributed lock (worker). If the lock is not obtained, that is, there is no consumer instance currently executing the task, the instance must immediately set a lock and use "instance IP address@instance MAC address" as the lock value. If the lock is obtained, the lock value must be extracted to check whether the corresponding instance survival mark exists to determine the other party's survival status. If the survival mark has disappeared, that is, the corresponding instance has gone offline, you can try to obtain the worker lock. If the survival mark exists, the corresponding instance is considered online, and this scheduling is completed. The open source scheduling platform XXL-JOB is used here, and other platforms with similar functions are also applicable. The distributed lock is implemented through Redis and does not specify an expiration time. Based on the above mechanism, it can be ensured that only one consumer instance is active.

[0031] Step 6: The consumer instance that successfully acquires the worker lock reads the earliest pending event from the message queue. If there are no pending events, the scheduling ends. Otherwise, the event status is updated to "processing" and the corresponding processing logic is completed according to the event type.

[0032] Step 7: After the consumer completes the processing, it exits successfully or throws an exception, updates the event status, and releases the worker lock.

[0033] A task queue is introduced to store events, and periodic polling scheduling is performed through the task scheduling platform. Each time an instance is called up, and the called instance needs to try to obtain the global distributed lock (worker). Only after successfully obtaining the lock can subsequent processing continue and ensure that the previous task is completed, to ensure that there is only one active consumer in the application at the same time, and then process the next task. At the same time, after persisting the message data to the database, if the client is accidentally interrupted or offline during the processing process, it can also quickly retry and resume the last unfinished task. In addition, the client that successfully obtains the lock (worker) only obtains the earliest unprocessed message each time, and releases the lock after processing (successful or throwing an exception exit), avoiding the repeated consumption of messages and the waste of client computing resources.

[0034] Among them, the distributed lock in step one is mutually exclusive and has an anti-deadlock function.

[0035] Mutual exclusion means that at any given time, only one client can hold the lock, ensuring that only one node can access shared resources at the same time, avoiding data conflicts and inconsistencies. The anti-deadlock function ensures that even if the client holding the lock fails, the lock will eventually be released, avoiding deadlock.

[0036] Among them, the content of the advertising contract in step two is legal, and the spokesperson of the advertising contract is legal and compliant.

[0037] The content of advertisements should comply with the relevant provisions of the Advertising Law on advertising content guidelines, and should avoid illegal or irregular announcement content. Spokespersons need to abide by laws and regulations, public order and good customs, and their speech and image should be positive and active, avoiding the promotion of unhealthy tendencies.

[0038] Among them, the unique key in step three is secure and ordered.

[0039] Security ensures that the generated ID should not expose system and business information, ensuring information security. Orderliness ensures that the ID is generated in an orderly manner according to certain rules, facilitating database writing and sorting operations.

[0040] Among them, the distributed architecture of step four has high availability and strong fault tolerance.

[0041] High availability allows the slave server to automatically take over services when the primary server fails, ensuring the normal and continuous operation of the system. Fault tolerance allows the distributed architecture to ensure that when a server fails, other servers can still provide data access services, ensuring the fault tolerance of the system.

[0042] Among them, the task scheduling platform in step five includes a scheduled scheduling time, and the task scheduling platform has a scheduling dependency.

[0043] Scheduling time can prevent tasks from running too long and wasting scheduling resources, while scheduling dependencies are used to define the upstream and downstream dependencies of tasks. Through the dependencies, upstream and downstream nodes are scheduled and run in an orderly manner to ensure the timely output of effective business data.

[0044] The instance survival flag in step 5 has a length limit and a special character limit.

[0045] The length limit means that the tag name cannot contain more than 400 characters to ensure that it meets the system requirements, and the special character limit is to ensure that the tag can be correctly identified and processed in the system to avoid recognition problems caused by the use of special characters.

[0046] Among them, XXL-JOB in step 5 contains the port number and the log path.

[0047] The port number is included because it is necessary to ensure that the port number is unique to avoid conflicts. The log path is used to store XXL-JOB's log files so that operators can monitor its operating status and performance, thereby discovering and resolving problems in a timely manner.

[0048] Among them, the Redis in step 4 has the function of data persistence, but Redis does not support custom identity authentication.

[0049] The data persistence function allows operators to regularly save data in memory to the hard disk to prevent data loss. The lack of support for custom identity authentication ensures that the Redis service is used in a secure network environment.

[0050] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0051] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for implementing a distributed blocking queue, characterized in that: The following steps are involved: Step 1: When a configuration item changes, each application instance updates its local configuration cache, competes for a distributed lock, and submits a global "data refresh" event to the message queue. The purpose of the distributed lock is to avoid duplicate events. Step 2: After the properties of the advertising contract are changed, the instance that executes the contract modification submits the contract "data refresh" event to the message queue; Step 3: Event data must include event type, trigger / creation time, data before change, event status, and event unique key; Step 4: Each consumer instance updates its survival flag at the same fixed interval through an independent background thread. This solution involves writing a key-value pair to Redis and setting an expiration time. The key structure is "instance IP address@instance MAC address", which uniquely identifies the instance in the distributed architecture; the value is an arbitrary value. If the instance unexpectedly goes offline, its survival flag disappears after a maximum of 30 seconds. Step 5: The task scheduling platform periodically polls each instance. The awakened instance first attempts to acquire a distributed lock. If the lock is not acquired, that is, there is no consumer instance currently executing the task, the instance must immediately set a lock and use "instance IP address@instance MAC address" as the lock value. If the lock is acquired, the lock value must be extracted and the corresponding instance survival mark must be checked to determine the other party's survival status. If the survival mark has disappeared, the corresponding instance is offline. An attempt is made to acquire the worker lock. If the survival mark exists, the corresponding instance is considered online, and the scheduling is completed. The open source scheduling platform XXL-JOB is used here. The distributed lock is implemented through Redis and no expiration time is specified to ensure that only one consumer instance is active. Step 6: The consumer instance that successfully acquires the worker lock reads the earliest pending event from the message queue. If there are no pending events, the scheduling ends. Otherwise, the event status is updated to "processing" and the corresponding processing logic is completed according to the event type. Step 7: After the consumer completes the processing, it exits successfully or throws an exception, updates the event status, and releases the worker lock.

2. The method for implementing a distributed blocking queue according to claim 1, wherein: The distributed lock described in step 1 is mutually exclusive and has an anti-deadlock function.

3. The method for implementing a distributed blocking queue according to claim 1, wherein: The content of the advertising contract described in step 2 is legal, and the spokesperson for the advertising contract is legal and compliant.

4. The method for implementing a distributed blocking queue according to claim 1, wherein: The unique key described in step 3 is secure and orderly.

5. The method for implementing a distributed blocking queue according to claim 1, wherein: The distributed architecture described in step 4 has high availability and strong fault tolerance.

6. The method for implementing a distributed blocking queue according to claim 1, wherein: The task scheduling platform described in step five includes a scheduled scheduling time, and the task scheduling platform has a scheduling dependency.

7. The method for implementing a distributed blocking queue according to claim 1, wherein: The instance survival flag described in step five has a length limit, and the instance survival flag has a special character limit.

8. The method for implementing a distributed blocking queue according to claim 1, wherein: The XXL-JOB mentioned in step 5 includes the port number and the log path.

9. The method for implementing a distributed blocking queue according to claim 1, wherein: The Redis described in step 4 has the function of data persistence, but does not support custom identity authentication.

Images