Implementation method of distributed blocking queue

Abstract

The invention belongs to the technical field of message processing, and discloses a method for realizing a distributed blocking queue, which comprises the following steps of: 1, after a configuration item is changed and each application instance updates a local configuration cache, scrambling for a distributed lock and submitting a global'data refresh 'event to a message queue, the distributed lock being used for avoiding repeated events; step 2, after the attribute of the advertisement contract is changed, submitting a contract'data refreshing 'event to a message queue by an instance executing contract modification; 3, the event data need to contain event types. According to the scheme, the MySQL data table and the redis distributed lock are utilized, the message queue is achieved, the continuation state of the consumer is updated in a fixed period through the ScheduleExecutorService, the worker distributed lock is introduced for the consumer, the requirements that the messages cannot be repeated and are sequentially processed one by one in a service scene are met, the implementation is simple and easy, the introduced middleware is relatively few, and the implementation efficiency is high. And the complexity of the system architecture is reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of message processing, and in particular relates to a method for realizing a distributed blocking queue. Background Art

[0002] As the functions of arbitrary applications become increasingly complex, the number of configuration items in the applications also increases. There is an increasing need for configuration management permissions and review mechanisms, as well as the effective time, release method, and effective scope granularity of the changes. Therefore, more and more applications or systems have introduced configuration management centers to meet the corresponding demands of configuration management.

[0003] Existing message queues are usually implemented using Kafka or Redis lists, and consumers use the corresponding Kafka clients and Redis clients. Although this can meet the "first in, first out" consumption strategy and avoid blocking message reading, for multi-instance scenarios, it is not easy for clients to block each other, and it is not easy to ensure that the next task is processed after the previous task is completed. In addition, if the client is accidentally interrupted or offline during the processing process, it is not easy to retry or recover the unfinished task. In the process of using Kafka to implement message queues, due to the uncertainty of task time consumption, client rebalance is very likely to occur, which can easily cause repeated consumption of messages and waste of client computing resources. Therefore, it needs to be improved. Summary of the invention

[0004] The purpose of the present invention is to provide a method for implementing a distributed blocking queue to solve the problems raised in the above background technology.

[0005] In order to achieve the above object, the present invention provides the following technical solution: a method for implementing a distributed blocking queue, comprising the following steps:

[0006] Step 1: When a configuration item changes, each application instance updates its local configuration cache, competes for a distributed lock, and submits a global "data refresh" event to the message queue. The purpose of the distributed lock is to avoid duplicate events.

[0007] Step 2: After the properties of the advertising contract are changed, the instance that executes the contract modification submits the contract "data refresh" event to the message queue;

[0008] Step 3: Event data must include event type, trigger / creation time, data before change (configuration / contract attributes), event status (default "pending"), and event unique key (optional UUID);

[0009] Step 4: Each consumer instance updates its survival mark through an independent thread in the background, such as ScheduledExecutorService in Java, at the same fixed period, such as once every 5 seconds. The solution adopted here is to write a key-value pair to redis and set an expiration time (30 seconds by default). The key structure is "instance IP address@instance MAC address", which can uniquely identify the instance in the distributed architecture; the value can be any value, such as the current timestamp when writing. If the instance goes offline unexpectedly, its survival mark will disappear after a maximum of 30 seconds;

[0010] Step 5: The task scheduling platform periodically polls each instance. The awakened instance first tries to obtain the distributed lock (worker). If the lock is not obtained, that is, there is no consumer instance currently executing the task, the instance needs to set the lock immediately and use "instance IP address@instance MAC address" as the lock value. If the lock is obtained, the lock value needs to be extracted to check whether the corresponding instance survival mark exists to determine the other party's survival status. If the survival mark has disappeared, that is, the corresponding instance has been offline, you can try to obtain the worker lock. If the survival mark exists, it is considered that the corresponding instance is online, and this scheduling is completed. The open source scheduling platform XXL-JOB is used here, and other platforms with similar functions are also applicable. The distributed lock is implemented through redis, and the expiration time is not specified. Based on the above mechanism, it can be ensured that only one consumer instance is active;

[0011] Step 6: The consumer instance that successfully obtains the worker lock reads the earliest pending event from the message queue. If there is no pending event, the scheduling ends. Otherwise, the event status is updated to "processing" and the corresponding processing logic is completed according to the event type.

[0012] Step 7: After the consumer completes the processing, it exits successfully or throws an exception, updates the event status, and releases the worker lock.

[0013] Preferably, the distributed lock described in step one is mutually exclusive, and the distributed lock has an anti-deadlock function.

[0014] Preferably, the advertising contract described in step 2 has legal content, and the spokesperson of the advertising contract is legal and compliant.

[0015] Preferably, the unique key described in step three is secure and orderly.

[0016] Preferably, the distributed architecture described in step four has high availability and strong fault tolerance.

[0017] Preferably, the task scheduling platform described in step five includes a scheduled scheduling time, and the task scheduling platform has a scheduling dependency.

[0018] Preferably, the instance survival mark described in step five has a length limit, and the instance survival mark has a special character limit.

[0019] Preferably, the XXL-JOB described in step five includes a port number, and the XXL-JOB includes a log path.

[0020] Preferably, the Redis described in step 4 has a data persistence function, and the Redis does not support custom identity authentication.

[0021] The beneficial effects of the present invention are as follows:

[0022] The solution adopted here uses MySQL data tables and redis distributed locks to implement message queues, and updates the consumer's survival status at a fixed period through ScheduledExecutorService, introducing worker distributed locks for consumers, which meets the business scenario's requirements for non-repeatable messages and sequential processing one by one. It is simple and easy to implement, and relatively few middleware are introduced, reducing the complexity of the system architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a functional architecture block diagram of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] like Figure 1 As shown, an embodiment of the present invention provides a method for implementing a distributed blocking queue, comprising the following steps:

[0026] Step 1: When a configuration item changes, each application instance updates its local configuration cache, competes for a distributed lock, and submits a global "data refresh" event to the message queue. The purpose of the distributed lock is to avoid duplicate events.

[0027] Step 2: After the properties of the advertising contract are changed, the instance that executes the contract modification submits the contract "data refresh" event to the message queue;

[0028] Step 3: Event data must include event type, trigger / creation time, data before change (configuration / contract attributes), event status (default "pending"), and event unique key (optional UUID);

[0029] Step 4: Each consumer instance updates its survival mark through an independent thread in the background, such as ScheduledExecutorService in Java, at the same fixed period, such as once every 5 seconds. The solution adopted here is to write a key-value pair to redis and set an expiration time (30 seconds by default). The key structure is "instance IP address@instance MAC address", which can uniquely identify the instance in the distributed architecture; the value can be any value, such as the current timestamp when writing. If the instance goes offline unexpectedly, its survival mark will disappear after a maximum of 30 seconds;

[0030] Step 5: The task scheduling platform periodically polls each instance. The awakened instance first tries to obtain the distributed lock (worker). If the lock is not obtained, that is, there is no consumer instance currently executing the task, the instance needs to set the lock immediately and use "instance IP address@instance MAC address" as the lock value. If the lock is obtained, the lock value needs to be extracted to check whether the corresponding instance survival mark exists to determine the other party's survival status. If the survival mark has disappeared, that is, the corresponding instance has been offline, you can try to obtain the worker lock. If the survival mark exists, it is considered that the corresponding instance is online, and this scheduling is completed. The open source scheduling platform XXL-JOB is used here, and other platforms with similar functions are also applicable. The distributed lock is implemented through redis, and the expiration time is not specified. Based on the above mechanism, it can be ensured that only one consumer instance is active;

[0031] Step 6: The consumer instance that successfully obtains the worker lock reads the earliest pending event from the message queue. If there is no pending event, the scheduling ends. Otherwise, the event status is updated to "processing" and the corresponding processing logic is completed according to the event type.

[0032] Step 7: After the consumer completes the processing, it exits successfully or throws an exception, updates the event status, and releases the worker lock.

[0033] A task queue is introduced to store events. The task scheduling platform performs periodic polling scheduling. Each time an instance is called up, and the called instance needs to try to obtain the global distributed lock (worker). Only after successfully acquiring the lock can subsequent processing continue and ensure that the previous task is completed. This ensures that there is only one active consumer in the application at the same time before processing the next task. At the same time, after persisting the message data to the database, if the client is accidentally interrupted or offline during the processing process, it can also quickly retry and resume the last unfinished task. In addition, the client that successfully obtains the lock (worker) only obtains the earliest unprocessed message each time, and releases the lock after processing (successful or throwing an exception to exit), avoiding the repeated consumption of messages and wasting client computing resources.

[0034] Among them, the distributed lock in step one is mutually exclusive, and the distributed lock has an anti-deadlock function.

[0035] Mutual exclusivity means that at any time, only one client can hold the lock, ensuring that only one node can access shared resources at the same time, avoiding data conflicts and inconsistencies. The anti-deadlock function ensures that the lock will eventually be released even if the client holding the lock fails, avoiding the occurrence of deadlock.

[0036] Among them, the content of the advertising contract in step two is legal, and the spokesperson of the advertising contract is legal and compliant.

[0037] The content of advertisements should comply with the relevant provisions of the Advertising Law on advertising content standards, and should avoid illegal or irregular announcement content. Spokespersons need to abide by laws and regulations and public order and good customs. Their speech and image should be positive and should avoid promoting unhealthy tendencies.

[0038] Among them, the unique key in step three is secure and ordered.

[0039] Security ensures that the generated ID should not expose system and business information, ensuring information security. Orderliness ensures that the ID is generated in an orderly manner according to certain rules, facilitating database writing and sorting operations.

[0040] Among them, the distributed architecture of step four has high availability and strong fault tolerance.

[0041] High availability means that when the primary server fails, the slave server can automatically take over the service to ensure the normal and continuous operation of the system. Fault tolerance means that when a server in the distributed architecture fails, other servers can still provide data access services to ensure the fault tolerance of the system.

[0042] Among them, the task scheduling platform of step five includes a scheduled scheduling time, and the task scheduling platform has a scheduling dependency.

[0043] Scheduling time can prevent tasks from running too long and wasting scheduling resources, while scheduling dependencies are used to define the upstream and downstream dependencies of tasks. Upstream and downstream nodes are scheduled and run in an orderly manner through dependencies to ensure timely output of effective business data.

[0044] The instance survival flag in step 5 has a length limit, and the instance survival flag has a special character limit.

[0045] The length limit means that the tag name cannot contain more than 400 characters to ensure compliance with system requirements, while the special character limit is to ensure that the tag can be correctly identified and processed in the system to avoid unrecognition problems caused by the use of special characters.

[0046] Among them, XXL-JOB in step 5 contains the port number and XXL-JOB contains the log path.

[0047] The port number is included because it is necessary to ensure that the port number is unique to avoid conflicts. The log path is used to store the log file of XXL-JOB so that operators can monitor its operating status and performance, so as to discover and solve problems in time.

[0048] Among them, the Redis in step 4 has the function of data persistence, but Redis does not support custom identity authentication.

[0049] The data persistence function allows operators to regularly save data in memory to the hard disk to prevent data loss. The lack of support for custom identity authentication ensures that the Redis service is used in a secure network environment.

[0050] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0051] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for implementing a distributed blocking queue, characterized in that: The following steps are involved: Step 1: When a configuration item changes, each application instance updates the local configuration cache, competes for the distributed lock, and submits the global "data refresh" event to the message queue. The purpose of the distributed lock is to avoid duplicate events. Step 2: After the properties of the advertising contract are changed, the instance that executes the contract modification submits the contract "data refresh" event to the message queue; Step 3: Event data must include event type, trigger / creation time, data before change (configuration / contract attributes), event status (default "pending"), and event unique key (optional UUID); Step 4: Each consumer instance updates its survival mark through an independent thread in the background, such as ScheduledExecutorService in Java, at the same fixed period, such as once every 5 seconds. The solution adopted here is to write a key-value pair to redis and set an expiration time (30 seconds by default). The key structure is "instance IP address@instance MAC address", which can uniquely identify the instance in the distributed architecture; the value can be any value, such as the current timestamp when writing. If the instance goes offline unexpectedly, its survival mark will disappear after a maximum of 30 seconds; Step 5: The task scheduling platform periodically polls each instance. The awakened instance first tries to obtain the distributed lock (worker). If the lock is not obtained, that is, there is no consumer instance currently executing the task, the instance needs to set the lock immediately and use "instance IP address@instance MAC address" as the lock value. If the lock is obtained, the lock value needs to be extracted to check whether the corresponding instance survival mark exists to determine the other party's survival status. If the survival mark has disappeared, that is, the corresponding instance has been offline, you can try to obtain the worker lock. If the survival mark exists, it is considered that the corresponding instance is online, and this scheduling is completed. The open source scheduling platform XXL-JOB is used here, and other platforms with similar functions are also applicable. The distributed lock is implemented through redis, and the expiration time is not specified. Based on the above mechanism, it can be ensured that only one consumer instance is active; Step 6: The consumer instance that successfully obtains the worker lock reads the earliest pending event from the message queue. If there is no pending event, the scheduling ends. Otherwise, the event status is updated to "processing" and the corresponding processing logic is completed according to the event type. Step 7: After the consumer completes the processing, it exits successfully or throws an exception, updates the event status, and releases the worker lock.

2. The method for implementing a distributed blocking queue according to claim 1, characterized in that: The distributed lock described in step 1 is mutually exclusive and has an anti-deadlock function.

3. The method for implementing a distributed blocking queue according to claim 1, characterized in that: The content of the advertising contract described in step 2 is legal, and the spokesperson of the advertising contract is legal and compliant.

4. The method for implementing a distributed blocking queue according to claim 1, characterized in that: The unique key described in step three is secure and orderly.

5. The method for implementing a distributed blocking queue according to claim 1, characterized in that: The distributed architecture described in step 4 has high availability and strong fault tolerance.

6. The method for implementing a distributed blocking queue according to claim 1, characterized in that: The task scheduling platform described in step five includes a scheduled scheduling time, and the task scheduling platform has a scheduling dependency.

7. The method for implementing a distributed blocking queue according to claim 1, characterized in that: The instance survival flag described in step five has a length limit, and the instance survival flag has a special character limit.

8. The method for implementing a distributed blocking queue according to claim 1, characterized in that: The XXL-JOB described in step five includes a port number, and the XXL-JOB includes a log path.

9. The method for implementing a distributed blocking queue according to claim 1, characterized in that: The Redis described in step 4 has the function of data persistence, but the Redis does not support custom identity authentication.

Images