An equilibrium method and system based on a time-series message consumption queue

Abstract

The present invention discloses a balancing method based on a time-series message consumption queue, which includes the following steps: S1, obtaining producer information, capable of obtaining messages sent by producers; S2, the message queue receiving and delivering messages, capable of evenly consuming time-series messages; S3, the consumer receiving messages, capable of passing the producer messages to downstream processing; No matter what strategy the producer uses to produce messages, the present invention can ensure that the messages in the message queue are evenly time-series messages. Through this mechanism, it is possible to maintain that the data in the IOT scenario does not have the phenomenon of message backlog and data time-series chaos.

Description

Technical Field

[0001] The present invention relates to the field, and in particular to a balancing method and system based on a time-sequential message consumption queue. Background Art

[0002] Under the background of the overall upgrade of the automotive industry, electrification and intelligence are important directions for the upgrade of the automotive industry. At the same time, as cars become more and more popular, users have higher and higher requirements for comfort and safety. This requires collecting more massive component data for user behavior analysis and component function analysis to improve the user experience and test the data throughput capacity of the platform. Currently, the MQTT message queue is widely used in the IOT field. As a buffering mechanism for messages, it can effectively reduce the pressure on the background service application.

[0003] Generally in the IOT field, especially in the field of vehicle networking, vehicle data collection with time-sequential characteristics is involved. Currently, the MQTT message queue is widely used in the IOT field. As a buffering mechanism for messages, it can effectively reduce the pressure on the background service application. There are two common working methods for the MQTT message queue, the polling message sending method and the fixed message sending according to the producer hash. Currently, the polling method solves the balance of messages among multiple consumers, but it causes the data to be distributed among different consumers and cannot handle time-sequential data services. And the data according to the producer hash can achieve that the messages of the same producer fall into the same consumer. However, there is a phenomenon of unbalanced consumption.

[0004] For example, a "consumption balancing method and system based on a message queue" disclosed in a Chinese patent document, with the publication number: CN105306552A, discloses that the message queue with the largest number of unconsumed messages is consumed in the next consumption to ensure consumption balance, but it cannot solve the phenomenon of message backlog and data time-sequential chaos. Summary of the Invention

[0005] Therefore, the present invention provides a balancing method and system based on a time-sequential message consumption queue, overcoming the deficiencies of the prior art and providing a method for achieving time-sequential balanced consumption of messages in the IOT field.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A balancing method based on a time-sequential message consumption queue includes the following steps:

[0008] S1. Obtain producer information, which can obtain the messages sent by the producer; the producer uses the device ID as the unique identifier for the message session;

[0009] S2. The message queue receives and delivers messages, and can evenly consume timing messages. The message queue implements a custom ordered hash table, where its hash key is the consumer ID and the value is the number of producers sending messages to the consumer. Each time the hash table is updated, it is sorted again according to the number of producers. After the producer session ends, the message queue also needs to rehash the table.

[0010] S3. The consumer receives messages and can pass the producer messages to downstream processing. It can implement both data timing and balanced message consumption.

[0011] Preferably, S2 includes using a message load algorithm to balance messages. Each time a new producer enters the message queue, it decides how to allocate to the consumer according to the current number of producers consumed by the consumer and maintains the hash table.

[0012] Preferably, in S3, the consumer performs multi-threaded consumption processing on the messages of a single producer. This can ensure that there is no message backlog and data timing chaos in the IOT scenario.

[0013] Preferably, in S3, it also includes merging messages according to the timing. This can solve the problem of data timing.

[0014] Preferably, in S2, the message queue removes inactive producers and consumers from the queue. The message queue needs to start a producer monitoring thread to check the current inactive producers and remove them from the hash table queue structure. Similarly, if there are inactive consumers, they are also removed. When removing consumers, the load calculation needs to be recalculated.

[0015] An equilibrium system based on a timing message consumption queue includes a producer module, a message queue module connected to the producer module. The message queue module has a hash table structure, and a consumer module connected to the message queue module. Among them, the message queue can manage consumers and allocate and send messages according to the load message balancing strategy.

[0016] Preferably, the consumer module is provided with a message thread and a sorting thread, which can process and sort messages. The message thread can perform multi-threaded consumption processing on the messages of a single producer, and the sorting thread sorts and merges the messages according to the timing and provides them to the downstream business system for processing.

[0017] Preferably, the message queue module includes a load monitoring thread and a load algorithm module. It can perform online statistics on the activities of consumers and producers, and the message balancing load algorithm calculates and updates the corresponding relationship between consumers and producers according to the above information data.

[0018] The embodiments of the present invention have the following advantages:

[0019] Regardless of the strategy adopted by the producer to produce messages, the present invention can ensure that the messages in the message queue are balanced in time sequence. Through this mechanism, it is possible to maintain that the data in the IOT scenario does not experience message backlog and data time sequence chaos. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings described below are merely exemplary, and for those of ordinary skill in the art, without creative efforts, other implementation drawings can also be obtained based on the provided drawings.

[0021] The structures, ratios, sizes, etc. depicted in this specification are only used to cooperate with the content disclosed in the specification for those familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention. Therefore, they do not have technical essence. Any modification of the structure, change in the proportional relationship, or adjustment of the size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope covered by the technical content disclosed in the present invention.

[0022] Figure 1 is the method flowchart of the present invention.

[0023] Figure 2 is the method logic schematic diagram of the present invention.

[0024] Figure 3 is the system block diagram of the present invention.

[0025] In the figure:

[0026] 1 - Producer module; 2 - Message queue module; 3 - Consumer module; 4 - Hash table structure; 5 - Load monitoring thread; 6 - Load algorithm module; 7 - Message thread; 8 - Sorting thread. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] The following specific embodiments illustrate the implementation manners of the present invention. Those familiar with this technology can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0028] As Figures 1-3 shown, in a preferred embodiment, the present invention discloses a method for balancing a time-sequence message consumption queue, including the following steps:

[0029] S1. Obtain producer information, capable of obtaining the messages sent by the producer; the producer uses the device ID as the unique identifier for the message session.

[0030] S2. The message queue receives and delivers messages, capable of evenly consuming sequential messages; the message queue implements a custom ordered hash table, where its hash key is the consumer ID and the value is the number of producers sent to the consumer; each time the hash table is updated, it is sorted again according to the number of producers; after the producer session ends, the message queue also needs to re - hash the table.

[0031] S3. The consumer receives messages and is capable of passing the producer messages to downstream processing. It can implement both data sequencing and balanced message consumption.

[0032] S2 includes using a message load algorithm to balance messages. Every time a new producer enters the message queue, it decides how to allocate to the consumers according to the current number of producers consumed by the consumers in a balanced manner and maintains the hash table.

[0033] S3 includes the consumer performing multi - threaded consumption processing on the messages of a single producer. It can maintain the data in the IOT scenario without message backlog and data sequencing chaos.

[0034] S3 also includes merging messages in sequence. It can solve the problem of data sequencing.

[0035] S2 includes the message queue removing inactive producers and consumers from the queue. The message queue needs to start a producer monitoring thread to check the current inactive producers and remove them from the hash table queue structure. Similarly, if there are inactive consumers, they are also removed. When removing consumers, the load calculation needs to be recalculated.

[0036] An equilibrium system based on a sequential message consumption queue, including a producer module, a message queue module connected to the producer module, the message queue module having a hash table structure, and a consumer module connected to the message queue module. Among them, the message queue can manage consumers and allocate and send messages according to the load message balancing strategy.

[0037] The consumer module is provided with a message thread and a sorting thread, capable of processing and sorting messages. The message thread can perform multi - threaded consumption processing on the messages of a single producer, and the sorting thread sorts and merges the messages in sequence and provides them to the downstream business system for processing.

[0038] The message queue module includes a load monitoring thread and a load algorithm module. It can perform online statistics on the activities of consumers and producers, and the message balancing load algorithm calculates and updates the corresponding relationship between consumers and producers according to the above information data.

[0039] The present invention is based on the IOT field, and characteristic devices are connected to continuously send service scenarios with time characteristics.

[0040] When the present invention is in use:

[0041] First, when a new producer arrives, it is first determined whether the producer session ID already exists on a certain consumer. If it exists, it is directly sent to that consumer. If it does not exist, the first consumer is selected from the ordered hash table, and the number of producers of this consumer in the hash table is incremented by 1. The sorting rule is based on the message load algorithm in ascending order of quantity.

[0042] In the next step, the consumer receives messages in time order. Here, multiple consumption threads need to be started for processing. Here, a message merging thread also needs to be started. The message merging thread is responsible for sorting the messages in time order and providing them to the downstream service for processing.

[0043] In the above steps, it is necessary to pay attention to the departure of producers and consumers. The MQTT message queue needs to start a producer monitoring thread to check the current inactive producers and remove the producers from the hash table queue structure. Similarly, if there are inactive consumers, the consumers are also removed. When removing consumers, the load calculation needs to be recalculated.

[0044] Although the present invention has been described in detail above with general descriptions and specific embodiments, based on the present invention, some modifications or improvements can be made, which are obvious to those skilled in the art. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of protection required by the present invention.

Claims

1. An equilibrium method based on a time-sequential message consumption queue, characterized in that, it includes the following steps: S1. Obtain producer information, capable of obtaining messages sent by producers; S2. The message queue receives and transfers messages, capable of evenly consuming time-sequential messages; the message queue implements a custom ordered hash table, whose hash key is the consumer ID and the hash value is the number of producers sent to the consumer; each time the hash table is updated, it is re-sorted according to the number of producers; after the producer session ends, the message queue also needs to update the hash table; First, when a new producer arrives, first determine whether the producer session ID already exists on a certain consumer. If it exists, it is directly sent to that consumer. If it does not exist, select the first consumer in the ordered hash table and increment the number of producers of this consumer in the hash table. The sorting rule is from smallest to largest according to the message load algorithm by quantity; S3. The consumer receives messages, capable of passing the producer messages to downstream processing, and also includes merging the messages in time sequence.

2. The equilibrium method based on a time-sequential message consumption queue according to claim 1, characterized in that, S2 includes using the message load algorithm to balance messages.

3. The equilibrium method based on a time-sequential message consumption queue according to claim 1 or 2, characterized in that, S3 includes the consumer performing multi-threaded consumption processing on the messages of a single producer.

4. The equilibrium method based on a time-sequential message consumption queue according to claim 1, characterized in that, S2 includes the message queue removing inactive producers and consumers from the queue.

5. An equilibrium system based on a time-sequential message consumption queue, applicable to the equilibrium method based on a time-sequential message consumption queue according to claims 1 to 4, characterized in that, it includes a producer module, a message queue module connected to the producer module, the message queue module having a hash table structure, and a consumer module connected to the message queue module.

6. The equilibrium system based on a time-sequential message consumption queue according to claim 5, characterized in that, the consumer module is provided with a message thread and a sorting thread, capable of processing and sorting messages.

7. The equilibrium system based on a time-sequential message consumption queue according to claim 5, characterized in that, the message queue module includes a load monitoring thread and a load algorithm module.

Images