Method and device for ensuring consumption sequence of distributed message queue

Abstract

The embodiment of the invention discloses a method and device for ensuring the consumption sequence of a distributed message queue. The method comprises the following steps: a producer side generates a to-be-sent message; the producer side generates parameter information contained in the message to be sent, and the parameter information at least comprises a timestamp and a first effective load of service data; the producer side sends a message to the agent side, and the agent side receives the message and stores the message in a message queue; and the consumer side obtains the messages in the message queue, performs sorting based on each message in the message queue and the corresponding parameter information to obtain a sorted message priority queue, and performs message consumption based on the first effective load corresponding to each message in the message priority queue. According to the technical scheme, the accuracy of the consumption sequence of the messages in the real-time data can be ensured, so that the error risk caused by message disorder is reduced.

Description

Technical Field

[0001] The present invention relates to the field of message queue consumption order, and in particular to a method and device for ensuring the consumption order of a distributed message queue. Background Art

[0002] In today's big data processing environment, high-throughput distributed message middleware is widely used. However, when processing real-time data messages containing multiple operation types such as new additions, its distributed nature and concurrent processing mechanism can lead to confusion in the order of message consumption, thus affecting data accuracy and consistency.

[0003] Existing technologies may use partition keys to partition messages in the hope of ensuring the order in which messages are consumed. However, due to the limited selection of partition keys, if the business identifier is not clear or complex, it may be difficult to accurately select the appropriate partition key, resulting in chaotic message distribution and affecting the order of consumption. When the number of messages in a partition is too large, disordered message processing may still occur, especially in the case of concurrent processing. For dynamically changing business needs, solutions based on partition keys may not be flexible enough, and the cost of adjustment and optimization is high. It is impossible to effectively handle the problem of message order across partitions. When the business logic involves message interaction across multiple partitions, it is difficult to guarantee the overall consumption order. Summary of the Invention

[0004] In view of this, the present invention provides a method and device for ensuring the consumption order of distributed message queues, which can ensure the consumption order of messages in real-time data, thereby effectively ensuring the data accuracy and correctness of business logic when processing business, and reducing the risk of errors caused by message disorder.

[0005] According to one aspect of the present invention, an embodiment of the present invention provides a method for ensuring the consumption order of a distributed message queue, the method comprising:

[0006] The producer generates a message to be sent; wherein the message to be sent is a message indicating a change in business data; the change includes at least the addition, deletion, and update of the business data; the message to be sent corresponds to a message topic, and each message topic is divided into at least two partitions;

[0007] The producer end generates parameter information included in the message to be sent, wherein the parameter information at least includes: a timestamp and a first payload of the business data;

[0008] The producer sends the message to be sent to the proxy, so that the proxy receives the message to be sent and stores the message to be sent as a message to be consumed in a message queue;

[0009] The consumer side obtains the messages to be consumed in the message queue, and sorts each message to be consumed in the message queue and its corresponding parameter information to obtain a sorted message priority queue;

[0010] The consumer end obtains the to-be-consumed messages in the message priority queue, and consumes the messages based on the first payload corresponding to each to-be-consumed message in the message priority queue.

[0011] According to another aspect of the present invention, an embodiment of the present invention further provides a device for ensuring the consumption order of a distributed message queue, the device comprising:

[0012] A message generation module is used for the producer to generate a message to be sent; wherein the message to be sent is a message indicating a change in business data; the message to be sent corresponds to a message topic, and each message topic is divided into at least two partitions;

[0013] A parameter generation module, configured to generate parameter information included in the message to be sent by the producer; wherein the parameter information includes at least: a timestamp and a first payload of the business data;

[0014] A message storage module is used for the producer to send the message to be sent to the agent, so that the agent receives the message to be sent and stores the message to be sent as a message to be consumed in a message queue;

[0015] A message sorting module is used for the consumer side to obtain the messages to be consumed in the message queue, and sort each message to be consumed in the message queue and its corresponding parameter information to obtain a sorted message priority queue;

[0016] The message consumption module is used for the consumer end to obtain the to-be-consumed messages in the message priority queue and to consume the messages based on the first payload corresponding to each to-be-consumed message in the message priority queue.

[0017] According to another aspect of the present invention, an embodiment of the present invention further provides an electronic device, comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for ensuring the consumption order of a distributed message queue as described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement the method of ensuring the consumption order of a distributed message queue as described in any embodiment of the present invention when executed.

[0022] The above-mentioned technical solution of the embodiment of the present invention generates parameter information (timestamp and the first payload of business data) contained in the message to be sent by the producer end, and the producer end sends the message to the agent end. The agent end receives the message and stores the message in the message queue. The consumer end obtains the message from the message queue and sorts it based on each message in the message queue and its corresponding parameter information to obtain a sorted message priority queue. Based on this, the consumer end obtains the message in the message priority queue and consumes the message based on the first payload corresponding to each message in the message priority queue. It can embed accurate timestamps in the messages and combine with the priority queue processing mechanism to ensure the consumption order of messages in real-time data, thereby effectively ensuring the data accuracy and correctness of business logic when processing business, and reducing the risk of errors caused by message disorder.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A flowchart of a method for ensuring the consumption order of a distributed message queue provided by one embodiment of the present invention;

[0026] Figure 2 A flowchart of another method for ensuring the consumption order of a distributed message queue provided by one embodiment of the present invention;

[0027] Figure 3 A schematic diagram of a message consumption process provided by one embodiment of the present invention;

[0028] Figure 4 A structural block diagram of a device for ensuring the consumption order of a distributed message queue provided by one embodiment of the present invention;

[0029] Figure 5A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] In one embodiment, Figure 1 This is a flowchart of a method for ensuring the consumption order of a distributed message queue provided by an embodiment of the present invention. This embodiment is applicable to situations where the consumption order of a distributed message queue is to be ensured. The method can be executed by a device for ensuring the consumption order of a distributed message queue. The device for ensuring the consumption order of a distributed message queue can be implemented in the form of hardware and / or software. The device for ensuring the consumption order of a distributed message queue can be configured in an electronic device. Figure 1 As shown, the method includes:

[0033] S110. The producer generates a message to be sent. The message to be sent is a message indicating a change in business data.

[0034] Changes in business data messages include at least the addition, deletion, and update of business data. Messages to be sent correspond to message topics, and each message topic is divided into at least two partitions. This means that messages are categorized by topic, and each topic corresponds to a message queue.

[0035] In this embodiment, the producer side generates a message indicating a change in business data and sends the message indicating the change in business data to the proxy side. This message can be understood as containing newly added, deleted, and updated business data. When business data changes, such as addition, deletion, and update, a corresponding message will be generated and sent downstream. In this embodiment, the message to be sent can be generated based on business logic. For example, the message is related user data. The trigger point for the generation of the message is that a user registers a certain software, resulting in a new user, which is equivalent to generating a piece of user data. At this time, the producer side sends a data message indicating that the corresponding data content has been newly added.

[0036] S120. The producer generates parameter information included in the message to be sent, wherein the parameter information includes at least: a timestamp and a first payload of business data.

[0037] In this embodiment, the producer side generates parameter information contained in the message to be sent, and the timestamp in the parameter information can be used to indicate the generation time of the message. In this embodiment, when a message is generated, a unique timestamp is generated to identify the generation time of the information. Specifically, a UTC timestamp is used to generate a timestamp accurate to the millisecond level for each message to be sent to ensure the uniqueness and accuracy of the timestamp. In this embodiment, in addition to the timestamp and the first payload of business data, the message to be sent may also include the producer's ID and the topic and partition where the message is located; in this embodiment, there are multiple consumers in the consumer group, and multiple consumers subscribe to a topic, and then consume the messages under the topic.

[0038] S130: The producer sends the message to be sent to the proxy, so that the proxy receives the message to be sent and stores the message to be sent in a message queue as a message to be consumed.

[0039] In this embodiment, the producer side sends the message to be sent to the proxy side, so that the proxy side receives the message to be sent and stores the message to be sent as a message to be consumed in the message queue. It can be understood that the proxy side decouples the production and consumption of messages, and the producer side and the consumer side do not need to communicate directly.

[0040] S140: The consumer side obtains the to-be-consumed messages from the message queue, and sorts each to-be-consumed message in the message queue and its corresponding parameter information to obtain a sorted message priority queue.

[0041] In this embodiment, the consumer side obtains the to-be-consumed messages from the message queue, and sorts the messages based on each to-be-consumed message in the message queue and its corresponding parameter information to obtain a sorted message priority queue. In this embodiment, the message priority queue can be understood as sorting in order of UTC timestamps, starting from the earliest UTC timestamp, to obtain a message priority queue from earliest to latest. It can be understood that the message with the earliest UTC timestamp has the highest priority, and from earliest to latest, the priority of the message decreases. It should be noted that messages with high priority will be dequeued before messages with low priority. The first payload can be understood as the specific content of the message to be sent, for example, the service identifier, the newly added data content of the service data, the deleted data content of the service data, and the updated data content of the service data.

[0042] S150: The consumer side obtains the to-be-consumed messages in the message priority queue, and consumes the messages based on the first payload corresponding to each to-be-consumed message in the message priority queue.

[0043] In this embodiment, the consumer side obtains the messages in the message priority queue in sequence according to the timestamps, and consumes the messages based on the first payload corresponding to each message in the message priority queue. In some embodiments, the consumer side reads the messages to be consumed in the message priority queue by a preset dynamic window polling method and starting from the earliest UTC timestamp according to the UTC timestamps of the messages to be consumed. Specifically, the consumer side can obtain messages from the message priority queue by a dynamic window polling method. The time window of the polling can be configured on demand according to the timeliness requirements of the data. It can be understood that the messages can be viewed and obtained in real time, or a time interval can be customized and obtained according to the customized time interval. In other embodiments, after configuring the initialization polling time window, each polling can slide the window according to the preset length and count the message processing throughput corresponding to the preset number of messages, and dynamically adjust the window according to the increase rate of the message processing throughput and the set threshold, as well as the decrease rate of the message processing throughput and the corresponding set threshold.

[0044] The above-mentioned technical solution of the embodiment of the present invention generates parameter information (timestamp and the first payload of business data) contained in the message to be sent by the producer end, and the producer end sends the message to the agent end. The agent end receives the message and stores the message in the message queue. The consumer end obtains the message from the message queue and sorts it based on each message in the message queue and its corresponding parameter information to obtain a sorted message priority queue. Based on this, the consumer end obtains the message in the message priority queue and consumes the message based on the first payload corresponding to each message in the message priority queue. It can embed accurate timestamps in the messages and combine with the priority queue processing mechanism to ensure the consumption order of messages in real-time data, thereby effectively ensuring the data accuracy and correctness of business logic when processing business, and reducing the risk of errors caused by message disorder.

[0045] In one embodiment, Figure 2 A flowchart of another method for ensuring the consumption order of distributed message queues provided for one embodiment of the present invention. Based on the above embodiments, this embodiment generates parameter information contained in the message to be sent on the producer side, and obtains the message to be consumed in the message queue on the consumer side, sorts each message to be consumed in the message queue and its corresponding parameter information to obtain a sorted message priority queue, and further refines the message consumption based on the first payload corresponding to each message to be consumed in the message priority queue.

[0046] like Figure 2 As shown, the method for ensuring the consumption order of the distributed message queue in this embodiment may specifically include the following steps:

[0047] S210. The producer generates a message to be sent. The message to be sent is a message indicating a change in business data. The message to be sent corresponds to a message topic, and each message topic is divided into at least two partitions.

[0048] S220: Use a preset timestamp conversion tool to convert the message generation time corresponding to the message to be sent into a UTC timestamp format to identify the generation time of the message.

[0049] The preset timestamp conversion tool is a preconfigured tool for time conversion. For example, it can be an Epoch Converter tool or a Timestamp Convert tool, which is not specifically limited in this embodiment. In this embodiment, the Epoch Converter tool supports conversion between Unix timestamps (seconds / milliseconds) and dates. The TimestampConvert tool supports conversion to local time zones, and the corresponding timestamp conversion tool can be selected based on needs.

[0050] In this embodiment, a preset timestamp conversion tool is used to convert the message generation time corresponding to the to-be-sent message into a UTC timestamp to identify the message generation time. Of course, the to-be-sent message can be one or at least two types of business data, such as additions, deletions, and updates. This means that multiple to-be-sent messages will be generated, each with a corresponding timestamp. Specifically, when a message is generated, the current time is obtained and converted to the corresponding timestamp using the timestamp conversion tool.

[0051] S230: Store the UTC timestamp in the message body as field information of the message.

[0052] In this embodiment, after the UTC timestamp is converted, the UTC timestamp is stored in the message body as the field information of the message. Similarly, the changes corresponding to the business data can also be stored in the message body. It can be understood that when the message is sent, the generated timestamp and specific data content are sent together with the message.

[0053] S240. Obtain the changed content corresponding to the business data, and store the changed content as the first payload as the field information of the message in the message body; wherein the changed content at least includes: newly added content, deleted content, and updated content of the business data.

[0054] It can be understood that, for the changes corresponding to the business data, in addition to writing the corresponding timestamp as field information into the message body, the changes must also be stored in the message body as the first payload as the field information of the message; wherein, the changes include at least: the new content, deleted content and updated content of the business data.

[0055] S250: The producer sends the message to be sent to the proxy, so that the proxy receives the message to be sent and stores the message to be sent as a message to be consumed in a message queue.

[0056] In this embodiment, the producer sends a message to the agent, and the agent stores the message in a message queue.

[0057] S260: The consumer side determines the time point corresponding to the UTC timestamp of each message to be consumed in the message queue.

[0058] In this embodiment, the time points corresponding to the UTC timestamps corresponding to the messages in the message queue are determined.

[0059] S270 , sort the messages to be consumed in each message queue according to the time point corresponding to the UTC timestamp, and write them into the local priority message queue in sequence.

[0060] In this embodiment, each to-be-sent message is sorted by the time point corresponding to the UTC timestamp and written to the local priority message queue in sequence. Sorting includes sorting by UTC timestamp, starting with the earliest UTC timestamp, to create a priority queue from earliest to latest. This means that data is read in the order of message timestamps. Messages with earlier timestamps are read and consumed earlier.

[0061] S280: The consumer side reads the messages to be consumed in the message priority queue in a preset dynamic window polling manner and starts reading the messages according to the UTC timestamps of the messages to be consumed, starting from the earliest UTC timestamp.

[0062] In this embodiment, the consumer reads messages from the message priority queue based on their UTC timestamps, starting with the earliest UTC timestamp. In this embodiment, a dynamic window polling method is preset, including: after initializing the polling time window, each polling performs a sliding window according to a preset length and calculates the message processing throughput corresponding to a preset number of messages; when the rate of increase in message processing throughput exceeds a first preset threshold, the polling time window is reduced by a preset first percentage each time; when the rate of decrease in message processing throughput exceeds a second preset threshold, the polling time window is expanded by a preset first percentage each time; wherein the first preset threshold is greater than the second preset threshold. Exemplarily, after initializing the polling time window (default is 1s), each polling calculates the throughput of the last five message processing times using a sliding window of length 5; if the rate of increase in message processing throughput exceeds a configured threshold (default is 20%), the polling time window is reduced by 10% each time; if the rate of decrease in message processing throughput exceeds a configured threshold (default is 15%), the polling time window is expanded by 10% each time.

[0063] S290. Obtain a first payload corresponding to each message to be consumed; wherein the first payload represents data content corresponding to the message to be consumed.

[0064] In this embodiment, the first payload corresponding to each message to be consumed is obtained; wherein the first payload represents the data content corresponding to the message to be consumed, and the data content includes at least: the newly added content, deleted content and updated content of the business data.

[0065] S2100: Extract the corresponding service identifier from the first payload, obtain the stored historical messages from a preset database, and use the service identifier to search the preset database for a target historical message corresponding to the first payload.

[0066] A historical message refers to a message indicating a change in historically stored business data. Similarly, changes in historical messages include at least the addition, deletion, and update of business data. A target historical message refers to the valid data content associated with the first payload found in the historical message. The business identifier can be a unique ID that represents the user.

[0067] In this embodiment, the corresponding service identifier is extracted from the first payload, the stored historical messages are obtained from the preset database, and the service identifier is used to search the preset database for valid data content contained in the historical messages associated with the first payload.

[0068] S2110. Compare the second payload contained in the target historical message with the first payload to obtain a first comparison result, and consume the message based on the first comparison result; wherein the first comparison result includes: the first payload has updated, deleted, and added data content compared with the second payload.

[0069] The second payload is a payload in the target history message that is associated with the first payload.

[0070] In this embodiment, the second payload contained in the target historical message is compared with the first payload to obtain a first comparison result, and the message consumption is performed based on the first comparison result. The first comparison result includes: data content in the first payload has been updated, deleted, or added compared to the second payload. This means that after receiving the message from the producer, it will be compared with the data in the database and executed based on the comparison result.

[0071] In one embodiment, consuming the message according to the first comparison result includes:

[0072] If the first comparison result indicates that the data content of the first payload is updated compared to the second payload, the field information to be updated in the first payload is obtained, and it is determined whether the field information to be updated exists in the second payload. If so, the field information to be updated in the first payload is directly updated to the second payload; if not, the field information to be updated in the first payload is directly inserted into the second payload;

[0073] If the first comparison result indicates that new data content has been added to the first payload compared to the second payload, then obtaining the field information to be added in the first payload and determining whether the field information to be added already exists in the second payload; if so, modifying the storage time of the field information to be added; if not, directly inserting the field information to be added in the first payload into the second payload;

[0074] If the first comparison result is that data content has been deleted between the first payload and the second payload, the deleted field information in the first payload is obtained, and it is determined whether the deleted field information already exists in the second payload. If so, the deleted field information is deleted; if not, the first payload is updated, the UTC timestamp of the first payload is regenerated, and it is re-added to the message priority queue, and the step of returning to the consumer side to obtain the message in the message priority queue is re-executed.

[0075] In this embodiment, if the first comparison result is that the data content of the first payload has been updated compared to the second payload, the field information to be updated in the first payload is obtained, and it is determined whether the field information to be updated exists in the second payload. If so, the field information to be updated in the first payload is directly updated to the second payload; if not, the field information to be updated in the first payload is directly inserted into the second payload. For example, the user name of user a is changed from user a to user b, and then the service identifier (user id of user a) is used to search the preset database to see if data a (user a's user name: user a) already exists. If so, data a can be changed to data b (a's user name: user b). If not, it is equivalent to directly adding a new data b (a's user name: user b).

[0076] In this embodiment, if the first comparison result is that new data content has been added between the first payload and the second payload, the field information to be added in the first payload is obtained, and it is determined whether the field information to be added already exists in the second payload. If so, the storage time of the field information to be added is modified; if not, the field information to be added in the first payload is directly inserted into the second payload.

[0077] In this embodiment, if the first comparison result shows that data content has been deleted compared to the first payload and the second payload, the deleted field information in the first payload is obtained, and it is determined whether the deleted field information already exists in the second payload. If so, the deleted field information is deleted; if not, the first payload is updated, the UTC timestamp of the first payload is regenerated, and the message is re-added to the message priority queue. The process returns to the consumer side and the step of obtaining the message in the message priority queue is re-executed. It can be understood that re-updating the first payload to the current timestamp and then processing this data is equivalent to adding it to the message queue and then processing it again.

[0078] It should be noted that if there is no updated data, new data or deleted data, disorder may occur. At this time, if disorder is found, if the updated data does not exist, it will be added immediately; if the newly added data already exists, only the update time will be modified; if the deleted data does not exist, it will be put back into the queue and the timestamp will be updated to the current time.

[0079] In one embodiment, to facilitate a better understanding of the method for ensuring the message queue consumption order based on timestamps, Figure 3 A flow chart of message consumption is provided for one embodiment of the present invention. In this embodiment, a method for ensuring the order of real-time data consumption based on timestamps and priority queues can effectively ensure the data accuracy and correctness of the business logic when the system processes business, and reduce the risk of errors caused by message disorder.

[0080] like Figure 3 As shown, the specific steps are as follows:

[0081] a1. Determine the type of change in data content between the first payload and the second payload.

[0082] a2. If the data content change is an update of the data content, obtain the field information to be updated in the first payload, and determine whether the field information to be updated exists in the second payload. If so, directly update the field information to be updated in the first payload to the second payload; if not, directly insert the field information to be updated in the first payload into the second payload.

[0083] a3. If the data content changes to include new data content between the first payload and the second payload, obtain the field information to be added in the first payload, and determine whether the field information to be added already exists in the second payload. If so, modify the storage time of the field information to be added; if not, directly insert the field information to be added in the first payload into the second payload.

[0084] a4. If the data content changes to deletion between the first payload and the second payload, obtain the deleted field information in the first payload and determine whether the deleted field information already exists in the second payload. If so, delete the deleted field information. If not, update the first payload, regenerate the UTC timestamp of the first payload, re-add it to the message priority queue, return to the consumer side, and re-execute the step of obtaining the message in the message priority queue.

[0085] In one embodiment, Figure 4This is a structural block diagram of a device for ensuring the consumption order of a distributed message queue provided by one embodiment of the present invention. The device is suitable for situations where the consumption order of a message queue is guaranteed. The device can be implemented by hardware / software. It can be configured in an electronic device to implement a method for ensuring the consumption order of a distributed message queue in an embodiment of the present invention. Figure 4 As shown, the device includes: a message generating module 410 , a parameter generating module 420 , a message storing module 430 , a message sorting module 440 and a message consuming module 450 .

[0086] The message generation module 410 is used by the producer to generate a message to be sent; wherein the message to be sent is a message indicating a change in business data; the change includes at least the addition, deletion, and update of the business data; the message to be sent corresponds to a message topic, and each message topic is divided into at least two partitions;

[0087] The parameter generation module 420 is configured to generate parameter information included in the message to be sent by the producer; wherein the parameter information includes at least: a timestamp and a first payload of the business data;

[0088] The message storage module 430 is used for the producer to send the message to be sent to the agent, so that the agent receives the message to be sent and stores the message to be sent as a message to be consumed in a message queue;

[0089] The message sorting module 440 is used for the consumer side to obtain the to-be-consumed messages in the message queue, and sort each to-be-consumed message in the message queue and its corresponding parameter information to obtain a sorted message priority queue;

[0090] The message consumption module 450 is used for the consumer end to obtain the to-be-consumed messages in the message priority queue and to consume the messages based on the first payload corresponding to each to-be-consumed message in the message priority queue.

[0091] In an embodiment of the present invention, a message generation module generates parameter information (timestamp and the first payload of business data) contained in a message to be sent, a message storage module receives messages and stores the messages to be consumed in a message queue, based on which a message consumption module obtains messages in the message queue, sorts each message to be consumed in the message queue and its corresponding parameter information to obtain a sorted message priority queue, and consumes messages based on the first payload corresponding to each message in the message priority queue. It can embed accurate timestamps in messages and combine with a priority queue processing mechanism to ensure the consumption order of messages in real-time data, thereby effectively ensuring data accuracy and the correctness of business logic when processing business, and reducing the risk of errors caused by message disorder.

[0092] In one embodiment, the parameter generation module 420 includes:

[0093] A conversion unit, configured to convert the message generation time corresponding to the to-be-sent message into a UTC timestamp format using a preset timestamp conversion tool to identify the generation time of the message;

[0094] A first storage unit, configured to store the UTC timestamp as field information of the message in a message body;

[0095] The second storage unit is used to obtain the changed content corresponding to the business data, and store the changed content as the first payload as the field information of the message in the message body; wherein the changed content at least includes: the newly added content, deleted content and updated content of the business data.

[0096] In one embodiment, the message sorting module 440 includes:

[0097] A timestamp determination unit, configured to determine the time points corresponding to the UTC timestamps of the messages to be consumed;

[0098] A writing unit is used to sort the messages to be consumed in each message queue according to the time point corresponding to the UTC timestamp, and write them into the local priority message queue in sequence; wherein, the sorting includes: sorting according to the UTC timestamp starting from the earliest UTC timestamp to obtain a message priority queue from the earliest to the latest.

[0099] In one embodiment, the message consumption module 450 includes:

[0100] The reading unit is used for the consumer side to read the messages to be consumed in the message priority queue through a preset dynamic window polling method and starting from the earliest UTC timestamp according to the UTC timestamp of each message to be consumed.

[0101] In one embodiment, the preset dynamic window polling method includes: after initializing the polling time window, each polling performs a sliding window according to a preset length and counts the message processing throughput corresponding to a preset number of messages;

[0102] When the rate of increase of the message processing throughput exceeds a first preset threshold, the polling time window is reduced each time by a preset first percentage;

[0103] When the rate of change of decrease in the message processing throughput exceeds a second preset threshold, the polling time window is expanded each time by a proportion of the preset first percentage; wherein the first preset threshold is greater than the second preset threshold.

[0104] In one embodiment, the message consumption module 450 includes:

[0105] A first payload determining unit, configured to obtain a first payload corresponding to each message to be consumed; wherein the first payload represents the data content corresponding to the message to be consumed;

[0106] a search unit, configured to extract a corresponding service identifier from the first payload, obtain stored historical messages from a preset database, and use the service identifier to search the preset database for a target historical message corresponding to the first payload;

[0107] A comparison unit is used to compare the second payload contained in the target historical message with the first payload to obtain a first comparison result, and to consume the message based on the first comparison result; wherein the first comparison result includes: the first payload has updated, deleted, and added data content compared with the second payload.

[0108] In one embodiment, the comparison unit includes:

[0109] a content update subunit, configured to, if the first comparison result indicates that data content of the first payload has been updated compared to the second payload, obtain field information to be updated in the first payload, and determine whether the field information to be updated exists in the second payload; if so, directly update the field information to be updated in the first payload into the second payload; if not, directly insert the field information to be updated in the first payload into the second payload;

[0110] a content adding subunit, configured to, if the first comparison result indicates that new data content has been added to the first payload compared to the second payload, obtain field information to be added in the first payload, and determine whether the field information to be added already exists in the second payload; if so, modify the storage time of the field information to be added; if not, directly insert the field information to be added in the first payload into the second payload;

[0111] A content deletion sub-unit is used to obtain the deleted field information in the first payload and determine whether the deleted field information already exists in the second payload if the first comparison result is that data content has been deleted between the first payload and the second payload. If so, the deleted field information is deleted; if not, the first payload is updated, the UTC timestamp of the first payload is regenerated, and the first payload is re-added to the message priority queue, and the step of returning to the consumer side to obtain the message in the message priority queue is re-executed.

[0112] The device for ensuring the consumption order of distributed message queues provided by an embodiment of the present invention can execute the method for ensuring the consumption order of message queues based on timestamps provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0113] In one embodiment, Figure 5 A schematic diagram of the structure of an electronic device provided for an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0114] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0115] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0116] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the method for ensuring the consumption order of a distributed message queue.

[0117] In some embodiments, the method for ensuring the order of consumption of distributed message queues may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for ensuring the order of consumption of message queues based on timestamps described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for ensuring the order of consumption of distributed message queues in any other appropriate manner (e.g., by means of firmware).

[0118] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] Computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device that ensures the order of consumption of distributed message queues, so that when executed by the processor, the computer programs implement the functions / operations specified in the flowcharts and / or block diagrams. The computer programs can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0120] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0122] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0123] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0124] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0125] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for ensuring the consumption order of a distributed message queue, characterized in that: The method comprises: The producer generates a message to be sent; wherein the message to be sent is a message indicating a change in business data; the change includes at least the addition, deletion, and update of the business data; the message to be sent corresponds to a message topic, and each message topic is divided into at least two partitions; The producer generates parameter information included in the message to be sent; wherein the parameter information includes at least: a timestamp and a first payload of the business data; The producer sends the message to be sent to the proxy, so that the proxy receives the message to be sent and stores the message to be sent as a message to be consumed in a message queue; The consumer side obtains the messages to be consumed in the message queue, and sorts each message to be consumed in the message queue and its corresponding parameter information to obtain a sorted message priority queue; The consumer end obtains the to-be-consumed messages in the message priority queue, and consumes the messages based on the first payload corresponding to each to-be-consumed message in the message priority queue.

2. The method according to claim 1, characterized in that The producer generates parameter information contained in the message to be sent, including: Use a preset timestamp conversion tool to convert the message generation time corresponding to the to-be-sent message into a UTC timestamp format to identify the generation time of the message; Storing the UTC timestamp in the message body as field information of the message; The changed content corresponding to the business data is obtained, and the changed content is stored as the first payload as the field information of the message in the message body; wherein the changed content at least includes: the newly added content, deleted content and updated content of the business data.

3. The method according to claim 1, characterized in that The consumer end obtains the to-be-consumed messages in the message queue, and sorts each to-be-consumed message in the message queue and its corresponding parameter information to obtain a sorted message priority queue, including: Determine the time point corresponding to the UTC timestamp of each to-be-consumed message in the message queue; The messages to be consumed in each message queue are sorted according to the time point corresponding to the UTC timestamp, and written into the local priority message queue in sequence; wherein, the sorting includes: sorting according to the UTC timestamp starting from the earliest UTC timestamp to obtain a message priority queue from the earliest to the latest.

4. The method according to claim 1, wherein The consumer end obtains the to-be-consumed message in the message priority queue, including: The consumer side reads the messages to be consumed in the message priority queue in a preset dynamic window polling manner and starts reading the messages according to the UTC timestamps of the messages to be consumed starting from the earliest UTC timestamp.

5. The method according to claim 4, characterized in that The preset dynamic window polling method includes: after initializing the polling time window, each polling performs a sliding window according to a preset length and counts the message processing throughput corresponding to the preset number of messages; When the rate of increase of the message processing throughput exceeds a first preset threshold, the polling time window is reduced each time by a preset first percentage; When the rate of change of decrease in the message processing throughput exceeds a second preset threshold, the polling time window is expanded each time by a proportion of the preset first percentage; wherein the first preset threshold is greater than the second preset threshold.

6. The method according to claim 1, characterized in that The consuming the message based on the first payload corresponding to each to-be-consumed message in the message priority queue includes: Obtaining a first payload corresponding to each message to be consumed; wherein the first payload represents the data content corresponding to the message to be consumed; Extracting a corresponding service identifier from the first payload, obtaining stored historical messages from a preset database, and searching the preset database for a target historical message corresponding to the first payload using the service identifier; The second payload contained in the target historical message is compared with the first payload to obtain a first comparison result, and the message is consumed based on the first comparison result; wherein the first comparison result includes: the first payload has updated, deleted, and added data content compared with the second payload.

7. The method according to claim 6, characterized in that The consuming the message according to the first comparison result includes: If the first comparison result indicates that the data content of the first payload is updated compared to the second payload, obtaining the field information to be updated in the first payload, and determining whether the field information to be updated exists in the second payload; if so, directly updating the field information to be updated in the first payload to the second payload; if not, directly inserting the field information to be updated in the first payload into the second payload; If the first comparison result indicates that new data content is added between the first payload and the second payload, obtaining field information to be added in the first payload, and determining whether the field information to be added already exists in the second payload; if so, modifying the storage time of the field information to be added; if not, directly inserting the field information to be added in the first payload into the second payload; If the first comparison result is that data content has been deleted compared to the second payload, the deleted field information in the first payload is obtained, and it is determined whether the deleted field information already exists in the second payload. If so, the deleted field information is deleted; if not, the first payload is updated, the UTC timestamp of the first payload is regenerated, and it is re-added to the message priority queue, and the step of returning to the consumer side to obtain the message in the message priority queue is re-executed.

8. A device for ensuring the consumption order of a distributed message queue, characterized in that: The device comprises: A message generation module is used for the producer to generate a message to be sent; wherein the message to be sent is a message indicating a change in business data; the change includes at least the addition, deletion, and update of the business data; the message to be sent corresponds to a message topic, and each message topic is divided into at least two partitions; A parameter generation module, configured to generate parameter information included in the message to be sent by the producer; wherein the parameter information includes at least: a timestamp and a first payload of the business data; A message storage module is used for the producer to send the message to be sent to the agent, so that the agent receives the message to be sent and stores the message to be sent as a message to be consumed in a message queue; A message sorting module is used for the consumer side to obtain the messages to be consumed in the message queue, and sort each message to be consumed in the message queue and its corresponding parameter information to obtain a sorted message priority queue; The message consumption module is used for the consumer end to obtain the to-be-consumed messages in the message priority queue and to consume the messages based on the first payload corresponding to each to-be-consumed message in the message priority queue.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method for ensuring the consumption order of a distributed message queue according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for ensuring the consumption order of a distributed message queue according to any one of claims 1 to 7 when executed.

Images