Message queue processing method and device

Abstract

The invention relates to the technical field of data processing, and discloses a message queue processing method and device, and the method comprises the steps: receiving a message queue sent by a producer; the messages in the message queue are analyzed, message parameters of the messages are obtained, and the message parameters comprise message themes, message primary keys and generation time; according to the message parameters of the currently processed first message, a target message is searched from a preset message log table, the message theme and the message main key of the target message are the same as the message theme and the message main key of the first message, and the generation time of the target message is after the generation time of the first message; and processing the target messages in sequence. According to the message processing method and device, orderliness processing of the message queue can be achieved through repeated processing of the out-of-order messages, the accuracy of consumer data is guaranteed, the message data quality and message processing efficiency of consumers are improved, and the message out-of-order problem existing in an asynchronous sending mechanism is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a message queue processing method and device. BACKGROUND

[0002] The communication methods between product applications in current commercial banks mainly include RPC (Remote Procedure Call) and message queue, etc. The communication method of message queue is usually used in the scene where the time efficiency requirement of system communication is not strict. In the distributed architecture, the consumer system consumes the messages pushed by the producer into the message queue through pulling. In order to prevent the messages of the same entity continuous operation from being consumed out of order due to being sent to different partitions, each product application specifies a Key (key business primary key) for message sending, so as to ensure that the messages with the same Key are sent to the same partition, and realize the order of the messages in the partition. However, in the extreme scene of network jitter or the scene of asynchronous re-sending of the messages with sending failure, the messages sent by the producer to the message queue are still out of order, which leads to the out-of-order consumption of the messages in the same partition by the consumer.

[0003] In order to solve the out-of-order consumption problem of the consumer, the following two schemes are adopted in the related technology: 1) The first scheme is that the producer uses a synchronous sending mechanism to push the messages. This message sending method is usually suitable for the scene with high message order requirement. The producer uses the synchronous sending method to send the messages. Although the performance of message delivery is lost, it can strictly ensure that the messages with the same Key of the same topic are sequentially sent to the same partition. The messages in the same partition are consumed by a single thread, so as to ensure that the data in the same partition is sequentially processed. However, because the message sending process will block and wait for the confirmation response of the server, the message sending performance is poor (especially in the scene with high network delay). If the Broker (message queue service end node) fails or the network is abnormal, the sending thread may be blocked for a long time. 2) The second scheme is that the producer sends the messages with the same Key to the same partition, and the consumer receives the messages, parses the messages, and writes the messages into a message consumption registration table for processing. The asynchronous message processing thread polls the message registration table, queries the messages in the table within a certain time, sorts the messages according to the message generation time, and then processes the messages in sequence, so as to realize the ordered consumption of the messages. That is, the present scheme ensures the order of the messages within a period of time based on the delay processing of the consumer. However, the message delay processing reduces the timeliness of message processing. In addition, the selection of the time window of delay processing needs to be carefully considered. If the time window is too large, the processing efficiency of the messages will be affected. If the time window is too small, the messages may be missed.

[0004] From the above, due to the uncertainty of message sending (such as network delay, retransmission after sending failure, etc.), the messages in the same topic, the same Key and the same partition are out of order, and in the related technology, in order to ensure the ordered processing of the consumer message, the processing scheme may exist the problems of message blocking, message delay processing and missing processing, which affects the message data quality of the consumer. SUMMARY

[0005] The main purpose of the present application is to provide a message queue processing method and device, so as to at least solve the technical problem of the related art that the message sent by the producer to the consumer in the same partition is out of order, which affects the message data quality of the consumer.

[0006] To solve the above technical problem, the present application provides a message queue processing method, comprising:

[0007] receiving a message queue sent by a producer;

[0008] parsing the message in the message queue to obtain a message parameter of the message, wherein the message parameter comprises a message topic, a message primary key and a generation time;

[0009] According to the message parameter of the first message currently processed, a target message is found in a preset message log table, wherein the message topic and the message primary key of the target message are the same as the message topic and the message primary key of the first message, and the generation time of the target message is located after the generation time of the first message;

[0010] processing the target message in order.

[0011] In some embodiments, after parsing the message in the message queue, the method further comprises:

[0012] processing each message in the message queue in order according to the parsing result of the message;

[0013] According to the message processing order, the processing parameter of each message is stored in the message log table, wherein the processing parameter comprises the message parameter.

[0014] In some embodiments, according to the message parameter of the first message currently processed, a target message is found in a preset message log table, comprising:

[0015] According to the message topic and the message primary key of the first message, the second message located before the generation time of the first message in a preset processing time window and / or a preset message quantity threshold range is determined as the target message.

[0016] In some embodiments, the second message located before the generation time of the first message within the preset processing time window is determined as the target message, comprising:

[0017] determining whether the time difference between the maximum generation time of the second message and the first generation time of the first message is less than the processing time window;

[0018] if yes, determining the second message as the target message; if no, determining the second message located within the processing time window as the target message.

[0019] In some embodiments, the method further comprises:

[0020] if the time difference between the maximum generation time of the second message and the first generation time of the first message is greater than or equal to the processing time window, a warning prompt is given to the second message exceeding the processing time window.

[0021] In some embodiments, the method further comprises:

[0022] if the number of the second messages exceeding the processing time window reaches a first preset number threshold, the size of the processing time window is determined according to the generation time of each second message.

[0023] In some embodiments, the target message is processed in sequence, comprising:

[0024] the target message is arranged in order according to the generation time of the message;

[0025] the arranged target message is processed in sequence based on the idempotent processing mode.

[0026] In some embodiments, the arranged target message is processed in sequence based on the idempotent processing mode, comprising:

[0027] if the type of the target message is an addition type, the target message is processed by message replacement;

[0028] if the target message is an update type, it is determined whether the first message body before the update is contained in the message log table, if yes, the first message body before the update is updated to a second message body; if no, the data in the second message body after the update is replaced;

[0029] if the target message is a deletion type, if the message log table contains the target message, the contract state of the target message is updated to an invalid state; if the message log table does not contain the target message, the target message is not processed.

[0030] In some embodiments, after the message in the message queue is parsed, the method further comprises:

[0031] determining whether a preset ordered consumption switch is on;

[0032] if yes, processing the message in the message queue, and after the first message is processed, searching the target message from a preset message log table;

[0033] if no, processing the message in the message queue, and after each message processing is completed, confirming that the message processing is completed.

[0034] The embodiments of the present application also provide a message queue processing device, comprising:

[0035] a receiving module configured to receive a message queue sent by a producer;

[0036] a parsing module configured to parse a message in the message queue to obtain a message parameter of the message, wherein the message parameter comprises a message topic, a message primary key and a generation time;

[0037] a searching module configured to search a target message from a preset message log table according to a message parameter of a first message currently processed, wherein the target message has the same message topic and message primary key as the first message, and the generation time of the target message is later than the generation time of the first message;

[0038] a processing module configured to process the target message in order.

[0039] The embodiments of the present application also provide an electronic device comprising at least a memory and a processor, wherein the memory stores a computer program, and the processor implements the above message queue processing method when executing the computer program stored in the memory.

[0040] The embodiments of the present application also provide a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the above message queue processing method.

[0041] The message queue processing method, apparatus, electronic device, and storage medium provided in this application embodiment receive a message queue sent by a producer; parse the messages in the message queue to obtain message parameters of the messages, wherein the message parameters include message topic, message primary key, and generation time; based on the message parameters of the currently processed first message, search for a target message in a preset message log table, wherein the message topic and message primary key of the target message are the same as those of the first message, and the generation time of the target message is after the generation time of the first message; process the target messages sequentially, allowing for processing of messages with the same topic and message primary key. When messages in the primary key become out of order, after consuming the current out-of-order message, the target messages that were selected before the generation time of the current out-of-order message are repeatedly processed. This achieves ordered processing of the message queue, ensures the accuracy of consumer data, improves the quality of consumer message data, and eliminates the need for message delay processing, thereby improving message processing efficiency. In addition, this application does not restrict the message delivery method of the producer. Regardless of whether the producer uses synchronous or asynchronous sending methods, it can correctly handle message out-of-order issues within the same partition. Especially when the producer uses asynchronous sending methods for message delivery, it can both ensure message delivery efficiency and solve the message out-of-order problem inherent in asynchronous sending mechanisms. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a first flowchart of a message queue processing method according to an embodiment of this application;

[0044] Figure 2 This is a second flowchart of a message queue processing method according to an embodiment of this application;

[0045] Figure 3 This is a third flowchart of a message queue processing method according to an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the consumer log table in the message queue processing method of this application embodiment;

[0047] Figure 5 This is a schematic diagram illustrating message processing based on idempotent processing in an embodiment of this application;

[0048] Figure 6This is a schematic diagram of the structure of a message queue processing device according to an embodiment of this application. Detailed Implementation

[0049] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0050] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0051] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0052] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0053] 1) Message Queues (MQ): A message queue is a communication method between applications used to transfer data between different applications or services. It acts as an intermediary between the sender and receiver, ensuring that messages are reliably delivered from the sender to the receiver. Message queue technology is widely used in distributed systems to achieve functions such as decoupling, asynchronous processing, and load balancing.

[0054] 1) Message: A message is a unit of information transmitted in a message queue, which includes the message body (the actual data to be transmitted) and related attributes (such as message ID, timestamp, etc.).

[0055] 3) Producer: The application or service responsible for creating and sending messages to the message queue.

[0056] 4) Consumer: An application or service that retrieves and processes messages from the message queue.

[0057] 5) Message Topic: A category identifier for messages, representing a message delivery method under a publish / subscribe model. Producers publish messages to specific topics, while consumers subscribe to topics to receive messages. Similar data will be stored in the same message topic.

[0058] 6) Partitioning: Topic data is stored in physical units. A Topic has multiple partitions, and the data of a Topic is distributed across different partitions. Data is not duplicated between partitions. Messages within the same partition are processed sequentially, while messages between different partitions are processed independently.

[0059] 7) Key: The key is an attribute value in the message that determines which partition the message is routed to. A specific algorithm (such as hashing) maps the key to the target partition, ensuring that messages with the same key always go to the same partition.

[0060] 8) Broker: The server node of the message queue, responsible for receiving messages from producers, storing messages, and pushing messages to consumers.

[0061] When producers send messages to the message queue, they use two methods: synchronous and asynchronous sending. Synchronous sending involves the producer sending a message and then blocking, waiting for confirmation from the server before sending the next message. Synchronous sending confirms the success of the previous message and is often used in high-reliability scenarios, but the blocking process significantly impacts performance. Asynchronous sending, on the other hand, involves the producer sending a message and then immediately processing the next message without waiting for confirmation, handling the sending result through a callback function. For failed messages, a compensation mechanism ensures reliability. Because asynchronous sending is non-blocking, a large number of messages can be sent concurrently, resulting in high throughput, and it is often used in scenarios requiring high throughput and low latency. Currently, to ensure the efficiency of data transmission from various product applications to consumer systems, producers mostly use asynchronous sending.

[0062] Each product application uses a key business primary key as the message key. A specific algorithm (such as hashing) maps messages with the same key value to the same target partition, ensuring that messages with the same key are ordered within the partition. However, in asynchronous producer sending mode, due to network jitter or compensation sending after asynchronous message sending failure, even if messages with the same key are sent to the designated partition, message out-of-order issues can occur within the message queue partition, affecting the accuracy of consumer data and reducing the quality of consumer message data.

[0063] In view of this, embodiments of this application provide a message queue processing method, apparatus, electronic device, and storage medium.

[0064] Therefore, as the consumer system for various product applications, it is necessary to develop an automated device that can process out-of-order messages within a partition, ensuring data accuracy and improving the data quality of the consumer system.

[0065] Example 1

[0066] Figures 1 to 3 A flowchart illustrating a message queue processing method according to an embodiment of this application is shown. Figures 1 to 3 As shown in the figure, this application embodiment provides a message queue processing method, including:

[0067] S101: Message queue for receiving messages sent by the producer.

[0068] The message queue processing method is applied to an ordered consumption device for consumers, which can receive message queues sent by producers in real time. Each producer (product application system) sends its own topic data through the message queue, delivering messages with the same topic and the same primary key to the same partition.

[0069] S102: Parse the messages in the message queue to obtain the message parameters of the messages, wherein the message parameters include message topic, message primary key and generation time.

[0070] like Figure 2 As shown, the ordered consumption device includes a message consumption module and a message processing module. After receiving the message queue, the message consumption module consumes messages from the message queue, parses each message in the message queue, and obtains the message parameters of each message.

[0071] S103: Based on the message parameters of the first message being processed, search for the target message in the preset message log table, wherein the message topic and message primary key of the target message are the same as the message topic and message primary key of the first message, and the generation time of the target message is after the generation time of the first message.

[0072] After parsing the messages in the message queue, the message consumption module sends the parsing results to the message processing module. The message processing module then processes the messages to ensure they are in the correct order. During the processing of messages in the message queue, if the message being processed is determined to be the first message in an out-of-order sequence, the message processing module searches a preset message log table for a target message with the same message topic and primary key as the first message, but located before the first message's creation time, based on the first message's message subject, primary key, and creation time.

[0073] like Figure 2 and Figure 4As shown, the message log table (hereinafter referred to as the message table) stores the processing parameters for each message. These parameters can include the message parameters obtained after parsing the message (such as the message generation time, message primary key, and message body), or specific message processing parameters, such as the message processing time (e.g., the message start processing time). The message body is the core data carrier of the message, containing the actual data that needs to be transmitted.

[0074] The target message has the same message topic and message key as the first message, indicating that the target message and the first message are in the same partition.

[0075] The target message is a message that the message processing module has already processed. However, the target message was generated after the first message was generated. That is, when the first message was processed, the target message generated later had already been processed. The first message is the currently out-of-order message, and the target message is a message that may need to be processed repeatedly (an out-of-order message to be processed).

[0076] S104: Process the target messages in sequence.

[0077] After the target message is identified, the message processing module performs repeated processing on the identified target message after the first message is processed, that is, it repeatedly consumes the target message, thereby achieving ordered processing of messages when the message queue is out of order.

[0078] The message queue processing method provided in this application embodiment involves receiving a message queue sent by a producer; parsing the messages in the message queue to obtain message parameters, wherein the message parameters include message topic, message primary key, and generation time; searching for a target message in a preset message log table based on the message parameters of the currently processed first message, wherein the message topic and message primary key of the target message are the same as those of the first message, and the generation time of the target message is after the generation time of the first message; and processing the target messages sequentially, allowing processing of messages with the same message topic and message primary key. When out-of-order messages occur, after consuming the current out-of-order message, the target messages that were selected before the generation time of the current out-of-order message are repeatedly processed. This ensures the ordered processing of the message queue, guarantees the accuracy of consumer data, improves the quality of consumer message data, and eliminates the need for message delay processing, thereby improving message processing efficiency. In addition, this application does not restrict the message delivery method of the producer. Regardless of whether the producer uses synchronous or asynchronous sending methods, it can correctly handle message out-of-order messages within the same partition. In particular, when the producer uses asynchronous sending methods for message delivery, it can both guarantee message delivery efficiency and solve the message out-of-order problem inherent in asynchronous sending mechanisms.

[0079] In some embodiments, after parsing the messages in the message queue in step S102, the method further includes:

[0080] S201: Based on the parsing result of the message, process each message in the message queue in sequence;

[0081] S202: Store the processing parameters of each message in the message log table according to the message processing order.

[0082] like Figures 2 to 4 As shown, the message consumption module parses the messages in the message queue one by one according to the message reception order, and obtains the parsing results. The message processing module processes each message in sequence (either the message reception order or the parsing order) according to the message parsing results, and stores (writes) the processing parameters to the message log table in the processing order.

[0083] It should be noted that the message consumption module can receive and parse multiple messages simultaneously, and the message processing module can process multiple parsed messages simultaneously. That is, the first message being processed can be one or more.

[0084] Consumers subscribe to various product application topics, and a single consumer thread consumes messages from a single partition, preventing out-of-order processing caused by multiple consumer threads consuming the same partition's data simultaneously. Each message log table can store processing parameters for messages within one partition, or it can store processing parameters for messages from multiple partitions. That is, within the same message log table, the message topic and message primary key for each message can be the same or different.

[0085] In some embodiments, step S103, searching for the target message from a preset message log table based on the message parameters of the currently processed first message, includes:

[0086] S1031: Based on the message topic and message key of the first message, determine the second message that is located before the generation time of the first message within the preset processing time window and / or within the preset message quantity threshold range as the target message.

[0087] Considering that a large number of messages in the message log table may be processed repeatedly in extreme scenarios, which may reduce message processing efficiency, in this embodiment, target messages can be filtered and searched from the message log table within a preset processing time window and / or within a preset message quantity threshold.

[0088] In practice, the process can be as follows: First, based on the generation time of the first message, find the second message that was generated before the generation time of the first message from the message log table; then, find the second message in the message list that is within the preset processing time window from the generation time of the current first message, and identify it as the target message; or find the second message in the message list that is within the preset message quantity threshold range, and identify it as the target message; or combine the preset processing time window and the preset message quantity threshold range to find the target message in the message list of the second message.

[0089] For example, if there are 8 second messages found in the message log table that are generated before the first message, and 3 of these second messages fall within the aforementioned processing time window (e.g., 20 minutes), then these 3 second messages are identified as target messages. If there are 8 second messages found in the message log table that are generated before the first message, and the preset message quantity threshold is 5, then the 5 second messages whose generation time is closest to the first message are identified as target messages. This eliminates the need to process second messages that are far removed from the generation time of the current first message, reducing the number of messages processed and improving the data processing performance of the message processing module.

[0090] Optional, such as Figure 4 As shown, in step S1031, the second message located before the generation time of the first message within the preset processing time window is determined as the target message, including:

[0091] S301: Determine whether the time difference between the maximum generation time of the second message and the first generation time of the first message is less than the processing time window;

[0092] S3021: If yes, determine that the second message is the target message; S3022: If no, determine that the second message located within the processing time window is the target message.

[0093] In this embodiment, when determining the target message according to the preset processing time window, it is first determined whether the time difference between the maximum generation time of the second message and the first generation time of the first message is less than the processing time window. If so, all second messages located before the generation time of the first message are determined as target messages, and then the target messages are processed repeatedly. If not, the second messages located within the processing time window are determined as target messages, and then the target messages are processed repeatedly, while the second messages outside the processing time window do not need to be processed repeatedly.

[0094] In some embodiments, the method further includes:

[0095] S3023: If the time difference between the maximum generation time of the second message and the first generation time of the first message is greater than or equal to the processing time window, a warning is issued for the second message that exceeds the processing time window.

[0096] If the time difference between the maximum generation time of the second message and the first generation time of the first message is greater than or equal to the processing time window, the message processing module can issue an early warning for the second message that exceeds the processing time window, prompting the user to manually intervene in the analysis.

[0097] Optionally, the method further includes:

[0098] S303: If the number of second messages exceeding the processing time window reaches a first preset number threshold, the size of the processing time window is determined according to the generation time of each second message.

[0099] The processing time window can be dynamically adjusted according to the actual out-of-order scenario. For example, if the current processing time window is set to 20 minutes, when the number of messages of the second message that exceeds the processing time window reaches the first preset number threshold, a large number of messages will trigger an alarm prompt that the processing time window has been exceeded. At this time, it is determined that there are a lot of out-of-order messages. At this time, the processing time window parameter can be dynamically adjusted according to the actual out-of-order time of the messages (for example, the processing time window can be increased to 30 minutes) to ensure the accuracy of data processing.

[0100] In some embodiments, the method further includes:

[0101] S401: Analyze messages in the production environment to determine the size of the processing time window and the size of the preset quantity threshold.

[0102] Before consuming messages received from producers, the ordered consumption device can analyze real out-of-order messages based on the generation time and / or quantity of real messages with the same consumption theme and primary key in the production environment. This analysis determines a reasonable out-of-order processing time window and / or out-of-order processing quantity threshold. Then, during the current first message processing, it filters target messages from a preset message log table that fall within the out-of-order processing time window and / or the out-of-order processing quantity threshold, preventing repeated processing of target messages and thus avoiding a significant drop in message processing performance. Simultaneously, during message processing, the size of the aforementioned processing time window and the preset quantity threshold can be dynamically optimized based on the target message filtering results.

[0103] In some embodiments, step S104 involves processing the target message sequentially, including:

[0104] S1041: Arrange the target messages in ascending order according to their message generation time;

[0105] S1042: Based on the idempotent processing method, the sorted target messages are processed in sequence.

[0106] After identifying the target messages that require repeated processing, the message processing module can arrange the target messages within the aforementioned processing time window and the preset quantity threshold range in an ordered manner according to their message generation time. Based on the idempotent mechanism of consumer message processing, repeated consumption of the ordered target messages can achieve ordered processing of out-of-order messages with the same consumption theme and the same consumption primary key. In this embodiment, through idempotent processing, processing the same message multiple times has the same effect as processing it once, and will not cause system state abnormalities, ensuring the accuracy of message processing.

[0107] In some embodiments, step S1042 involves processing the sorted target messages sequentially based on an idempotent processing method, including:

[0108] S501: If the target message is a new type, the target message is processed by message replacement;

[0109] S502: If the target message is an update type, determine whether the message log table contains the first message body before the update. If it does, update the first message body before the update to the second message body; if it does not, replace the data in the updated second message body.

[0110] S503: If the target message is of the deletion type, and if the message log table contains the target message, then update the contract status of the target message to the invalid status; if the message log table does not contain the target message, then do not process the target message.

[0111] When consumers lack idempotent message processing capabilities, processing the first message first and then repeatedly processing the selected target messages may not necessarily achieve an ordered processing result.

[0112] To ensure the accuracy of message processing, this embodiment processes target messages using an idempotent approach. For insert-type target messages, a replace method is used to overwrite the insert. For update-type target messages, if the message log table contains both the before message body (first message body) and the after message body (second message body), if the target message record exists in the before message body, the data in the before message body is updated to the data in the after message body; if the target message record does not exist in the before message body, the data in the after message body is replaced to update the data. For delete-type target messages, if the target message record exists in the message body, the contract status of the message is updated to invalid; if the target message record does not exist in the message body, no processing is performed.

[0113] Here, Before and After refer to the data state before and after data processing. The difference between Before and After is:

[0114] Before (for insert and update) can modify newly added data, while After cannot. Neither can modify existing data. For insert statements, only new is valid; for delete statements, only old is valid; for update statements, both new and old can be used simultaneously.

[0115] When message out-of-order delivery occurs, messages arrive before messages arrive. Although the earlier messages can be processed normally, the final state of the earlier messages is not updated correctly due to the delay in their arrival. Repeatedly executing the earlier messages can correctly correct the state of the earlier messages.

[0116] like Figure 5 As shown, taking two ordered INSERT messages as an example: when no out-of-order message occurs, the two messages are processed in order; when out-of-order message occurs (INSERT1 is processed later), the data of INSERT1 message will overwrite the data of INSERT2 message. By repeatedly processing INSERT2 messages whose message generation time is greater than the current message generation time, the data of INSERT2 message can be retrieved again, thus avoiding data processing errors.

[0117] In some embodiments, such as Figure 2 and Figure 3 As shown, after parsing the messages in the message queue, the method further includes:

[0118] S601: Determine whether the preset ordered consumption switch is turned on;

[0119] S602: If so, process the messages in the message queue, and after processing the first message, search for the target message in the preset message log table;

[0120] S603: If not, process the messages in the message queue, and confirm that message processing is complete after each message has been processed.

[0121] Before processing messages in the current message queue, the message processing module can first determine whether the ordered consumption switch is on. If the ordered consumption switch is off, it consumes the current message (current consumption list), processes the messages in the message queue, and stores or writes the processing parameters of the current message to the message log table (write message table), then ends the consumption of the current message (current consumption list). If the ordered consumption switch is on, it first consumes the current message (current consumption list), stores or writes the processing parameters of the current message to the message log table, then queries the message log table for target messages with the same message topic, the same message primary key, and a message generation timestamp greater than the generation timestamp of the current first message, and then processes the target messages repeatedly to achieve ordered message processing. After the current message (current consumption list) is processed, the message processing process ends.

[0122] In this embodiment, by judging the state of the ordered consumption switch, when an emergency exception occurs in the ordered consumption of messages that cannot be quickly recovered, the ordered consumption switch can be turned off in a timely manner to handle the exception and ensure the reliability of message processing.

[0123] Example 2

[0124] Figure 6 A schematic diagram of the message queue processing method according to an embodiment of this application is shown. Figure 6 As shown in the illustration, this application also provides a message queue processing apparatus, including:

[0125] Receiver module 20 is configured to receive message queues sent by the producer;

[0126] The parsing module 20 is configured to parse messages in the message queue to obtain message parameters of the messages, wherein the message parameters include message topic, message primary key and generation time;

[0127] The search module 30 is configured to search for a target message from a preset message log table based on the message parameters of the first message currently being processed, wherein the message topic and message primary key of the target message are the same as the message topic and message primary key of the first message, and the generation time of the target message is after the generation time of the first message;

[0128] The processing module 40 is configured to process the target message sequentially.

[0129] In some embodiments, after parsing the messages in the message queue, the processing module 40 is further configured to:

[0130] Based on the parsing results of the messages, the messages in the message queue are processed in sequence;

[0131] According to the message processing order, the processing parameters of each message are stored in the message log table, wherein the processing parameters include the message parameters.

[0132] In some embodiments, the lookup module 30 is further configured to:

[0133] Based on the message topic and message key of the first message, the second message located before the generation time of the first message within a preset processing time window and / or within a preset message quantity threshold range is determined as the target message.

[0134] In some embodiments, the lookup module 30 is further configured to:

[0135] Determine whether the time difference between the maximum generation time of the second message and the first generation time of the first message is less than the processing time window;

[0136] If yes, determine that the second message is the target message; if no, determine that the second message located within the processing time window is the target message.

[0137] In some embodiments, the lookup module 30 is further configured to:

[0138] If the time difference between the maximum generation time of the second message and the first generation time of the first message is greater than or equal to the processing time window, a warning is issued for the second message that exceeds the processing time window.

[0139] In some embodiments, the lookup module 30 is further configured to:

[0140] If the number of second messages exceeding the processing time window reaches a first preset threshold, the size of the processing time window is determined based on the generation time of each second message.

[0141] In some embodiments, the processing module 40 is further configured to:

[0142] The target messages are arranged in ascending order of message generation time;

[0143] Based on the idempotent processing method, the sorted target messages are processed sequentially.

[0144] In some embodiments, the processing module 40 is further configured to:

[0145] If the target message is a new type, the target message is processed by message replacement;

[0146] If the target message is an update type, determine whether the message log table contains the first message body before the update. If it does, update the first message body before the update to the second message body; if it does not, replace the data in the updated second message body.

[0147] If the target message is of the deletion type, and the target message is included in the message log table, then the contract status of the target message is updated to invalid; if the target message is not included in the message log table, then no action is taken on the target message.

[0148] In some embodiments, after the parsing module 20 parses the messages in the message queue, the processing module 40 is further configured to:

[0149] Determine whether the preset orderly consumption switch is turned on;

[0150] If so, process the messages in the message queue, and after processing the first message, search for the target message in the preset message log table;

[0151] If not, process the messages in the message queue, and confirm that message processing is complete after each message has been processed.

[0152] The message queue processing apparatus provided in this application corresponds to the message queue processing method in the above embodiments. Any option in the message queue processing method embodiments is also applicable to the embodiments of the message queue processing apparatus, and will not be repeated here.

[0153] Example 3

[0154] This application also provides an electronic device, which includes at least a memory and a processor. The memory stores a computer program, and the processor implements the message queue processing method described above when executing the computer program in the memory.

[0155] In some embodiments, the processor executing a computer program may be a processing device that includes one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc. More specifically, the processor may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor that runs other instruction sets, or a processor that runs a combination of instruction sets. The processor may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), system-on-a-chip (SoCs), etc.

[0156] Memory can be read-only memory (ROM), random access memory (RAM), phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), electrically erasable programmable read-only memory (EEPROM), other types of random access memory (RAM), flash drives or other forms of flash memory, cache, registers, static memory, optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape cassette or other magnetic storage devices, or any other possible non-transitory medium used to store information or instructions that can be accessed by computer equipment. Electronic devices may also include input / output interfaces, network interfaces, and memory, etc.

[0157] The electronic devices in this application embodiment may include, but are not limited to, fixed terminal devices such as servers, desktop computers, and digital TVs, as well as mobile terminal devices such as in-vehicle devices (e.g., head-up displays), handheld devices (e.g., mobile phones, tablets, etc.), and wearable devices (e.g., smartwatches, smart bracelets, etc.).

[0158] Example 4

[0159] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the message queue processing method described above.

[0160] The computer-readable storage medium in this application embodiment can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. In this application embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device; for example, it can be the aforementioned memory.

[0161] Those skilled in the art will understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0162] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0167] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A message queue processing method, characterized in that, include: A message queue that receives messages sent by the producer; The messages in the message queue are parsed to obtain the message parameters of the messages, wherein the message parameters include message topic, message primary key and generation time; Based on the message parameters of the first message being processed, the target message is searched from the preset message log table. The message topic and message key of the target message are the same as those of the first message, and the generation time of the target message is after the generation time of the first message. The target messages are processed sequentially.

2. The method according to claim 1, characterized in that, After parsing the messages in the message queue, the method further includes: Based on the parsing results of the messages, the messages in the message queue are processed in sequence; According to the message processing order, the processing parameters of each message are stored in the message log table, wherein the processing parameters include the message parameters.

3. The method according to claim 1, characterized in that, Based on the message parameters of the first message being processed, the target message is retrieved from the preset message log table, including: Based on the message topic and message key of the first message, the second message located before the generation time of the first message within a preset processing time window and / or within a preset message quantity threshold range is determined as the target message.

4. The method according to claim 3, characterized in that, The second message, occurring before the generation time of the first message within a preset processing time window, is identified as the target message, including: Determine whether the time difference between the maximum generation time of the second message and the first generation time of the first message is less than the processing time window; If yes, determine that the second message is the target message; if no, determine that the second message located within the processing time window is the target message.

5. The method according to claim 4, characterized in that, The method further includes: If the time difference between the maximum generation time of the second message and the first generation time of the first message is greater than or equal to the processing time window, a warning is issued for the second message that exceeds the processing time window.

6. The method according to claim 4, characterized in that, The method further includes: If the number of second messages exceeding the processing time window reaches a first preset threshold, the size of the processing time window is determined based on the generation time of each second message.

7. The method according to claim 1, characterized in that, The target message is processed sequentially, including: The target messages are arranged in ascending order of message generation time; Based on the idempotent processing method, the sorted target messages are processed sequentially.

8. The method according to claim 7, characterized in that, Based on the idempotent processing method, the sorted target messages are processed sequentially, including: If the target message is a new type, the target message is processed by message replacement; If the target message is an update type, determine whether the message log table contains the first message body before the update. If it does, update the first message body before the update to the second message body; if it does not, replace the data in the updated second message body. If the target message is of the deletion type, and the target message is included in the message log table, then the contract status of the target message is updated to invalid; if the target message is not included in the message log table, then no action is taken on the target message.

9. The method according to claim 1, characterized in that, After parsing the messages in the message queue, the method further includes: Determine whether the preset orderly consumption switch is turned on; If so, process the messages in the message queue, and after processing the first message, search for the target message in the preset message log table; If not, process the messages in the message queue, and confirm that message processing is complete after each message has been processed.

10. A message queue processing apparatus, characterized in that, include: The receiving module is configured to receive message queues sent by the producer; The parsing module is configured to parse messages in the message queue to obtain message parameters of the messages, wherein the message parameters include message topic, message primary key and generation time; The search module is configured to search for a target message from a preset message log table based on the message parameters of the first message currently being processed, wherein the message topic and message primary key of the target message are the same as the message topic and message primary key of the first message, and the generation time of the target message is after the generation time of the first message; The processing module is configured to process the target messages sequentially.

Images