A sequential message processing method, apparatus, device, and storage medium

Abstract

This application discloses a sequential message processing method, apparatus, device, and storage medium. The technical solution provided by this application includes: retrieving a first message from a first message queue at a message management terminal; determining whether the first message meets preset consumption conditions based on key information of the first message; wherein messages configured with the same key information are stored in the same first message queue according to a preset consumption order; when the first message meets the consumption conditions, consuming the first message, and sending a first transfer instruction to the message management terminal when the first message consumption fails, so that the message management terminal creates a first message node corresponding to the key information, and transfers the first message from the first message queue to a preset message queue through the first message node. This technical solution addresses the problems of low message consumption efficiency and message congestion in existing technologies, improving message consumption efficiency and ensuring the stability of the consumption system.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a sequential message processing method, apparatus, device, and storage medium. Background Technology

[0002] In a messaging system, message producers generate sequential messages for message consumers to consume. Sequential messages consist of multiple messages that are consumed in a preset order. Message consumers must consume sequential messages in the order they are consumed to avoid information confusion in the messaging system.

[0003] Current sequential message processing methods store messages in the same message queue according to the consumption order. Message consumers then pull and consume messages from the head of the queue according to the first-in, first-out (FIFO) principle. If a message consumer fails to consume the message from the head of the queue, it will retry until successful. However, when the order of non-head messages in the message queue is unrelated to the order of head messages, if the consumption time of such messages is long, it will severely impact the consumption efficiency of subsequent messages, leading to message blocking and other anomalies in the message system, thus affecting the stability of the message system. Summary of the Invention

[0004] This application provides a sequential message processing method, apparatus, device, and storage medium to solve the problems of low message consumption efficiency and message congestion in the prior art, thereby improving the consumption efficiency of sequential messages and ensuring the stability of the consumption system.

[0005] Firstly, this application provides a sequential message processing method, including:

[0006] Retrieve the first message from the first message queue in the message management console;

[0007] Based on the key information of the first message, determine whether the first message meets the preset consumption conditions; wherein, messages with the same key information are stored in the same first message queue according to the preset consumption order;

[0008] When the first message meets the consumption conditions, the first message is consumed. If the consumption of the first message fails, a first transfer instruction is sent to the message management terminal so that the message management terminal creates a first message node corresponding to the key information and transfers the first message from the first message queue to a preset message queue through the first message node.

[0009] Secondly, this application provides a sequential message processing method, including:

[0010] Based on the key information of the first message, the first message is added to the corresponding first message queue according to the preset consumption order;

[0011] Upon receiving a first transfer instruction from the message consumer, a first message node corresponding to the key information of the first message is created, and the first message is transferred from the first message queue to a preset message queue through the first message node.

[0012] Thirdly, this application provides a sequential message processing apparatus, comprising:

[0013] The first message retrieval module is configured to retrieve the first message from the first message queue of the message management terminal.

[0014] The condition judgment module is configured to determine whether the first message meets the preset consumption conditions based on the key information of the first message; wherein messages with the same key information are stored in the same first message queue according to the preset consumption order.

[0015] The first instruction sending module is configured to consume the first message when it is determined that the first message meets the consumption conditions, and to send a first transfer instruction to the message management terminal when the consumption of the first message fails, so that the message management terminal creates a first message node corresponding to the key information, and transfers the first message from the first message queue to a preset message queue through the first message node.

[0016] Fourthly, this application provides a sequential message processing apparatus, comprising:

[0017] The message adding module is configured to add the first message to the corresponding first message queue according to a preset consumption order based on the key information of the first message;

[0018] The first message transfer module is configured to, upon receiving a first transfer instruction sent by the message consumer, create a first message node corresponding to the key information of the first message, and transfer the first message from the first message queue to a preset message queue through the first message node.

[0019] Fifthly, this application provides a sequential message processing device, including: a memory and one or more processors;

[0020] The memory is used to store one or more programs;

[0021] When the one or more programs are executed by the one or more processors, the one or more processors implement the sequential message processing method as described in the first or second aspect.

[0022] In a sixth aspect, this application provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the sequential message processing method as described in the first or second aspect.

[0023] This application determines whether consuming the first message satisfies its corresponding consumption order based on key information of the first message. If not, the first message is not consumed to adhere to the consumption order of sequential messages and avoid consumption errors. If the first message satisfies the order, it is consumed. In the event of consumption failure, a corresponding message node is created to record the key information of the first message for subsequent checks on whether messages belonging to the same sequential message satisfy the consumption order. The first message is then transferred to a preset message queue via the message node, preventing it from blocking subsequent messages in the queue, improving the efficiency of sequential message consumption, and ensuring the stability of the consumption system. Attached Figure Description

[0024] Figure 1 This is a flowchart of a sequential message processing method provided in an embodiment of this application;

[0025] Figure 2 This is a flowchart provided in an embodiment of the present application for a message consumer to determine whether a first message meets the consumption conditions;

[0026] Figure 3 This is a flowchart of the consumption management terminal transferring the first message according to the first transfer instruction provided in the embodiments of this application;

[0027] Figure 4 This is a schematic diagram of the index tree structure provided in an embodiment of this application;

[0028] Figure 5 This is a flowchart illustrating how the message management terminal, according to a second transfer instruction, transfers the first message based on the second transfer instruction, as provided in this application embodiment.

[0029] Figure 6 This is a flowchart illustrating another sequential message processing method provided in an embodiment of this application;

[0030] Figure 7 This is a flowchart illustrating how the message management terminal, as provided in this application embodiment, adds a third message from the head of a third message queue to the tail of a second message queue;

[0031] Figure 8 This is a schematic diagram of the structure of a sequential message processing device provided in an embodiment of this application;

[0032] Figure 9 This is a schematic diagram of another sequential message processing device provided in an embodiment of this application;

[0033] Figure 10 This is a schematic diagram of the structure of a sequential message processing device provided in an embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0035] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0036] The sequential message processing method provided in this embodiment can be executed by a sequential message processing device, which can be implemented by software and / or hardware. The sequential message processing device can consist of two or more physical entities, or it can consist of a single physical entity. In this embodiment, the sequential message processing device can be a message system, or it can be a message management end and a message consumption end in a consumer system. Taking the Kakfa message subscription system as an example, the Kakfa message subscription system includes Producers, Brokers, and Consumers. Brokers act as the message management end in the message system, Consumers act as the message consumption end, and Producers act as the message production end. After producing a message, the Producers send it to the Brokers. The Brokers store the message body in the storage space and store the message's address information in that storage space at the tail of the message queue. Consumers pull the message's address information from the head of the message queue, parse the address information, pull the message body from the corresponding storage space, and consume the message body. In short, the message management end manages the messages generated by the message production end, and the message consumption end pulls messages from the message management end and consumes them. It should be noted that the messaging system in this embodiment is not limited to the Kafka message subscription system; other messaging systems involving message management and message consumption are also applicable to this embodiment.

[0037] In one embodiment, sequential messages include multiple messages consumed in a preset consumption order. For example, sequential messages could be order messages from a shopping platform, including order placement messages, packing messages, and shipping messages. The message consumer needs to process the order placement messages before processing the packing and shipping messages; processing the packing messages first would disrupt the order messages on the shopping platform. Traditional sequential message processing methods involve the message management end storing sequential messages in the same message queue according to the consumption order. The message consumer, following the first-in, first-out (FIFO) principle of the message queue, uses a single thread to pull the message at the head of the queue and acquire the queue lock to prevent other threads from pulling the message at the head of the queue, thus preventing duplicate message consumption. When a single thread fails to consume the message, it re-consumes the message after a certain interval until successful consumption. This interval is generally set relatively short to avoid blocking the application. However, if the message consumer experiences a network anomaly or the program restarts, the interval may be insufficient for the message consumer to recover, and messages may require a longer delay before being consumed. The message queue also stores other sequential messages. Since the single thread has locked the header message, other threads cannot consume the header message and other sequential messages in the message queue, which seriously affects the consumption efficiency of sequential messages, causing abnormal situations such as message blocking in the message system and affecting the stability of the message system.

[0038] To address the aforementioned issues, this application provides a sequential message processing method to prevent messages from blocking the message queue and improve message consumption efficiency.

[0039] Figure 1 A flowchart of a sequential message processing method provided in an embodiment of this application is given. The sequential message processing method provided in this application can be executed by a messaging system.

[0040] The following description uses a messaging system as the main example to illustrate sequential message processing. (Reference) Figure 1 The sequential message processing method includes:

[0041] S110. The message management terminal adds the first message to the corresponding first message queue according to the preset consumption order based on the key information of the first message.

[0042] Here, the first message refers to the message produced by the message producer. Each first message record has corresponding key information, which is used to mark the sequence of messages corresponding to the first message. First messages belonging to the same sequence have the same key information. In one embodiment, when defining the message structure of the first message, the message producer reserves a field to store the key information of the first message. For example, the message structure of the first message is {"mqKey": "12345", "orderId": "12345", "amount": 100}, where mqKey is the key information of the first message and orderId is the message identifier of the first message.

[0043] In one embodiment, the key information can be a keyword. Using the same keyword, first messages belonging to the same message sequence are stored in the same first message queue, improving indexing efficiency and reducing grouping difficulty. The key information can also be a hash value. Using the same hash value, first messages belonging to the same message sequence are stored in the same first message queue.

[0044] Furthermore, after receiving the first message sent by the message producer, the message management terminal stores the message body of the first message in a preset storage space and stores the address information of the message body in the storage space in the first message queue. The first message queue refers to the ordinary message queue in the message management terminal, which is used to store the first messages produced by the message producer. In this embodiment, according to the order in which the first messages were produced, first messages with the same key information are stored sequentially at the end of the same first message queue, with the first message produced earlier being stored first, ensuring that first messages belonging to the same sequence are stored in the same first message queue according to their corresponding consumption order.

[0045] S120, The message consumer pulls the first message from the first message queue of the message management terminal.

[0046] For example, the message consumer retrieves the address information of the first message from the head of the first message queue and locks the first message queue. Based on this address information, the message consumer retrieves the message body of the corresponding first message from the storage space, parses the fields of the message body to obtain the key information of the first message.

[0047] S130. The message consumer determines whether the first message meets the preset consumption conditions based on the key information of the first message; among them, messages with the same key information are stored in the same first message queue according to the preset consumption order.

[0048] The consumption condition can be understood as the conditions that must be met for the first message to be consumed in accordance with the corresponding consumption order. For example, when the first message meets the consumption condition, it indicates that the message consumer is consuming the first message in accordance with its consumption order, and therefore the message consumer can consume the first message. If the first message does not meet the consumption condition, it indicates that the message consumer is not consuming the first message in accordance with its consumption order, and therefore the message consumer cannot consume the first message.

[0049] In one embodiment, Figure 2 This is a flowchart illustrating how a message consumer determines whether a first message meets consumption conditions, as provided in an embodiment of this application. Figure 2 As shown, the steps for the message consumer to determine whether the first message meets the consumption conditions specifically include S1301-S1303:

[0050] S1301. The message consumer determines whether the preset index tree includes the first message node corresponding to the key information based on the key information of the first message.

[0051] S1302. When the message consumer determines that the index tree includes the first message node, it determines that the first message does not meet the consumption conditions.

[0052] S1303. When the message consumer determines that the index tree does not include the first message node, it determines that the first message meets the consumption conditions.

[0053] The index tree can be understood as a data structure configured by the message management end, used to record key information of failed sequential messages. For example, the index tree has multiple index nodes, each recording key information for a sequential message. When the consumption of a message in the sequence fails, the message management end records the key information of that message in an index node of the index tree, generating the message node corresponding to that key information. The message consumer can then determine the failed sequential message by querying the message nodes in the index tree.

[0054] In this embodiment, the message node in the index tree is queried based on the key information of the first message. If the message node corresponding to the key information is found in the index tree, it can be determined that the message to be consumed first in the sequential messages corresponding to the first message has failed to be consumed. In this case, consuming the first message will not follow the corresponding consumption order, thus determining that the first message does not meet the consumption conditions. Conversely, if the message node corresponding to the key information is not found in the index tree, it can be determined that the message to be consumed first in the sequential messages corresponding to the first message has been consumed successfully, or that the first message is the message to be consumed first in the sequential messages. In this case, the first message can follow the corresponding consumption order, thus determining that the first message meets the consumption conditions.

[0055] S140. When the message consumer determines that the first message meets the consumption conditions, it consumes the first message and sends a first transfer instruction to the message management end when the consumption of the first message fails.

[0056] For example, if the first message meets the consumption conditions, the message consumer consumes it. After successfully consuming the first message, the message consumer sends a third removal command to the message management terminal. Upon receiving the third removal command, the message management terminal removes the first message from the head of the first message queue, and all first messages in the first message queue move forward. After the message management terminal completes the third removal command, it notifies the message consumer that the third removal command has been completed, and the message consumer unlocks the first message queue so that other message consumers can pull and consume new first messages from the head of the first message queue.

[0057] Furthermore, if the message consumer cannot consume the first message normally due to network anomalies or program restarts, the message consumer will retry consuming the first message after a certain period of time. If the message consumer attempts to retrace a certain number of times, the consumption of the first message is deemed to have failed. When the consumption of the first message fails, the message consumer sends a first transfer instruction to the message management terminal. This first transfer instruction notifies the message consumer of the failure, enabling the message management terminal to create a first message node corresponding to the key information and transfer the first message from the first message queue to a preset message queue through the first message node. After completing the second transfer instruction, the message management terminal notifies the message consumer that the first transfer instruction has been completed, and the message consumer unlocks the first message queue so that other message consumers can pull and consume messages from the first message queue.

[0058] In one embodiment, the first preset number of times can be set according to the message quantity level of the consumption system. If there are few messages in the message system, the first preset number of times can be set to a larger value, and the failure to consume the first message will not seriously block the message system. If there are many messages in the message system, the first preset number of times can be set to a smaller value, and the failure to consume the first message will seriously block the message system.

[0059] S150. When the message management terminal receives the first transfer instruction sent by the message consumer terminal, it creates a first message node corresponding to the key information of the first message, and transfers the first message from the first message queue to the preset message queue through the first message node.

[0060] In one embodiment, the message management terminal determines the key information of the first message that failed to be consumed by the message consumer according to the first transfer instruction. Based on the key information, it creates a corresponding first message node on the index tree and transfers the first message to the tail of the second message queue and the tail of the third message queue corresponding to the first message node. Here, the first message node refers to the message node of the sequential message corresponding to the first message. As can be seen from the foregoing, the first message can be understood as the first message that failed to be consumed in the corresponding sequential message. The key information of the sequential message is recorded by the index node on the index tree to generate the message node corresponding to the key information. This message node represents the failure of the sequential message consumption corresponding to the recorded key information. Further, the message management terminal creates a third message queue corresponding to the message node at the leaf node of the message node. The message management terminal removes the address information of the first message from the first message queue and stores the address information of the first message in the tail of the third message queue corresponding to the message node, so as to record the message that failed to be consumed in the sequential message through the third message queue. It can be understood that the first message is the first message stored in the third message queue, therefore the address information of the first message is stored at the head of the third message queue. Furthermore, the message management system adds the message at the head of the third message queue to the tail of the second message queue. The second message queue can be understood as a non-blocking message queue, which stores the messages at the head of the third message queue corresponding to each message node. The message consumer pulls these failed-to-consume messages from the non-blocking message queue for secondary consumption.

[0061] In this embodiment, Figure 3 This is a flowchart illustrating how the consumer management terminal, according to a first transfer instruction, transfers a first message based on a first transfer instruction, as provided in this application embodiment. Figure 3 As shown, the steps for the consumer management terminal to transfer the first message according to the first transfer instruction specifically include S1501-S1503:

[0062] S1501, Create an index tree and a second message queue.

[0063] For example, if the first message is the first message to fail to be consumed in the messaging system, the consumption management end first creates an index tree and a second message queue. The index tree records key information of the failed sequential messages, ensuring that sequential messages are consumed in the corresponding message order. The second message queue caches the failed sequential messages so that they can be consumed again in subsequent calls.

[0064] S1502. Based on the key information of the first message, create the first message node corresponding to the key information on the index tree, and create the third message queue corresponding to the first message node.

[0065] In one embodiment, the index tree is a B+ tree data structure. Figure 4 This is a schematic diagram of the index tree structure provided in an embodiment of this application. For example... Figure 4 As shown, assuming the key information of the first message is "First Key Information," an index node is added to the index tree to store the first key information, thus completing the creation of the first message node. In the leaf nodes of this index node, a corresponding third message queue, a field recording the first key information, and a flag field are created. The flag field stores a flag that indicates whether the consumer management terminal is currently transferring messages from the first message queue to the corresponding third message queue. (Reference) Figure 4 The storage space adopts a commitLog message storage structure, which is used to store the message body. The message address information refers to the displacement of the message body in the commitLog. The first displacement is the displacement of the message body of the first message stored in the third message queue in the commitLog. In another embodiment, the index tree can also adopt a red-black tree, AVL tree, or other data structures with a time complexity of less than log2N.

[0066] Similarly, refer to Figure 4 Sequential messages with key information mqKey2 and mqKey3 are stored in the index tree just like sequential messages with key information mqKey1.

[0067] S1503. Transfer the first message to the tail of the second message queue and the tail of the third message queue corresponding to the first message node.

[0068] For example, after removing the address information of the first message from the head of the first message queue, the message management terminal adds the address information of the first message to the tail of the corresponding third message queue. The first message from the head of the third message queue is then added to the tail of the second message queue.

[0069] S160. When the message consumer determines that the first message does not meet the consumption conditions, it sends a second transfer instruction to the message management end.

[0070] For example, if the first message does not meet the consumption conditions, then the priority consumption message in the corresponding sequential messages will fail to be consumed. Consuming the first message in this case would disrupt the consumption order of the sequential messages, so it is not consumed. A second transfer instruction is sent to the message management terminal, notifying it that the first message is a sequential message that failed to be consumed. This allows the message management terminal to determine the first message node corresponding to the key information and transfer the first message from the first message queue to a preset message queue through the first message node. After completing the second transfer instruction, the message management terminal notifies the message consumer that the second transfer instruction has been completed, and the message consumer unlocks the first message queue so that other message consumers can pull and consume messages from the first message queue.

[0071] S170. When the message management terminal receives the second transfer instruction sent by the message consumer terminal, it determines the first message node corresponding to the key information of the first message based on the key information of the first message, and transfers the first message from the first message queue to the preset message queue through the first message node.

[0072] In one embodiment, Figure 5 This is a flowchart illustrating how the message management terminal, according to a second transfer instruction, transfers the first message based on the second transfer instruction, as provided in this application embodiment. Figure 5 As shown, the steps for the message management terminal to transfer the first message according to the second transfer instruction specifically include S1701-S1702:

[0073] S1701. Determine the corresponding first message node on the index tree based on the key information of the first message.

[0074] S1702, Transfer the first message to the tail of the third message queue corresponding to the first message node.

[0075] For example, as described above, if the index tree already contains message nodes with key information about the first message, then there is no need to create new message nodes. Instead, the corresponding message node is determined based on the key information, and the address information of the first message is added to the tail of the third message queue. (See reference) Figure 4 Assuming the key information of the first message is the first key information, and the first message is the second message in the corresponding sequential message list to be stored in the third message queue, then the second offset of the message body of the first message in the commitLog is stored at the end of the third message queue. At the same time, the first message at the head of the first message queue is removed so that other message consumers can continue to pull and consume other messages from the first message queue.

[0076] In one embodiment, after receiving the second transfer instruction, the message management terminal sets the flag bit corresponding to the first message node to the second flag information, and sets the flag bit to the first flag information again after the first message is added to the tail of the third message queue. Here, the flag bit corresponding to the first message node refers to the flag bit field of the leaf node of the first message node. When the flag bit field is the first flag information, it indicates that the consumption management terminal is not currently transferring the message from the first message queue to the corresponding third message queue. When the flag bit field is the second flag information, it indicates that the consumption management terminal is currently transferring the message from the first message queue to the corresponding third message queue. For example, after receiving the second transfer instruction, the message management terminal sets the flag bit corresponding to the first message node to the second flag information to inform the message consumer that is about to consume the sequential message corresponding to the first message node that the consumption management terminal is currently transferring the message from the first message queue to the corresponding third message queue, thus preventing the message consumer from consuming the sequential message. After the first message is successfully added to the tail of the third message queue, the message management terminal sets the flag corresponding to the first message node to the first flag information. This first flag information informs the message consumer that is about to consume the sequential message corresponding to the first message node that the consumption management terminal has not yet transferred the message from the first message queue to the corresponding third message queue, so that the message consumer can consume the sequential message.

[0077] In summary, the sequential message processing method provided in this application involves the message consumer determining whether consuming the first message satisfies its corresponding consumption order based on its key information. If not, the first message is not consumed to adhere to the sequential message consumption order and avoid consumption errors. If the first message satisfies the order, it is consumed. In the event of consumption failure, a corresponding message node is created to record the key information of the first message, which can then be used to subsequently check whether messages belonging to the same sequential message order satisfy the consumption order. The message management end transfers the first message to a preset message queue through the message node, preventing the first message from blocking subsequent messages in the queue, improving message consumption efficiency, and ensuring the stability of the consumption system.

[0078] Based on the above embodiments, Figure 6 This is a flowchart illustrating another sequential message processing method provided in an embodiment of this application. This sequential message processing method is a specific implementation of the above-described sequential message processing method. For example... Figure 6 As shown, the sequential message processing method includes:

[0079] S210. The message consumer pulls the second message from the head of the second message queue and determines the corresponding second message node on the index tree based on the key information of the second message.

[0080] In one embodiment, the message management terminal is configured with multiple second message queues, which store the header messages of the third message queue in the index tree. Since the key information of the messages stored in the second message queues differs, the message management terminal can randomly store messages at the tail of the second message queue. Here, the second message refers to the message stored within the second message queue. The message consumer can use multiple threads to pull the corresponding second messages from the heads of the multiple second message queues, and concurrently consume failed messages from the non-blocking message queues, thereby improving message consumption efficiency.

[0081] In this embodiment, after the message consumer retrieves the offset of the second message in the commitLog from the head of the second message queue, it retrieves the message body of the second message from the commitLog based on this offset. The message consumer parses the fields of the message body to obtain the key information of the second message. Based on the key information of the second message, it queries the corresponding second message node in the index tree.

[0082] S220. The message consumer determines whether the flag bit of the second message node is the first flag information or the second flag information; when the flag bit of the second message node is determined to be the first flag information, step S230 is executed; when the flag bit of the second message node is determined to be the second flag information, step S310 is executed.

[0083] For example, if the message consumer determines that the flag field of the leaf node of the second message node contains the first flag information, it determines that the consumption management end has not yet transferred messages from the first message queue to the corresponding third message queue. Therefore, it can pull third messages from this third message queue for consumption. Conversely, if the message consumer determines that the flag field of the leaf node of the second message node contains the second flag information, it determines that the consumption management end is currently transferring messages from the first message queue to the corresponding third message queue. If a third message is pulled from this third message queue for consumption at this time, messages that the consumption management end is currently transferring from the first message queue to the third message queue may be missed, leading to consumption errors.

[0084] S230. The message consumer pulls the third message from the head of the third message queue corresponding to the second message node and consumes the third message. When the third message is successfully consumed, step S240 is executed. When the third message is not consumed, step S290 is executed.

[0085] S240, The message consumer sends the first removal command to the message management end.

[0086] S250. When the message management terminal receives the first removal instruction sent by the message consumer terminal, it removes the third message from the head of the third message queue.

[0087] S260. The message consumer determines whether the third message queue is empty. If the third message queue is empty, proceed to step S270. If the third message queue is not empty, proceed to step S230.

[0088] Here, the third message refers to the message in the third message queue. For example, assume that the third message queue corresponding to the second message M1 stores third messages M1_1 to M1_n from head to tail, where the second message M1 and the third message M1_1 are identical. After determining that the flag bit of the second message node is the first flag information, the message consumer retrieves the third message M1_1 from the head of the corresponding third message queue and consumes it. If the third message M1_1 is successfully consumed, the message consumer sends a first removal instruction to the message management end, notifying the message management end that the message at the head of the third message queue has been consumed. Upon receiving the first removal instruction, the message management end removes the third message M1_1 from the head of the third message queue, updating the head of the third message queue to the third message M1_2. If the message consumer determines that the third message queue is not empty, it continues to retrieve the third message M1_2 from the head of the third message queue and consumes it. When the message consumer successfully consumes the third message M1_n, the message management terminal removes the third message M1_n from the head of the third message queue, making the third message queue empty. At this point, the message consumer, knowing that the third message queue is empty and that all sequential messages corresponding to the second message have been consumed, can send a second removal command to the message management terminal to inform it that all sequential messages corresponding to the second message have been consumed.

[0089] For example, if the message consumer fails to consume the third message M1_i due to network fluctuations, it sends a fourth transfer instruction to the message management terminal to notify the message management terminal that the consumption of the sequential messages M1_i to M1_n corresponding to the second message M1 has failed.

[0090] S270, The message consumer sends a second removal command to the message management end.

[0091] S280. When the consumer management terminal receives the second removal instruction sent by the message consumer terminal, it removes the second message node on the index tree and removes the second message from the head of the second message queue.

[0092] For example, the message management terminal determines, based on the second removal instruction, that the sequential messages M1_1 to M1_n corresponding to the second message M1 have been completely consumed. It can then remove the second message M1 from the second message queue, allowing the message consumer to retrieve the second message M2 from the second message queue. Since the computing power required for the message consumer to access the second message queue is greater than that required to access the first message queue, the second message node on the index tree can be removed. If the message consumer retrieves a first message from the first message queue with the same key information as the second message M1, querying the index tree can determine that the sequential messages corresponding to this first message have been successfully consumed, thus allowing the first message to be consumed and saving computing power.

[0093] Furthermore, after the message management end removes the second message M1 from the head of the second message queue, the consumer end returns to the execution step S230.

[0094] S290, The message consumer sends the fourth transfer instruction to the message management end.

[0095] S300 When the message management terminal receives the fourth transfer instruction sent by the message consumer terminal, it removes the second message from the head of the second message queue and adds the third message from the head of the third message queue to the tail of the second message queue.

[0096] For example, the message management terminal determines, based on the fourth transfer instruction, that the sequential messages M1_i to M1_n corresponding to the second message M1 have not yet been consumed. The message management terminal then adds the third message M1_i from the head of the third message queue to the tail of the second message queue. This ensures that when the subsequent message consumer retrieves M1_i from the head of the second message queue, it can re-consume M1_i to M1_n, guaranteeing that the sequential messages M1_i to M1_n can be consumed again. Furthermore, the message management terminal removes the second message M1 from the head of the second message queue to prevent the sequential messages M1_i corresponding to the second message M1 from blocking the second message queue due to consumption failure.

[0097] In one embodiment, Figure 7 This is a flowchart illustrating how the message management terminal, as provided in this embodiment, adds a third message from the head of a third message queue to the tail of a second message queue. Figure 7 As shown, the steps for the message management terminal to add the third message from the head of the third message queue to the tail of the second message queue specifically include S3001-S3002:

[0098] S3001. Create a delayed queue corresponding to the second message queue and add the third message from the head of the third message queue to the delayed queue.

[0099] S3002. Set the trigger time of the delay queue so that the third message in the delay queue is added to the tail of the corresponding second message queue at the trigger time.

[0100] For example, since the message management end only needs to add the third message M1_i from the head of the third message queue to the tail of the second message queue, and the second message queue does not store any messages with the same key information as the third message M1_i, the third message M1_i can be regarded as a normal message and does not need to follow the consumption order. In this case, the time for adding the third message M1_i to the tail of the second message queue can be set according to actual needs. If the current message consumer fails to consume the third message M1_i due to network abnormalities or program restart, a longer trigger time can be set so that the delay queue will add the third message M1_i to the tail of the second message queue after a longer interval.

[0101] S310, The message consumer sends a third transfer instruction to the message management end.

[0102] For example, when the message consumer determines that the consumption management end is currently transferring messages from the first message queue to the corresponding third message queue, it sends a third transfer instruction to the message management end to avoid missing message consumption. The third transfer instruction notifies the message consumer that it cannot currently consume the sequential messages M1_i to M1_n corresponding to the second message M1.

[0103] In another embodiment, if the time for the message management terminal to transfer the message from the first message queue to the third message queue is short, the message consumer terminal can query the flag bit of the second message node corresponding to the second message M1 at regular intervals, and execute step S230 when the flag bit is found to be the first flag information.

[0104] S320. When the message management terminal receives the third transfer instruction sent by the message consumer terminal, it removes the second message from the head of the second message queue and adds the third message from the head of the third message queue to the tail of the second message queue.

[0105] For example, after determining from the third transfer instruction that the message consumer cannot currently consume the sequential messages M1_1 to M1_n corresponding to the second message M1, the message management terminal removes the second message M1 from the head of the second message queue to prevent the second message M1 from blocking the second message queue. Further, the message management terminal adds the message M1_1 from the head of the third message queue to the tail of the second message queue after a certain delay using a delay queue, so that the subsequent message consumer can consume the sequential messages M1_1 to M1_n. The specific steps of the message management terminal delaying the addition of messages to the tail of the second message queue using a delay queue can be found in steps S3001-S3002.

[0106] In one embodiment, after determining that the flag bit of the second message node is the first flag information, the message consumer retrieves the displacement of the third message on the commitLog from the head of the third message queue corresponding to the second message node, and determines whether the displacement of the third message is the same as the displacement of the second message. Since the second message stored in the second message queue is generally the third message corresponding to the head of the third message queue, if the displacement of the third message is different from the displacement of the second message, it can be determined that an error has occurred in a certain consumption stage. Consumption error information can be sent to the staff so that the staff can check which stage of consumption has failed and ensure the correctness of consumption.

[0107] In summary, the message consumer pulls the second message from the second message queue, and determines the third message queue containing the corresponding sequential messages based on the key information of the second message. The sequential messages in the third message queue are then consumed, allowing for the re-consumption of failed sequential messages, thus improving the efficiency of sequential message consumption and ensuring the stability of the consumption system.

[0108] Based on the above embodiments, Figure 8 This is a schematic diagram of the structure of a sequential message processing device provided in an embodiment of this application. Figure 8 As shown, the sequential message processing device includes a first message retrieval module 41, a condition judgment module 42, and a first instruction sending module 43.

[0109] Among them, the first message retrieval module is configured to retrieve the first message from the first message queue of the message management terminal;

[0110] The condition judgment module is configured to determine whether the first message meets the preset consumption conditions based on the key information of the first message; wherein messages with the same key information are stored in the same first message queue according to the preset consumption order.

[0111] The first instruction sending module is configured to consume the first message when it is determined that the first message meets the consumption conditions, and to send a first transfer instruction to the message management terminal when the consumption of the first message fails, so that the message management terminal creates a first message node corresponding to the key information, and transfers the first message from the first message queue to a preset message queue through the first message node.

[0112] Based on the above embodiments, Figure 9 This is a schematic diagram of another sequential message processing device provided in an embodiment of this application. For example... Figure 9 As shown, the sequential message processing device includes a message adding module 51 and a first message transfer module 52.

[0113] The message adding module is configured to add the first message to the corresponding first message queue according to a preset consumption order based on the key information of the first message.

[0114] The first message transfer module is configured to, upon receiving a first transfer instruction sent by the message consumer, create a first message node corresponding to the key information of the first message, and transfer the first message from the first message queue to a preset message queue through the first message node.

[0115] In summary, the sequential message processing apparatus provided in this application allows the message consumer to determine whether consuming the first message satisfies its corresponding consumption order based on the key information of the first message. If not, the first message is not consumed to adhere to the consumption order of sequential messages and avoid consumption errors. If the first message satisfies the order, it is consumed. In the event of consumption failure, a corresponding message node is created to record the key information of the first message for subsequent detection of whether messages belonging to the same sequential message order satisfy the consumption order. The message management end transfers the first message to a preset message queue through the message node, preventing the first message from blocking other messages in the queue, improving the efficiency of sequential message consumption, and ensuring the stability of the consumption system.

[0116] This application also provides a sequential message processing device, which can integrate the sequential message processing apparatus provided in this application. Figure 10 This is a schematic diagram of the structure of a sequential message processing device provided in an embodiment of this application. (Reference) Figure 10 The sequential message processing device includes: an input device 63, an output device 64, a memory 62, and one or more processors 61; the memory 62 is used to store one or more programs; when the one or more programs are executed by the one or more processors 61, the one or more processors 61 implement the sequential message processing method provided in the above embodiments. The sequential message processing device, apparatus, and computer provided above can be used to execute the sequential message processing method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0117] This application also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the sequential message processing method provided in the above embodiments. Of course, the computer-executable instructions in the storage medium containing computer-executable instructions provided in this application are not limited to the sequential message processing method described above; they can also perform related operations in the sequential message processing method provided in any embodiment of this application. The sequential message processing apparatus, device, and storage medium provided in the above embodiments can execute the sequential message processing method provided in any embodiment of this application. Technical details not described in detail in the above embodiments can be found in the sequential message processing method provided in any embodiment of this application.

[0118] The above are only preferred embodiments of the present application and the technical principles employed. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that are possible for those skilled in the art will not depart from the scope of protection of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments and may include more other equivalent embodiments without departing from the concept of the present application. The scope of the present application is determined by the scope of the claims.

Claims

1. A sequential message processing method, characterized in that, include: Retrieve the first message from the first message queue in the message management console; Based on the key information of the first message, determine whether the first message meets the preset consumption conditions; wherein, messages configured with the same key information are stored in the same first message queue according to the preset consumption order, and the consumption condition is that the preset index tree does not contain the first message node corresponding to the key information; When the first message meets the consumption conditions, the first message is consumed. If the consumption of the first message fails, a first transfer instruction is sent to the message management terminal so that the message management terminal creates a first message node corresponding to the key information on the index tree and transfers the first message from the first message queue to a preset message queue through the first message node.

2. The sequential message processing method according to claim 1, characterized in that, The sequential message processing method further includes: When it is determined that the first message does not meet the consumption conditions, a second transfer instruction is sent to the message management terminal so that the message management terminal can determine the first message node corresponding to the key information and transfer the first message from the first message queue to a preset message queue through the first message node.

3. The sequential message processing method according to claim 2, characterized in that, Determining whether the first message meets the consumption conditions based on the key information of the first message includes: Based on the key information of the first message, determine whether the preset index tree includes the first message node corresponding to the key information; When it is determined that the index tree includes the first message node, it is determined that the first message does not meet the consumption condition; If it is determined that the index tree does not include the first message node, then the first message is determined to meet the consumption condition.

4. The sequential message processing method according to claim 3, characterized in that, Sending a first transfer instruction to the message management terminal to enable the message management terminal to create a first message node corresponding to the key information, and transferring the first message from the first message queue to a preset message queue through the first message node, includes: The first transfer instruction is sent to the message management terminal so that the message management terminal creates a corresponding first message node on the index tree based on the key information, and transfers the first message to the tail of the second message queue and the tail of the third message queue corresponding to the first message node.

5. The sequential message processing method according to claim 3, characterized in that, Sending a second transfer instruction to the message management terminal, so that the message management terminal determines the message node corresponding to the key information, and transfers the first message from the first message queue to a preset message queue through the first message node, includes: The message management terminal sends the second transfer instruction to the message management terminal so that the message management terminal determines the corresponding first message node on the index tree based on the key information and transfers the first message to the tail of the third message queue corresponding to the first message node.

6. The sequential message processing method according to claim 4 or 5, characterized in that, After the message management terminal creates a corresponding first message node on the index tree based on the key information, and transfers the first message to the tail of the second message queue and the tail of the third message queue corresponding to the first message node, the process further includes: The second message is retrieved from the head of the second message queue, and the corresponding second message node on the index tree is determined based on the key information of the second message. The third message is retrieved from the head of the third message queue corresponding to the second message node, and the third message is consumed. After the third message is successfully consumed, a new third message is pulled from the head of the third message queue and consumed until the third message queue is empty.

7. The sequential message processing method according to claim 6, characterized in that, Before retrieving the third message from the head of the third message queue corresponding to the second message node, the method further includes: When the flag bit of the second message node is determined to be the first flag information, the third message is pulled from the head of the third message queue corresponding to the second message node; When the flag bit of the second message node is determined to be the second flag information, a third transfer instruction is sent to the message management terminal so that the message management terminal removes the second message from the head of the second message queue and adds the third message from the head of the third message queue to the tail of the second message queue.

8. The sequential message processing method according to claim 6, characterized in that, After consuming the third message, the process further includes: When the third message consumption fails, a fourth transfer instruction is sent to the message management terminal to cause the message management terminal to remove the second message from the head of the second message queue and add the third message from the head of the third message queue to the tail of the second message queue. When the third message is successfully consumed, a first removal command is sent to the message management terminal so that the message management terminal removes the third message from the head of the third message queue.

9. The sequential message processing method according to claim 6, characterized in that, After consuming the third message, the process further includes: When the third message queue is empty, a second removal instruction is sent to the message management terminal so that the message management terminal removes the second message node on the index tree and removes the second message at the head of the second message queue.

10. A sequential message processing method, characterized in that, include: Based on the key information of the first message, the first message is added to the corresponding first message queue according to the preset consumption order; When a first transfer instruction is received from the message consumer, a first message node corresponding to the key information of the first message is created on a preset index tree, and the first message is transferred from the first message queue to the preset message queue through the first message node. The index tree is used to record message nodes of sequential messages that have failed to be consumed.

11. The sequential message processing method according to claim 10, characterized in that, The sequential message processing method further includes: When the second transfer instruction sent by the message consumer is received, the first message node corresponding to the key information of the first message is determined according to the key information of the first message, and the first message is transferred from the first message queue to the preset message queue through the first message node.

12. The sequential message processing method according to claim 11, characterized in that, The step of creating a first message node corresponding to the key information of the first message, and transferring the first message from the first message queue to a preset message queue through the first message node, includes: Create an index tree and a second message queue; Based on the key information of the first message, a first message node corresponding to the key information is created on the index tree, and a third message queue corresponding to the first message node is created. The first message is moved to the tail of the second message queue and the tail of the third message queue corresponding to the first message node.

13. The sequential message processing method according to claim 12, characterized in that, The step of determining the first message node corresponding to the key information based on the key information of the first message, and transferring the first message from the first message queue to a preset message queue through the first message node, includes: The first message node on the index tree is determined based on the key information of the first message. The first message is moved to the tail of the third message queue corresponding to the first message node.

14. The sequential message processing method according to claim 12 or 13, characterized in that, The step of transferring the first message to the tail of the third message queue corresponding to the first message node includes: Set the flag corresponding to the first message node to the second flag information, and after the first message is added to the tail of the third message queue, set the flag to the first flag information.

15. The sequential message processing method according to claim 12 or 13, characterized in that, After transferring the first message to the tail of the second message queue and the tail of the third message queue corresponding to the first message node, the method further includes: Upon receiving a third or fourth transfer instruction from the message consumer, the second message at the head of the second message queue is removed, and the third message at the head of the third message queue is added to the tail of the second message queue.

16. The sequential message processing method according to claim 15, characterized in that, Adding the third message from the head of the third message queue to the tail of the second message queue includes: Create a delayed queue corresponding to the second message queue and add the third message from the head of the third message queue to the delayed queue; Set the trigger time for the delay queue so that the third message in the delay queue is added to the tail of the corresponding second message queue at the trigger time.

17. The sequential message processing method according to claim 12 or 13, characterized in that, After transferring the first message to the tail of the second message queue and the tail of the third message queue corresponding to the first message node, the method further includes: Upon receiving the first removal instruction sent by the message consumer, the third message is removed from the head of the third message queue.

18. The sequential message processing method according to claim 12 or 13, characterized in that, After transferring the first message to the tail of the second message queue and the tail of the third message queue corresponding to the first message node, the method further includes: Upon receiving the second removal instruction sent by the message consumer, the second message node on the index tree and the second message at the head of the second message queue are removed.

19. A sequential message processing apparatus, characterized in that, include: The first message retrieval module is configured to retrieve the first message from the first message queue of the message management terminal. The condition judgment module is configured to determine whether the first message meets the preset consumption conditions based on the key information of the first message; wherein, messages with the same key information are stored in the same first message queue in a preset consumption order, and the consumption condition is that the preset index tree does not contain the first message node corresponding to the key information; The first instruction sending module is configured to consume the first message when it is determined that the first message meets the consumption conditions, and to send a first transfer instruction to the message management terminal when the first message consumption fails, so that the message management terminal creates a first message node corresponding to the key information on the index tree, and transfers the first message from the first message queue to a preset message queue through the first message node.

20. A sequential message processing apparatus, characterized in that, include: The message adding module is configured to add the first message to the corresponding first message queue according to a preset consumption order based on the key information of the first message; The first message transfer module is configured to, upon receiving a first transfer instruction sent by the message consumer, create a first message node corresponding to the key information of the first message on a preset index tree, and transfer the first message from the first message queue to the preset message queue through the first message node. The index tree is used to record message nodes of sequential messages that have failed to be consumed.

21. A sequential message processing device, characterized in that, include: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the sequential message processing method as described in any one of claims 1-18.

22. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the sequential message processing method as described in any one of claims 1-18.

Images