Message queue consumption method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a message queue consumption method and device, electronic equipment and a storage medium. The method comprises the following steps: pulling at least one to-be-consumed message in a to-be-consumed message queue, and determining message type information of each to-be-consumed message; wherein the to-be-consumed messages of each message type are arranged in a to-be-consumed message queue according to a corresponding preset consumption sequence; for each to-be-consumed message, determining a matched blocking queue corresponding to the to-be-consumed message based on the message type information; according to the message arrangement sequence, sequentially storing each message to be consumed into a corresponding matching blocking queue, and consuming the messages to be consumed in the matching blocking queues corresponding to threads through the threads in the thread pool; wherein the threads in the thread pool are in one-to-one correspondence with the matching blocking queues. According to the technical scheme provided by the embodiment of the invention, the effect that the to-be-consumed messages of the same message type can meet the requirement of the preset consumption sequence for ordered consumption during consumption is realized.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of communication technology, and in particular to a message queue consumption method, device, electronic device, and storage medium. Background Art

[0002] In the prior art, business messages generated in the current business scenario are all added to a message queue. To speed up the consumption of the message queue, a thread pool is usually used to perform multi-threaded consumption of the message queue.

[0003] However, in the process of implementing the present invention, it was found that the existing technology has at least the following technical problems: due to the different consumption speeds of each message, in a multi-threaded environment, the messages in the message queue are in a state of disorderly execution, and the message queue cannot be consumed according to the execution order requirements of the current business scenario, resulting in confusion in the message execution order. Summary of the Invention

[0004] The embodiments of the present invention provide a message queue consumption method, device, electronic device and storage medium to achieve the purpose of orderly consumption of to-be-consumed messages of the same message type that can meet the requirements of a preset consumption order when consuming.

[0005] According to one aspect of the present invention, a message queue consumption method is provided, comprising:

[0006] Pulling at least one to-be-consumed message from a to-be-consumed message queue, and determining message type information of each to-be-consumed message; wherein the to-be-consumed messages of each message type are arranged in the to-be-consumed message queue according to a corresponding preset consumption order;

[0007] For each of the messages to be consumed, determining a matching blocking queue corresponding to the message to be consumed based on the message type information;

[0008] According to the order in which the messages are arranged, each of the messages to be consumed is stored in the corresponding matching blocking queue in turn, and the threads in the thread pool consume the messages to be consumed in the matching blocking queue corresponding to the thread; wherein, the threads in the thread pool correspond one to one to the matching blocking queue.

[0009] According to another aspect of the present invention, a message queue consumption device is provided, the device comprising:

[0010] A message pulling module is used to pull at least one message to be consumed from the message queue to be consumed and determine the message type information of each message to be consumed; wherein the messages to be consumed of each message type are arranged in the message queue to be consumed according to the corresponding preset consumption order;

[0011] A matching blocking queue determining module is configured to determine, for each of the messages to be consumed, a matching blocking queue corresponding to the message to be consumed based on the message type information;

[0012] The message storage module is used to store each of the to-be-consumed messages into the corresponding matching blocking queue in sequence according to the message arrangement order, and consume the to-be-consumed messages in the matching blocking queue corresponding to the thread through the thread in the thread pool; wherein, the threads in the thread pool correspond one-to-one to the matching blocking queue.

[0013] According to another aspect of the present invention, an electronic device is provided, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to execute the message queue consumption method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the message queue consumption method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention pulls at least one message to be consumed from the message queue to be consumed and determines the message type information of each message to be consumed; for each message to be consumed, determines the matching blocking queue of the message to be consumed based on the message type information, so that the messages to be consumed of the same type can be stored in the same matching blocking queue; and, since the messages to be consumed of each message type are arranged according to a preset consumption order, the message arrangement order of the messages to be consumed of each message type meets the requirements of the corresponding preset consumption order, and the messages to be consumed are stored in the matching blocking queue based on the message arrangement order, and according to the one-to-one correspondence between the threads in the thread pool and the matching blocking queue, the messages to be consumed in the corresponding matching blocking queue are consumed by the threads according to the first-in-first-out principle, thereby achieving the effect of orderly consumption of the messages to be consumed of the same message type in accordance with the requirements of the preset consumption order; at the same time, a multi-threaded approach is adopted to efficiently complete the processing of the messages to be consumed.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a flowchart of a message queue consumption method provided according to an embodiment of the present invention;

[0022] Figure 2 is a flowchart of another message queue consumption method provided according to an embodiment of the present invention;

[0023] Figure 3 2 is a schematic diagram of the structure of a task processing thread pool provided according to an embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of an abnormality feedback process provided according to an embodiment of the present invention;

[0025] Figure 5 1 is a schematic structural diagram of a message queue consumption device provided according to an embodiment of the present invention;

[0026] Figure 6 The present invention is a schematic diagram of the structure of an electronic device that implements the message queue consumption method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "etc." and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solutions disclosed herein all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken with respect to user personal information to prevent unauthorized access to user personal information data and to safeguard the security of user personal information, network security, and national security.

[0030] Before introducing the technical solution, an example application scenario can be first described. This technical solution can be applied to business fields that require the order of message consumption, and scenarios where business messages are consumed in an orderly manner according to a preset consumption order. For example, in the logistics industry, business messages may include package collection and delivery messages, sorting messages, sealing messages, transportation messages, and delivery messages. According to logistics and transportation requirements, the corresponding messages need to be consumed in the order of collection, sorting, sealing, transportation, and delivery. In the ride-hailing service industry, business messages may include order placement messages, order delivery messages, order receipt messages, etc., and messages also need to be consumed in an orderly manner according to ride-hailing requirements; in the e-commerce field, business messages may include order submission messages, order payment messages, and order outbound messages, and messages are consumed in the order of order submission, order payment, and order outbound. Based on the solution of this embodiment, the threads in the thread pool can consume the messages to be consumed in the corresponding matching blocking queue in an orderly manner according to the preset order requirements, and execute multiple threads in parallel to improve the consumption efficiency of the messages to be consumed.

[0031] Figure 1 This is a flowchart of a message queue consumption method provided according to an embodiment of the present invention. This embodiment is applicable to the orderly consumption of business messages according to a preset consumption order. The method can be executed by a message queue consumption device, which can be implemented in hardware and / or software.

[0032] like Figure 1 As shown, the method of this embodiment may specifically include:

[0033] S110. Pull at least one message to be consumed from the message queue to be consumed, and determine the message type information of each message to be consumed; wherein, the messages to be consumed of each message type are arranged in the message queue to be consumed according to the corresponding preset consumption order.

[0034] The pending message queue is used to store unconsumed messages generated by the message generation system. Message type information reflects the category of the pending message. For example, in the logistics field, message type information reflects the package category, and the package's unique identifier, such as the waybill number, can be used as the message type information. In the e-commerce field, message type information reflects the category of the ordered item, such as the order number.

[0035] In this embodiment, at least one message can be pulled out from the to-be-consumed message queue as a to-be-consumed message according to a first-in-first-out principle. For example, the number of to-be-consumed messages to be pulled out can be determined based on the number of threads in a pre-built thread pool.

[0036] Furthermore, after pulling out the pending messages, the consumption type information corresponding to each pending message can be determined. Specifically, when the pending messages are stored in the pending message queue, the consumption type information of each pending message can be stored accordingly. After pulling the pending messages from the pending message queue, the pre-stored message type information of each pending message can be directly obtained.

[0037] It should be noted that each message type may correspond to a preset consumption order. When storing a message in the message queue to be consumed, the preset consumption order corresponding to the message type information may be determined based on the message type information of the message to be stored. The message may be stored in the message queue to be consumed according to the preset consumption order, so that messages of the same message type are arranged in the message queue to be consumed according to the preset consumption order. Then, when pulling messages to be consumed from the message queue to be consumed, they may be pulled according to the preset consumption order. For example, if the message type of the message to be stored in the message queue to be consumed is type 1, the preset consumption order corresponding to type 1 is message a, message b, message d, and message c, and after the message is stored in the message queue to be consumed according to the preset consumption order, the order in which the messages corresponding to type 1 are arranged in the message queue to be consumed is message a, message b, message d, and message c.

[0038] S120 : For each message to be consumed, determine a matching blocking queue corresponding to the message to be consumed based on the message type information.

[0039] In this embodiment, multiple blocking queues for storing messages to be consumed may be pre-built. For each message to be consumed, a matching blocking queue for storing the message to be consumed may be determined in each blocking queue based on the message type information of the message to be consumed.

[0040] Optionally, a specific implementation method for determining a matching blocking queue corresponding to a message to be consumed based on message type information may include: determining a matching blocking queue corresponding to a message of the same type as the message to be consumed, and determining the matching blocking queue corresponding to the message of the same type as the matching blocking queue corresponding to the message to be consumed. Alternatively, if no matching blocking queue is determined for messages of the same type, determining any blocking queue as the matching blocking queue for the message to be consumed. Alternatively, determining a hash value of the message type information based on a preset hash algorithm, and determining a matching blocking queue corresponding to the message to be consumed based on the hash value.

[0041] In order to evenly distribute the number of to-be-consumed messages contained in each matching blocking queue and improve the consumption speed of to-be-consumed messages, in this embodiment, a specific implementation method for determining the matching blocking queue corresponding to each to-be-consumed message based on the message type information further includes: for each preset matching method, matching an alternative blocking queue for each to-be-consumed message according to the preset matching method and the message type information of each to-be-consumed message; determining the number of to-be-consumed messages corresponding to each alternative blocking queue based on the matching result, and determining message distribution data corresponding to the preset matching method based on the number of messages; determining a target matching method in each preset matching method based on the message distribution data, and determining the matching blocking queue corresponding to each to-be-consumed message based on the target matching method.

[0042] There are multiple preset matching methods; each preset matching method can determine a corresponding matching blocking queue for each message to be consumed. Exemplarily, the preset matching method can be a hash algorithm matching method, where a matching blocking queue corresponding to the message to be consumed is determined by determining a hash value corresponding to the message type information. For example, the preset matching method can include a message digest algorithm matching method and / or a secure hash algorithm matching method.

[0043] In a specific implementation, for each preset matching method, the message type information of each to-be-consumed message can be processed using the preset matching method. Based on the information processing results, a candidate blocking queue matching each to-be-consumed message is determined in each blocking queue. The correspondence between each to-be-consumed message and the candidate blocking queue is used as the matching result, and the number of to-be-consumed messages matching each candidate blocking queue is determined. Based on the number of messages, message distribution data corresponding to the preset matching method is determined.

[0044] The message distribution data is used to reflect the distribution of each pending message in each candidate blocking queue after queue matching is performed using a preset matching method. Exemplarily, the message distribution data includes the number of pending messages matched by each candidate blocking queue; or, the message distribution data includes the ratio of the number of messages corresponding to each candidate blocking queue to the total number of pending messages; or, the message distribution data includes the difference between the number of messages corresponding to each candidate blocking queue and the total number of pending messages.

[0045] In a specific implementation, each preset matching method corresponds to a piece of message distribution data. Based on the message distribution data, a preset matching method that evenly distributes messages can be determined as a target matching method. A corresponding matching blocking queue is then determined for each pending message based on the target matching method. The target matching method is the matching method actually used to determine the matching blocking queue for a pending message.

[0046] Exemplarily, when the message distribution data is the ratio of the number of messages corresponding to each alternative blocking queue to the total number of messages to be consumed, the maximum ratio and the minimum ratio in each ratio can be determined, and the ratio difference between the maximum ratio and the minimum ratio can be determined. The smaller the ratio difference, the more evenly the messages to be consumed are distributed in each alternative blocking queue, and the preset matching method with the smallest ratio difference is used as the target matching method. Alternatively, when the message distribution data includes the number of messages to be consumed that are matched by each alternative blocking queue, the maximum and minimum values ​​in the number of messages can be determined, and the quantity difference between the maximum and the minimum values ​​can be determined. The smaller the quantity difference, the more evenly the messages to be consumed are distributed in each alternative blocking queue, and the preset matching method with the smallest quantity difference is used as the target matching method.

[0047] In this embodiment, multiple preset matching methods are provided in advance, and the corresponding message distribution data after message matching according to each preset matching method is determined. Therefore, under the premise of fully considering the message distribution factors, the target matching method is determined in the preset matching method, which is conducive to determining a more evenly distributed matching method to match the consumption messages, thereby improving consumption efficiency and saving consumption time when using multi-threading to consume the consumption messages.

[0048] In this embodiment, the method of determining the message distribution data corresponding to the preset matching method based on the number of messages may include: determining the maximum and minimum values ​​of the number of messages in each alternative blocking queue, and determining the message distribution data of the preset matching method based on the maximum and minimum values; based on each message distribution data, determining the target matching method in each preset matching method, including: respectively determining the difference between each message distribution data and the preset distribution threshold, and determining the preset matching method corresponding to the minimum difference as the target matching method.

[0049] To make the message distribution data more intuitive and clear, we can determine the maximum and minimum number of messages in each candidate blocking queue. The ratio of the maximum to the minimum value is used as the message distribution data; alternatively, the difference between the maximum and minimum values ​​is used as the message distribution data.

[0050] Furthermore, when the message distribution data is the ratio of the maximum value divided by the minimum value, the preset distribution threshold may be 1. The closer the ratio is to the preset distribution threshold, the more even the message distribution is; the greater the difference between the ratio and the preset distribution threshold, the more uneven the message distribution is. Alternatively, when the message distribution data is the difference between the maximum value and the minimum value, the preset distribution threshold may be 0. The closer the difference is to the preset distribution threshold, the more even the message distribution is; the greater the difference between the difference and the preset distribution threshold, the more uneven the message distribution is.

[0051] After determining the message distribution data, in order to determine the degree of closeness between the message distribution data and the preset distribution threshold, the difference between the message distribution data and the preset distribution threshold can be determined. The smaller the difference, the more uniform the distribution of the messages to be consumed after message matching. The preset matching method corresponding to the minimum difference can be determined as the target matching method.

[0052] In this embodiment, the ratio or difference between the maximum and minimum values ​​in the number of messages is determined as message distribution data, so as to intuitively and clearly reflect the message distribution corresponding to each preset matching method, and the preset matching method corresponding to the minimum difference is determined as the target matching method, so as to determine the preset matching method with the most uniform distribution as the target matching method, which is conducive to improving consumption efficiency.

[0053] In a specific implementation, in order to quickly determine the alternative blocking queue corresponding to each message to be consumed, the preset matching method adopts a hash algorithm matching method. Optionally, according to the preset matching method and the message type information, the alternative blocking queue corresponding to each message to be consumed is determined, including: for each message to be consumed, determining the hash value of the message type information based on the preset hash algorithm, performing a modulo operation on the hash value with the number of threads included in the thread pool, and determining the alternative blocking queue corresponding to the message to be consumed based on the modulo result.

[0054] In this embodiment, a hash value of the message type information of each to-be-consumed message is determined using a preset hash algorithm. For example, when the to-be-consumed message is a package message, the message type information includes the waybill number corresponding to the package, and the hash value of the waybill number is determined based on the preset hash algorithm.

[0055] Furthermore, a modulo operation is performed on the thread data included in the thread pool using the hash value to obtain a modulo result. In order to accurately and quickly determine the alternative blocking queue corresponding to the message to be consumed, the blocking queue can be numbered in advance. The method of determining the alternative blocking queue corresponding to the message to be consumed based on the modulo result may include: performing a modulo operation on the queue number of the blocking queue again based on the modulo result, determining a number that is the same as the value obtained after performing a modulo operation on the queue number, and using the blocking queue corresponding to the number as the alternative blocking queue. Alternatively, when the number of threads is consistent with the number of queues of the blocking queue, determining a number that is the same as the value obtained after performing a modulo operation on the number of threads, and using the blocking queue corresponding to the number as the alternative blocking queue.

[0056] For example, when the number of threads is consistent with the number of queues in the blocking queue, the blocking queue may include 4, such as blocking queue 1, blocking queue 2, blocking queue 3 and blocking queue 4; after the hash value is modulo the number of threads, a value of 1, 2, 3 or 4 can also be obtained as the modulo result. If the modulo result is 1, blocking queue 1 is determined as the alternative blocking queue.

[0057] This embodiment can intuitively and quickly determine the candidate blocking queue corresponding to the message to be consumed by performing a modulo operation on the hash value with the number of threads included in the thread pool, and effectively complete the matching process with the candidate blocking queue.

[0058] Optionally, the preset hash algorithm includes a message digest algorithm (MD) and / or a secure hash algorithm (SHA). Exemplarily, at least one hash algorithm among MD5, SHA-1 and SHA-2 can be used as the preset hash algorithm.

[0059] In this embodiment, three different preset hash algorithms may be provided, that is, three preset matching methods are provided, which is conducive to maximizing the message consumption speed.

[0060] S130. Store each to-be-consumed message into the corresponding matching blocking queue in sequence according to the message arrangement order, and consume the to-be-consumed message in the matching blocking queue corresponding to the thread through the thread in the thread pool.

[0061] Among them, the threads in the thread pool correspond one to one with the matching blocking queues.

[0062] It should be noted that since the to-be-consumed messages of each message type are arranged in the to-be-consumed message queue according to the corresponding preset consumption order, the arrangement order of the to-be-consumed messages in the to-be-consumed message queue meets the requirements of the preset consumption order. In order to ensure that the to-be-consumed messages can still be consumed in the preset consumption order after being stored in the matching blocking queue, based on the first-in-first-out principle of the matching blocking queue, each to-be-consumed message can be stored in the corresponding matching blocking queue in the order in which the messages are arranged. Since the matching blocking queue is determined based on the message type information of the to-be-consumed messages, the to-be-consumed messages with the same message type correspond to the same matching blocking queue.

[0063] In this embodiment, after each pending message is stored, a thread in the thread pool can consume the pending messages in the matching blocking queue corresponding to that thread. Specifically, during thread pool initialization, a one-to-one correspondence between threads and matching blocking queues can be pre-established. Based on this established one-to-one correspondence, each thread can determine the matching blocking queue corresponding to that thread and consume the pending messages in that matching blocking queue.

[0064] For example, when all pending messages are stored in the corresponding matching blocking queues, each thread in the thread pool can be triggered to consume the corresponding pending messages. Alternatively, when the thread pool is initialized, the threads can be set to consume the pending messages in the corresponding matching blocking queues in an active processing manner.

[0065] The technical solution of the embodiment of the present invention pulls at least one message to be consumed from the message queue to be consumed and determines the message type information of each message to be consumed; for each message to be consumed, determines the matching blocking queue of the message to be consumed based on the message type information, so that the messages to be consumed of the same type can be stored in the same matching blocking queue; and, since the messages to be consumed of each message type are arranged according to a preset consumption order, the message arrangement order of the messages to be consumed of each message type meets the requirements of the corresponding preset consumption order, and the messages to be consumed are stored in the matching blocking queue based on the message arrangement order, and according to the one-to-one correspondence between the threads in the thread pool and the matching blocking queue, the messages to be consumed in the corresponding matching blocking queue are consumed by the threads according to the first-in-first-out principle, thereby achieving the effect of orderly consumption of the messages to be consumed of the same message type in accordance with the requirements of the preset consumption order; at the same time, a multi-threaded approach is adopted to efficiently complete the processing of the messages to be consumed.

[0066] Figure 2This is a flowchart of another message queue consumption method provided in accordance with an embodiment of the present invention. Based on the above embodiment, this embodiment further includes: detecting the message processing status of each thread, and generating exception information and feeding back the exception information to the message queue client when the message processing status of at least one thread is abnormal. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 2 As shown, the method includes:

[0067] S210. Pull at least one message to be consumed from the message queue to be consumed, and determine the message type information of each message to be consumed; wherein, the messages to be consumed of each message type are arranged in the message queue to be consumed according to the corresponding preset consumption order.

[0068] S220 . For each message to be consumed, determine a matching blocking queue corresponding to the message to be consumed based on the message type information.

[0069] S230. Store each to-be-consumed message into the corresponding matching blocking queue in sequence according to the message arrangement order, and consume the to-be-consumed message in the matching blocking queue corresponding to the thread through the thread in the thread pool.

[0070] In this embodiment, after consuming the to-be-consumed messages in the matching blocking queue corresponding to the thread, it also includes: after each thread finishes consuming the to-be-consumed messages in the matching blocking queue, based on the number of pulled to-be-consumed messages, updating the offset of the to-be-consumed message queue.

[0071] The offset is used to identify the position of the last message in the queue of pending messages. It should be noted that in this embodiment, the offset can be used to record the progress of message consumption in the queue of pending messages. This facilitates the next message pull by determining the position of unconsumed messages using the offset and directly pulling the unconsumed messages as pending messages, avoiding repeated pulling of already consumed messages.

[0072] In a specific implementation, when consuming the to-be-consumed messages in the corresponding matching blocking queues through a thread, the consumption status of each matching blocking queue can be detected. Specifically, the consumption status of the to-be-consumed messages in each matching blocking queue can be periodically detected according to a preset detection cycle. After all the to-be-consumed messages in each matching blocking queue are consumed, it means that all the pulled to-be-consumed messages are consumed. Then, the number of pulled to-be-consumed messages can be determined. Based on the number of pulled, the offset of the to-be-consumed message queue is updated so that the updated offset can reflect the position of the last message to be consumed in the to-be-consumed message queue.

[0073] This embodiment ensures the accuracy of the offset of the message queue to be consumed by updating the offset of the message queue to be consumed based on the pulled data, which facilitates the rapid completion of message positioning when the messages in the message queue to be consumed are subsequently pulled, thereby improving the convenience of the message consumption process.

[0074] S240: Detect the message processing status of each thread. If the message processing status of at least one thread is abnormal, generate abnormal information and feed the abnormal information back to the message queue client.

[0075] A monitoring thread can be pre-created, for example, when the thread pool is initialized, to monitor the message processing status of each thread in the thread pool. The message processing status reflects whether a message has been successfully consumed. If a thread successfully consumes a message, the message processing status is determined to be normal; if a thread fails to consume a message, the message processing status is determined to be abnormal. For example, if a database error occurs while consuming a pending message, the pending message cannot be successfully consumed, and the message processing status can be determined to be abnormal.

[0076] In a specific implementation, the message processing status of each thread can be detected by a pre-created monitoring thread. For example, the message processing status of each thread can be detected periodically; alternatively, the message processing status of each thread in the thread pool can be detected upon receiving a status detection instruction. Those skilled in the art can determine the method for detecting the message processing status of each thread based on actual application circumstances, and this embodiment does not limit this method.

[0077] In this embodiment, if at least one thread's message processing status is abnormal, an exception message is generated to ensure that each pending message is processed normally. This exception message includes information such as the thread ID of the abnormal thread. By feeding this exception information back to the message queue client, timely feedback on the abnormality is provided, facilitating timely detection and correction of the abnormality, ensuring that each pending message is processed normally, and improving message consumption efficiency.

[0078] Furthermore, after the abnormal information is fed back to the message queue client, the method further includes: interrupting the consumption operation of each thread on the message to be consumed, obtaining the current offset corresponding to the message queue to be consumed, and re-pulling the message to be consumed based on the current offset for consumption.

[0079] It should be noted that the current offset is used to reflect the position of the message that has been consumed in the queue of messages to be consumed at the current moment.

[0080] In the specific implementation, after detecting that the thread is in an abnormal state, the consumption operation of the message to be consumed cannot be completed normally. In order to complete the consumption operation of all the messages to be consumed, it is necessary to re-consume the messages to be consumed corresponding to the thread in the abnormal state, and interrupt the consumption operation of the thread without abnormality in advance.

[0081] Specifically, after detecting that at least one thread is in an abnormal state, the consumption operation of each thread on the message to be consumed can be interrupted, and the current offset corresponding to the queue of messages to be consumed can be obtained. The position of the last consumed message recorded is determined by the current offset, and the position of the unconsumed messages in the queue to be consumed is determined, the unconsumed messages are pulled, and the pulled unconsumed messages are determined as messages to be consumed, and the messages to be consumed are stored in the matching blocking queue, and the consumption operation on the messages to be consumed in the matching blocking queue is performed by the thread.

[0082] In this embodiment, after an exception occurs in the thread, the thread's consumption operation can be interrupted in a timely manner, and based on the current offset, the location of the message that needs to be pulled can be accurately determined, and the message pulling and message consumption operations can be quickly performed, which is conducive to efficiently completing the consumption of messages in the message queue to be consumed.

[0083] The embodiments corresponding to the message queue consumption method are described in detail above. In order to make those skilled in the art further understand the technical solution of this method, a specific application scenario is given below.

[0084] In this embodiment, the task processing thread pool, hash grouping thread pool and monitoring thread can be pre-built and initialized. Figure 3 As shown, the number of threads included in the task processing thread pool can be set according to the actual application situation, and one thread corresponds to one blocking queue; wherein n represents a positive integer greater than 1. The number of threads can be set according to the number of blocking queues. The number of threads in the hash grouping thread pool can be 3, corresponding to the MD5 algorithm, SHA-1 algorithm and SHA-2 algorithm respectively. The monitoring thread is used to monitor the task processing thread pool using the observer mode to determine the message processing status when the thread executes the consumption task of the message to be consumed. If it is determined based on the message processing status that there is an exception in at least one task, all tasks in the task processing thread pool are interrupted and an exception is thrown. When all consumption tasks in the task processing thread pool are completed, the offset can be updated based on the number of pulled messages to be consumed.

[0085] In this embodiment, batches of messages can be pulled from the queue of messages to be consumed as messages to be consumed, and the number of messages to be pulled can be defined according to the actual application. Before storing each message to be consumed, the matching blocking queue of each message to be consumed needs to be determined.

[0086] Specifically, the process of determining the matching blocking queue is as follows:

[0087] 1. Construct an array of length N in each hash algorithm thread in the hash group thread pool; where N is the number of blocking queues created.

[0088] 2. Traverse the pending messages. For each pending message, hash the key in each message using each hash algorithm thread. Modulo N is used to calculate the modulo value of the hash value obtained for each hash algorithm thread. Based on the modulo result, the corresponding blocking queue for each pending message is determined. The array corresponding to each hash algorithm thread records the number of pending messages corresponding to each blocking queue when matching the pending messages using that hash algorithm thread. The key in the pending message reflects the message type.

[0089] 3. By determining the ratio of the minimum to maximum value in each array, we determine the distribution of messages in the blocking queue when matching messages to be consumed by different hash algorithm threads. Since a larger ratio indicates a more even distribution of messages, the hash algorithm thread with the largest ratio is selected as the target algorithm thread. The matching blocking queue for each message to be consumed is determined based on the matching method of the target algorithm thread.

[0090] In this embodiment, the task processing thread pool processes the message and throws an exception. When an exception is thrown, the exception is returned to the monitoring thread. The monitoring thread interrupts the thread in the task processing thread pool and throws an exception back to the message queue client. The exception feedback process is as follows: Figure 4 After all threads have finished consuming, the offset is returned, and the next message is pulled and consumed. The above process is repeated until all messages are consumed. During the message consumption process, a long connection is maintained.

[0091] This embodiment addresses the consumption order issues caused by thread pool consumption by improving the thread pool. Each thread is assigned a blocking queue, making it suitable for consumption with strict order requirements. Furthermore, three hashing algorithms are used to determine the most even message distribution method and the matching message queue for each message to be consumed, maximizing consumption speed. To address the thread pool's inability to detect exceptions, an observer pattern is used to monitor the thread pool and provide a callback method.

[0092] Figure 5This is a schematic diagram of the structure of a message queue consumption device provided in accordance with an embodiment of the present invention. The device is used to execute the message queue consumption method provided in any of the above embodiments. The device and the message queue consumption method of the above embodiments belong to the same inventive concept. For details not fully described in the embodiment of the message queue consumption device, please refer to the embodiment of the message queue consumption method. Figure 5 As shown, the device includes:

[0093] The message pulling module 10 is used to pull at least one message to be consumed from the message queue to be consumed and determine the message type information of each message to be consumed; wherein the messages to be consumed of each message type are arranged in the message queue to be consumed according to the corresponding preset consumption order;

[0094] A matching blocking queue determining module 11 is configured to determine, for each message to be consumed, a matching blocking queue corresponding to the message to be consumed based on the message type information;

[0095] The message storage module 12 is used to store each of the to-be-consumed messages into the corresponding matching blocking queue in sequence according to the message arrangement order, and consume the to-be-consumed messages in the matching blocking queue corresponding to the thread through the thread in the thread pool; wherein the threads in the thread pool correspond one-to-one to the matching blocking queue.

[0096] Based on any optional technical solution in the embodiment of the present invention, optionally, the matching blocking queue determination module 11 includes:

[0097] A message quantity determination submodule is configured to match each to-be-consumed message with an alternative blocking queue according to the preset matching method and the message type information of each to-be-consumed message for each preset matching method; determine the number of to-be-consumed messages corresponding to each alternative blocking queue based on the matching result; and determine message distribution data corresponding to the preset matching method based on the number of messages; wherein the number of preset matching methods is multiple;

[0098] The target matching method determination submodule is used to determine the target matching method among various preset matching methods based on the distribution data of each message, and determine the matching blocking queue corresponding to each message to be consumed based on the target matching method.

[0099] Based on any optional technical solution in the embodiments of the present invention, optionally, the message quantity determination submodule includes:

[0100] a message distribution data determining unit, configured to determine a maximum value and a minimum value of the number of messages in each candidate blocking queue, and determine message distribution data of a preset matching method based on the maximum value and the minimum value;

[0101] The target matching method determination submodule includes:

[0102] The difference determination unit is used to respectively determine the difference between each message distribution data and a preset distribution threshold, and determine the preset matching mode corresponding to the minimum difference as the target matching mode.

[0103] Based on any optional technical solution in the embodiments of the present invention, optionally, the message quantity determination submodule includes:

[0104] The alternative blocking queue determination unit is used to determine the hash value of the message type information for each message to be consumed based on a preset hash algorithm, perform a modulo operation on the hash value with the number of threads included in the thread pool, and determine the alternative blocking queue corresponding to the message to be consumed based on the modulo result.

[0105] Based on any optional technical solution in the embodiments of the present invention, optionally, the preset hash algorithm includes a message digest algorithm and / or a secure hash algorithm.

[0106] Based on any optional technical solution in the embodiments of the present invention, the following may be optionally included:

[0107] The offset update module is used to update the offset of the queue of messages to be consumed based on the number of pulled messages to be consumed after each thread finishes consuming the messages to be consumed in the matching blocking queue.

[0108] Based on any optional technical solution in the embodiments of the present invention, the following may be optionally included:

[0109] The exception information generation module is used to detect the message processing status of each thread. If the message processing status of at least one thread is abnormal, it generates exception information and feeds the exception information back to the message queue client.

[0110] Based on any optional technical solution in the embodiments of the present invention, the following may be optionally included:

[0111] The current offset acquisition module is used to interrupt the consumption operation of each thread on the message to be consumed after the exception information is fed back to the message queue client, obtain the current offset corresponding to the message queue to be consumed, and re-pull the message to be consumed based on the current offset for consumption.

[0112] The message queue consumption device provided by the embodiment of the present invention pulls at least one message to be consumed from the message queue to be consumed and determines the message type information of each message to be consumed; for each message to be consumed, determines the matching blocking queue of the message to be consumed based on the message type information, so that the messages to be consumed of the same type can be stored in the same matching blocking queue; and, since the messages to be consumed of each message type are arranged according to a preset consumption order, the message arrangement order of the messages to be consumed of each message type meets the requirements of the corresponding preset consumption order, and the messages to be consumed are stored in the matching blocking queue based on the message arrangement order, and according to the one-to-one correspondence between the threads in the thread pool and the matching blocking queue, the threads consume the messages to be consumed in the corresponding matching blocking queue according to the first-in-first-out principle, thereby achieving the effect of orderly consumption of the messages to be consumed of the same message type in accordance with the requirements of the preset consumption order; at the same time, a multi-threaded approach is adopted to efficiently complete the processing of the messages to be consumed.

[0113] It is worth noting that in the embodiment of the above-mentioned message queue consumption device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0114] Figure 6 Schematic diagram of the structure of an electronic device that implements the message queue consumption method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0115] like Figure 6As shown, the electronic device 20 includes at least one processor 21, and a memory connected to the at least one processor 21, such as a read-only memory (ROM) 22, a random access memory (RAM) 23, etc., wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 21 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 22 or the computer program loaded from the storage unit 28 to the random access memory (RAM) 23. Various programs and data required for the operation of the electronic device 20 can also be stored in the RAM 23. The processor 21, ROM 22 and RAM 23 are connected to each other via a bus 24. An input / output (I / O) interface 25 is also connected to the bus 24.

[0116] Multiple components in the electronic device 20 are connected to the I / O interface 25, including an input unit 26, such as a keyboard, a mouse, etc.; an output unit 27, such as various types of displays, speakers, etc.; a storage unit 28, such as a magnetic disk, an optical disk, etc.; and a communication unit 29, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 29 allows the electronic device 20 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0117] The processor 21 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of the processor 21 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors for running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 21 executes the various methods and processes described above, such as the message queue consumption method.

[0118] In some embodiments, the message queue consumption method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 28. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 20 via the ROM 22 and / or the communication unit 29. When the computer program is loaded into the RAM 23 and executed by the processor 21, one or more steps of the message queue consumption method described above can be performed. Alternatively, in other embodiments, the processor 21 can be configured to execute the message queue consumption method in any other appropriate manner (for example, by means of firmware).

[0119] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0121] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0124] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0125] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0126] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A message queue consumption method, characterized in that: include: Pulling at least one to-be-consumed message from a to-be-consumed message queue, and determining message type information of each to-be-consumed message; wherein the to-be-consumed messages of each message type are arranged in the to-be-consumed message queue according to a corresponding preset consumption order; For each of the messages to be consumed, determining a matching blocking queue corresponding to the message to be consumed based on the message type information; According to the order in which the messages are arranged, each of the messages to be consumed is stored in the corresponding matching blocking queue in turn, and the threads in the thread pool consume the messages to be consumed in the matching blocking queue corresponding to the thread; wherein, the threads in the thread pool correspond one to one to the matching blocking queue.

2. The method according to claim 1, characterized in that The step of determining, for each of the messages to be consumed, a matching blocking queue corresponding to the message to be consumed based on the message type information, includes: For each preset matching method, matching each to-be-consumed message with an alternative blocking queue according to the preset matching method and the message type information of each to-be-consumed message; determining the number of to-be-consumed messages corresponding to each alternative blocking queue based on the matching result, and determining message distribution data corresponding to the preset matching method based on the number of messages; wherein the number of the preset matching methods is multiple; Based on each of the message distribution data, a target matching method is determined in each of the preset matching methods, and based on the target matching method, a matching blocking queue corresponding to each of the to-be-consumed messages is determined.

3. The method according to claim 2, characterized in that The determining, based on the number of messages, message distribution data corresponding to the preset matching method includes: Determine a maximum value and a minimum value of the number of messages in each candidate blocking queue, and determine message distribution data of the preset matching method based on the maximum value and the minimum value; The step of determining a target matching method in each of the preset matching methods based on each of the message distribution data includes: The difference between each message distribution data and a preset distribution threshold is determined respectively, and the preset matching mode corresponding to the minimum difference is determined as the target matching mode.

4. The method according to claim 2, characterized in that Determining, according to the preset matching method and the message type information, a candidate blocking queue corresponding to each message to be consumed, includes: For each of the messages to be consumed, a hash value of the message type information is determined based on a preset hash algorithm, the hash value is modulo the number of threads included in the thread pool, and the alternative blocking queue corresponding to the message to be consumed is determined based on the modulo result.

5. The method according to claim 4, characterized in that The preset hash algorithm includes a message digest algorithm and / or a secure hash algorithm.

6. The method according to claim 1, characterized in that Also includes: After each of the threads finishes consuming the to-be-consumed messages in the matching blocking queue, the offset of the to-be-consumed message queue is updated based on the number of the pulled to-be-consumed messages.

7. The method according to claim 1, characterized in that Also includes: The message processing state of each of the threads is detected, and when the message processing state of at least one of the threads is abnormal, abnormal information is generated and fed back to the message queue client.

8. The method according to claim 7, characterized in that After feeding back the abnormal information to the message queue client, the method further includes: Interrupt the consumption operation of each thread on the message to be consumed, obtain the current offset corresponding to the queue of messages to be consumed, and re-pull the message to be consumed based on the current offset.

9. A message queue consumption device, characterized in that: include: A message pulling module is used to pull at least one message to be consumed from the message queue to be consumed and determine the message type information of each message to be consumed; wherein the messages to be consumed of each message type are arranged in the message queue to be consumed according to the corresponding preset consumption order; A matching blocking queue determining module is configured to determine, for each of the messages to be consumed, a matching blocking queue corresponding to the message to be consumed based on the message type information; The message storage module is used to store each of the to-be-consumed messages into the corresponding matching blocking queue in sequence according to the message arrangement order, and consume the to-be-consumed messages in the matching blocking queue corresponding to the thread through the thread in the thread pool; wherein, the threads in the thread pool correspond one-to-one to the matching blocking queue.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to execute the message queue consumption method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the message queue consumption method according to any one of claims 1 to 8 when executed.

Images