Message queue processing method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of big data collection, and discloses a message queue processing method and device, computer equipment and a storage medium. The method comprises the following steps: storing received messages to be processed into a message queue according to a time sequence; wherein the to-be-processed message comprises a collector identifier, and when the to-be-processed message is the last message collected by the target collector, an ending identifier is added to a message identifier bit of the to-be-processed message; reading the to-be-processed message in the message queue through a consumer, and determining whether the to-be-processed message is the last message or not; and when the to-be-processed message is the last message, whether the service logic corresponding to the to-be-processed message is executed or not is determined according to the related information of the message which is not executed completely in the to-be-processed message list, and the related information comprises the collector identifier. According to the invention, through distributed extensible acquisition and consumption, the metadata acquisition speed is improved, and it is ensured that previous metadata messages are executed when acquisition of the same acquirer is completed.

Description

Technical Field

[0001] The present invention relates to the technical field of big data collection, and in particular to a message queue processing method, device, computer equipment and storage medium. Background Art

[0002] When there is message retransmission, multiple duplicate messages, or multiple different messages, if the consumer executes multiple duplicate messages, or incorrectly executes the business logic of the last message when the previous message has not been processed, it will cause incorrect business logic and even cause the business process to be interrupted.

[0003] Therefore, there is an urgent need for a sequential and highly reliable message processing method. Summary of the invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the problems of poor sequentiality and poor reliability of message processing in the related art.

[0005] In order to solve the above technical problems, the present invention provides a message queue processing method, the message queue processing method comprising:

[0006] Receive a message to be processed collected by a target collector; the message to be processed includes a collector identifier of the target collector; the collector identifier of the target collector is used to distinguish the target collector from other collectors; the message to be processed is metadata;

[0007] The received messages to be processed are stored in a message queue according to the time sequence; wherein, when the message to be processed is the last message collected by the target collector, an end mark is added to the message mark of the message to be processed;

[0008] The consumer reads the to-be-processed message in the message queue, and determines whether the to-be-processed message is the last message according to the message identification bit of the to-be-processed message;

[0009] When the to-be-processed message is the last message, query the to-be-processed message list for related information of the uncompleted messages; the to-be-processed message list is used to store the uncompleted messages; the related information includes a collector identification;

[0010] Determine whether to execute the business logic corresponding to the pending message according to the relevant information of the uncompleted message in the pending message list.

[0011] In an optional implementation, determining whether the message to be processed is the last message according to the message identification bit of the message to be processed includes:

[0012] When the message to be processed is not the last message, the business logic corresponding to the message to be processed is executed, and a response result is returned when the business logic corresponding to the message to be processed is completed.

[0013] In an optional implementation, the determining whether to execute the business logic corresponding to the pending message according to the relevant information of the uncompleted message in the pending message list includes:

[0014] Determine whether there are uncompleted messages collected by the target collector in the list of messages to be processed according to relevant information of the uncompleted messages in the list of messages to be processed;

[0015] When there is no unfinished message collected by the target collector in the list of pending messages, the business logic corresponding to the pending message is executed, and a response result is returned when the business logic corresponding to the pending message is executed.

[0016] In an optional implementation, determining whether there are uncompleted messages collected by the target collector in the list of messages to be processed according to relevant information of the uncompleted messages in the list of messages to be processed includes:

[0017] When there are uncompleted messages collected by the target collector in the list of messages to be processed, the business logic corresponding to the messages to be processed is skipped.

[0018] In an optional implementation manner, the relevant information further includes: consumption time and number of group changes, and the method further includes:

[0019] Reading the consumption time of the unexecuted messages in the list of pending messages at a preset time interval;

[0020] Determine whether there is a first target message in the list of pending messages according to the consumption time and the preset time of the uncompleted messages in the list of pending messages; the first target message is an uncompleted message whose consumption time is greater than the preset time;

[0021] When there is a first target message in the list of messages to be processed, a group transfer operation is performed on the first target message, the number of group transfers is increased by 1, and the consumption time of the first target message is reset to 0.

[0022] In an optional implementation, determining whether the first target message exists in the list of messages to be processed according to the consumption time and the preset time of the uncompleted messages in the list of messages to be processed includes:

[0023] When there is no first target message in the list of pending messages, determining whether there is a second target message in the list of pending messages according to the number of group transfers and the preset number of the uncompleted messages in the list of pending messages; the second target message is an uncompleted message whose number of group transfers is greater than the preset number;

[0024] When there is no second target message in the list of pending messages, the pending messages in the message queue are read again by the consumer, and it is determined whether the pending message is the last message according to the message identification bit of the pending message.

[0025] In an optional implementation, determining whether there is a second target message in the list of messages to be processed according to the number of group transfers of the uncompleted messages in the list of messages to be processed and a preset number of times includes:

[0026] When there is a second target message in the list of messages to be processed, it is determined that the collector message collection corresponding to the second target message fails.

[0027] In a second aspect, the present invention provides a message queue processing device, the message queue processing device comprising:

[0028] A first processing module is used to receive a message to be processed collected by a target collector; the message to be processed includes a collector identifier of the target collector; the collector identifier of the target collector is used to distinguish the target collector from other collectors; the message to be processed is metadata;

[0029] The second processing module is used to store the received to-be-processed messages into a message queue according to the time sequence; wherein, when the to-be-processed message is the last message collected by the target collector, an end mark is added to the message mark of the to-be-processed message;

[0030] A third processing module is used to read the to-be-processed message in the message queue through a consumer, and determine whether the to-be-processed message is the last message according to a message identification bit of the to-be-processed message;

[0031] The third processing module includes a first processing unit, and the first processing unit includes a first sub-processing unit and a second sub-processing unit;

[0032] The first sub-processing unit is used to query the relevant information of the unfinished messages in the list of unfinished messages when the unfinished message is the last message; the list of unfinished messages is used to store the unfinished messages; the relevant information includes the collector identification;

[0033] The second sub-processing unit is used to determine whether to execute the business logic corresponding to the pending message according to the relevant information of the uncompleted message in the pending message list.

[0034] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the message queue processing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium, a single computer-readable storage medium storing computer instructions, and the computer instructions are used to enable a computer to execute the message queue processing method of the first aspect or any corresponding embodiment thereof.

[0036] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the message queue processing method of the first aspect or any corresponding embodiment thereof.

[0037] The technical solution provided by the present invention has the following technical effects:

[0038] The technical solution of the embodiment of the present invention ensures the order of messages by storing the pending messages collected by the target collector in the message queue in chronological order. This order is very critical in subsequent processing, especially when processing metadata with sequential dependencies. For example, in some data processing processes, if the metadata is information about different stages of a business process, storing it in chronological order can ensure that the processing order is consistent with the actual order of occurrence of the business process, avoiding confusion in data processing. Each pending message contains a collector identifier, which enables metadata collected by different collectors to be clearly distinguished. In a complex multi-collector environment, it helps to accurately track the source of the message. When a problem occurs (such as data anomalies or collection failures), it can be quickly located to which collector has caused the problem, which is convenient for targeted investigation and resolution. Adding an end identifier to the message identifier of the last message collected by the target collector provides a clear mark for the subsequent processing process. When a consumer reads a pending message in a message queue, it determines whether it is the last message based on the message identifier. This identifier-based judgment method makes the message processing logic more flexible and accurate. For messages that are not the last one, they can be processed according to conventional business logic, while the last message is specially processed, which improves the pertinence of message processing. When the pending message is the last message, query the relevant information (including collector identifier) ​​of the unfinished messages in the pending message list. This step ensures the integrity of the entire collection process before processing the last message. For example, if the previous message cannot be processed due to some reason (such as a temporary failure of the consumer), it can be known through the query to avoid erroneously executing the business logic of the last message when the previous message has not been processed. Determine whether to execute the business logic corresponding to the pending message based on the relevant information of the unfinished messages in the pending message list, thereby ensuring the consistency and correctness of the entire message processing process. The business logic of the last message is only executed when all the previous messages have been processed, avoiding problems such as data inconsistency or business process interruption, and improving the reliability of message processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flowchart of a message queue processing method according to an embodiment of the present invention;

[0041] Figure 2It is a schematic diagram of the overall flow of the message queue processing method according to an embodiment of the present invention;

[0042] Figure 3 is a flow chart of another message queue processing method according to an embodiment of the present invention;

[0043] Figure 4 It is a schematic diagram of the overall flow of another message queue processing method according to an embodiment of the present invention;

[0044] Figure 5 It is a structural schematic diagram of a message queue processing device according to an embodiment of the present invention;

[0045] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0047] The embodiments of the present invention provide a message queue processing method, device, computer equipment and storage medium to solve the problems of poor sequentiality and poor reliability of message processing in related technologies.

[0048] According to an embodiment of the present invention, a message queue processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer device such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] Figure 1 It is a flowchart of a message queue processing method according to an embodiment of the present invention.

[0050] like Figure 1As shown, a message queue processing method is provided in an embodiment of the present invention, and the message queue processing method is implemented based on redis stream. Redisstream is a data type introduced in version 5.0 of the redis database, and is mainly used to build a message queue system. It allows the collector to add messages to it, and the consumer reads and processes the messages from it, which is similar to a message pipeline. In redisstream, messages exist in the form of key-value pairs. Each message has a message identifier ID. For example, a message identifier ID may be "1695646219053-0". The content of the message is a key-value pair containing multiple fields, which can be defined according to specific business needs. For example, in the scenario of metadata collection, there may be fields such as collector identifier (collectorid) and end flag (endFlag). Redisstream stores messages in a way similar to logs. It arranges messages in the order in which they are added, and can store a large number of messages. It can be imagined as a growing message list, and new messages are added to the end of the list. For example, when there are multiple metadata collection messages that need to be stored, they will be added to redisstream in sequence according to the time order of collection.

[0051] The message queue processing method includes:

[0052] S101: receiving a to-be-processed message collected by a target collector.

[0053] In this embodiment, the message to be processed includes the collector identification of the target collector. The collector identification of the target collector is used to distinguish the target collector from other collectors. The message to be processed is metadata, specifically metadata table information. The target collector collects the message to be processed in the database.

[0054] S102: The received messages to be processed are stored in a message queue according to time sequence.

[0055] In this embodiment, when the message to be processed is the last message collected by the target collector, an end mark is added to the message mark bit of the message to be processed.

[0056] In this embodiment, during the metadata collection and consumption process, a collector identifier (eg, id) is added to the collected messages to be processed for unique identification, and the last message collected or sent by the collector needs to be added with an end identifier, for example, endFlag=true.

[0057] In this embodiment, the message queue is a redis queue, which stores messages sequentially. It is an ordered message queue that can record the generation order of messages.

[0058] S103: The consumer reads the to-be-processed message in the message queue, and determines whether the to-be-processed message is the last message according to the message identification bit of the to-be-processed message.

[0059] In this embodiment, the pending messages in the message queue read by the consumer are stored in a pending message list (Pending Entries List, PEL), referred to as PEL queue.

[0060] Considering that redis stream consumes through consumer groups, that is, multiple consumers, although the order of message consumption can be guaranteed, it cannot be guaranteed that the order is also followed when consumption is completed. For example, messages a, b, and c are sent to the message queue in sequence. There are 3 consumers 1, 2, and 3 under the consumer group. It can be guaranteed that 1 is consumed first, and then 2 is consumed, and then 3 is consumed. However, since there are 3 consumers, it cannot be guaranteed that when consumer 3 executes the business logic corresponding to message c, the business logic corresponding to message a under consumer 1 has been executed. In order to avoid erroneously executing the business logic of the last message when the preceding message has not been processed, it is necessary to determine whether the message to be processed is the last message according to the technical solution of S103. Whether the message to be processed is the last message can be determined based on whether the message identification bit of the message to be processed is the end identification bit. When the message identification bit of the message to be processed is the end identification bit (for example, endFlag=true), it is determined that the message to be processed is the last message. When the message identification bit of the message to be processed is not the end identification bit, when the message identification bit is false (or not set to true), it is determined that the message to be processed is not the last message.

[0061] S104: When the to-be-processed message is the last message, query the to-be-processed message list for related information of the uncompleted messages.

[0062] In this embodiment, the list of pending messages is used to store messages that have not been executed. The relevant information includes a collector identifier.

[0063] In this embodiment, when the message to be processed is not the last message, the business logic corresponding to the message to be processed is executed, and a response result is returned when the business logic corresponding to the message to be processed is completed. If the message is not the last message, the consumer will process the message according to the conventional business logic. For example, in metadata collection, operations such as storage and analysis may be performed on the collected single metadata table information.

[0064] In this embodiment, after the consumer reads the pending message in the message queue, it stores the pending message in the pending message list. The pending message list is specifically a PEL queue. When the message is consumed but all business logic processing has not been completed (that is, the response result has not been returned, and the ask has not been submitted), the unfinished message will be stored in the PEL queue.

[0065] For example, in the metadata collection scenario, when using redisstream as the message queue, the message will be stored in the PEL queue during the consumption process. The relevant information in the PEL queue can be used to determine whether the previous messages have been successfully consumed, thereby ensuring the order and reliability of consumption. For example, when encountering a message with endFlag=true (the last message), it will check whether there are uncompleted messages collected by the same collector in the PEL queue, so as to decide whether to return a response result and submit the ask, ensuring that when the same collector completes the collection, the previous messages have been executed.

[0066] S105: Determine whether to execute the business logic corresponding to the pending message according to the relevant information of the uncompleted message in the pending message list.

[0067] In this embodiment, when the endFlag value is true, the consumer will realize that this may be the last message in a series of related messages (such as a set of metadata collection messages from the same collector). At this time, the consumer will not directly process the business logic of this message, but will first check whether the business logic of the message consumed before the collector identifier id of the target collector has been executed. This is to ensure that all related preceding messages have been successfully processed before executing the business logic of the last message.

[0068] In this embodiment, determining whether to execute the business logic corresponding to the pending message according to the relevant information of the uncompleted message in the pending message list in S105 specifically includes:

[0069] It is determined whether there are uncompleted messages collected by the target collector in the list of messages to be processed according to the relevant information of the uncompleted messages in the list of messages to be processed.

[0070] When there are no unfinished messages collected by the target collector in the list of pending messages, the business logic corresponding to the pending messages is executed, and the response result is returned when the business logic corresponding to the pending messages is completed, and the pending messages in the list of pending messages are removed. Specifically, when the collector identifiers of the unfinished messages in the list of pending messages are not the collector identifiers of the target collector, it is determined that there are no unfinished messages collected by the target collector in the list of pending messages. The message will be removed from the PEL queue only after the consumer successfully processes the message and returns a response result (ask). This mechanism ensures that the message will not be lost due to consumer failures and other situations, and ensures the reliability of message processing.

[0071] When there are uncompleted messages collected by the target collector in the list of pending messages, the business logic corresponding to the pending messages is skipped. Specifically, when the collector identifier of the uncompleted message in the list of pending messages is the collector identifier of the target collector, it is determined that there are uncompleted messages collected by the target collector in the list of pending messages.

[0072] In this embodiment, considering that if the business logic corresponding to the message is executed, the message will be removed from the list of pending messages, it is possible to determine whether the business logic corresponding to the message previously consumed by the target collector has been executed, based on whether the collector identifier of the uncompleted message in the list of pending messages is the collector identifier of the target collector. When the collector identifier of the uncompleted message in the list of pending messages does not include the collector identifier of the target collector, it is determined that the business logic corresponding to the message previously consumed by the target collector has been executed. At this time, the business logic corresponding to the last message is executed, which can avoid the erroneous execution of the business logic of the last message when the preceding message has not been processed.

[0073] In this embodiment, the collector identification id of the corresponding collector that has not returned a response result in the PEL queue can be obtained through the relevant information of the pending message. If the target collector id does not exist in the PEL queue before, the business logic corresponding to the pending message with endFlag=true is executed, and ask is submitted, and the collection and consumption ends. If the target collector id exists before, the business logic corresponding to the pending message with endFlag=true is skipped, and ask is not submitted.

[0074] In the present invention, the collector identification id can be a specific value, and the collector identification id can be generated according to the order of the collection or transmission time. The earlier the collection or transmission time, the smaller the collector identification id generated, and the later generated collector identification id is larger than the previously generated collector identification id. The pending message also includes information such as the message identifier id and the message queue name, which can be stored together in the pending message list. Each message has a unique message identifier id, which is a string consisting of a timestamp and a sequence number. Similarly, the message identifier id of the message generated later is larger than the message identifier id of the message generated earlier.

[0075] When determining whether to execute the business logic corresponding to the pending message based on the relevant information of the unfinished message in the pending message list, it can be determined whether the smallest message identifier id in the pending message list is smaller than the message identifier id of the pending message. For example, when the smallest message identifier id1 in the pending message list is smaller than the message identifier id7 (current id) of the pending message, determine the messages in the pending message list with message identifier id1 to message identifier id6 (the previous one of message identifier id7), and query whether the collector identifier (collector id) of the target collector exists in the relevant information of the messages with message identifier id1 to message identifier id6. Taking the example of a consumer group including three consumers, the overall flow diagram of the message queue processing method of an embodiment of the present invention is as follows: Figure 2 shown.

[0076] Figure 3 It is a flowchart of another message queue processing method according to an embodiment of the present invention.

[0077] like Figure 3 As shown, an embodiment of the present invention provides a message queue processing method, the message queue processing method comprising:

[0078] S201: Reading the consumption time of the uncompleted messages in the list of pending messages at a preset time interval.

[0079] In this embodiment, the relevant information of the unfinished messages in the pending message list includes: collector identification, consumption time and number of group transfers. The unfinished messages in the pending message list can also be read by the consumer at a preset time interval. The preset time interval can be set and modified according to actual needs, for example, it can be 30 seconds. The consumption time is the residence time of the message in the PEL queue.

[0080] S202: Determine whether there is a first target message in the list of messages to be processed according to the consumption time of the uncompleted messages in the list of messages to be processed and the preset time.

[0081] In this embodiment, the first target message is an unfinished message whose consumption time is greater than the preset time. The preset time can be set and modified according to actual needs, for example, it can be 2 minutes or 120 seconds. The PEL queue is used to store messages that have been read by consumers but have not yet confirmed (ask) that the processing is completed. The consumption time (PEL time) of a message in the PEL queue is an important indicator. When the consumption time is greater than the set threshold (here is 2 minutes), it is determined that there may be a problem with the message consumption, such as a consumer failure in the process of processing the message, a long network delay, or an infinite loop in the execution of the business logic, resulting in the message not being processed within a reasonable time. Therefore, it is necessary to determine such a message, that is, the first target message, perform a group transfer operation on the first target message, complete the processing of the previous message as soon as possible, and execute the business logic corresponding to the last message as soon as possible.

[0082] S203: When the first target message exists in the list of messages to be processed, a group transfer operation is performed on the first target message, the number of group transfers is increased by 1, and the consumption time of the first target message is reset to 0.

[0083] In this embodiment, the number of group transfers indicates the number of times the message is reallocated to other consumers. If this number is large, it may mean that the consumer often has problems processing the message, causing the message to be reallocated to other consumers for processing. A consumer group is an organizational form in which multiple consumers collaboratively process messages. A consumer group can contain multiple consumers, who read and process messages from a message queue or a list of pending messages to achieve distributed consumption and improve processing efficiency. Message group transfer operation: When it is determined that the message consumption fails (the consumption time is greater than 2 minutes), the message is transferred to another consumer in the consumer group for processing. The message is transferred from the consumer currently processing it (which may not be able to complete the message processing normally due to problems with the consumer) to other consumers in the consumer group for processing. Specifically, during the group transfer process, the deliveryCount parameter of the message in the PEL queue will automatically increase by 1, indicating that the number of times the message is reallocated has increased once. At the same time, the elapsedTime (the time the message stays in the PEL queue) is reset to 0, and after the next scheduled task is triggered, the new consumer will re-consume the message. By transferring messages to groups, the reliability of message processing can be improved. Even if a problem occurs with a single consumer, the message can be reprocessed by other consumers in the consumer group, preventing the message from being lost or not being processed for a long time due to a failure of a certain consumer. This mechanism helps to ensure the stability of the entire business process and the success rate of message processing.

[0084] S204: When the first target message does not exist in the list of pending messages, determine whether the second target message exists in the list of pending messages according to the number of group transfers of the uncompleted messages in the list of pending messages and a preset number of times.

[0085] In this embodiment, the second target message is an uncompleted message whose number of group transfers is greater than a preset number.

[0086] S2041: When there is no second target message in the list of messages to be processed, the above technical solutions of S103-S105 are executed again. The preset number of times can be set and modified according to actual needs, for example, it can be 3 times.

[0087] In this embodiment, the detailed contents of the technical solutions of S103-S105 above refer to the relevant descriptions of S102-S105 above, which will not be repeated here.

[0088] S2042: When there is a second target message in the list of messages to be processed, it is determined that the collector message collection corresponding to the second target message fails.

[0089] The overall flow diagram of another message queue processing method is as follows Figure 4 As shown, the technical solution of the present invention also includes starting a timed task, reading the uncompleted messages in the PEL queue by periodic polling at a preset time interval, and judging whether the consumption time elapsedTime of the uncompleted messages in the PEL queue is greater than the preset time. For example, the preset time interval is set to 30 seconds, giving a sufficient time, such as 2 minutes, to ensure that the consumption can be completed within the preset time (2 minutes). If it is found that the consumption time of the message is greater than 2 minutes, there is reason to determine that the message consumption has failed, and the message is transferred to a group. The deliveryCount parameter of the number of transfers of the message in the PEL queue will automatically increase by 1, and elapsedTime will be reset to 0. After waiting for the next timed task to be triggered, the message is consumed again. After re-consumption, the message that has not submitted ack is subjected to a whole logical judgment (executing the above-mentioned technical solution of S103-S105). At this point, the logical design that ensures that the message before endFlag=true can be successfully consumed is completed. In order to relieve the CPU and avoid meaningless execution, you can also set the maximum number of consumption times, because each consumption failure will cause a group transfer. For example, if the group transfer number deliveryCount parameter is greater than 3 times and still fails, it is directly determined that the collector message collection has failed.

[0090] The specific commands executed by the technical solution of the present invention are as follows:

[0091] >XPENDING metadata_stream default_group-+5. This command is used to query the information about pending messages (Pending Entries) under a specific consumer group (default_group) in redisstream. Where metadata_stream is the name of redisstream, - and + are parameters used to specify the range of message identifiers (here querying all messages), and 5 is a parameter to limit the number of response results returned.

[0092] 1695646219053-0 (message id). This is the message identifier of the message. The message identifier is used to distinguish different messages in RedisStream. It is a string consisting of a timestamp and a sequence number. A message can be uniquely located through the message identifier.

[0093] default_consumer@0 (consumer) indicates the ID of the consumer that is processing this message. This helps track which consumer is processing the message and can be used to locate possible problems in a multi-consumer environment, such as a consumer processing messages too slowly or failing.

[0094] 23477 (elapsedTime) represents the length of time that the message has been consumed (consumption time), in milliseconds. The consumption time can help determine whether the message processing time is too long, and thus determine whether there may be performance problems or consumer failures.

[0095] 1 (deliveryCount) indicates the number of times this message has been redistributed to other consumers. If this number is large, it may mean that consumers often have problems processing this message, causing the message to be redistributed to other consumers for processing.

[0096] The query result of the xpending command shows that there is a message before this message that has not been completed and submitted to ack. The elapsedTime is 23477 milliseconds and the number of group transfers is 1. The relevant information of the messages that have not been executed can be viewed through the xrange command.

[0097] >XRANGE hive_metadata_collection 1695646219053-0+COUNT 1. This command is used to obtain relevant information about messages in a specified message range in redisstream, such as the message queue name queueName, collector identifier collectorid, and message flag endFlag.

[0098] 1695646219053-0

[0099] queueName

[0100] "metadata_stream"

[0101] collectorid 570

[0103] endFlag

[0104] true

[0105] When the second target message does not exist in the list of pending messages, when the technical solution of S103-S105 is re-executed, the XPENDING command is mainly used to query the messages in the pending state. In the scenario of using redis stream as a message queue, the message is in the pending state when it is consumed but has not yet submitted for confirmation (ack), and these messages are stored in the PEL queue. The XPENDING command can obtain detailed information about these pending messages, such as the consumer group to which the message belongs, the range of the message (from which message identifier id to which message identifier id), the consumer corresponding to each pending message, the time the message has been waiting for processing (elapsedTime), and the number of group transfers of the message (deliveryCount). For example, the command "XPENDING metadata_stream default_group-+5 1695646219053-0(message id)default_consumer@0(consumer)23477(consumption time elapsedTime)1(number of group transfers deliveryCount)" shows the output of the XPENDING command. Among them, "metadata_stream" is the name of the PEL queue, and "default_group" is the consumer group name. "-+" indicates the range of querying all messages, "5" may be a parameter related to the number of messages or the range limit (depending on the system implementation), and "1695646219053-0" is the range of message IDs, followed by consumer information "default_consumer@0", consumption time "23477" milliseconds, and number of group transfers "1". Through this information, we can know that in this PEL queue and consumer group, there is a message that has been waiting for 23477 milliseconds and has not been processed yet, and has been transferred to the group once, which is very useful for monitoring the processing status of messages and discovering potential problems.

[0106] The XRANGE command is used to obtain information about a specified range of messages in a PEL queue. Messages can be retrieved based on the range of message identifiers, and information about the messages, including business-related data such as collectorid and endFlag, can be returned. For example, you can determine whether a message is the last message (through endFlag) and the collector it belongs to (through collectorId). For example, the command "XRANGE hive_me tadata_collection 1695646219053-0+COUNT 1 1695646219053-0queue Name"metadata_stream"collectorId 570endFlagtrue" shows the output of the XRANGE command. Among them, "hive_metadata_collection" is the collection name of the message storage (similar to the table name in the database), "1695646219053-0+" is the range of the message identifier id (you can specify the message identifier id from the beginning to the end), "COUNT 1" limits the number of returned messages to 1, and the following information includes the queue name "metadata_stream" to which the message belongs, the collector identifier "collectorId570" and the value of `endFlag` "true". Through the XRANGE command, you can get these key message contents, so as to decide how to process this message according to business rules, such as whether to submit ack and other operations. In general, the XPENDING command focuses on querying the processing status of the message (such as consumption time, number of group transfers, etc.), while the XRANGE command focuses on querying the specific content of the message (such as business data and identification fields in the message). It plays an important role in ensuring the order and reliability of the message queue and correctly processing messages.

[0107] In metadata collection, distributed consumption is achieved by using lightweight redis stream as a message queue to improve consumption speed and reliability.

[0108] The message queue currently used in the big data field generally uses Kafka. Kafka is a high-throughput distributed publish-subscribe messaging system. It is a commonly used component in the big data and streaming fields. However, for metadata collection, the number of tables collected each time is relatively small. The existence of 100,000-level tables in the data warehouse is already quite large. Many companies often cannot reach this level, and this level does not require the use of Kafka, which increases the component's dependence on operation and maintenance costs. Redis stream can be selected as a more suitable solution. Redis itself also provides the Redis publish-subscribe (pub / sub) function, but there is a problem of Redis crashing and the message being lost, and the reliability is insufficient. At the same time, it is impossible to establish a consumer group to increase the consumption speed, and it is impossible to trace back the consumption history data.

[0109] To this end, the present invention proposes a technical solution for message queue processing based on redis stream.

[0110] Redis Stream is a lightweight message queue newly added in Redis 5.0. As an upgraded version of Redis pub / sub, it solves the following major problems of Redis pub / sub:

[0111] Reliability issues: Messages are not persistent. If the network is disconnected or redis crashes after sending, the subscriber cannot consume the data, resulting in message loss. There is no ack mechanism. If the subscriber crashes, the message will also be lost.

[0112] The same data cannot be consumed repeatedly, and messages cannot be consumed based on keys.

[0113] Due to the shortcomings of Redis pub / sub, Redis stream was introduced. Compared with other message queue types, Redis is lighter. The following are the core concepts of Redis stream.

[0114] Stream: Stream is the queue concept of Redis, which is equivalent to the topic concept in Kafka. The message identifier id of the message in stream is unique, and the format is timestamp-sequence number. The message identifier id generated later is required to be larger than the previous one.

[0115] Consumer: A consumer consumes messages from a stream.

[0116] Consumer group: A consumer group can contain multiple consumers. Multiple consumers consume the same stream together to increase the consumption speed.

[0117] PEL queue: When a consumer consumes a message, the message will be stored in the PEL queue. When the consumer submits ack, the message will be removed from the PEL queue and can be queried through the xpending command.

[0118] Metadata collection is an important function of the data center. It collects metadata from different systems and manages them in a unified manner, making it convenient to view metadata information in a unified manner, such as the database's Schema, Table, and Column information.

[0119] This paper proposes a metadata collection solution using redis stream as a lightweight message queue. Aiming at the consumption sequence and reliability of redis stream and distributed metadata collection, the following innovations are made to improve the collection speed:

[0120] By sending the tables under the database for metadata collection to the message queue, distributed and scalable collection and consumption can be achieved, thereby improving the speed of metadata collection.

[0121] Implement the order of redis stream consumption messages to ensure that when the same collector completes the collection, the previous metadata information has been executed.

[0122] When the producer sends a message, an identification field with endFlag=true is added to the last collected message.

[0123] When a consumer consumes a message, it first determines whether it is a message with endFlag=true. If not, it executes normally. If so, it uses the redis-stream xpending function to check whether there are messages in the PEL queue in the historical consumption records that have the collector identifier id of the target collector. If so, it does not submit ack. If not, it executes the collector end logic and submits ack.

[0124] When the technical solution of the present invention is applied, a timed task is started to regularly consume message records in the PEL queue, and messages that have not been executed for more than a certain period of time are re-consumed.

[0125] It should be noted that the contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

[0126] In the present embodiment, a message queue processing device is also provided, and a single device is used to implement the above-mentioned embodiment and optional implementation mode, and the description that has been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0127] Figure 5 It is a structural diagram of a message queue processing device according to an embodiment of the present invention.

[0128] The present invention provides a message queue processing device, such as Figure 5 As shown, the message queue processing device includes:

[0129] The first processing module 11 is used to receive a message to be processed collected by a target collector. The message to be processed includes a collector identifier of the target collector. The collector identifier of the target collector is used to distinguish the target collector from other collectors. The message to be processed is metadata.

[0130] The second processing module 12 is used to store the received messages to be processed into the message queue according to the time sequence. When the message to be processed is the last message collected by the target collector, an end mark is added to the message mark of the message to be processed.

[0131] The third processing module 13 is used to read the to-be-processed message in the message queue through the consumer, and determine whether the to-be-processed message is the last message according to the message identification bit of the to-be-processed message.

[0132] The third processing module 13 includes a first processing unit 131 , and the first processing unit 131 includes a first sub-processing unit 1311 and a second sub-processing unit 1312 .

[0133] The first sub-processing unit 1311 is used to query the relevant information of the uncompleted messages in the list of uncompleted messages when the unprocessed message is the last message. The list of unprocessed messages is used to store the uncompleted messages. The relevant information includes the collector identification.

[0134] The second sub-processing unit 1312 is used to determine whether to execute the business logic corresponding to the pending message according to the relevant information of the uncompleted message in the pending message list.

[0135] In an optional implementation, the third processing module 13 further includes a second processing unit.

[0136] The second processing unit is used to execute the business logic corresponding to the message to be processed when the message to be processed is not the last message, and return a response result when the business logic corresponding to the message to be processed is completed.

[0137] In an optional implementation, the second sub-processing unit 1312 is specifically configured to determine whether there are any uncompleted messages collected by the target collector in the list of messages to be processed according to relevant information of the uncompleted messages in the list of messages to be processed.

[0138] When there is no unfinished message collected by the target collector in the list of pending messages, the business logic corresponding to the pending message is executed, and a response result is returned when the business logic corresponding to the pending message is completed.

[0139] In an optional implementation, the second sub-processing unit 1312 is further configured to skip the business logic corresponding to the message to be processed when there are uncompleted messages collected by the target collector in the message list to be processed.

[0140] In an optional implementation, the relevant information further includes: consumption time and number of group transfers, and the message queue processing device further includes: a polling module.

[0141] The polling module includes a first processing unit and a second processing unit.

[0142] The first processing unit is used to read the consumption time of the unexecuted messages in the to-be-processed message list at a preset time interval.

[0143] The second processing unit is used to determine whether there is a first target message in the list of pending messages according to the consumption time of the uncompleted messages in the list of pending messages and the preset time. The first target message is an uncompleted message whose consumption time is greater than the preset time. When the first target message exists in the list of pending messages, a group transfer operation is performed on the first target message, the number of group transfers is increased by 1, and the consumption time of the first target message is reset to 0.

[0144] In an optional embodiment, the second processing unit under the polling module is also used to determine whether there is a second target message in the pending message list based on the number of group transfers and the preset number of uncompleted messages in the pending message list when the first target message does not exist in the pending message list. The second target message is an uncompleted message whose number of group transfers is greater than the preset number. When the second target message does not exist in the pending message list, the pending message in the message queue is read again by the consumer, and it is determined whether the pending message is the last message based on the message identification bit of the pending message.

[0145] In an optional implementation, the second processing unit under the polling module is further configured to determine that the collector message collection corresponding to the second target message fails when there is a second target message in the list of messages to be processed.

[0146] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0147] The message queue processing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0148] The embodiment of the present invention also provides a computer device having the above Figure 5 The message queue processing device shown.

[0149] See also Figure 6 , Figure 6 Schematic diagram of the hardware structure of the computer device according to the embodiment of the present invention. Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In an optional embodiment, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor device). Figure 6 A processor 10 is taken as an example.

[0150] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0151] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0152] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating device, an application required for at least one function. The data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In an optional embodiment, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0153] The memory 20 may include a volatile memory, such as a random access memory. The memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive. The memory 20 may also include a combination of the above-mentioned types of memory.

[0154] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0155] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc. Further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0156] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0157] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A message queue processing method, characterized in that: include: Receive the pending messages collected by the target collector; The message to be processed includes the collector identification of the target collector; The collector identification of the target collector is used to distinguish the target collector from other collectors; the message to be processed is metadata; The received messages to be processed are stored in a message queue according to the time sequence; wherein, when the message to be processed is the last message collected by the target collector, an end mark is added to the message mark of the message to be processed; The consumer reads the to-be-processed message in the message queue, and determines whether the to-be-processed message is the last message according to the message identification bit of the to-be-processed message; When the to-be-processed message is the last message, query the to-be-processed message list for related information of the uncompleted messages; the to-be-processed message list is used to store the uncompleted messages; the related information includes a collector identification; Determine whether to execute the business logic corresponding to the pending message according to the relevant information of the uncompleted message in the pending message list.

2. The method according to claim 1, characterized in that The determining, according to the message identification bit of the message to be processed, whether the message to be processed is the last message includes: When the message to be processed is not the last message, the business logic corresponding to the message to be processed is executed, and a response result is returned when the business logic corresponding to the message to be processed is completed.

3. The method according to claim 1, characterized in that The determining whether to execute the business logic corresponding to the pending message according to the relevant information of the uncompleted message in the pending message list includes: Determine whether there are uncompleted messages collected by the target collector in the list of messages to be processed according to relevant information of the uncompleted messages in the list of messages to be processed; When there is no unfinished message collected by the target collector in the list of pending messages, the business logic corresponding to the pending message is executed, and a response result is returned when the business logic corresponding to the pending message is executed.

4. The method according to claim 1, characterized in that The determining, according to the relevant information of the uncompleted messages in the list of messages to be processed, whether there are uncompleted messages collected by the target collector in the list of messages to be processed includes: When there are uncompleted messages collected by the target collector in the list of messages to be processed, the business logic corresponding to the messages to be processed is skipped.

5. The method according to claim 1, characterized in that The relevant information also includes: consumption time and number of group transfers. The method also includes: Reading the consumption time of the unexecuted messages in the list of pending messages at a preset time interval; Determine whether there is a first target message in the list of pending messages according to the consumption time and the preset time of the uncompleted messages in the list of pending messages; the first target message is an uncompleted message whose consumption time is greater than the preset time; When there is a first target message in the list of messages to be processed, a group transfer operation is performed on the first target message, the number of group transfers is increased by 1, and the consumption time of the first target message is reset to 0.

6. The method according to claim 5, characterized in that The determining whether there is a first target message in the list of messages to be processed according to the consumption time and the preset time of the uncompleted messages in the list of messages to be processed includes: When there is no first target message in the list of pending messages, determining whether there is a second target message in the list of pending messages according to the number of group transfers and the preset number of the uncompleted messages in the list of pending messages; the second target message is an uncompleted message whose number of group transfers is greater than the preset number; When there is no second target message in the list of pending messages, the pending messages in the message queue are read again by the consumer, and it is determined whether the pending message is the last message according to the message identification bit of the pending message.

7. The method according to claim 6, characterized in that The determining whether there is a second target message in the list of messages to be processed according to the number of group transfers of the uncompleted messages in the list of messages to be processed and the preset number of times includes: When there is a second target message in the list of messages to be processed, it is determined that the collector message collection corresponding to the second target message fails.

8. A message queue processing device, characterized in that: include: A first processing module, used for receiving the to-be-processed message collected by the target collector; The message to be processed includes the collector identification of the target collector; The collector identification of the target collector is used to distinguish the target collector from other collectors; the message to be processed is metadata; The second processing module is used to store the received to-be-processed messages into a message queue according to the time sequence; wherein, when the to-be-processed message is the last message collected by the target collector, an end mark is added to the message mark of the to-be-processed message; A third processing module is used to read the to-be-processed message in the message queue through a consumer, and determine whether the to-be-processed message is the last message according to a message identification bit of the to-be-processed message; The third processing module includes a first processing unit, and the first processing unit includes a first sub-processing unit and a second sub-processing unit; The first sub-processing unit is used to query the relevant information of the unfinished messages in the list of unfinished messages when the unfinished message is the last message; the list of unfinished messages is used to store the unfinished messages; the relevant information includes the collector identification; The second sub-processing unit is used to determine whether to execute the business logic corresponding to the pending message according to the relevant information of the uncompleted message in the pending message list.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the message queue processing method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the message queue processing method according to any one of claims 1 to 7.

Images