Message processing methods, apparatus and equipment

Abstract

This application provides a message processing method, apparatus, and device. The method includes: dynamically configuring the message storage quota of each message queue topic when a consumer in a consumption unit pulls messages from a corresponding message queue topic in a message queue unit; the message queue topic stores messages to be transmitted sent by an upstream system; the message queue unit includes at least one message queue topic, each message queue topic corresponds to at least one consumer, and the consumption unit includes at least one consumer; and transmitting the stored messages to be transmitted to the corresponding consumer. In this technical solution, by setting different message storage quotas for multiple message topics, the topic quotas can be dynamically adjusted according to the real-time message consumption and processing situation, adapting to different message scenarios, thereby achieving reasonable resource utilization. Quotas can be allocated from top to bottom and collected from bottom to top based on the quota implementation situation of each topic, providing greater flexibility.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a message processing method, apparatus, and device. Background Technology

[0002] A message queue is a container that stores messages during message transmission. The main purpose of a message queue is to provide routing and ensure message delivery. If the receiver is unavailable when a message is sent, the message queue will retain the message until it can be successfully delivered.

[0003] In existing technologies, message delivery based on message queues mainly involves setting up multiple message queue topics according to pre-defined business rules, storing different messages in different topics, setting quotas for each topic, and having the upstream system send messages to each topic according to the quotas.

[0004] However, when the upstream system has no messages to send, there is a backlog of one or more topics with no messages, causing the consumer to pull less messages than the maximum amount allocated at the beginning. This results in the consumer frequently pulling small batches of messages, leading to a waste of resources. Summary of the Invention

[0005] This application provides a message processing method, apparatus, and device to solve the problem of resource waste caused by consumers frequently pulling small batches of messages.

[0006] In a first aspect, embodiments of this application provide a message processing method applied to a message processing device, the message processing device including at least a message queue unit and a consumption unit, the method comprising:

[0007] When a consumer in the consumption unit pulls a message from the corresponding message queue topic in the message queue unit, the message storage quota of each message queue topic is dynamically configured. The message queue topic stores messages to be transmitted sent by the upstream system. The message queue unit includes at least one message queue topic, and each message queue topic corresponds to at least one consumer. The consumption unit includes at least one consumer.

[0008] Transmit the stored messages to be transmitted to the corresponding consumers.

[0009] Secondly, embodiments of this application provide a message processing apparatus, including:

[0010] The quota configuration module is used to dynamically configure the message storage quota of each message queue topic when a consumer in the consumption unit pulls messages from the corresponding message queue topic in the message queue unit. The message queue topic stores messages to be transmitted sent by the upstream system. The message queue unit includes at least one message queue topic, and each message queue topic corresponds to at least one consumer. The consumption unit includes at least one consumer.

[0011] The message transmission module is used to transmit stored messages to the corresponding consumers.

[0012] Thirdly, embodiments of this application provide a message processing device, including: a message processing device and a dynamic flow control device, a dynamic quota control device, and a status refresh device respectively connected to the message processing device, wherein the status refresh device is connected to the dynamic quota control device, and the dynamic quota control device is connected to the dynamic flow control device;

[0013] The message processing device includes a message queue unit and a consumption unit, and the message queue unit is connected to the consumption unit.

[0014] The message queue unit is used to create message queue topics of different priorities based on the messages to be transmitted sent by the upstream system. Each message queue topic is used to store messages of the corresponding type to be transmitted.

[0015] The consumption unit includes a special consumer and a consumer corresponding to the message queue topic. Each consumer is used to pull messages from the corresponding message queue topic and send them to the special consumer. The special consumer is used to perform unified message sending.

[0016] The dynamic quota controller is used to initialize the quota for each message queue topic and dynamically adjust the quota for each message queue topic according to the message backlog in the message queue topic and the message consumption status of the consumer.

[0017] The dynamic flow control device is used to dynamically configure the processing rate of each message queue topic under a preset total flow control rate;

[0018] The status refresh device is used to obtain the status information of each message queue topic from the message queue unit and send it to the dynamic quota control device.

[0019] The message processing method, apparatus, and device provided in this application embodiment can dynamically adjust the topic quota according to the real-time consumption and processing of messages by setting different message storage quotas for multiple message topics, adapting to different message scenarios, thereby achieving reasonable utilization of resources. The quota can be allocated from top to bottom and collected from bottom to top according to the quota implementation of each topic, which is more flexible. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application;

[0021] Figure 1 This is a schematic diagram of a message processing scenario provided in an embodiment of this application;

[0022] Figure 2 A flowchart illustrating the message processing method provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of the structure of the message processing apparatus provided in the embodiments of this application;

[0024] Figure 4 This is a schematic diagram of the workflow of the message processing device provided in the embodiments of this application;

[0025] Figure 5 This is a schematic diagram of the state setting process provided in the embodiments of this application;

[0026] Figure 6 A schematic diagram of the storage quota reduction process provided in this application embodiment;

[0027] Figure 7 A schematic diagram illustrating the process of collecting storage quotas as provided in this application embodiment;

[0028] Figure 8 This is a schematic diagram of the working process of the dynamic flow control device in the embodiments of this application;

[0029] Figure 9 A schematic diagram illustrating the workflow of the state refresh device provided in this application embodiment;

[0030] Figure 10 This is a schematic diagram of the structure of the message processing device provided in the embodiments of this application.

[0031] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] First, let me explain the terms used in this application:

[0034] A message queue is a container that stores messages during message transmission. The main purpose of a message queue is to provide routing and ensure message delivery. If the receiver is unavailable when a message is sent, the message queue will retain the message until it can be successfully delivered.

[0035] Message topic: This is a classification of messages based on business rules. Different types of messages correspond to different topics, and messages within the same topic have the same priority.

[0036] Product code: This represents each type of SMS message, such as transaction SMS, verification code SMS, marketing SMS, etc. Multiple product codes may correspond to the same theme, in which case they have the same priority.

[0037] Producers: These are the entities that place messages into different topics in the message queue according to business rules.

[0038] Consumers retrieve and process messages from different topics in the message queue.

[0039] Special consumers: This refers to messages consumed by multiple consumers with different priorities, which are then sent in a unified manner.

[0040] Quota: The maximum capacity of messages that each topic in a message queue can hold.

[0041] Dynamic quotas: These are used to dynamically configure different quotas for messages of different priorities based on actual circumstances.

[0042] Dynamic flow control: Dynamic flow control refers to the dynamic allocation of processing rates for each priority level under a given total flow control rate.

[0043] Status refresh: This triggers dynamic quotas and dynamic flow control within a given time interval and updates the processing status of each priority level in real time.

[0044] Online SMS: These are messages generated in real time by the upstream system, processed into SMS messages by the message service platform, and then sent to the client terminal.

[0045] Batch SMS: This refers to the upstream system sending SMS messages of the same service type to multiple users simultaneously, where timeliness requirements are not high.

[0046] With the rapid development of internet technology, the scale of messaging services has also expanded rapidly. As a system that directly faces customers, SMS services have extremely high requirements for timeliness and accuracy, especially for sensitive messages, which directly affect user experience and various transaction processes. For SMS sending scenarios, each type of SMS corresponds to a product code, such as account transaction SMS, verification code SMS, marketing SMS, loan overdue SMS, interest settlement SMS, etc. Based on time, they can be divided into online SMS and batch SMS, and based on timeliness, they can be divided into extremely time-sensitive SMS and moderately time-sensitive SMS. Online SMS has higher priority than batch SMS, and extremely time-sensitive SMS has higher priority than moderately time-sensitive SMS. SMS messages are received and sent through message queues, which treat each topic partition as a sequentially ordered log. The upstream system sends messages to the topics in the message queue for storage, and consumers retrieve messages from the topics, process them, and then send them. Although message queues have high throughput, their sequential write mechanism does not allow for priority queuing. In traditional message queues, messages are consumed in the order they arrive. For example, if marketing SMS messages enter the message queue before verification code SMS messages, the marketing SMS will be sent first. This approach doesn't consider message priority and cannot achieve the "queue-jumping" effect, exhibiting a significant flaw. Furthermore, setting different message queues based on different priorities, with these queues isolated and not interfering with each other, can lead to high-priority message queues being blocked and waiting, while low-priority queues may be idle, still failing to truly achieve priority or "queue-jumping" effects. Against this backdrop, configuring different message quotas for multiple queue units to send messages becomes possible, enabling high-priority messages to "jump the queue" and be sent first.

[0047] Current priority control schemes based on message queues mainly include: First, setting up multiple message queue topics according to pre-defined business rules, with different messages corresponding to different topics. Quotas for each topic are set based on historical experience and remain unchanged. Ideally, upstream messages are always sent according to the quota. Second, setting up multiple message topics with different quotas based on scheme one. The difference is that the quota for each message topic can be dynamically adjusted according to the number of messages pulled. If a high-priority topic has excess quota, it can be allocated to a lower-priority topic. Third, setting up different message queues based on different priorities, with higher-priority message queues having higher quotas and lower-priority message queues having lower quotas. Compared to scheme one, where multiple topics are set up in a single queue and messages from all topics are aggregated and sent in the same batch, scheme three sets up multiple message queues, with each priority queue independent of the others and without interference.

[0048] For Option 1, different priority quotas are pre-set based on historical experience and remain unchanged. This option is feasible if the upstream system consistently sends messages according to the quota. However, for multiple priority topics, consider situations where the upstream system has no messages to send. In such cases, one or more topics may have a backlog of unsent messages. The consumer will pull fewer messages than the initial maximum pull amount, leading to frequent small-batch pulls and wasted resources. For example, a privileged consumer might be initially specified to pull 50 messages at a time, but the actual pull amount might be 30. When lower-priority messages accumulate while higher-priority messages are idle, the lower-priority messages will still be pulled at a lower quota ratio. For Option 2, dynamic adjustment can allocate excess quota from high-priority topics to lower-priority topics. However, because the allocation is unidirectional, after high-priority quotas are exhausted, message volume surges, and the remaining quotas become insufficient. Lower-priority topic quotas cannot be reassigned, leading to a backlog of high-priority messages, creating a performance bottleneck and affecting system stability. For Option 3, consumption is isolated between priorities, with different priorities corresponding to different processing flows. This fundamentally fails to achieve priority or "queue-jumping" effects. Low-priority messages have smaller quotas, allowing them to immediately fetch the next batch after consuming the current batch; high-priority messages have larger quotas and are fetched less frequently. Extending the time window will cause the total number of messages fetched by high and low priority messages to become more consistent. For example, if a message queue with a quota of 30 fetches messages once and a message queue with a quota of 5 fetches messages six times within a certain period, then the total number of messages fetched by both queues is equal, at 30.

[0049] To address the aforementioned issues, embodiments of this application provide a message processing method, apparatus, and device. By setting different message storage quotas for multiple message topics, the topic quotas can be dynamically adjusted based on real-time message consumption and processing, adapting to different message scenarios and thus achieving rational resource utilization. Quotas can be allocated from top to bottom and collected from bottom to top based on the quota implementation status of each topic, offering greater flexibility.

[0050] The technical solution of this application will now be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0051] Figure 1 This is a schematic diagram of a message processing scenario provided in an embodiment of this application, such as... Figure 1 As shown, the upstream system 10 can be a message producer, which sends messages to the message queue 11, where the message queue 11 caches them, and then the consumer 12 pulls them to complete the consumption of the messages.

[0052] Figure 2This is a flowchart illustrating a message processing method provided in an embodiment of this application. This method can be applied to a message processing device, which can be integrated into an electronic device, such as a computer. The message processing device includes at least a message queue unit and a consumer unit. Figure 2 As shown, the method may specifically include the following steps:

[0053] Step S201: When a consumer in a consumer unit pulls messages from the corresponding message queue topic in a message queue unit, the message storage quota for each message queue topic is dynamically configured. Here, a message queue topic stores messages to be transmitted from the upstream system; a message queue unit includes at least one message queue topic; each message queue topic corresponds to at least one consumer; and a consumer unit includes at least one consumer.

[0054] Step S202: Transmit the stored message to be transmitted to the corresponding consumer.

[0055] In this embodiment, each message queue topic corresponds to at least one consumer. Each consumer pulls a preset number of messages from the message queue topic. The messages in the message queue topic originate from the upstream system (i.e., the message producer). Different consumers can pull different amounts of messages from the message queue topic. For example, each message queue topic can be preset with an initial message storage quota. For instance, message queue topic X1 can be set to an initial message storage quota of E1, and message queue topic X2 can be set to an initial message storage quota of E2. The consumer corresponding to message queue topic X1 can be consumer F1, and the consumer corresponding to message queue topic X2 can be consumer F2. Consumer F1 can pull 30 messages from message queue topic X1 each time (if the actual number of stored messages is less than 30, then all stored messages are pulled), and consumer F2 can pull 10 messages from message queue topic X2 each time (if the actual number of stored messages is less than 30, then all stored messages are pulled).

[0056] Specifically, when the number of messages stored in message queue topic X1 or message queue topic X2 is insufficient to meet the pull volume, the message storage quota of message queue topic X1 or message queue topic X2 needs to be dynamically adjusted so that the adjusted message storage quota is greater than or equal to the pull volume, thereby avoiding consumers frequently pulling small batches of messages and wasting resources.

[0057] This application embodiment sets different message storage quotas for multiple message topics, which can dynamically adjust the topic quotas according to the real-time consumption and processing of messages, adapting to different message scenarios, thereby achieving reasonable utilization of resources. The quotas can be allocated from top to bottom and collected from bottom to top according to the quota implementation of each topic, which is more flexible.

[0058] In some embodiments, Figure 3 This is a schematic diagram of the structure of the message processing device provided in the embodiments of this application, such as... Figure 3 As shown, the message processing device 300 includes a message queue unit 310, a consumer unit 320, a dynamic flow control device 330, a dynamic quota control device 340, and a status refresh device 350, in addition to these. The message queue unit 310 includes several message queue topics, and the consumer unit 320 includes a number of consumers corresponding to each message queue topic, as well as a special consumer. Each consumer is connected to a message queue topic, and the special consumer is connected to each consumer. The special consumer is used for unified message delivery.

[0059] The dynamic flow control device 330 also includes a flow control switch and a flow control configuration unit; the dynamic quota control device 340 also includes an active reduction unit and a passive collection unit; and the status refresh device 350 also includes a status update unit and a fixed trigger unit.

[0060] The status refresh device 350 is connected to the dynamic quota controller 340, the consumer unit 320, and the dynamic flow control device 330. The dynamic quota controller 340 is connected to the consumer unit 320 and the dynamic flow control device 330. The dynamic flow control device 330 is connected to the consumer unit 320. The message queue unit 330 is connected to the consumer unit 320.

[0061] Furthermore, based on the above embodiments, in some embodiments, the above method further includes the following steps: configuring the priority of each message queue topic in the message queue unit, wherein different message queue topics have different priorities, and message storage quotas for message queue topics with different priorities are different.

[0062] In this embodiment, messages are categorized by priority based on all message types in the upstream system. The message queue unit creates a corresponding number of message queue topics based on the number of categories to store messages. A priority coefficient is assigned to each message queue topic; the higher the priority, the larger the coefficient. The dynamic quota controller mentioned above initializes a quota for each message topic in the message queue unit. After initialization, the number of messages stored in each message topic cannot exceed the set quota.

[0063] Furthermore, different priority message queue topics correspond to different message pull volumes for consumers. For example, there are three message queue topics: topic 1 has the highest priority (priority coefficient 2), topic 2 has a relatively high priority (priority coefficient 1), and topic 3 has the lowest priority (priority coefficient 0). Assuming a particular consumer pulls 50 messages each time, the quota for topic 1 is set to 30, topic 2 to 15, and topic 3 to 5, respectively, according to their priorities. Consumers for these topics will then consume messages according to these quotas.

[0064] For example, Figure 4 This is a schematic diagram of the workflow of the message processing device provided in the embodiments of this application, such as... Figure 4 As shown, it includes the following steps: Step S401, the message queue unit creates multiple message queue topics and configures corresponding priorities. Step S402, multiple consumers in the consumption unit pull messages from the corresponding message queue topics. Step S403, a special consumer aggregates messages from multiple consumers and sends them uniformly through a centralized message distribution system.

[0065] The message queue unit contains multiple message queue topics, each corresponding to a different type of SMS message. The upstream system sends messages to different message queue topics for queuing according to business rules. For SMS scenarios, one type of SMS message corresponds to one product code. SMS messages with multiple product codes may have the same or similar priorities; in this case, they reside in the same message queue topic and will be consumed by the same consumer. Topic priorities are pre-defined; for example, messages in message queue topic 1 have the highest priority, and messages in message queue topic n have the lowest priority. Once the priority setting for a message queue topic is fixed, it will not be modified. The consumption unit contains multiple consumer processes. Each consumer process consumes messages from a fixed message queue topic. If a message queue topic has no message backlog, the corresponding consumer process is in an idle waiting state. Special consumers aggregate messages consumed by multiple consumers with different priorities and send them uniformly through a single point of contact.

[0066] This application implements different quotas and stores messages of different priorities in different message topics. High-priority messages can be consumed in advance, which can truly achieve the effect of "jumping the queue" and allow high-priority messages to be sent out first.

[0067] Based on the above embodiments, in some other embodiments, the "dynamically configuring the message storage quota of each message queue topic" in step S201 can be implemented through the following steps: determining the message storage quota of each topic and the message pull amount when each consumer in the consumption unit pulls messages from the corresponding message queue unit; dynamically configuring the message storage quota of each message queue topic according to the message storage quota and the pull amount.

[0068] In this embodiment, the dynamic quota controller initializes the quota (i.e., initializes the message storage quota) for each message queue topic in the message queue unit. After initialization, the number of messages stored in each message queue topic cannot exceed the set quota. In addition, based on the message backlog and message consumption in the message queue topics, the dynamic quota controller can dynamically adjust the quota setting (i.e., message storage quota) for each message queue topic.

[0069] This application embodiment adjusts the message storage quota of each message queue topic through a dynamic quota controller, adapting to different message scenarios and avoiding consumers frequently pulling small batches of messages from the message queue topic, thereby achieving reasonable utilization of resources.

[0070] Furthermore, in some other embodiments, the above step of "dynamically configuring the message storage quota of each message queue topic based on the message storage quantity and the pull quantity" can be implemented through the following steps: determining whether there is a first target message queue topic or a second target message queue topic in each message queue topic, wherein the difference between the message storage quantity and the pull quantity of the first target message queue topic is greater than a first preset threshold, and the difference between the message storage quantity and the pull quantity of the second target message queue topic is less than a second preset threshold; when the first target message queue topic exists, increasing the message storage quota of the first target message queue topic; when the second target message queue topic exists, decreasing the message storage quota of the second target message queue topic.

[0071] In this embodiment, the total message storage quota of all message queue topics can be regarded as a fixed value. When the message storage quota of one of the message queue topics needs to be adjusted, its message storage quota can be allocated to other message queue topics or obtained from other message queue topics.

[0072] For example, taking message queue topics F1, F2, and F3 as examples, their priorities are as follows: F1 > F2, and F2 > F3. The current message storage quota for message queue topic F1 is FE1. When a sudden surge of messages from the upstream system causes message backlog in message queue topic F1, the message storage quota for F1 can be increased. Specifically, some message storage quota can be borrowed from message queue topics F2 and F3. For example, first borrow K1 from message queue topic F3. If the borrowed message storage quota (FE1 + K1) still cannot resolve the message backlog, then message storage quota can be borrowed from message queue topic F2.

[0073] This application embodiment achieves rapid message sending while keeping the total message storage limit constant by dynamically configuring the message storage quantity for each message queue topic, thus avoiding message backlog and improving message sending efficiency.

[0074] Furthermore, based on the above embodiments, in some other embodiments, the above step "increasing the message storage quota of the message queue topic corresponding to the target consumer" can be implemented through the following steps: determining the priority of the first target message queue topic and the lowest message queue topic, wherein the priority of the lowest message queue topic is the lowest among all message queue topics; allocating the target quota from the message storage quota of the lowest message queue topic to the first target message queue topic.

[0075] In this embodiment, when a high-priority message queue topic allocates excess message storage quota to other message queue topics, resulting in a smaller message storage quota for itself, if the message volume of the upstream system surges within a certain period of time, the message storage quota of the high-priority message queue topic will no longer meet the capacity requirements. It will need to solicit excess quota from the lowest-priority topic. If the lowest-priority topic does not meet the requirements, it will continue to solicit quotas from higher-priority topics to meet the requirements.

[0076] For example, taking message queue topics F1, F2, and F3 as examples, their priorities are as follows: message queue topic F1 > message queue topic F2, and message queue topic F2 > message queue topic F3. When the current message storage quota FE1 of message queue topic F1 is insufficient to meet the current demand and more message storage quota is needed, message storage quota is first solicited from the lowest priority message queue topic F3 (for example, the solicited message storage quota is α). At this time, the message storage quota of message queue topic F1 becomes (FE1 + α). If the message storage quota of message queue topic F1 becomes (FE1 + α) and still does not meet the current demand, then message storage quota is continued to be solicited from message queue topic F2.

[0077] This application embodiment increases the message storage limit of the message queue topic. When a sudden message outbreak occurs in the upstream system, the message storage limit of the message queue topic can be adjusted in a timely manner to avoid message storage failure due to insufficient storage limit and ensure the reliability of message processing.

[0078] Furthermore, based on the above embodiments, in some other embodiments, the above step "reducing the message storage quota of the message queue topic" can be implemented through the following steps: determining the priority of the second target message queue topic and the next message queue topic, wherein the priority of the next message queue topic is lower than the priority of the second target message queue topic; allocating the target quota from the message storage quota of the second target message queue topic to the next message queue topic.

[0079] In this embodiment, when the message storage quota of a certain message queue topic is too high and a certain amount of message storage quota needs to be allocated, it can be allocated from the highest priority to the next lowest priority. When the next lowest priority message queue topic does not need the extra quota, it can continue to be allocated downwards until a message queue topic that meets the conditions is found. In special cases, the message storage quota of the highest priority message queue topic can be allocated to the lowest priority message queue topic.

[0080] For example, taking message queue topics F1, F2, and F3 as examples, their priorities are as follows: message queue topic F1 > message queue topic F2, and message queue topic F2 > message queue topic F3. When the message storage quota FE1 of message queue topic F1 is too large and needs to be allocated to other message queue topics with K (K less than FE1) message storage quotas, it will be allocated to message queue topic F2 in order. If message queue topic F2 does not need the extra quota, it will then be allocated to message queue topic F3.

[0081] In addition, in this embodiment, if the upstream system message volume surges within a certain period after a high-priority message queue topic has been allocated excess quota, and the current message storage quota of the high-priority message queue topic is no longer sufficient to meet the capacity requirements, then excess message storage quota will be solicited from the lowest-priority message queue topic first. If the lowest-priority message queue topic is insufficient, the quota will continue to be solicited from the next higher-priority message queue topic to meet the requirements.

[0082] This application reduces the message storage limit of a message queue topic, thereby allocating more message storage limit to other message queue topics, improving the flexibility of message storage, and preventing other message queue topics from being unable to store messages in a timely manner due to message storage limit limitations.

[0083] In some embodiments, the above method may further include the following steps: determining the relationship between the message pull volume and the current message storage quota of the message queue topic when each consumer pulls messages from the corresponding message queue topic, wherein the message storage quota is between a preset upper limit and a preset lower limit; if the message pull volume is greater than or equal to the message storage quota, then setting the status of the message queue topic to busy; if the message pull volume is greater than the lower limit but less than the message storage quota, then setting the status of the message queue topic to semi-busy; if the message pull volume is less than or equal to the lower limit, then setting the status of the message queue topic to idle; and dynamically configuring the message storage quota of each message queue topic according to the status of each message queue topic.

[0084] In this embodiment, when dynamically adjusting the message storage quota for each message queue topic, the dynamic quota needs to have upper and lower limits set. Initialization requires configuring an "upper limit" and a "lower limit" for each priority message queue topic according to business rules. The message storage quota for each priority message queue topic fluctuates within the upper and lower limits and cannot exceed these limits.

[0085] Each priority message queue topic needs to be set with a processing status. When the consumer's message pull volume equals the current message storage limit, the corresponding message queue topic is in a "busy" state. When the preset lower limit is less than the message pull volume and the preset upper limit is less than the preset upper limit, the corresponding message queue topic is in a "semi-busy" state. If the pull volume reaches or falls below the preset upper limit, the corresponding message queue topic is in an "idle" state.

[0086] For example, Figure 5 This is a schematic diagram of the state setting process provided in the embodiments of this application, such as... Figure 5As shown, the method includes the following steps: Step S501, setting the upper and lower limits for each message queue topic according to business rules. Step S502, determining whether the message pull volume is greater than or equal to the current message storage limit. Step S503, setting this message queue topic to a busy state. Step S504, determining whether the message pull volume is less than the current message storage limit but greater than the lower limit. Step S505, setting this message queue topic to a semi-busy state. Step S506, setting this message queue topic to an idle state.

[0087] This application embodiment enables real-time detection of whether the message storage quota of each message queue topic needs to be changed by setting the status of each message queue topic, thus ensuring the timeliness of dynamic quota allocation for each message queue topic.

[0088] For example, Figure 6 This is a schematic diagram of the storage quota reduction process provided in the embodiments of this application, as follows: Figure 6 As shown, it includes the following steps: Step S601, whether the message pull amount is less than the current message storage quota. Step S602, the difference between the current message storage quota and the pull amount is allocated to the next lower priority message queue topic. Step S603, whether the next lower priority message queue topic has reached its upper limit limit. Step S604, whether the quota-restricted message queue topic is in a busy state. Step S605, allocate to the next lower priority message queue topic, until the lowest priority message queue topic.

[0089] In this embodiment, for message queue topics whose pull volume is insufficient to meet the current message storage quota, their storage quota is reduced. The difference between the current message storage quota and the pull volume is used as the available quota for allocation. Capacity is allocated only to the next lower priority level. If the capacity of the next lower priority level reaches the "upper limit" limit, the excess capacity continues to be allocated to the next lower priority level. The message queue topic subject to the quota needs to be in a "busy" state. If all message queue topics are not in a "busy" state, the quota will eventually be concentrated on the lowest priority message queue topic. The lowest priority message queue topic is not subject to the "upper limit" limit.

[0090] For example, Figure 7 A schematic diagram of the storage quota collection process provided in this application embodiment is shown below. Figure 7As shown, it includes the following steps: Step S701, calculate the amount to be replenished. Step S702, check if the current pull amount is greater than the upper limit of the message queue topic. Step S703, assign the amount to be replenished to the pull amount. Step S704, assign the amount to be replenished to the difference between the upper limit and the current message storage amount. Step S705, solicit quota from the lowest priority message queue topic. Step S706, check if the low-priority message queue topic has excess quota. Step S707, solicit quota from the message queue topic of the next higher priority than the lowest priority.

[0091] In this embodiment, if a message queue topic initially had no message backlog but then experiences a backlog due to message production, the previously reduced quota for that topic needs to be replenished. If the pull volume is greater than or equal to the dynamically adjusted message storage quota, the quota is increased. This increased quota needs to be requisitioned from other priority message queue topics. The requisition strategy prioritizes requisitioning the quota requirement from the lowest priority message queue topic. The currently available quota for the lowest priority topic needs to be calculated. If the currently available quota is lower than the required quota, the remaining quota is recursively requisitioned from the next higher priority message queue topic until the required quota is reached. The requisitioned message queue topic must be in an "idle" state. The lowest priority requisition capacity is not limited by the "busy" state. The requisition quota size is limited not only by the "upper limit" of the current lowest priority message queue topic but also by the available quota provided by the other priority message queue topics. The available quota size is equal to the available quota of all message queue topics in a "busy" state, thus maintaining a consistent total dynamic quota.

[0092] In some embodiments, the above method further includes the following steps: in response to dynamic quota and flow control requests, obtaining the ratio of the current message storage quota of each message queue topic to a preset baseline quota; and configuring the flow control rate of each message queue topic according to the ratio and the preset total flow control rate.

[0093] In this embodiment, the dynamic flow control device dynamically allocates the processing rate of each message queue topic under a given total flow control rate. When a message queue topic has no message backlog, its rate is dynamically allocated to other message queue topics. Dynamic flow control relies on dynamic quotas. Each message queue topic pulls messages synchronously, meaning the processing time of each message queue topic is consistent. Flow control rate is controlled by controlling the quotas. Specifically, with the dynamic quota controller and flow control switch unit enabled, after each quota update by the flow control configuration unit, the ratio of the current quota to the baseline quota for each message queue topic is calculated. Based on this ratio, the flow control rate of each message queue topic under a given total flow control rate can be calculated. The dynamic quota and dynamic flow control use the same ratio, ensuring that the message processing time for a specific message queue topic is consistent before and after dynamic allocation.

[0094] For example, Figure 8 This is a schematic diagram of the working process of the dynamic flow control device in the embodiments of this application, as shown below. Figure 8 As shown, it includes the following steps: Step S801, whether to enable flow control and dynamic quota. Step S802, calculate the ratio of the current quota to the base quota for each message queue topic. Step S803, calculate the total flow control rate. Step S804, calculate the flow control rate for each message queue topic based on the quota ratio and the total flow control rate.

[0095] The embodiments of this application can dynamically allocate the processing rate of each message queue unit through a dynamic quota controller, so that high-priority messages can be processed and sent.

[0096] Figure 9 This is a schematic diagram of the workflow of the state refresh device provided in the embodiments of this application, as shown below. Figure 9 As shown, it includes the following steps: Step S901, collect data for each message queue topic, including the initial message storage quota, current message storage quota, upper limit, lower limit, and status. Step S902, check if the triggering unit meets the triggering conditions. Step S903, poll and wait. Step S904, update the status of each message queue topic and implement dynamic quota and dynamic flow control.

[0097] In this embodiment, the status update unit collects data, including the initial message storage quota, the current message storage quota, upper limit, lower limit, and status. The amount of data pulled by each consumer corresponding to each message queue topic is written as metadata to the status update unit, which then updates the message queue topic status. A fixed trigger unit updates the status of each message queue topic through the status update unit within a given time interval, triggering dynamic quotas and dynamic flow control. The fixed trigger unit's time interval affects the real-time performance of dynamic quotas and dynamic flow control and can be set as needed based on actual conditions.

[0098] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0099] Figure 10 This is a schematic diagram of the structure of the message processing device provided in the embodiments of this application, such as... Figure 10 As shown, the message processing device 1000 may specifically include a quota configuration module 1010 and a message transmission module 1020. The quota configuration module 1010 is used to dynamically configure the message storage quota of each message queue topic when a consumer in a consumer unit pulls messages from a corresponding message queue topic in a message queue unit. Each message queue topic stores messages to be transmitted sent by the upstream system. Each message queue unit includes at least one message queue topic, and each message queue topic corresponds to at least one consumer. The message transmission module 1020 is used to transmit the stored messages to be transmitted to the corresponding consumers.

[0100] Optionally, the above message processing device further includes: a priority configuration module, used to configure the priority of each message queue topic in the message queue unit, wherein different message queue topics have different priorities and different message storage quotas for message queue topics with different priorities.

[0101] Optionally, the quota configuration module can be used to: determine the message storage quota for each topic and the message pull amount when each consumer in the consumption unit pulls messages from the corresponding message queue unit; and dynamically configure the message storage quota for each message queue topic based on the message storage quota and the pull amount.

[0102] Optionally, the quota configuration module can be used to: determine whether there is a first target message queue topic or a second target message queue topic in each message queue topic, wherein the difference between the message storage quantity and the pull quantity of the first target message queue topic is greater than a first preset threshold, and the difference between the message storage quantity and the pull quantity of the second target message queue topic is less than a second preset threshold; when the first target message queue topic exists, increase the message storage quota of the first target message queue topic; when the second target message queue topic exists, decrease the message storage quota of the second target message queue topic.

[0103] Optionally, the quota configuration module can be used to: determine the priority of the first target message queue topic and the lowest message queue topic, wherein the priority of the lowest message queue topic is the lowest among all message queue topics; and allocate the target quota from the message storage quota of the lowest message queue topic to the first target message queue topic.

[0104] Optionally, the quota configuration module can be used to: determine the priority of the second target message queue topic and the next message queue topic, wherein the priority of the next message queue topic is lower than the priority of the second target message queue topic; and allocate the target quota from the message storage quota of the second target message queue topic to the next message queue topic.

[0105] Optionally, a status configuration module is also included to determine the relationship between the message pull volume and the current message storage quota of the message queue topic when each consumer pulls messages from the corresponding message queue topic. The message storage quota is between a preset upper limit and a preset lower limit. If the message pull volume is greater than or equal to the message storage quota, the status of the message queue topic is set to busy. If the message pull volume is greater than the lower limit but less than the message storage quota, the status of the message queue topic is set to semi-busy. If the message pull volume is less than or equal to the lower limit, the status of the message queue topic is set to idle. The message storage quota of each message queue topic is dynamically configured according to its status.

[0106] Optionally, a flow rate configuration module is also included, which is used to: obtain the ratio of the current message storage quota of each message queue topic to the preset baseline quota in response to dynamic quota and flow control requests; and configure the flow control rate of each message queue topic according to the ratio and the preset total flow control rate.

[0107] The apparatus provided in this application embodiment can be used to execute the methods in the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0108] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the quota configuration module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and its functions can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through the integrated logic circuits in the hardware of the processor element or through software instructions.

[0109] For example, please refer to the above. Figure 3 This application provides a message processing device, comprising: a message processing device and a dynamic flow control device, a dynamic quota control device, and a status refresh device respectively connected to the message processing device. The status refresh device is connected to the dynamic quota control device, and the dynamic quota control device is connected to the dynamic flow control device. The message processing device includes a message queue unit and a consumption unit, which are connected to each other. The message queue unit is used to create message queue topics of different priorities based on messages to be transmitted sent by the upstream system. Each message queue topic is used to store messages of the corresponding type to be transmitted. The consumption unit includes special consumers and consumers corresponding to message queue topics. Each consumer is used to pull messages from the corresponding message queue topic and send them to the special consumer. The special consumer is used to perform unified message transmission. The dynamic quota control device is used to initialize the quota for each message queue topic and dynamically adjust the quota of each message queue topic based on the message backlog status in the message queue topic and the message consumption status of the consumer. The dynamic flow control device is used to dynamically configure the processing rate of each message queue topic under a preset total flow control rate. The status refresh device is used to obtain the status information of each message queue topic from the message queue unit and send it to the dynamic quota control device.

[0110] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the preceding and following related objects; in formulas, the character " / " indicates a "division" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0111] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. In the embodiments of this application, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A message processing method, characterized in that, Applied to a message processing device, the message processing device including at least a message queue unit and a consumption unit, the method includes: When a consumer in the consumption unit pulls messages from the corresponding message queue topic in the message queue unit, the message storage limit of each message queue topic and the message pull amount of each consumer in the consumption unit when pulling messages from the corresponding message queue topic are determined; the message queue topic stores messages to be transmitted sent by the upstream system, the message queue unit includes at least one message queue topic, each message queue topic corresponds to at least one consumer, and the consumption unit includes at least one consumer; the message storage limit of the message queue topic has an upper limit and a lower limit; Based on the message storage quota and the pull volume, dynamically configure the message storage quota for each message queue topic; Transmit the stored message to the corresponding consumer; The step of dynamically configuring the message storage quota for each message queue topic based on the message storage quota and the pull volume includes: For any message queue topic, if the amount of messages pulled by a consumer from that message queue topic exceeds the upper limit of the message storage quota of that message queue topic, a quota to be replenished is determined; the quota to be replenished is the difference between the upper limit of the message storage quota of that message queue topic and the current message storage quota of that message queue topic. Allocate a target amount from the message storage quota of the lowest priority message queue topic to that message queue topic; the target amount is less than or equal to the quota to be replenished; If the target quota is less than the quota to be supplemented, quota is solicited from the message queue topic with the next higher priority than the lowest priority; wherein, the message queue topic to be solicited is in an idle state. If the message pull volume is less than or equal to the lower limit of the message storage quota of the corresponding message queue topic, the message queue topic is in an idle state. The method further includes: The relationship between the message pull volume of each consumer when pulling messages from the corresponding message queue topic and the current message storage quota of the message queue topic is determined, where the message storage quota is between the upper limit and the lower limit. If the message pull volume is greater than or equal to the message storage quota, the status of the message queue topic is set to busy. If the message pull volume is greater than the lower limit but less than the message storage quota, the status of the message queue topic is set to semi-busy. If the message pull volume is less than or equal to the lower limit, the status of the message queue topic is set to idle. The message storage quota of each message queue topic is dynamically configured according to its status.

2. The method according to claim 1, characterized in that, The method further includes: Configure the priority of each message queue topic in the message queue unit. Different message queue topics have different priorities, and message storage quotas for message queue topics with different priorities are different.

3. The method according to claim 1, characterized in that, The step of dynamically configuring the message storage quota for each message queue topic based on the message storage quota and the pull volume includes: Determine whether there is a first target message queue topic or a second target message queue topic in each message queue topic, wherein the difference between the message storage quota of the first target message queue topic and the pull amount is greater than a first preset threshold, and the difference between the message storage quota of the second target message queue topic and the pull amount is less than a second preset threshold; When the first target message queue topic exists, increase the message storage quota of the first target message queue topic; If the second target message queue topic exists, reduce the message storage quota of the second target message queue topic.

4. The method according to claim 3, characterized in that, The reduction of the message storage limit for the second target message queue topic includes: Determine the priority of the second target message queue topic and the next message queue topic, wherein the priority of the next message queue topic is lower than the priority of the second target message queue topic; Allocate the target amount from the message storage quota of the second target message queue topic to the next message queue topic.

5. The method according to claim 1, characterized in that, The method further includes: In response to dynamic quota and flow control requests, obtain the ratio of the current message storage quota to the preset baseline quota for each message queue topic; Configure the flow control rate of each message queue topic according to the ratio and the preset total flow control rate.

6. A message processing device, characterized in that, include: The quota configuration module is used to determine the message storage quota of each message queue topic and the message pull amount of each consumer in the consumption unit when pulling messages from the corresponding message queue topic in the message queue unit. The message queue topic stores messages to be transmitted sent by the upstream system. The message queue unit includes at least one message queue topic, and each message queue topic corresponds to at least one consumer. The consumption unit includes at least one consumer. The message storage quota of each message queue topic has an upper limit and a lower limit. The message storage quota of each message queue topic is dynamically configured based on the message storage quota and the pull amount. The message transmission module is used to transmit the stored messages to be transmitted to the corresponding consumers; When dynamically configuring the message storage quota for each message queue topic based on the message storage quota and the pull volume, the quota configuration module is specifically used for: For any message queue topic, if the amount of messages pulled by a consumer from that message queue topic exceeds the upper limit of the message storage quota of that message queue topic, a quota to be replenished is determined; the quota to be replenished is the difference between the upper limit of the message storage quota of that message queue topic and the current message storage quota of that message queue topic. Allocate the target amount from the message storage quota of the lowest priority message queue topic to this message queue topic; The target amount is less than or equal to the amount to be supplemented; If the target quota is less than the quota to be supplemented, quota is solicited from the message queue topic with the next higher priority than the lowest priority; wherein, the message queue topic to be solicited is in an idle state. If the message pull volume is less than or equal to the lower limit of the message storage quota of the corresponding message queue topic, the message queue topic is in an idle state. The status configuration module is used to determine the relationship between the message pull volume of each consumer when pulling messages from the corresponding message queue topic and the current message storage quota of the message queue topic, wherein the message storage quota is between the upper limit and the lower limit. If the message pull volume is greater than or equal to the message storage quota, the status of the message queue topic is set to busy. If the message pull volume is greater than the lower limit but less than the message storage quota, the status of the message queue topic is set to semi-busy. If the message pull volume is less than or equal to the lower limit, the status of the message queue topic is set to idle. The message storage quota of each message queue topic is dynamically configured according to its status.

7. A message processing device, characterized in that, include: A message processing device and a dynamic flow control device, a dynamic quota control device, and a status refresh device, which are respectively connected to the message processing device, wherein the status refresh device is connected to the dynamic quota control device, and the dynamic quota control device is connected to the dynamic flow control device. The message processing device includes a message queue unit and a consumption unit, and the message queue unit is connected to the consumption unit. The message queue unit is used to create message queue topics of different priorities based on the messages to be transmitted sent by the upstream system. Each message queue topic is used to store messages of the corresponding type to be transmitted. The consumption unit includes a special consumer and a consumer corresponding to the message queue topic. Each consumer is used to pull messages from the corresponding message queue topic and send them to the special consumer. The special consumer is used to perform unified message sending. The dynamic quota controller is used to initialize quotas for each message queue topic, and when a consumer in the consumer unit pulls messages from the corresponding message queue topic in the message queue unit, it determines the message storage quota for each message queue topic and the message pull amount when each consumer in the consumer unit pulls messages from the corresponding message queue topic. The message queue topic stores messages to be transmitted sent by the upstream system. The message queue unit includes at least one message queue topic, and each message queue topic corresponds to at least one consumer. The consumer unit includes at least one consumer. The message storage quota of the message queue topic has an upper limit and a lower limit. The message storage quota for each message queue topic is dynamically configured based on the message storage quota and the pull amount. When the dynamic quota controller dynamically configures the message storage quota of each message queue topic based on the message storage quota and the pull volume, it is specifically used for: for any message queue topic, if the message pull volume from the consumer to the message queue topic is greater than the upper limit of the message storage quota of the message queue topic, determining the quota to be supplemented; the quota to be supplemented is the difference between the upper limit of the message storage quota of the message queue topic and the current message storage quota of the message queue topic; allocating a target quota from the message storage quota of the lowest priority message queue topic to the message queue topic; the target quota is less than or equal to the quota to be supplemented; if the target quota is less than the quota to be supplemented, then soliciting quota from the message queue topic of the next higher priority than the lowest priority; wherein, the solicited message queue topic is in an idle state, and if the message pull volume is less than or equal to the lower limit of the message storage quota of the corresponding message queue topic, then the state of the message queue topic is an idle state; The dynamic quota controller is further configured to: determine the relationship between the message pull volume when each consumer pulls messages from the corresponding message queue topic and the current message storage quota of the message queue topic, wherein the message storage quota is between the upper limit and the lower limit; if the message pull volume is greater than or equal to the message storage quota, then the status of the message queue topic is set to busy; if the message pull volume is greater than the lower limit but less than the message storage quota, then the status of the message queue topic is set to semi-busy; if the message pull volume is less than or equal to the lower limit, then the status of the message queue topic is set to idle; and dynamically configure the message storage quota of each message queue topic according to the status of each message queue topic. The dynamic flow control device is used to dynamically configure the processing rate of each message queue topic under a preset total flow control rate; The status refresh device is used to obtain the status information of each message queue topic from the message queue unit and send it to the dynamic quota control device.

Images