Message processing method and device, computer equipment, readable storage medium and program product
By calculating the target consumption rate of each consumer group, giving priority to ensuring full-speed consumption of high-priority consumers, the problem of inability to flexibly allocate resources in traditional technologies is solved, and efficient message processing and resource utilization is achieved.
Patent Information
- Application Number
- CN202411737086.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-14
AI Technical Summary
Traditional technologies cannot flexibly configure resources when high-priority messages are accumulated, resulting in the inability to effectively consume high-priority messages at full speed.
By obtaining multiple message queues with different priority and their current consumption rates, the target consumption rate of each consumer group is calculated to prioritize the full-speed consumption of high-priority consumers and keep the total consumption rate within the preset range.
When high-priority messages are backlogged, the target consumption rate of high-priority consumers is increased and the target consumption rate of low-priority consumers is reduced, thereby ensuring that high-priority messages can be consumed by consumers at full speed, improving resource utilization, and avoiding overload and platform overload.
Smart Images
Figure CN119629236A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a message processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art
[0002] With the increasing use of message queues in software development and the continuous increase in business scenarios, backend consumers are increasingly demanding message processing.
[0003] In traditional technology, multiple message queues are introduced to describe the priorities of consumers corresponding to different message queues, so as to send different consumption rates, i.e., message quantities, to consumers of different priorities. However, this solution cannot flexibly allocate resources when high-priority messages accumulate. Summary of the invention
[0004] Based on this, it is necessary to provide a message processing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can flexibly configure resources in response to the above technical problems.
[0005] In a first aspect, the present application provides a message processing method, comprising:
[0006] Obtain multiple message queues of different priorities in the platform, obtain the preset total consumption rate threshold and the current consumption rate of each consumer in each consumer group; one message queue corresponds to one consumer group, the priority of each consumer group corresponds to the priority of its corresponding message queue, and the priority of each consumer corresponds to the priority of its consumer group;
[0007] According to the total consumption rate threshold, the current consumption rate of each consumer and the priority corresponding to each consumer, the target consumption rate corresponding to each consumer is obtained with the goal of giving priority to full-speed consumption of consumers with high priority; the difference between the sum of the target consumption rates of multiple consumer groups and the total consumption rate threshold is within a preset range;
[0008] Each of the target consumption rates is sent to a corresponding consumer, so that each of the consumers pulls messages from a corresponding message queue according to the received target consumption rate.
[0009] In one embodiment, before sending each of the target consumption rates to the corresponding consumers respectively, the method further includes:
[0010] Obtaining a rate adjustment value according to a current consumption rate of the consumer and a corresponding target consumption rate;
[0011] The target consumption rate is modified according to a preset rate adjustment threshold corresponding to the consumer priority and the rate adjustment value.
[0012] In one embodiment, the priority of each of the consumers includes a highest priority, a lowest priority, and other priorities;
[0013] The method of obtaining the target consumption rate corresponding to each consumer according to the total consumption rate threshold, the current consumption rate of each consumer, and the priority corresponding to each consumer, with the goal of giving priority to full-speed consumption of consumers with high priority, includes:
[0014] When the consumer has the highest priority, the target consumption rate of the consumer with the highest priority is obtained according to the sub-consumption rate threshold and the minimum consumption rate corresponding to consumers with lower priority than the current consumer; the sub-consumption rate threshold is obtained based on the total consumption rate threshold, and the sub-consumption rate threshold is the threshold of the sum of the consumption rates of individual consumers of different priorities.
[0015] In one embodiment, the target consumption rate corresponding to each consumer is obtained based on the total consumption rate threshold, the current consumption rate of each consumer, and the priority corresponding to each consumer, with the goal of giving priority to full-speed consumption of consumers with high priority, including:
[0016] When the consumer is of other priority levels, the target consumption rate of the consumers of other priority levels is obtained according to the sub-consumption rate threshold, the current consumption rate of consumers with higher priority levels than the current consumer, and the minimum consumption rate corresponding to consumers with lower priority levels than the current consumer.
[0017] In one embodiment, the target consumption rate corresponding to each consumer is obtained based on the total consumption rate threshold, the current consumption rate of each consumer, and the priority corresponding to each consumer, with the goal of giving priority to full-speed consumption of consumers with high priority, including:
[0018] In the case where the consumer is of the lowest priority, the target consumption rate of the consumer of the lowest priority is obtained according to the sub-consumption rate threshold and the current consumption rate of consumers with a higher priority than the current consumer.
[0019] In one embodiment, the step of obtaining the sub-consumption rate threshold comprises:
[0020] A sub-consumption rate threshold is obtained according to the preset total consumption rate threshold and the number of consumers in each of the consumer groups.
[0021] In a second aspect, the present application also provides a message processing device, including:
[0022] A data acquisition module is used to acquire multiple message queues of different priorities in the platform, and to acquire a preset total consumption rate threshold and the current consumption rate of each consumer in each consumer group; one message queue corresponds to one consumer group, the priority of each consumer group corresponds to the priority of its corresponding message queue, and the priority of each consumer corresponds to the priority of its consumer group;
[0023] A rate calculation module, configured to obtain a target consumption rate corresponding to each consumer according to the total consumption rate threshold, the current consumption rate of each consumer and the priority corresponding to each consumer, with the goal of giving priority to full-speed consumption of consumers with high priority; the difference between the sum of the target consumption rates of multiple consumer groups and the total consumption rate threshold is within a preset range;
[0024] The rate sending module is used to send each of the target consumption rates to the corresponding consumers respectively, so that each of the consumers pulls messages from the corresponding message queue according to the received target consumption rate.
[0025] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.
[0026] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.
[0027] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.
[0028] The above-mentioned message processing method, device, computer equipment, computer-readable storage medium and computer program product, the platform obtains multiple message queues of different priorities in the platform, obtains a preset total consumption rate threshold and the current consumption rate of each consumer in each consumer group; one message queue corresponds to a consumer group, the priority of each consumer group corresponds to the priority of the corresponding message queue, and the priority of each consumer corresponds to the priority of the consumer group to which it belongs; then the platform obtains the target consumption rate corresponding to each consumer based on the total consumption rate threshold, the current consumption rate of each consumer and the priority corresponding to each consumer, with the goal of giving priority to full-speed consumption of consumers with high priority; the difference between the sum of the target consumption rates of multiple consumer groups and the total consumption rate threshold is within a preset range; finally, the platform sends each target consumption rate to the corresponding consumer respectively, so that each consumer pulls messages from the corresponding message queue according to the received target consumption rate. In the present application, the target consumption rate corresponding to each consumer is calculated by giving priority to ensuring full-speed consumption of high-priority consumers. In this way, when high-priority queue messages are accumulated, the platform can increase the target consumption rate of high-priority consumers and reduce the target consumption rate of low-priority consumers, thereby ensuring that high-priority important messages can be consumed at full speed by consumers and processed in a timely manner; at the same time, the target consumption rate obtained and allocated by the above calculation, after adjusting the consumer rate, needs to keep the total consumption rate of the entire platform within the preset range of the preset total consumption rate threshold, which can not only improve the resource utilization of the platform, but also avoid overload, and reduce the risk of platform overload or even collapse. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0030] Figure 1 A schematic diagram of a flow chart of a message processing method in an embodiment;
[0031] Figure 2 It is a flowchart of the step of correcting the target consumption rate in one embodiment;
[0032] Figure 3 is a structural block diagram of a message processing device in one embodiment;
[0033] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0035] In an exemplary embodiment, Figure 1 As shown, a message processing method is provided, which is applied to the platform and includes the following steps S202 to S206. Among them:
[0036] Step S202, obtaining a plurality of message queues of different priorities in the platform, obtaining a preset total consumption rate threshold and a current consumption rate of each consumer in each consumer group.
[0037] Among them, one message queue corresponds to one consumer group, the priority of each consumer group corresponds to the priority of its corresponding message queue, and the priority of each consumer corresponds to the priority of its consumer group.
[0038] Among them, the platform may refer to a system for receiving, storing, processing and distributing messages, which may be composed of multiple components, such as a message queue, consumers and producers; consumers may refer to various applications or services that can pull messages from the message queue; the consumption rate may refer to the rate at which consumers pull messages from the message queue; the current consumption rate may refer to the rate at which consumers pull messages from the message queue at the current moment.
[0039] The consumer group may refer to a group of consumers, wherein each consumer in the same consumer group consumes the same type of message queue together, for example, each consumer in the first priority consumer group consumes the first priority message queue together.
[0040] Exemplarily, multiple message queues can be defined in the platform, each message queue has a different priority (such as high, medium, and low), and then corresponding consumer groups can be created. The number of consumers in each group is consistent with the number of fixed partitions of the corresponding message queue; the platform obtains the current consumption rate of each consumer in each consumer group from each message queue, and obtains the preset total consumption rate threshold saved in the platform.
[0041] Step S204, according to the total consumption rate threshold, the current consumption rate of each consumer and the priority level of each consumer, the target consumption rate corresponding to each consumer is obtained with the goal of giving priority to full-speed consumption of consumers with high priority levels.
[0042] Among them, the difference between the sum of the target consumption rates of the multiple consumer groups and the total consumption rate threshold is within a preset range.
[0043] The total consumption rate threshold may refer to the maximum consumption rate allowed by the platform. Exceeding this rate may cause the platform to be overloaded and response delayed.
[0044] The target consumption rate may refer to the calculated ideal consumption rate that the corresponding consumer should achieve.
[0045] Exemplarily, the platform can calculate the target consumption rate for each consumer based on the total consumption rate threshold, the current consumption rate of each consumer, and the corresponding priority of each consumer, with the goal of ensuring full-speed consumption for consumers with high priority. For example, high-priority consumers can be given priority to ensure that they can consume at full speed. At the same time, it is also necessary to ensure that the difference between the sum of the target consumption rates of all consumers and the total consumption rate threshold is within a preset range to ensure that the platform does not suffer from serious overload and response delays.
[0046] Step S206: Send each target consumption rate to the corresponding consumer, so that each consumer pulls messages from the corresponding message queue according to the received target consumption rate.
[0047] Exemplarily, the platform sends the calculated target consumption rates to the corresponding consumers, and each consumer adjusts its message pulling rate according to the received target consumption rate, and then pulls messages from the corresponding message queue. In some examples, the platform can also regularly monitor the consumption of each consumer and dynamically adjust the target consumption rate according to the actual situation to cope with changes in message volume or fluctuations in consumer performance.
[0048] In the above message processing method, the target consumption rate corresponding to each consumer is calculated by giving priority to ensuring full-speed consumption of high-priority consumers. In this way, when high-priority queue messages are accumulated, the platform can increase the target consumption rate of high-priority consumers and reduce the target consumption rate of low-priority consumers, thereby ensuring that high-priority important messages can be consumed at full speed by consumers and processed in time; at the same time, according to the target consumption rate obtained and allocated by the above calculation, after adjusting the consumer rate, it is necessary to keep the total consumption rate of the entire platform within the preset range of the preset total consumption rate threshold, which can not only improve the resource utilization of the platform, but also avoid overload, and reduce the risk of platform overload or even collapse.
[0049] In an exemplary embodiment, Figure 2 As shown, before the above step S206 is executed, the following steps S302 to S304 may also be included. Among them:
[0050] Step S302, obtaining a rate adjustment value according to the current consumption rate of the consumer and the corresponding target consumption rate.
[0051] The rate adjustment value may refer to the difference between the current consumption rate and the target consumption rate, reflecting the amount that the consumer needs to adjust in order to achieve the target consumption rate.
[0052] Step S304: modify the target consumption rate according to the preset rate adjustment threshold and the rate adjustment value corresponding to the consumer priority.
[0053] The preset rate adjustment threshold may refer to a rate change limit preset for consumers of different priorities, which is used to ease the rate adjustment process. For example, a high-priority consumer may be set with a smaller preset rate adjustment threshold, that is, a high-priority consumer may be set with a smaller adjustment range; a low-priority consumer may be set with a larger preset rate adjustment threshold, that is, a low-priority consumer may be set with a larger adjustment range. For example, a high-priority preset rate adjustment threshold may be 0.5, 1, etc., and a low-priority preset rate adjustment threshold may be 1.5, 2, etc.; in some application scenarios, the preset rate adjustment thresholds of all priority consumers may be set to the same according to actual needs.
[0054] For example, if the preset rate adjustment threshold of consumer A is 1, the current consumption rate of consumer A is 5, and the target consumption rate is 10, then its rate adjustment value is 5, which exceeds the preset rate adjustment value 1. Therefore, the target consumption rate 10 is corrected according to the preset rate adjustment value 1 and the rate adjustment value 5, that is, the target consumption rate is corrected to 5+1=6. For another example, if the preset rate adjustment value is 1, the current consumption rate is 10, and the target consumption rate is 10, then there is no need to correct the target consumption rate; the above values are all examples and can be set according to actual conditions, and are not specifically limited to the embodiment herein.
[0055] In some examples, the preset rate adjustment threshold does not target scenarios where the consumer rate is reduced. For example, consumer B's current consumption rate is 12, the preset rate adjustment threshold is 1, but the target consumption rate is 10. The adjustment from the current consumption rate of 12 to the target consumption rate of 10 is a rate reduction process. Since the rate reduction is a release of platform resources and will not cause platform overload, when the rate adjustment value indicates that the consumer's rate is reduced, the target consumption rate can be corrected without referring to the preset rate adjustment threshold, that is, the final target consumption rate is still 10.
[0056] In this embodiment, the target consumption rate is corrected by presetting the rate adjustment threshold and the rate adjustment value, which can ensure that consumers have a higher smoothness when adjusting the consumption rate and reduce the possibility of platform overload due to excessive rate adjustment range.
[0057] In an exemplary embodiment, the priority of each consumer includes the highest priority, the lowest priority and other priorities; the above step S204 may specifically include: when the consumer is the highest priority, the target consumption rate of the consumer with the highest priority is obtained according to the sub-consumption rate threshold and the minimum consumption rate corresponding to the consumers with lower priority than the current consumer.
[0058] The sub-consumption rate threshold is obtained based on the total consumption rate threshold, and the sub-consumption rate threshold is the threshold of the sum of consumption rates of individual consumers of different priorities.
[0059] Among them, the priority of each consumer includes the highest priority, the lowest priority and other priorities. The descriptions of "highest" and "lowest" for the priority are relative among the multiple set priorities; the highest priority is applicable to important tasks or urgent messages; the lowest priority is applicable to processing non-urgent or minor messages; other priorities can be consumers between the highest priority and the lowest priority, which can be flexibly set according to system requirements. For example, the highest priority can be level one, the lowest priority can be level five, and other priorities can be level two, level three, and level four.
[0060] The minimum consumption rate may be a minimum rate pre-set for consumers of different priorities, which is used to ensure that the consumers are in an active or online state.
[0061] Exemplarily, the platform obtains a sub-consumption rate threshold based on a preset total consumption rate threshold, where the sub-consumption rate threshold is a threshold of the sum of consumption rates of individual consumers of different priorities. For example, the priority of the message queue has five levels, and the priority of the consumer group also corresponds to five levels. Based on the preset total consumption rate threshold, a threshold of the sum of consumption rates of individual consumers corresponding to the five priorities is calculated, that is, a threshold of the sum of consumption rates of a first-level consumer, a second-level consumer, a third-level consumer, a fourth-level consumer, and a fifth-level consumer, wherein the consumption rates of each consumer in a consumer group are consistent; when the consumer is of the highest priority, the platform can obtain the target consumption rate of the highest priority by subtracting the sum of the minimum consumption rates corresponding to consumers with a lower priority than the current consumer from the sub-consumption rate threshold; for example, the highest priority consumer is a first-level consumer, and the consumers below the first-level consumer are second to fifth-level consumers, then the target consumption rate of the first-level consumer can be calculated as: sub-consumption rate threshold - (minimum consumption rate of the second-level consumer + minimum consumption rate of the third-level consumer + minimum consumption rate of the fourth-level consumer + minimum consumption rate of the fifth-level consumer).
[0062] In this embodiment, when the consumer has the highest priority, the target consumption rate of the consumer with the highest priority is obtained based on the sub-consumption rate threshold and the minimum consumption rate corresponding to consumers with lower priority than the current consumer. The target consumption rate obtained by the above calculation method can prioritize the full-speed consumption of high-priority consumers.
[0063] In an exemplary embodiment, the above step S204 may also include: when the consumer is of other priority levels, obtaining the target consumption rate of consumers of other priority levels based on the sum of the sub-consumption rate threshold, the current consumption rate of consumers with higher priority levels than the current consumer, and the minimum consumption rate corresponding to consumers with lower priority levels than the current consumer,
[0064] Exemplarily, there are five levels corresponding to the priorities of different consumers, with the highest priority being level one, the lowest priority being level five, and the other priorities being level two, level three, and level four; taking level three consumers as an example, the calculation process of their target consumption rate can be: sub-consumption rate threshold - (current consumption rate of level one consumers + current consumption rate of level two consumers) - (minimum consumption rate of level four consumers + minimum consumption rate of level five consumers).
[0065] In this embodiment, when the consumer is of other priority levels, the target consumption rate of consumers of other priority levels is obtained based on the sum of the sub-consumption rate threshold, the current consumption rate of consumers with higher priority levels than the current consumer, and the minimum consumption rate corresponding to consumers with lower priority levels than the current consumer. Through the above calculation method, it can be obtained that the target consumption rate of consumers of other priority levels is limited by the current consumption rate of consumers with higher priority levels. It can be seen that the target consumption rate of consumers of other priority levels is calculated on the premise of giving priority to satisfying high-priority consumers.
[0066] In an exemplary embodiment, the above step S204 may further include: when the consumer is of the lowest priority, obtaining the target consumption rate of the consumer of the lowest priority according to the sub-consumption rate threshold and the current consumption rate of consumers with a higher priority than the current consumer.
[0067] Exemplarily, there are five levels corresponding to the priorities of different consumers, with the highest priority being level one, the lowest priority being level five, and the other priorities being level two, level three, and level four; taking level five consumers as the lowest priority consumers as an example, the calculation process of their target consumption rate can be: sub-consumption rate threshold - (current consumption rate of level one consumers + current consumption rate of level two consumers + current consumption rate of level four consumers + current consumption rate of level five consumers).
[0068] In this embodiment, when the consumer is of the lowest priority, the target consumption rate of the lowest priority consumer is obtained based on the sub-consumption rate threshold and the current consumption rate of consumers with higher priority than the current consumer. The target consumption rate of the lowest priority consumer obtained by the above calculation method takes into account the current consumption rates of consumers of all priorities higher than its priority, so that the platform's resources can be allocated to high-priority consumers, thereby improving resource utilization efficiency.
[0069] In an exemplary embodiment, the step of obtaining the sub-consumption rate threshold in the above embodiment includes: obtaining the sub-consumption rate threshold according to a preset total consumption rate threshold and the number of consumers in each consumer group.
[0070] Exemplarily, there are three levels of priority corresponding to consumer groups, high priority, medium priority and low priority. There are A consumers in the high priority consumer group, B consumers in the medium priority consumer group, and C consumers in the low priority consumer group. The number A, the number B and the number C may be the same or different. The consumption rate of each consumer in a consumer group should be consistent. According to the preset total consumption rate threshold and the number of high priority consumers A, the number of medium priority consumers B, and the number of low priority consumers C, a consumption rate threshold corresponding to one high priority consumer, one medium priority consumer and one low priority consumer as a whole can be obtained as a sub-consumption rate threshold, where the number of priorities can be set according to actual conditions. The three levels of priority described here are only examples.
[0071] In an exemplary embodiment, a message processing method is provided, which may specifically include:
[0072] Step S1, obtain multiple message queues of different priorities in the platform, obtain the preset total consumption rate threshold and the current consumption rate of each consumer in each consumer group; one message queue corresponds to one consumer group, the priority of each consumer group corresponds to the priority of the corresponding message queue, and the priority of each consumer corresponds to the priority of the consumer group to which it belongs.
[0073] Step S2, according to the total consumption rate threshold, the current consumption rate of each consumer and the priority corresponding to each consumer, with the goal of giving priority to full-speed consumption of consumers with high priority, obtain the target consumption rate corresponding to each consumer; the difference between the sum of the target consumption rates of multiple consumer groups and the total consumption rate threshold is within a preset range; wherein the priority of each consumer includes the highest priority, the lowest priority and other priorities; in the case of the highest priority consumer, the target consumption rate of the highest priority consumer is obtained according to the sub-consumption rate threshold and the minimum consumption rate corresponding to the consumer lower than the current consumer priority; the sub-consumption rate threshold is obtained based on the total consumption rate threshold, and the sub-consumption rate threshold is the threshold of the sum of the consumption rates of single consumers of different priorities. In the case of consumers of other priorities, the target consumption rate of consumers of other priorities is obtained according to the sub-consumption rate threshold, the current consumption rate of consumers higher than the current consumer priority and the minimum consumption rate corresponding to consumers lower than the current consumer priority. In the case of the lowest priority consumer, the target consumption rate of the lowest priority consumer is obtained according to the sub-consumption rate threshold and the current consumption rate of consumers higher than the current consumer priority.
[0074] During the application process, the target consumption rates of consumers of different priorities can be calculated in order of priority from high to low, referring to the above calculation process. When the target consumption rate of each priority consumer is obtained, the target consumption rates of other consumers of the same priority are also known, and then the target consumption rates of all priority consumers can be waited for to be obtained before being sent to the corresponding consumers respectively; in another example, while the target consumption rates of consumers of different priorities are calculated in order of priority from high to low, referring to the above calculation process, the target consumption rates are immediately sent to the corresponding consumers, so that when calculating the target consumption rates of subsequent lower priority consumers, the current consumption rate of the high priority consumer after being adjusted according to the target consumption rate is used as a reference, so that dynamic resource adjustment can be achieved.
[0075] The step of obtaining the sub-consumption rate threshold includes: obtaining the sub-consumption rate threshold according to a preset total consumption rate threshold and the number of consumers in each consumer group.
[0076] Step S3, obtaining a rate adjustment value according to the current consumption rate of the consumer and the corresponding target consumption rate; and modifying the target consumption rate according to a preset rate adjustment threshold and the rate adjustment value corresponding to the consumer priority.
[0077] Step S4: Send each target consumption rate to the corresponding consumer, so that each consumer pulls messages from the corresponding message queue according to the received target consumption rate.
[0078] In an application example, the message processing method can be applied to the Kafka platform, and the message queues of different priorities can be obtained from the Kafka message queue architecture. The steps of using the method may include: defining five message queues, each corresponding to a group of consumers, representing different priorities; defining a coordinator, the function of which is to ensure that the total consumption capacity of the five consumers is approximately maintained at a constant rate, so that the backend support resources are not wasted as much as possible, but not overloaded. The coordinator records the current consumption rate of each node of each consumer group (how many records are polled at a time), and calculates the rate at which each node should consume next (the change of this consumption rate is guaranteed to be slow and gentle), and then sends it to each node, and each node uses the new rate for the next consumption. The coordinator ensures that the sent rate fully considers the priority of each consumer group, and gives priority to allocating the consumption rate to the high-priority consumer group. If the high-priority group can no longer consume at full speed, the consumption rate is gradually increased for the low-priority group, but the total consumption rate must not exceed the predetermined value (that is, the preset total consumption rate threshold); the consumers in the consumer groups of each priority level pull up messages according to the received consumption capacity signals, and immediately call the backend resources in parallel for business processing.
[0079] Among them, the method to ensure the total consumption rate of the five consumers is: each priority message queue has a fixed number of partitions, corresponding to a consumer group, and the number of consumers in each priority consumer group is consistent with the number of partitions. The total pull capacity of the k-th consumer (k is the number of partitions) of all priority consumer groups is sum_p. Calculate the k-th consumer of all priority consumer groups in turn (k ranges from 0 to the maximum partition number --- max_num), and calculate the maximum pull quantity of the current consumer sum_p-SUM (the actual amount obtained by the k-th consumer with a higher priority than the current one, that is, the current consumption rate)-the number of k-th consumers with a lower priority than the current one * the minimum single consumption of each consumer (that is, the minimum consumption rate). The next consumer rate adjustment shall not exceed this number. Since the preset rate adjustment threshold is set, the consumer rate adjustment is slow, and the overall rate sum generally does not exceed sum_p. Even if it exceeds, it will be adjusted back quickly, and it will not exceed too much, ensuring that the total load of the system does not exceed the standard. The above example uses multiple message queues to describe the priority of messages, and introduces a coordinator to issue the consumption rate to the consumer group of each message queue, and the consumer group consumes according to the issued rate. A coordinator is introduced to control the consumption rate of messages according to priority, so that the platform messages can be prioritized while ensuring business throughput and not wasting resources, and the system will not be overloaded. In addition, Kafka does not natively support the definition of message priority, and cannot give priority to important messages. This solution can support the definition of message priority without affecting system throughput as much as possible, so that high-priority messages can be consumed first.
[0080] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0081] Based on the same inventive concept, the embodiment of the present application also provides a message processing device for implementing the message processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more message processing device embodiments provided below can refer to the limitations on the message processing method above, and will not be repeated here.
[0082] In an exemplary embodiment, Figure 3 As shown, a message processing device 900 is provided, including: a data acquisition module 901, a rate calculation module 902 and a rate sending module 903, wherein:
[0083] The data acquisition module 901 is used to acquire multiple message queues of different priorities in the platform, and acquire the preset total consumption rate threshold and the current consumption rate of each consumer in each consumer group; one message queue corresponds to one consumer group, the priority of each consumer group corresponds to the priority of the corresponding message queue, and the priority of each consumer corresponds to the priority of the consumer group to which it belongs;
[0084] The rate calculation module 902 is used to obtain the target consumption rate corresponding to each consumer according to the total consumption rate threshold, the current consumption rate of each consumer and the priority corresponding to each consumer, with the goal of giving priority to full-speed consumption of consumers with high priority; the difference between the sum of the target consumption rates of multiple consumer groups and the total consumption rate threshold is within a preset range;
[0085] The rate sending module 903 is used to send each target consumption rate to the corresponding consumer, so that each consumer pulls messages from the corresponding message queue according to the received target consumption rate.
[0086] In an exemplary embodiment, the message processing device also includes a rate correction module for obtaining a rate adjustment value based on the consumer's current consumption rate and the corresponding target consumption rate; and correcting the target consumption rate based on a preset rate adjustment threshold and a rate adjustment value corresponding to the consumer priority.
[0087] In an exemplary embodiment, the priority of each consumer includes the highest priority, the lowest priority and other priorities; the above-mentioned rate calculation module 902 is also used to obtain the target consumption rate of the consumer with the highest priority according to the sub-consumption rate threshold and the minimum consumption rate corresponding to consumers with lower priority than the current consumer when the consumer has the highest priority; the sub-consumption rate threshold is obtained based on the total consumption rate threshold, and the sub-consumption rate threshold is the threshold of the sum of the consumption rates of individual consumers of different priorities.
[0088] In an exemplary embodiment, the above-mentioned rate calculation module 902 is also used to obtain the target consumption rate of consumers of other priorities based on the sub-consumption rate threshold, the current consumption rate of consumers with higher priority than the current consumer, and the minimum consumption rate corresponding to consumers with lower priority than the current consumer when the consumer is of other priorities.
[0089] In an exemplary embodiment, the rate calculation module 902 is also used to obtain the target consumption rate of the lowest priority consumer based on the sub-consumption rate threshold and the current consumption rate of consumers with higher priority than the current consumer when the consumer is the lowest priority.
[0090] In an exemplary embodiment, the rate calculation module 902 is further configured to obtain a sub-consumption rate threshold according to a preset total consumption rate threshold and the number of consumers in each consumer group.
[0091] Each module in the above message processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.
[0092] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a message processing method is implemented.
[0093] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0094] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0095] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0096] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0097] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0098] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0099] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A message processing method, characterized in that: Applied to the platform, the method comprises: Obtain multiple message queues of different priorities in the platform, obtain the preset total consumption rate threshold and the current consumption rate of each consumer in each consumer group; one message queue corresponds to one consumer group, the priority of each consumer group corresponds to the priority of its corresponding message queue, and the priority of each consumer corresponds to the priority of its consumer group; According to the total consumption rate threshold, the current consumption rate of each consumer and the priority corresponding to each consumer, the target consumption rate corresponding to each consumer is obtained with the goal of giving priority to full-speed consumption of consumers with high priority; the difference between the sum of the target consumption rates of multiple consumer groups and the total consumption rate threshold is within a preset range; Each of the target consumption rates is sent to a corresponding consumer, so that each of the consumers pulls messages from a corresponding message queue according to the received target consumption rate.
2. The method according to claim 1, characterized in that Before sending each of the target consumption rates to the corresponding consumers respectively, the method further includes: Obtaining a rate adjustment value according to a current consumption rate of the consumer and a corresponding target consumption rate; The target consumption rate is modified according to a preset rate adjustment threshold corresponding to the consumer priority and the rate adjustment value.
3. The method according to claim 1, characterized in that The priority of each of the consumers includes a highest priority, a lowest priority, and other priorities; The method of obtaining the target consumption rate corresponding to each consumer according to the total consumption rate threshold, the current consumption rate of each consumer, and the priority corresponding to each consumer, with the goal of giving priority to full-speed consumption of consumers with high priority, includes: When the consumer has the highest priority, the target consumption rate of the consumer with the highest priority is obtained according to the sub-consumption rate threshold and the minimum consumption rate corresponding to consumers with lower priority than the current consumer; the sub-consumption rate threshold is obtained based on the total consumption rate threshold, and the sub-consumption rate threshold is the threshold of the sum of the consumption rates of individual consumers of different priorities.
4. The method according to claim 3, characterized in that The method of obtaining the target consumption rate corresponding to each consumer according to the total consumption rate threshold, the current consumption rate of each consumer, and the priority corresponding to each consumer, with the goal of giving priority to full-speed consumption of consumers with high priority, includes: When the consumer is of other priority levels, the target consumption rate of the consumers of other priority levels is obtained according to the sub-consumption rate threshold, the current consumption rate of consumers with higher priority levels than the current consumer, and the minimum consumption rate corresponding to consumers with lower priority levels than the current consumer.
5. The method according to claim 3, characterized in that: The method of obtaining the target consumption rate corresponding to each consumer according to the total consumption rate threshold, the current consumption rate of each consumer, and the priority corresponding to each consumer, with the goal of giving priority to full-speed consumption of consumers with high priority, includes: In the case where the consumer is of the lowest priority, the target consumption rate of the consumer of the lowest priority is obtained according to the sub-consumption rate threshold and the current consumption rate of consumers with a higher priority than the current consumer.
6. The method according to claim 3, characterized in that The step of obtaining the sub-consumption rate threshold comprises: A sub-consumption rate threshold is obtained according to the preset total consumption rate threshold and the number of consumers in each of the consumer groups.
7. A message processing device, characterized in that: The device comprises: A data acquisition module is used to acquire multiple message queues of different priorities in the platform, and to acquire a preset total consumption rate threshold and the current consumption rate of each consumer in each consumer group; one message queue corresponds to one consumer group, the priority of each consumer group corresponds to the priority of its corresponding message queue, and the priority of each consumer corresponds to the priority of its consumer group; A rate calculation module, configured to obtain a target consumption rate corresponding to each consumer according to the total consumption rate threshold, the current consumption rate of each consumer and the priority corresponding to each consumer, with the goal of giving priority to full-speed consumption of consumers with high priority; the difference between the sum of the target consumption rates of multiple consumer groups and the total consumption rate threshold is within a preset range; The rate sending module is used to send each of the target consumption rates to the corresponding consumers respectively, so that each of the consumers pulls messages from the corresponding message queue according to the received target consumption rate.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.