Consumer load balancing method and device, computer equipment and readable storage medium
In the consumer-side load balancing method, dynamic allocation is performed based on the gap in the number of unconsumed messages between the partition and the consumer, the problem of load imbalance in the prior art is solved, and a more effective load balancing effect is achieved.
Patent Information
- Application Number
- CN202411693145.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-02-18
AI Technical Summary
The existing consumer-side load balancing method can only achieve balance at the partition level, and cannot effectively deal with the situation where the number of messages that have not been consumed between different partitions is too large, resulting in load imbalance between consumers.
By determining the partition set composed of partitions that need to be allocated when the load adjustment conditions are met, selecting the target partition from the partition set, and according to the number of messages that have not been consumed in the allocated partition of each consumer, from the consumer Select the target consumer from the set and assign the target partition to the target consumer until the end condition is met.
Load balancing is achieved in the number of unconsumed messages, and by prioritizing the allocation to consumers with smaller loads, the gap in the number of messages between partitions is minimized, thereby improving the effect of load balancing.
Smart Images
Figure CN119473624A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of distributed systems, and in particular to a consumer-side load balancing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art
[0002] Message middleware is a subsystem in a distributed system, responsible for sending and receiving data. Message middleware uses an efficient and reliable asynchronous information transmission mechanism to integrate data from various subsystems in a distributed system, such as asynchronous processing, application decoupling, traffic peak shaving, log processing and other scenarios. Currently, user behavior data and behavior logs can be transmitted and aggregated through the open source stream processing platform Kafka. The message sources (feeds of messages) of Kafka processing resources can be classified into different categories, and each classified topic can be divided into multiple partitions, and each partition is an ordered queue.
[0003] In order to improve performance, Kafka performs load balancing on the consumer side, but the existing solution can only ensure that consumers are balanced at the partition level. When the number of unconsumed messages between different partitions is too different, it may lead to load imbalance between consumers. Summary of the invention
[0004] Based on this, it is necessary to provide a consumer-side load balancing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve load balancing in response to the above technical problems.
[0005] In a first aspect, the present application provides a consumer-side load balancing method, comprising:
[0006] Under the condition that the load adjustment condition is met, determining a partition set consisting of partitions that need to be allocated;
[0007] Select the target partition from the partition set;
[0008] Select the target consumer from the consumer set based on the number of unconsumed messages in the partition assigned to each consumer;
[0009] Assign the target partition to the target consumer, delete the target partition from the partition set, return to the step of selecting the target partition from the partition set and continue executing until the end condition is met.
[0010] In one of the embodiments, the load adjustment condition includes a change in the number of consumers in the set of consumers.
[0011] In one embodiment, selecting a target partition from a set of partitions includes:
[0012] Select the target partition from the partition set based on the number of unconsumed messages in each partition in the partition set.
[0013] In one embodiment, selecting a target partition from a partition set according to the number of unconsumed messages in each partition in the partition set includes:
[0014] The partition with the largest number of unconsumed messages in the partition set is determined as the target partition.
[0015] In one embodiment, selecting a target consumer from a consumer set according to the number of unconsumed messages in a partition assigned to each consumer includes:
[0016] The consumer with the least number of unconsumed messages in the assigned partition in the consumer set is determined as the target consumer.
[0017] In one of the embodiments, the termination condition includes the partition set being empty.
[0018] In a second aspect, the present application also provides a consumer-side load balancing device, including:
[0019] A determination module, used for determining a partition set consisting of partitions to be allocated when a load adjustment condition is met;
[0020] A selection module, used to select a target partition from a partition set;
[0021] The selection module is also used to select a target consumer from the consumer set according to the number of unconsumed messages in the partition assigned to each consumer;
[0022] The allocation module is used to allocate the target partition to the target consumer, delete the target partition from the partition set, return to the step of selecting the target partition from the partition set and continue to execute until the end condition is met.
[0023] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0024] Under the condition that the load adjustment condition is met, determining a partition set consisting of partitions that need to be allocated;
[0025] Select the target partition from the partition set;
[0026] Select the target consumer from the consumer set based on the number of unconsumed messages in the partition assigned to each consumer;
[0027] Assign the target partition to the target consumer, delete the target partition from the partition set, return to the step of selecting the target partition from the partition set and continue executing until the end condition is met.
[0028] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:
[0029] Under the condition that the load adjustment condition is met, determining a partition set consisting of partitions that need to be allocated;
[0030] Select the target partition from the partition set;
[0031] Select the target consumer from the consumer set based on the number of unconsumed messages in the partition assigned to each consumer;
[0032] Assign the target partition to the target consumer, delete the target partition from the partition set, return to the step of selecting the target partition from the partition set and continue executing until the end condition is met.
[0033] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:
[0034] Under the condition that the load adjustment condition is met, determining a partition set consisting of partitions that need to be allocated;
[0035] Select the target partition from the partition set;
[0036] Select the target consumer from the consumer set based on the number of unconsumed messages in the partition assigned to each consumer;
[0037] Assign the target partition to the target consumer, delete the target partition from the partition set, return to the step of selecting the target partition from the partition set and continue executing until the end condition is met.
[0038] The above-mentioned consumer-side load balancing method, device, computer equipment, computer-readable storage medium and computer program product determine a partition set consisting of partitions that need to be allocated when the load adjustment condition is met. Select a target partition from the partition set, and select a target consumer from the consumer set according to the number of unconsumed messages in the partition allocated to each consumer. Assign the target partition to the target consumer, delete the target partition from the partition set, return to the step of selecting the target partition from the partition set and continue to execute until the end condition is met. Since the target consumer can be selected from the consumer set according to the number of unconsumed messages in the partition allocated to each consumer, instead of selecting consumers alternately, it is possible to select consumers with fewer unconsumed messages in the allocated partitions as much as possible, that is, to give priority to allocating partitions to consumers with smaller loads, so that a balance can be achieved in the number of unconsumed messages to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0040] FIG1 is a schematic diagram of a process of alternately allocating partitions to consumers in one embodiment;
[0041] FIG2 is a schematic diagram showing the result of alternately allocating partitions to consumers in one embodiment;
[0042] FIG3 is an application environment diagram of a consumer-side load balancing method in one embodiment;
[0043] FIG4 is a schematic diagram of a process flow of a consumer-side load balancing method in one embodiment;
[0044] FIG5 is a schematic diagram of a flow chart of a consumer-side load balancing method in another embodiment;
[0045] FIG6 is a schematic diagram of a process of allocating partitions to consumers in one embodiment;
[0046] FIG7 is a schematic diagram showing the result of allocating partitions to consumers in one embodiment;
[0047] FIG8 is a structural block diagram of a load balancing device at a consumer end in one embodiment;
[0048] FIG. 9 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0050] At present, when load balancing is performed on the consumer side, the main purpose is to ensure that consumers are balanced at the partition level. For the specific solution, please refer to Figure 1. Poll different partitions and assign each partition to consumers in sequence. For example, there are two consumers in Figure 1, namely Consumer 1 and Consumer 2; and the partitions are Partitions 1 to Partition 6. Assign Partition 1 to Consumer 1, Partition 2 to Consumer 2, Partition 3 to Consumer 1, Partition 4 to Consumer 2, Partition 5 to Consumer 1, and Partition 6 to Consumer 2. By alternating the allocation in this way, load balancing at the partition level can be achieved, that is, the number of partitions mounted by each consumer is balanced. For the specific allocation results, please refer to Figure 2.
[0051] However, the number of unconsumed messages in different partitions may be different. Some partitions may have more unconsumed messages, while others may have less. When the difference in the number of unconsumed messages between different partitions is too large, it may cause load imbalance among consumers.
[0052] In response to the above-mentioned problems, a consumer-side load balancing method is provided in an embodiment of the present application, which can be applied to the application environment shown in Figure 3. Among them, the consumer end 302 communicates with the server 304 through the network. The data storage system can store the data that the server 304 needs to process. The data storage system can be integrated on the server 304, or it can be placed on the cloud or other network servers. The server 304 can determine a partition set consisting of partitions that need to be allocated when the load adjustment conditions are met. Select the target partition from the partition set. Select the target consumer from the consumer set based on the number of unconsumed messages in the allocated partition of each consumer. Assign the target partition to the target consumer, that is, assign it to the consumer end 302. Delete the target partition from the partition set, return to the step of selecting the target partition from the partition set and continue to execute until the end condition is met.
[0053] The consumer end 302 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality devices, smart glasses, etc. The server 304 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.
[0054] In an exemplary embodiment, as shown in FIG4 , a consumer-side load balancing method is provided, which is described by taking the method applied to the server in FIG3 as an example, and includes the following steps 402 to 408. Among them:
[0055] Step 402: When the load adjustment condition is met, determine a partition set consisting of partitions that need to be allocated.
[0056] Among them, the partition may refer to the topic partition, and the partition to be allocated refers to the partition that has not yet been allocated to the consumer. The load balancing condition refers to the condition under which the load needs to be redistributed. For example, it may include a change in the number of partitions, or an increase in the number of unconsumed messages in certain partitions and the number exceeds a preset threshold, etc., which is not specifically limited in the embodiments of the present application.
[0057] Step 404: Select a target partition from the partition set.
[0058] Among them, the target partition can be selected randomly, or the partition with the largest number of unconsumed messages can be selected as the target partition. The embodiment of the present application does not make any specific limitations on this.
[0059] Step 406 , select a target consumer from the consumer set according to the number of unconsumed messages in the partition assigned to each consumer.
[0060] It is understandable that for a certain consumer, the number of unconsumed messages in its allocated partition can represent whether the load of the consumer is too large, too small, or moderate. Therefore, when selecting target consumers, consumers with fewer unconsumed messages in their allocated partitions can be selected as much as possible, that is, consumers with a relatively small load. Prioritizing the allocation of partitions to such consumers can achieve a balance in the number of unconsumed messages to a certain extent.
[0061] Step 408, assign the target partition to the target consumer, delete the target partition from the partition set, return to the step of selecting the target partition from the partition set and continue to execute until the end condition is met.
[0062] Assign the target partition to the target consumer. At this time, the number of partitions to be assigned in the partition set will be reduced by one, which means that the target partition needs to be deleted from the partition set. At this time, there may still be unassigned partitions in the partition set.
[0063] In the above-mentioned consumer-side load balancing method, a partition set consisting of partitions to be allocated is determined while satisfying the load adjustment conditions. A target partition is selected from the partition set, and a target consumer is selected from the consumer set according to the number of unconsumed messages in the partitions allocated to each consumer. The target partition is assigned to the target consumer, the target partition is deleted from the partition set, and the step of selecting the target partition from the partition set is returned and continued until the end condition is met. Since the target consumer can be selected from the consumer set according to the number of unconsumed messages in the partitions allocated to each consumer, instead of selecting consumers alternately, it is possible to select consumers with fewer unconsumed messages in the allocated partitions as much as possible, that is, partitions are allocated preferentially to consumers with smaller loads, so that a balance can be achieved in the number of unconsumed messages to a certain extent.
[0064] In an exemplary embodiment, the load adjustment condition includes a change in the number of consumers in the set of consumers.
[0065] Among them, the change in the number of consumers means that more or fewer consumers can be used to process the messages in the partition. Under this condition, reallocating the partitions is conducive to achieving load balancing.
[0066] In this embodiment, since the partition redistribution can be initiated when the number of consumers in the consumer set changes, it is beneficial to achieve load balancing.
[0067] In an exemplary embodiment, selecting a target partition from a set of partitions includes:
[0068] Select the target partition from the partition set based on the number of unconsumed messages in each partition in the partition set.
[0069] It is understandable that the number of unconsumed messages in different partitions may vary. If a partition with a large number of unconsumed messages is assigned to a certain consumer, the load of the consumer will be increased to a greater extent; while if a partition with a small number of unconsumed messages is assigned to a certain consumer, the load of the consumer will be increased to a lesser extent. Therefore, according to the number of unconsumed messages in each partition in the partition set, the target partition is selected from the partition set, and the partition with a large number of unconsumed messages can be selected as much as possible, and preferentially assigned to consumers with a small number of unconsumed messages in the assigned partitions, that is, consumers with a smaller load, so as to achieve load balancing on the number of unconsumed messages.
[0070] In this embodiment, since the target partition is selected from the partition set according to the number of unconsumed messages in each partition in the partition set, the partition with a large number of unconsumed messages can be selected as much as possible, and preferentially allocated to consumers with a small number of unconsumed messages in the allocated partitions, that is, consumers with a smaller load, which is conducive to achieving load balancing on the number of unconsumed messages.
[0071] In an exemplary embodiment, selecting a target partition from the partition set according to the number of unconsumed messages in each partition in the partition set includes:
[0072] The partition with the largest number of unconsumed messages in the partition set is determined as the target partition.
[0073] In this embodiment, the partition with the largest number of unconsumed messages in the partition set can be determined as the target partition. That is, the partition with the largest number of unconsumed messages can be selected and preferentially allocated to consumers with smaller loads, so that consumers with smaller loads can preferentially bear more loads, which is conducive to achieving load balancing on the number of unconsumed messages.
[0074] In an exemplary embodiment, selecting a target consumer from a consumer set according to the number of unconsumed messages in the partition assigned to each consumer includes:
[0075] The consumer with the least number of unconsumed messages in the assigned partition in the consumer set is determined as the target consumer.
[0076] In this embodiment, the consumer with the least number of unconsumed messages in the allocated partitions in the consumer set can be determined as the target consumer. That is, the partition with a large number of unconsumed messages can be preferentially allocated to the consumer with the smallest load, so that the consumer with the smallest load can bear more load first, which is conducive to achieving load balancing on the number of unconsumed messages.
[0077] In an exemplary embodiment, the termination condition includes the partition set being empty.
[0078] Among them, the partition set is empty, which means that all partitions have been allocated. Of course, in actual implementation, there can be other termination conditions, such as the load formed by the allocated partitions of each consumer has exceeded the corresponding load limit. At this time, the allocation can also be terminated.
[0079] In this embodiment, since a cyclic process can be used, each time a partition is allocated, the partition with the largest number of unconsumed messages is selected and preferentially allocated to the consumer with the smallest load until all partitions are allocated. Thus, the entire allocation process is conducive to achieving load balancing in the number of unconsumed messages.
[0080] For ease of understanding, the complete method flow provided by the embodiment of the present application is now described. Referring to FIG5 , the specific implementation can be shown in the following process steps:
[0081] Step 502: when the number of consumers in the consumer set changes, determine a partition set consisting of partitions that need to be allocated.
[0082] Step 504: determine the partition with the largest number of unconsumed messages in the partition set as the target partition.
[0083] Step 506: The consumer with the least number of unconsumed messages in the allocated partition in the consumer set is determined as the target consumer.
[0084] Step 508, assigning the target partition to the target consumer, deleting the target partition from the partition set, returning the partition with the largest number of unconsumed messages in the partition set, determining it as the target partition and continuing to execute until the partition set is empty.
[0085] The above allocation process can be referred to Figures 6 and 7. As can be seen in Figure 6, each time when allocating partitions, the partition with the largest number of unconsumed messages is selected and allocated to the consumer with the smallest load first; Figure 7 shows the corresponding allocation result.
[0086] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0087] Based on the same inventive concept, the embodiment of the present application also provides a consumer-end load balancing device for implementing the above-mentioned consumer-end load balancing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more consumer-end load balancing device embodiments provided below can refer to the limitations of the consumer-end load balancing method above, and will not be repeated here.
[0088] In an exemplary embodiment, as shown in FIG8 , a consumer-side load balancing device is provided, including: a determination module 802 , a selection module 804 , and an allocation module 806 , wherein:
[0089] A determination module 802, configured to determine a partition set consisting of partitions to be allocated when a load adjustment condition is met;
[0090] A selection module 804 is used to select a target partition from the partition set;
[0091] The selection module 804 is further used to select a target consumer from the consumer set according to the number of unconsumed messages in the partition assigned to each consumer;
[0092] The allocation module 806 is used to allocate the target partition to the target consumer, delete the target partition from the partition set, return to the step of selecting the target partition from the partition set and continue to execute until the end condition is met.
[0093] In an exemplary embodiment, the load adjustment condition includes a change in the number of consumers in the set of consumers.
[0094] In an exemplary embodiment, the selection module 804 is used to select a target partition from the partition set according to the number of unconsumed messages in each partition in the partition set.
[0095] In an exemplary embodiment, the selection module 804 is used to determine the partition with the largest number of unconsumed messages in the partition set as the target partition.
[0096] In an exemplary embodiment, the selection module 804 is used to determine the consumer with the least number of unconsumed messages in the allocated partition in the consumer set as the target consumer.
[0097] In an exemplary embodiment, the termination condition includes the partition set being empty.
[0098] Each module in the above-mentioned consumer-side load balancing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0099] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG9. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store messages. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a consumer-side load balancing method is implemented.
[0100] Those skilled in the art will understand that the structure shown in FIG. 9 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0101] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0102] Under the condition that the load adjustment condition is met, determining a partition set consisting of partitions that need to be allocated;
[0103] Select the target partition from the partition set;
[0104] Select the target consumer from the consumer set based on the number of unconsumed messages in the partition assigned to each consumer;
[0105] Assign the target partition to the target consumer, delete the target partition from the partition set, return to the step of selecting the target partition from the partition set and continue executing until the end condition is met.
[0106] In one embodiment, when the processor executes the computer program, the load adjustment condition includes a change in the number of consumers in the consumer set.
[0107] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0108] Select the target partition from the partition set based on the number of unconsumed messages in each partition in the partition set.
[0109] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0110] The partition with the largest number of unconsumed messages in the partition set is determined as the target partition.
[0111] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0112] The consumer with the least number of unconsumed messages in the allocated partition in the consumer set is determined as the target consumer.
[0113] In one embodiment, when the processor executes the computer program, the termination condition includes the partition set being empty.
[0114] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0115] Under the condition that the load adjustment condition is met, determining a partition set consisting of partitions that need to be allocated;
[0116] Select the target partition from the partition set;
[0117] Select the target consumer from the consumer set based on the number of unconsumed messages in the partition assigned to each consumer;
[0118] Assign the target partition to the target consumer, delete the target partition from the partition set, return to the step of selecting the target partition from the partition set and continue executing until the end condition is met.
[0119] In one embodiment, when the computer program is executed by a processor, the load adjustment condition includes a change in the number of consumers in the consumer set.
[0120] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0121] Select the target partition from the partition set based on the number of unconsumed messages in each partition in the partition set.
[0122] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0123] The partition with the largest number of unconsumed messages in the partition set is determined as the target partition.
[0124] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0125] The consumer with the least number of unconsumed messages in the assigned partition in the consumer set is determined as the target consumer.
[0126] In one embodiment, when the computer program is executed by a processor, the termination condition includes the partition set being empty.
[0127] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0128] Under the condition that the load adjustment condition is met, determining a partition set consisting of partitions that need to be allocated;
[0129] Select the target partition from the partition set;
[0130] Select the target consumer from the consumer set based on the number of unconsumed messages in the partition assigned to each consumer;
[0131] Assign the target partition to the target consumer, delete the target partition from the partition set, return to the step of selecting the target partition from the partition set and continue executing until the end condition is met.
[0132] In one embodiment, when the computer program is executed by a processor, the load adjustment condition includes a change in the number of consumers in the consumer set.
[0133] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0134] Select the target partition from the partition set based on the number of unconsumed messages in each partition in the partition set.
[0135] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0136] The partition with the largest number of unconsumed messages in the partition set is determined as the target partition.
[0137] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0138] The consumer with the least number of unconsumed messages in the assigned partition in the consumer set is determined as the target consumer.
[0139] In one embodiment, when the computer program is executed by a processor, the termination condition includes the partition set being empty.
[0140] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0141] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0142] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A consumer-side load balancing method, characterized in that: The method comprises: Under the condition that the load adjustment condition is met, determining a partition set consisting of partitions that need to be allocated; Selecting a target partition from the set of partitions; Select the target consumer from the consumer set based on the number of unconsumed messages in the partition assigned to each consumer; The target partition is assigned to the target consumer, the target partition is deleted from the partition set, and the step of selecting the target partition from the partition set is returned to and continued to be executed until an end condition is met.
2. The method according to claim 1, characterized in that The load adjustment condition includes a change in the number of consumers in the consumer set.
3. The method according to claim 1, characterized in that The selecting a target partition from the partition set includes: A target partition is selected from the partition set according to the number of unconsumed messages in each partition in the partition set.
4. The method according to claim 3, characterized in that The selecting a target partition from the partition set according to the number of unconsumed messages in each partition in the partition set includes: The partition with the largest number of unconsumed messages in the partition set is determined as the target partition.
5. The method according to claim 1, characterized in that The step of selecting a target consumer from a consumer set according to the number of unconsumed messages in the partition assigned to each consumer includes: The consumer with the least number of unconsumed messages in the assigned partition in the consumer set is determined as the target consumer.
6. The method according to claim 1, characterized in that The termination condition includes that the partition set is empty.
7. A consumer-end load balancing device, characterized in that: The device comprises: A determination module, used for determining a partition set consisting of partitions to be allocated when a load adjustment condition is met; A selection module, used for selecting a target partition from the partition set; The selection module is further used to select a target consumer from the consumer set according to the number of unconsumed messages in the partition assigned to each consumer; The allocation module is used to allocate the target partition to the target consumer, delete the target partition from the partition set, return to the step of selecting the target partition from the partition set and continue to execute until the end condition is met.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.