Kafka load balancing method, electronic equipment and storage medium

Abstract

The invention provides a Kafka load balancing method, electronic equipment and a storage medium. The method comprises the following steps: determining a manager from n consumers; the manager sends an average consumption partition number to each consumer, wherein the average consumption partition number is the number of partitions which each consumer should consume at present; and each consumer adjusts the actual consumption partition number according to the average consumption partition number, so that the actual consumption partition number is smaller than or equal to the average consumption partition number. The consumer with the minimum load is determined as a manager in the n consumers, so that collapse caused by the fact that consumers with high loads bear extra loads can be avoided, and load balance is guaranteed; besides, the manager only needs to send the average consumption partition number to the consumer, and the consumer adjusts the actual consumption partition number, so that the load of the manager is reduced, and the load of each consumer is balanced.

Description

Technical Field

[0001] This application relates to the technical field of message processing, and particularly to a Kafka load balancing method, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of modern Internet, the number of users of the vehicle networking information system has increased exponentially. The functions of the vehicle networking information system include, but are not limited to, receiving status information sent by the in-vehicle unit, receiving control instructions sent by users through the APP, and controlling the in-vehicle unit according to the control instructions. The huge user requests have put forward higher requirements for the message processing ability of the vehicle networking information system, and the application of distributed message systems is becoming more and more extensive. The distributed message system is a distributed architecture based on the message middleware mechanism. Currently, the core mode of message processing in the distributed message system is the publish-subscribe mode, and Kafka is a leader in the distributed message system based on the publish-subscribe mode.

[0003] In related technologies, when Kafka consumers consume messages, the administrator evenly distributes all partitions under the target topic to the consumers within the consumer group (i.e., achieving load balancing among consumers).

[0004] However, because the mechanism for selecting the administrator in related technologies is almost randomly selected, it may cause consumers with high load themselves to bear additional load and collapse; and each time load balancing is required, the administrator needs to re-evenly distribute all partitions to the consumers within the consumer group, which will cause additional system resource overhead.

[0005] It should be noted that the information disclosed in the background art part of this application is only intended to deepen the understanding of the general background art of this application, and should not be regarded as an admission or any form of suggestion that this information constitutes the prior art known to those skilled in the art. Summary of the Invention

[0006] In view of this, this application provides a Kafka load balancing method, an electronic device, and a storage medium, which are conducive to solving the problems in the prior art that because the mechanism for selecting the administrator in related technologies is almost randomly selected, it may cause consumers with high load themselves to bear additional load and collapse; and each time load balancing is required, the administrator needs to re-evenly distribute all partitions to the consumers within the consumer group, which will cause additional system resource overhead.

[0007] In a first aspect, an embodiment of this application provides a Kafka load balancing method, including: Determine an administrator from n consumers, where n is an integer greater than or equal to 1; The manager sends the average number of consumption partitions to each of the consumers, where the average number of consumption partitions is the number of partitions that each consumer should consume currently; Each of the consumers adjusts the actual number of consumption partitions according to the average number of consumption partitions, such that the actual number of consumption partitions is less than or equal to the average number of consumption partitions.

[0008] In an embodiment of the present application, determining the consumer with the lowest load among n consumers as the manager can prevent a consumer with a high load from taking on additional load and crashing, thus ensuring load balancing; in addition, the manager only needs to send the average number of consumption partitions to the consumers, and the consumers adjust the actual number of consumption partitions, reducing the load on the manager and thus balancing the load of each consumer.

[0009] In a possible implementation manner, the method further includes: Each consumer sends load information to the Redis database; Each of the consumers looks up the load information corresponding to other consumers in the Redis database; If the load information corresponding to any one of the consumers is the lowest, then any one of the consumers is the monitor, and the monitor is used to monitor other consumers.

[0010] In an embodiment of the present application, taking the consumer with the lowest load among n consumers as the monitor to monitor other consumers. It can be understood that taking the consumer with the lowest load as the monitor can prevent the load of a consumer from being too large, thus balancing the load.

[0011] In a possible implementation manner, the manager is the monitor.

[0012] In an embodiment of the present application, the manager is the monitor, which is the consumer with the lowest load in the entire consumer group. It can be understood that when the manager and the monitor are the same consumer, and when this consumer monitors that the actual number of consumption partitions of other consumers is abnormal, it directly calculates the average number of consumption partitions and thus adjusts the actual number of consumption partitions.

[0013] In a possible implementation manner, before the manager sends the average number of consumption partitions to each of the consumers, it further includes: The manager calculates the average number of consumption partitions according to the number of partitions and the number of consumers in the consumer group, where the consumer group is a cluster including all consumers.

[0014] In the embodiments of the present application, the administrator calculates the average number of consumption partitions according to the number of partitions and the number of consumers in the consumer group. It can be understood that the administrator only needs to calculate the quantity and does not need to adjust the load of each consumer, which can save system resources.

[0015] In a possible implementation manner, the administrator sending the average number of consumption partitions to each of the consumers includes: When the actual number of consumption partitions is greater than the average number of consumption partitions, or when the number of consumers in the consumer group changes, the administrator sends the average number of consumption partitions to each of the consumers.

[0016] In the embodiments of the present application, when the actual number of consumption partitions is greater than the average number of consumption partitions, that is, the consumer load surges. When the consumer load surges or the number of consumers changes, the administrator enables the load balancing function.

[0017] In a possible implementation manner, the method further includes: When the actual number of consumption partitions is less than or equal to the average number of consumption partitions, and the number of consumers in the consumer group remains unchanged, the administrator does not send the average number of consumption partitions to each of the consumers.

[0018] In the embodiments of the present application, when the consumer group remains stable, there is no need for the administrator to execute the load balancing function, thereby avoiding excessive load on the administrator.

[0019] In a possible implementation manner, the partitions consumed by each of the consumers are sorted in reverse order. Each of the consumers adjusts the actual number of consumption partitions according to the average number of consumption partitions, including: Each of the consumers adjusts the actual number of consumption partitions from the tail of the partition sequence according to the average number of consumption partitions.

[0020] In the embodiments of the present application, the partitions corresponding to each consumer are sorted in reverse order. When adjusting the actual number of consumption partitions, it starts from the tail of the partition sequence. It can be understood that since the programming order is last-in-first-out, sorting in reverse order and starting from the tail can ensure that the partitions that enter first also come out first, improving the liquidity of each partition.

[0021] In a possible implementation manner, the method further includes: Each of the consumers inserts a status message into the Redis database at a preset time interval, and the status message is used to indicate that the consumer is running normally.

[0022] In the embodiments of the present application, it is determined whether the consumer is operating normally based on the status information sent by the consumer. If the consumer does not send status information within a preset time period, it can be determined that the consumer has a fault. The fault condition of the consumer is discovered in a timely manner, so as to perform load balancing in a timely manner.

[0023] In a second aspect, an embodiment of the present application provides an electronic device, including: A processor; A memory; And a computer program, wherein the computer program is stored in the memory, and the computer program includes instructions that, when executed by the processor, cause the electronic device to execute the method described in any one of the first aspects.

[0024] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored program, wherein, when the program runs, it controls the device where the computer-readable storage medium is located to execute the method described in any one of the first aspects.

[0025] It can be understood that the electronic device provided in the second aspect and the computer-readable storage medium provided in the third aspect are used to execute the method provided by the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0027] Figure 1 It is a schematic diagram of a Kafka architecture in the related art; Figure 2 It is a schematic diagram of a Kafka architecture provided by an embodiment of the present application; Figure 3 It is a schematic flowchart of a Kafka load balancing method provided by an embodiment of the present application; Figure 4 It is a schematic diagram of a load balancing method provided by an embodiment of the present application; Figure 5 It is a schematic diagram of another load balancing method provided by an embodiment of the present application; Figure 6 It is a schematic diagram of another load balancing method provided by an embodiment of the present application; Figure 7A schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0028] To better understand the technical solution of the present application, the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0029] It should be clear that the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.

[0030] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "the" and "said" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0031] It should be understood that the term " / and / " used herein is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after.

[0032] With the rapid development of the modern Internet, the number of users of the vehicle networking information system has increased exponentially. The functions of the vehicle networking information system include but are not limited to receiving the status information sent by the vehicle head unit, receiving the control instructions sent by the user through the APP, and controlling the vehicle head unit according to the control instructions. The huge user requests have put forward higher requirements for the message processing ability of the vehicle networking information system, and the application of distributed message systems is also becoming more and more extensive. The distributed message system is a distributed architecture based on the message middleware mechanism. At present, the core mode of message processing in the distributed message system is the publish-subscribe mode, and Kafka is a leader in the distributed message system based on the publish-subscribe mode.

[0033] Kafka is a distributed publish-subscribe message system, with characteristics such as high throughput, persistence, replica set mechanism, and distributed support for horizontal expansion. The design of producers and consumers is completely decoupled, the traffic can be peak-shaved, and the message processing can be asynchronous. Now it has been used by many companies as various types of data pipelines and message systems.

[0034] To better illustrate and understand the technical solution of the present invention, the basic concepts of Kafka are introduced as follows: 1. Producer and Consumer For Kafka, there are two basic types of clients, including: producers and consumers. Producers (also known as publishers) are responsible for creating messages and then delivering them to Kafka, while consumers (also known as subscribers) are responsible for consuming messages.

[0035] 2. Topic and Partition In Kafka, messages are classified by topic. Each topic corresponds to a "message queue", which is similar to a table in a database. However, if all messages of the same type are stuffed into a "central" queue, scalability will surely be lacking. Whether it is an increase in the number of producers / consumers or an increase in the number of messages, it may exhaust the system's performance or storage. To solve this problem, the concept of partition is introduced in this solution to achieve horizontal expansion.

[0036] 3. Consumer Group and Broker A consumer group consists of multiple consumers. Each consumer within a consumer group is responsible for consuming data from different partitions. A partition can only be consumed by one consumer within the group, and consumer groups do not affect each other. A broker is a service node in the Kafka system. Usually, a server host in the Kafka system is called a broker of the Kafka system. The main role of a broker is to be responsible for storing message data, providing services to clients, and ensuring the normal operation of the Kafka system, etc. A broker is the smallest unit for forming a cluster in the Kafka system. A Kafka cluster consists of one broker or multiple brokers.

[0037] In the related art, when Kafka consumers consume messages, the administrator evenly distributes all partitions under the target topic to the consumers within the consumer group (i.e., to achieve load balancing among consumers).

[0038] For ease of understanding, an embodiment of this application provides a schematic diagram of a Kafka architecture in the related art.

[0039] See Figure 1 , which is a schematic diagram of a Kafka architecture in the related art. Figure 1 shows 3 producers (i.e., Producer A, Producer B, and Producer C), 4 brokers (i.e., Broker A, Broker B, Broker C, and Broker D), and 3 consumers (i.e., Consumer A, Consumer B, and Consumer C). Producers deliver messages to Kafka, consumers consume messages through Kafka, and brokers are used to process messages.

[0040] However, since the mechanism for selecting a manager in the related art is almost random selection, it may cause consumers with high load themselves to bear additional load and collapse; moreover, each time load balancing is performed, the manager needs to re - evenly distribute all partitions to the consumers within the consumer group, which will cause additional system resource overhead.

[0041] In view of the above problems, the embodiment of the present application provides a Kafka load balancing method. Determining the consumer with the smallest load among n consumers as the manager can avoid consumers with high load themselves from bearing additional load and collapsing, thus ensuring load balancing; in addition, the manager only needs to send the average number of consumed partitions to the consumers, and the consumers adjust the actual number of consumed partitions, reducing the load on the manager, thereby balancing the load of each consumer. It will be described in detail below in combination with the accompanying drawings and specific embodiments.

[0042] In the embodiment of the present application, on the basis of the original structure of the Kafka system, the connection interaction between Kafka and Redis is newly added. Since the Redis in - memory database uses the way of accessing memory, its speed is extremely fast and will not affect the system throughput rate; moreover, all Redis operations are atomic and will not be interrupted once started, which can ensure that when multiple consumers access Redis simultaneously, they can all obtain correct results; finally, Redis supports multiple data types and is suitable for multiple use cases such as caching and message queues. For the convenience of understanding, the embodiment of the present application provides a schematic diagram of the Kafka system structure.

[0043] See Figure 2 , which is a schematic diagram of the Kafka system structure provided by the embodiment of the present application. As Figure 2 shown, all information of the target topic and information of all consumers consuming this topic will be stored in the Redis database. By reading and writing this information, the manager of the system can adjust the partitions consumed by all consumers during load balancing, and the monitor of the system can also monitor the working status of all consumers.

[0044] See Figure 3 , which is a schematic flowchart of the Kafka load balancing method provided by the embodiment of the present application. As Figure 3 shown, it mainly includes the following steps.

[0045] Step S301: Determine a manager from n consumers.

[0046] Specifically, determine a manager from n consumers, where n is an integer greater than or equal to 1. In the embodiment of the present application, the n consumers are all consumers in the same consumer group, and each topic is a consumer group respectively.

[0047] In a possible implementation, the manager is the monitor, and the monitor is a consumer with the smallest load in a consumer group. Specifically, the method for determining the monitor is: each consumer sends load information to the Redis database; each consumer searches for the load information corresponding to other consumers in the Redis database; if the load information corresponding to any consumer is the smallest, then the consumer is the monitor, and the monitor is used to monitor other consumers.

[0048] It can be understood that each consumer traverses the load information corresponding to other consumers in the Redis database and compares its own load information with the load information of other consumers. If its own load information is the smallest, it will be regarded as the monitor. If its own load information is not the smallest, it will determine which consumer is the monitor. The monitor can monitor other consumers, and similarly, other consumers also need to monitor the monitor.

[0049] In an embodiment of the present application, the manager and the monitor are the same consumer, and the load of this consumer is the smallest. When this consumer monitors that there is an abnormality in the actual number of consumption partitions of other consumers, the average number of consumption partitions is directly calculated to adjust the actual number of consumption partitions.

[0050] In addition, in a possible implementation, the consumer is not a monitor. When the load of each consumer is high, if the same consumer is used as a monitor and a manager at the same time, the load of the consumer may be too high, and load imbalance may occur. The consumer with the smallest load and the consumer with the second smallest load are respectively used as managers and monitors. Exemplarily, when a new consumer joins the consumer group, the newly added consumer is used as a manager, but the monitor in the consumer group is still the original monitor. It can be understood that the load of the newly added consumer is 0, and the consumer is used as a manager to ensure that the load of other consumers is balanced and stable.

[0051] Step S302: The manager sends the average number of consumption partitions to each consumer.

[0052] Specifically, the manager calculates the average number of consumption partitions based on the number of partitions and the number of consumers in the consumer group, and sends the average number of consumption partitions to each consumer. The average number of consumption partitions is the number of partitions that each consumer should currently consume.

[0053] In a possible implementation, all partitions of the target topic are sorted and stored in the Redis data table. The number of partitions is denoted as p. All consumers in the consumer group corresponding to this topic are sorted in reverse order, and the number of consumers is denoted as c. The manager calculates the average number of consumed partitions through a = p / c, and a is the average number of consumed partitions. It can be understood that sorting the consumers in reverse order is to ensure that when adjusting the partitions of the consumers, the partitions adjusted first are the partitions that the consumers have consumed most recently.

[0054] In the embodiment of the present application, a is rounded up. And regardless of whether a is an integer, when the partitions are allocated for the last time, the remaining partitions are uniformly allocated to the last consumer. Exemplarily, when the number of partitions p is 15 and the number of consumers c is 4, the average number of consumed partitions a = 4. When 15 partitions are allocated to the last consumer, only 3 partitions are left, so the number of partitions allocated to the last consumer is 3. Of course, when the number of partitions p is 15 and the number of consumers c is 5, the average number of consumed partitions a = 3, and all the partitions can be evenly distributed among 3 consumers.

[0055] In a possible implementation, before the manager sends the average number of consumed partitions to the consumers, it will determine whether the current number of consumed partitions is less than or equal to the average number of consumed partitions. When the actual number of consumed partitions is greater than the average number of consumed partitions, or when the number of consumers in the consumer group changes, the manager will send the average number of consumed partitions to each consumer. It can be understood that when the actual number of consumed partitions is greater than the average number of consumed partitions, that is, the consumer load surges. When the consumer load surges or the number of consumers changes, the manager enables the load balancing function.

[0056] When the actual number of consumed partitions is less than or equal to the average number of consumed partitions, and the number of consumers in the consumer group remains unchanged, the manager does not send the average number of consumed partitions to each consumer. It can be understood that when the consumer group remains stable, there is no need for the manager to execute the load balancing function, thus avoiding excessive load on the manager.

[0057] Step S303: Each consumer adjusts the actual number of consumed partitions according to the average number of consumed partitions.

[0058] Specifically, after each consumer receives the average number of consumed partitions sent by the manager, it adjusts the actual number of consumed partitions according to the average number of consumed partitions, so that the actual number of consumed partitions is less than or equal to the average number of consumed partitions.

[0059] In the embodiment of the present application, since the partitions of the consumers are sorted in reverse order, when each consumer adjusts the actual number of consumed partitions, it starts to adjust the partitions from the end of the partition sequence. It can be understood that since the programming order is last-in-first-out, sorting in reverse order and starting to adjust from the end can ensure that the partitions that enter first also come out first, improving the liquidity of each partition.

[0060] For ease of understanding, the embodiments of the present application provide load balancing methods in multiple scenarios.

[0061] See Figure 4 , which is a schematic diagram of a load balancing method provided by the embodiments of the present application. As Figure 4 shown, taking 4 consumers as an example, the number of partitions consumed by consumer A is 3, the number of partitions consumed by consumer B is 3, the number of partitions consumed by consumer C is 3, and the number of partitions consumed by consumer D is 7. Obviously, the load of consumer D is too large. At this time, first determine whether consumer D is a monitor. If consumer D is not a monitor, the original monitor acts as a manager to initiate and control load balancing; if consumer D is a monitor, first elect a new monitor according to the latest information in the Redis database, and then the new monitor acts as a manager to determine whether load balancing is required.

[0062] When performing load balancing, the manager calculates the average number of consumed partitions corresponding to each consumer and sends the average number of consumed partitions to other consumers.

[0063] Assume that consumer D is not the manager. After consumer D receives the average number of consumed partitions, it removes the excess partitions from the tail (i.e., Figure 4 partitions d, h, and l in Figure 4 ). The manager will add these topic partitions to the idle consumed partitions of other consumers, so that other consumers (consumer A, consumer B, and consumer C) consume these partitions. As Figure 4 shown, partition d is added to the idle consumed partitions of consumer A, partition h is added to the idle consumed partitions of consumer B, and partition l is added to the idle consumed partitions of consumer C. After adjusting the consumed partitions of the 4 consumers, the number of partitions consumed by each consumer is 4 partitions, and there is no situation where a certain partition has too large a load, achieving load balancing.

[0064] In addition, the embodiments of the present application also provide a load balancing method in another scenario.

[0065] See Figure 5 , which is another schematic diagram of a load balancing method provided by the embodiments of the present application. As Figure 5 shown, a consumer group originally included 3 consumers, and then consumer D was newly added. Before adjusting the load, the number of partitions consumed by consumer A, consumer B, and consumer C was 4 partitions each. When consumer D joins, the monitor checks whether the actual number of consumed partitions of each consumer in the consumer group is less than or equal to the average number of consumed partitions after adding this consumer. If so, the newly added consumer (consumer D) enters the idle state; if not, the newly added consumer (consumer D) acts as a manager to initiate and control load balancing.

[0066] When performing load balancing, the manager calculates the average number of consumption partitions corresponding to each consumer and sends the average number of consumption partitions to other consumers.

[0067] After other consumers receive the average number of consumption partitions, they remove the excess partitions from the tail (i.e., Figure 4 partition d, partition h, and partition l in Figure 5 ). The manager will add these partitions to its own consumption partitions for consumption. As

[0068] shown, consumer A removes partition d, consumer B removes partition h, and consumer C removes partition l. After adjusting the consumption partitions of the 4 consumers, the number of partitions consumed by each consumer is 3 partitions, achieving load balancing.

[0069] In addition, the embodiment of the present application also provides a load balancing method in another case. Figure 6 Refer to Figure 6 , which is a schematic diagram of another load balancing method provided by the embodiment of the present application. As

[0070] shown, a consumer group originally includes 4 consumers, and then consumer D exits. Before adjusting the load, the number of partitions consumed by consumer A, consumer B, consumer C, and consumer D is 3 partitions respectively. When the number of consumers in the consumer group decreases, all consumers determine whether the exited consumer is a monitor. If the exited consumer is not a monitor, the monitor acts as the manager to initiate and control the load balancing; if the exited consumer is a monitor, a new monitor is first elected according to the latest information in the Redis database, and then the new monitor acts as the manager to determine whether load balancing is required.

[0071] When performing load balancing, the manager calculates the average number of consumption partitions corresponding to each consumer and sends the average number of consumption partitions to other consumers.

[0072] The manager distributes the partitions of the exited consumer to other consumers according to the calculated average number of consumption partitions. As Figure 6 shown, partition d of consumer D is assigned to consumer A, partition h of consumer D is assigned to consumer B, and partition l of consumer D is assigned to consumer C. After adjusting the consumption partitions of the 3 consumers, the number of partitions consumed by each consumer is 4 partitions, achieving load balancing.

[0073] As described above, when a consumer exits normally, it will log out its own information in the Redis database and send a logout message in the Redis database, so that the monitor and other consumers can timely detect the decrease in the number of consumers and determine whether the exiting consumer is the monitor.

[0074] However, if there is a monitoring failure of the consumer, that is, when the consumer cannot exit normally, it cannot actively log out its own information, creating an illusion that the consumer is still running, which makes it impossible for the consumer group to perform load balancing in a timely manner. Therefore, to avoid the problem of inability to perform load balancing in a timely manner caused by consumer failures, the embodiments of the present application monitor the running states of all consumers.

[0075] Specifically, during the running of the consumer, a status message is inserted into the Redis database at a preset time interval as a heartbeat, indicating that the consumer is running normally. By accessing the heartbeat information of each consumer in Redis, the monitor is responsible for monitoring the running states of all the other consumers, and the running state of the monitor is monitored by all the other consumers together. If the monitor finds that a certain consumer has not updated its heartbeat information within a preset time period, it is considered that the consumer has failed. At this time, the monitor, as the manager, logs out the information of the consumer and initiates load balancing. If it is found that the monitor has failed, a new monitor is immediately elected according to the information in the Redis database, and then the new monitor, as the manager, logs out the information of the original failed monitor and initiates load balancing.

[0076] In a possible implementation manner, the preset time interval is 10 minutes, and the preset time period is 2 preset time intervals, that is, 20 minutes. Because a consumer lacking a status message may be due to network problems, but if a consumer continuously lacks two status messages, it can be determined that the consumer has failed. Of course, the preset time interval and the preset time period in the embodiments of the present application are only exemplary descriptions and should not be regarded as limitations on the protection scope of the present application.

[0077] In summary, by selecting a manager among the consumers to uniformly coordinate and allocate the matching situation between consumers and partitions, the problem of relative independence of consumers is effectively solved; by using the Redis in-memory database, the monitors in the system can efficiently monitor all consumers, greatly improving the rate of system load balancing; by optimizing the strategy for electing the master among consumers, the situation of providing additional load for high-load consumers is avoided; by selecting a manager among the consumers to uniformly coordinate and allocate the matching situation between consumers and partitions, incorrect load balancing attempts can be avoided.

[0078] Corresponding to the above embodiments, the present application also provides an electronic device.

[0079] See Figure 7, which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 7 shown, the electronic device 700 may include: a processor 701, a memory 702, and a communication unit 703. These components communicate through one or more buses. Those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiments of the present application. It can be a bus structure, a star structure, and may also include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0080] Among them, the communication unit 703 is used to establish a communication channel so that the electronic device can communicate with other devices. Receive user data sent by other devices or send user data to other devices.

[0081] The processor 701 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and lines. By running or executing software programs, instructions, and / or modules stored in the memory 702, and by calling data stored in the memory, it executes various functions of the electronic device and / or processes data. The processor may be composed of an integrated circuit (IC). For example, it may be composed of a single packaged IC, or may be composed of multiple packaged ICs with the same or different functions connected together. For example, the processor 701 may only include a central processing unit (CPU). In the embodiment of the present application, the CPU may be a single arithmetic core or may include multiple arithmetic cores.

[0082] The memory 702 is used to store the execution instructions of the processor 701. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc.

[0083] When the execution instructions in the memory 702 are executed by the processor 701, the electronic device 700 is enabled to execute Figure 3 some or all of the steps in the shown embodiments.

[0084] In specific implementation, an embodiment of the present application further provides a computer storage medium. The computer storage medium may store a program, and when the program is executed, it may include some or all of the steps in the embodiments of the simulation scenario generation method provided by the embodiments of the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), or the like.

[0085] In specific implementation, an embodiment of the present application further provides a computer program product. The computer program product includes executable instructions, and when the executable instructions are executed on a computer, the computer is caused to execute some or all of the steps in the embodiments of the simulation scenario generation method provided by the embodiments of the present application.

[0086] In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent the cases of A existing alone, A and B existing simultaneously, and B existing alone. Where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one of the following" and its similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, and c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0087] Those of ordinary skill in the art can realize that the various units and algorithm steps described in the embodiments disclosed herein can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0088] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0089] In several embodiments provided by the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or a part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0090] For the same or similar parts among the various embodiments in this specification, reference can be made to each other. In particular, for the device embodiments and the terminal embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for the relevant parts, reference can be made to the descriptions in the method embodiments.

Claims

1. A Kafka load balancing method, characterized in that: include: Determine a manager from n consumers, where n is an integer greater than or equal to 1; The manager sends an average number of consumption partitions to each of the consumers, where the average number of consumption partitions is the number of partitions that each of the consumers should currently consume; Each of the consumers adjusts the actual number of consumption partitions according to the average number of consumption partitions, so that the actual number of consumption partitions is less than or equal to the average number of consumption partitions.

2. The method according to claim 1, characterized in that The method further comprises: Each consumer sends load information to the Redis database; Each of the consumers searches the Redis database for load information corresponding to other consumers; If the load information corresponding to any of the consumers is the smallest, then any of the consumers is a monitor, and the monitor is used to monitor other consumers.

3. The method according to claim 2, characterized in that The manager is the monitor.

4. The method according to claim 1, characterized in that: Before the manager sends the average number of consumption partitions to each of the consumers, the method further includes: The manager calculates the average number of consumer partitions based on the number of partitions and the number of consumers in the consumer group, where the consumer group is a cluster including all consumers.

5. The method according to claim 1, characterized in that The manager sends the average number of consumption partitions to each consumer, including: When the actual number of consumption partitions is greater than the average number of consumption partitions, or the number of consumers in the consumer group changes, the manager sends the average number of consumption partitions to each of the consumers, and the consumer group is a cluster including all consumers.

6. The method according to claim 5, characterized in that The method further comprises: When the actual number of consumption partitions is less than or equal to the average number of consumption partitions, and the number of consumers in the consumer group remains unchanged, the manager does not send the average number of consumption partitions to each of the consumers.

7. The method according to claim 1, characterized in that The partitions consumed by each consumer are sorted in descending order, and each consumer adjusts the actual number of consumption partitions according to the average number of consumption partitions, including: Each of the consumers adjusts the actual number of consumption partitions from the end of the partition sequence according to the average number of consumption partitions.

8. The method according to claim 1, characterized in that The method further comprises: Each of the consumers inserts a piece of status information into the Redis database at a preset time interval, and the status information is used to indicate that the consumer is running normally.

9. An electronic device, characterized in that: include: processor; Memory; A communication unit, used for establishing a communication channel; And a computer program, wherein the computer program is stored in the memory, and the computer program includes instructions, and when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 8.

Images