Message partition adjustment method and electronic equipment

Abstract

The invention discloses a message partition adjustment method and electronic equipment. The method comprises the steps of monitoring and obtaining related data in the process that each consumer side pulls a message from each message partition corresponding to a target subscription theme in a kafka server; the related data comprises the number of requests arrived by the message partition per second and the number of requests which can be processed by the message partition per second; determining performance information corresponding to each message partition based on the related data; the performance information at least comprises average response time; determining consumption capability information corresponding to each consumption end based on the performance information corresponding to each message partition; based on the consumption capability information, determining message partition quantity information corresponding to each consumption end; and based on the information of the number of the message partitions, adjusting the message partitions corresponding to the consumption ends.

Description

Technical Field

[0001] This application relates to the field of communication technologies, and in particular, to a method for adjusting message partitions and an electronic device. Background Art

[0002] Kafka is a high-throughput distributed publish-subscribe message system. Currently, when Kafka allocates message partitions to the consumer side, it usually allocates consumer sides according to the strategy that the consumer sides allocated to each message partition should be as uniform as possible. However, due to differences in the consumption capabilities of consumer sides, the consumption speed of messages in each message partition will be different, and the strategy of evenly allocating consumer sides will result in low message consumption efficiency. Summary of the Invention

[0003] In view of this, embodiments of this application provide a method for adjusting message partitions and an electronic device.

[0004] According to the first aspect of this application, embodiments of this application provide a method for adjusting message partitions, including:

[0005] Monitoring and obtaining relevant data during the process of each consumer side pulling messages from each message partition corresponding to the target subscription topic in the kafka server; the relevant data includes the number of requests arriving at the message partition per second and the total number of requests that the message partition can process per second;

[0006] Determining performance information corresponding to each message partition based on the relevant data; the performance information includes at least the average response time;

[0007] Determining consumption capacity information corresponding to each consumer side based on the performance information corresponding to each message partition;

[0008] Determining the number of message partitions corresponding to each consumer side based on the consumption capacity information;

[0009] Adjusting the message partitions corresponding to each consumer side based on the number of message partitions information.

[0010] Optionally, determining performance information corresponding to each message partition based on the relevant data includes:

[0011] Calculating the average response time corresponding to each message partition through the average response time calculation formula based on the number of requests arriving at the message partition per second and the total number of requests that the message partition can process per second.

[0012] Optionally, the performance information further includes the system load corresponding to the message partition;

[0013] Determining performance information corresponding to each message partition based on the relevant data includes:

[0014] Based on the number of requests arriving per second for each message partition and the total number of requests that each message partition can process per second, calculate the system utilization rate corresponding to each message partition through the system utilization formula, and obtain the system load corresponding to each message partition.

[0015] Optionally, the performance information further includes the minimum number of servers required to maintain system stability corresponding to each message partition;

[0016] Determine the performance information corresponding to each message partition based on relevant data, including:

[0017] Based on the number of requests arriving per second for each message partition and the total number of requests that each message partition can process per second, calculate the minimum number of servers required to maintain system stability corresponding to each message partition through the resource requirement formula.

[0018] Optionally, based on the performance information corresponding to each message partition, determine the consumption capacity information corresponding to each consumer end, including:

[0019] Determine each consumer end corresponding to each message partition, and use the performance information corresponding to each message partition as the performance information of each consumer end corresponding to each message partition;

[0020] Based on the performance information and the corresponding relationship between the performance information and the consumption capacity information, determine the consumption capacity information corresponding to each consumer end.

[0021] Optionally, the relevant data further includes the central processing unit performance information of the consumer end itself;

[0022] Determine the performance information corresponding to each message partition based on relevant data, including:

[0023] Determine each consumer end corresponding to each message partition;

[0024] Based on each consumer end corresponding to each message partition and the central processing unit performance information of each consumer end itself, determine the performance information corresponding to each message partition.

[0025] Optionally, obtain the central processing unit performance information of each consumer end itself, including:

[0026] Obtain the request task complexity information and the remaining memory information corresponding to each consumer end;

[0027] Based on the request task complexity information and the remaining memory information corresponding to each consumer end, and the corresponding relationship between the request task complexity information, the remaining memory information, and the central processing unit performance information, determine the central processing unit performance information of each consumer end itself.

[0028] Optionally, based on the consumption capacity information, determine the number of message partitions information corresponding to each consumer end, including:

[0029] Normalize the consumption ability information corresponding to each consumer end to obtain the normalized consumption ability information;

[0030] Based on the normalized consumption ability information, determine the percentage of the number of message partitions that can be allocated to each consumer end.

[0031] Optionally, the method for adjusting message partitions further includes:

[0032] Obtain the consumption timing data of each consumer end;

[0033] Detect whether each consumer end is abnormal based on the consumption timing data;

[0034] If an abnormal consumer end is detected, adjust the message partitions of the abnormal consumer end.

[0035] According to the second aspect of the present application, an embodiment of the present application provides an electronic device, including:

[0036] At least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for adjusting message partitions as in the first aspect or any implementation manner of the first aspect.

[0037] The above description is only an overview of the technical solution of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present application more obvious and understandable, the specific implementation manners of the present application are specifically exemplified below. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic flowchart of a method for adjusting message partitions in an embodiment of the present application;

[0039] Figure 2 It is a schematic flowchart of another method for adjusting message partitions in an embodiment of the present application;

[0040] Figure 3 It is a schematic flowchart of another method for adjusting message partitions in an embodiment of the present application;

[0041] Figure 4 It is a schematic flowchart of another method for adjusting message partitions in an embodiment of the present application;

[0042] Figure 5 It is a schematic flowchart of another method for adjusting message partitions in an embodiment of the present application;

[0043] Figure 6 It is a schematic hardware structure diagram of an electronic device in an embodiment of the present application. Specific Embodiments

[0044] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0045] The embodiments of the present application provide a method for adjusting message partitioning, as Figure 1 shown, including:

[0046] S101, monitoring and obtaining relevant data during the process of each consumer pulling messages from each message partition corresponding to the target subscription topic in the kafka server; the relevant data includes the number of requests arriving at the message partition per second and the total number of requests that the message partition can process per second.

[0047] In this embodiment, Kafka is a distributed, partitioned, multi-replica, multi-subscriber distributed logging system coordinated by zookeeper, and is commonly used for web / nginx logs, access logs, message services, etc. The kafka server may refer to a kafka cluster, and the kafka cluster includes one or more servers.

[0048] In this embodiment, when the kafka server saves messages, they are classified according to the topic. A Topic can be regarded as a type of message, and each Topic will be divided into multiple message partitions.

[0049] In this embodiment, a subscriber who subscribes to messages from the kafka server becomes a consumer, also known as a consumer end. A consumer end can subscribe to messages of multiple topics, and messages of one topic can also be consumed by multiple consumer ends. When a consumer end wants to subscribe to messages under the target subscription topic, it can send the parameters of the consumer end to the kafka server, and the kafka server sets the allocation strategy of the message partitions of the consumer end under the target subscription topic based on the parameters of the consumer end. A consumer end can be configured to correspond to at least one message partition under the target subscription topic, and then the consumer end can pull messages from the at least one message partition of the target subscription topic.

[0050] In this embodiment, the adjustment of the message partitions is applied to the scenario where multiple consumer ends simultaneously consume messages in multiple message partitions under the same topic.

[0051] In this embodiment, each message partition corresponding to the target subscription topic can be allocated to each consumer based on general technologies, and then each message partition corresponding to the target subscription topic is regarded as a queuing system, so as to monitor and obtain the number of requests λ arriving at each message partition per second, each consumer corresponding to each message partition, the number of servers s corresponding to each message partition, the number of requests μ that each server can process per second, etc., so as to obtain the relevant data in the process of each consumer pulling messages from each message partition corresponding to the target subscription topic in the kafka server. Among them, a request refers to a request initiated when a consumer pulls a message from a message partition; the number of requests arriving at a message partition per second refers to the total number of requests received by the message partition per second. The total number of requests that a message partition can process per second can be calculated through the number of servers s corresponding to the message partition and the number of requests μ that the server can process per second, that is, the total number of requests that a message partition can process per second = sμ.

[0052] S102. Determine the performance information corresponding to each message partition based on the relevant data; the performance information includes at least the average response time.

[0053] In this embodiment, after obtaining the number of requests arriving at each message partition per second and the total number of requests that each message partition can process per second, performance information such as the average response time corresponding to each message partition can be calculated. For example, the average waiting time corresponding to each message partition is determined based on queuing theory, and the average response time corresponding to each message partition is obtained.

[0054] S103. Determine the consumption capacity information corresponding to each consumer based on the performance information corresponding to each message partition.

[0055] In this embodiment, since each message partition corresponds to multiple consumers, the performance information of each message partition represents the consumption capacity information of each consumer pulling messages in the message partition. Therefore, based on the performance information of each message partition, the consumption capacity information corresponding to each consumer can be determined.

[0056] In this embodiment, the consumption capacity information may refer to the score of the consumption capacity, the level of the consumption capacity, etc.

[0057] S104. Determine the message partition quantity information corresponding to each consumer based on the consumption capacity information. The message partition quantity information may be the specific number of message partitions or the proportion of the number of message partitions.

[0058] In this embodiment, after determining the consumption capacity information of each consumer, the number of message partitions corresponding to each consumer can be re-determined. For example, the stronger the consumption capacity, the more message partitions can be corresponding; the weaker the consumption capacity, the fewer message partitions can be corresponding. Thus, the consumption speed of each message partition under the target subscription topic can be increased, and the accumulation of messages in the message partitions can be reduced.

[0059] S105. Adjust the message partitions corresponding to each consumer based on the message partition number information.

[0060] In this embodiment, after determining the message partition number information, all the message partitions corresponding to the target subscription topic can be re-allocated based on the message partition number information to adjust the message partitions corresponding to each consumer.

[0061] In this embodiment, the execution subject of the message partition adjustment method can be a kafka server or an external model / system. If the execution subject is an external model / system, after determining the message partitions corresponding to each adjusted consumer, the message partitions corresponding to each adjusted consumer can be directly sent to the kafka server. Then the kafka server can notify the consumer, or the consumer can determine the adjusted message partitions through automatic subscription.

[0062] The message partition adjustment method provided in the embodiment of the present application monitors and obtains relevant data during the process of each consumer pulling messages from each message partition corresponding to the target subscription topic in the kafka server; the relevant data includes the number of requests arriving at the message partition per second and the total number of requests that the message partition can process per second; determines the performance information corresponding to each message partition based on the relevant data; the performance information at least includes the average response time; determines the consumption capacity information corresponding to each consumer based on the performance information corresponding to each message partition; determines the message partition number information corresponding to each consumer based on the consumption capacity information; adjusts the message partitions corresponding to each consumer based on the message partition number information; in this way, the message partitions corresponding to each consumer are adjusted according to the consumption capacity of each consumer, so that the consumer with strong consumption capacity corresponds to more message partitions, and the consumer with weak consumption capacity corresponds to fewer message partitions, so that the messages under the target subscription topic can be consumed faster and the efficiency of message consumption can be improved.

[0063] In an optional embodiment, as Figure 2 shown, step S102, determining the performance information corresponding to each message partition based on the relevant data, includes:

[0064] S1021. Calculate the average response time corresponding to each message partition through the average response time calculation formula based on the number of requests arriving per second in the message partition and the total number of requests that the message partition can process per second.

[0065] In this embodiment, the average response time calculation formula is as follows:

[0066] E[T] = 1 / (sμ - λ); where E[T] is the average response time, sμ is the total number of requests that the message partition can process per second, and λ is the number of requests arriving per second in the message partition.

[0067] In this embodiment, through the average response time calculation formula, the average response time corresponding to each message partition can be quickly calculated, thereby determining the performance information corresponding to each message partition.

[0068] In an alternative embodiment, the performance information further includes the system load corresponding to the message partition.

[0069] As Figure 3 shown, step S102 of determining the performance information corresponding to each message partition based on relevant data includes:

[0070] S1022. Calculate the system utilization rate corresponding to each message partition through the system utilization rate formula based on the number of requests arriving per second in the message partition and the total number of requests that the message partition can process per second, and obtain the system load corresponding to each message partition.

[0071] In this embodiment, based on the number of requests arriving per second in the message partition and the total number of requests that the message partition can process per second, the average response time corresponding to each message partition can be calculated through the average response time calculation formula. At the same time, based on the number of requests arriving per second in the message partition and the total number of requests that the message partition can process per second, the system utilization rate corresponding to each message partition is calculated through the system utilization rate formula, and the system load corresponding to each message partition is obtained.

[0072] In this embodiment, the system utilization rate formula is as follows:

[0073] ρ = λ / sμ; where ρ is the system utilization rate, that is, the system load; λ refers to the number of requests arriving per second; and sμ is the total number of requests that the message partition can process per second.

[0074] In this embodiment, through the system utilization rate formula, the system load corresponding to each message partition can be quickly calculated.

[0075] It should be noted that in this embodiment, only the example of first calculating the average response time corresponding to each message partition and then calculating the system load corresponding to each message partition is used for illustration, but it does not limit the calculation order of the average response time and the system load. The average response time and the system load can be calculated simultaneously, or the system load can be calculated first and then the average response time.

[0076] In an alternative embodiment, the performance information further includes the minimum number of servers for maintaining system stability corresponding to the message partition.

[0077] As Figure 4 shown, step S102, determining the performance information corresponding to each message partition based on relevant data, includes:

[0078] S1023, calculating the minimum number of servers for maintaining system stability corresponding to each message partition through the resource requirement formula based on the number of requests arriving at the message partition per second and the total number of requests that the message partition can process per second.

[0079] In this embodiment, the average response time corresponding to each message partition can be calculated through the average response time calculation formula based on the number of requests arriving at the message partition per second and the total number of requests that the message partition can process per second. At the same time, the system utilization rate corresponding to each message partition can be calculated through the system utilization rate formula based on the number of requests arriving at the message partition per second and the total number of requests that the message partition can process per second, and the system load corresponding to each message partition can be obtained. At the same time, the minimum number of servers for maintaining system stability corresponding to each message partition can be calculated through the resource requirement formula based on the number of requests arriving at the message partition per second and the total number of requests that the message partition can process per second.

[0080] In this embodiment, the resource requirement formula is as follows:

[0081] R = λ / μ, where R is the resource requirement, that is, the minimum number of servers for maintaining system stability, λ refers to the number of requests arriving per second; μ is the number of requests that each server of the message partition can process per second, μ = the total number of requests that the message partition can process per second / s, and s is the number of servers corresponding to the message partition.

[0082] In this embodiment, through the resource requirement formula, the minimum number of servers for maintaining system stability corresponding to each message partition can be quickly calculated.

[0083] It should be noted that in this embodiment, only an example is given where the average response time corresponding to each message partition is calculated first, then the system load corresponding to each message partition is calculated, and finally the minimum number of servers to maintain system stability is calculated. However, it does not limit the calculation sequence of the average response time, system load, and the minimum number of servers to maintain system stability. The average response time, system load, and the minimum number of servers to maintain system stability can be calculated simultaneously, or the minimum number of servers to maintain system stability and system load can be calculated first, and then the average response time can be calculated.

[0084] In an alternative embodiment, in step S103, based on the performance information corresponding to each message partition, the consumption capacity information corresponding to each consumer is determined, as Figure 5 shown, including:

[0085] S1031, determine each consumer corresponding to each message partition, and use the performance information corresponding to each message partition as the performance information of each consumer corresponding to each message partition.

[0086] S1032, based on the performance information and the corresponding relationship between the performance information and the consumption capacity information, determine the consumption capacity information corresponding to each consumer.

[0087] In this embodiment, for step S1031, since the messages in the same message partition are consumed by multiple consumers, the performance information of the message partition represents the average performance of the consumers of the message partition. Therefore, the performance information corresponding to the message partition can be used as the performance information of each consumer of the message partition.

[0088] In this embodiment, for step S1032, the corresponding relationship between the performance information and the consumption capacity information can be preset. For example, if the average response time is less than the first threshold, the consumption capacity is the first score; if the average response time is greater than the first threshold and less than the second threshold, the consumption capacity is the second score; if the average response time is greater than the second threshold, the consumption capacity is the third score; or, if the average response time is greater than the first threshold and less than the second threshold, and the system load is less than the third threshold, the consumption capacity is the third score.

[0089] After determining the performance information of each consumer corresponding to each message partition, the corresponding consumption capacity information can be found from the corresponding relationship between the performance information and the consumption capacity information, so that the consumption capacity information of each consumer corresponding to each message partition can be determined.

[0090] In some embodiments, since the consumer side may correspond to one or more message partitions, if the consumer side corresponds to one message partition, the consumption capacity information of the consumer side can be uniquely determined. If the consumer side corresponds to multiple message partitions, the maximum consumption capacity corresponding to the consumer side can be selected as the consumption capacity of the consumer side; or the minimum consumption capacity corresponding to the consumer side can be selected as the consumption capacity of the consumer side; or the average value of the multiple consumption capacities corresponding to the consumer side can be taken as the consumption capacity of the consumer side, so as to obtain the consumption capacity information of the consumer side.

[0091] In this embodiment, by using the performance information corresponding to each message partition as the performance information of each consumer side corresponding to each message partition; and presetting the corresponding relationship between the performance information and the consumption capacity information, the consumption capacity information corresponding to each consumer side can be quickly determined.

[0092] In an alternative embodiment, the relevant data further includes the central processing unit performance information of the consumer side itself.

[0093] Step S102, determining the performance information corresponding to each message partition based on the relevant data, includes:

[0094] Determining each consumer side corresponding to each message partition; based on each consumer side corresponding to each message partition and the central processing unit performance information of each consumer side itself, determining the performance information corresponding to each message partition.

[0095] In this embodiment, in addition to being able to monitor and obtain the number of requests λ arriving per second for each message partition, each consumer side corresponding to each message partition, the number of servers s corresponding to each message partition, the number of requests μ that each server can process per second, etc., the central processing unit performance information of the consumer side itself can also be obtained. The central processing unit performance information can be the time for the central processing unit to process request tasks, the resources occupied, etc.

[0096] In some embodiments, the consumer side can feedback its own central processing unit performance information to the kafka server based on the requests of the kafka server.

[0097] In some embodiments, during the process of the consumer side pulling information from the kafka server, it can actively feedback its own central processing unit performance information.

[0098] In this embodiment, since one message partition corresponds to multiple different consumer sides and the central processing unit performance information of the consumer side itself is determined, for each message partition, by determining each consumer side corresponding to each message partition, then counting the central processing unit performance information of each consumer side, and then comparing the corresponding relationship between the central processing unit performance information and the message partition performance information, the performance information corresponding to the message partition can be determined.

[0099] In this embodiment, by partitioning the central processing unit performance information of each consumer, the performance information corresponding to the message partition can also be determined.

[0100] In an alternative embodiment, obtaining the central processing unit performance information of each consumer includes:

[0101] Obtaining the request task complexity information and remaining memory information corresponding to each consumer; based on the request task complexity information and remaining memory information corresponding to each consumer, and the corresponding relationship between the request task complexity information, remaining memory information, and central processing unit performance information, determining the central processing unit performance information of each consumer.

[0102] In this embodiment, the request task complexity information corresponding to each consumer can be determined by evaluating the request task complexity of the requests sent by each consumer.

[0103] In some embodiments, the remaining memory information of the consumer can be obtained by sending a request to the consumer.

[0104] In this embodiment, the corresponding relationship between the request task complexity information, remaining memory information, and central processing unit performance information can be pre-stored. After obtaining the request task complexity information and remaining memory information corresponding to each consumer, the central processing unit performance information of each consumer can be found.

[0105] In this embodiment, by obtaining the request task complexity information and remaining memory information corresponding to each consumer, and pre-storing the corresponding relationship between the request task complexity information, remaining memory information, and central processing unit performance information, the central processing unit performance information of each consumer can be quickly determined.

[0106] In an alternative embodiment, step S104 of determining the message partition quantity information corresponding to each consumer based on the consumption capacity information includes:

[0107] Normalizing the consumption capacity information corresponding to each consumer to obtain the normalized consumption capacity information; based on the normalized consumption capacity information, determining the percentage of the number of message partitions that each consumer can be assigned.

[0108] Specifically, the consumption capacity information corresponding to each consumer can be normalized by a normalization formula to obtain the normalized consumption capacity information. Then, the value corresponding to the normalized consumption capacity information is used as the percentage of the number of message partitions that each consumer can be assigned.

[0109] In this embodiment, the consumption capacity information corresponding to each consumer is normalized to facilitate determining the percentage of the number of message partitions that each consumer can be assigned, that is, to facilitate determining the message partition number information corresponding to each consumer.

[0110] In an alternative embodiment, the method for adjusting message partitions further includes:

[0111] Obtain the consumption timing data of each consumer; detect whether each consumer is abnormal based on the consumption timing data; if an abnormal consumer is detected, adjust the message partitions of the abnormal consumer.

[0112] In specific implementation, the respective message partitions corresponding to each consumer can be determined, and then the consumption timing data of each consumer can be obtained. The consumption timing data of a consumer includes the timing of the message partitions when the consumer pulls messages from multiple message partitions. If the consumption timing data indicates that the consumer does not pull messages from any message partition, or only pulls information from the corresponding partial message partitions, it can be determined that the consumer is abnormal. If the consumer is abnormal, the message partitions of the abnormal consumer can be adjusted.

[0113] In this embodiment, by detecting abnormal consumers, the message partitions corresponding to the abnormal consumers can be reallocated, thereby increasing the speed at which the messages in the message partitions are consumed and improving the data transmission efficiency.

[0114] According to the embodiments of the present application, the present application also provides an electronic device and a computer-readable storage medium.

[0115] Figure 6 FIG. shows a schematic block diagram of an exemplary electronic device 800 that can be used to implement the embodiments of the present application. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present application described and / or claimed herein.

[0116] As Figure 6As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to computer programs stored in a read-only memory (ROM) 802 or computer programs loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0117] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disc, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0118] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 executes the various methods and processes described above, such as the method for adjusting message partitioning. For example, in some embodiments, the method for adjusting message partitioning can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the method for adjusting message partitioning described above can be executed. Alternatively, in other embodiments, the computing unit 801 can be configured to execute the method for adjusting message partitioning in any other appropriate manner (e.g., by means of firmware).

[0119] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0120] The program code for implementing the methods of this application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing device, such that the program codes, when executed by the processor or controller, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on the remote machine or server.

[0121] In the context of this application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input received from the user can be in any form (including acoustic input, speech input, or tactile input).

[0123] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

[0124] A computer system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, or a server of a distributed system, or a server incorporating blockchain.

[0125] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in this application can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved, and no limitations are imposed herein.

[0126] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" can explicitly or implicitly include at least one of such features. In the description of this application, "a plurality of" means two or more, unless otherwise specifically defined.

[0127] As described above, it is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims described above.

Claims

1. A method for adjusting message partitions, comprising: Monitoring and obtaining relevant data during the process of each consumer pulling messages from each message partition corresponding to the target subscribed topic in the Kafka server; The relevant data includes the number of requests arriving at the message partition per second and the total number of requests that the message partition can process per second; Determining the performance information corresponding to each message partition based on the relevant data; The performance information at least includes the average response time; Determining the consumption capacity information corresponding to each consumer based on the performance information corresponding to each message partition; Determining the message partition quantity information corresponding to each consumer based on the consumption capacity information; Adjusting the message partitions corresponding to each consumer based on the message partition quantity information.

2. The method for adjusting message partitions according to claim 1, wherein determining the performance information corresponding to each message partition based on the relevant data includes: Calculating the average response time corresponding to each message partition through the average response time calculation formula based on the number of requests arriving at the message partition per second and the total number of requests that the message partition can process per second.

3. The method for adjusting message partitions according to claim 1, wherein the performance information further includes the system load corresponding to the message partition; Determining the performance information corresponding to each message partition based on the relevant data includes: Calculating the system utilization rate corresponding to each message partition through the system utilization formula based on the number of requests arriving at the message partition per second and the total number of requests that the message partition can process per second, and obtaining the system load corresponding to each message partition.

4. The method for adjusting message partitions according to any one of claims 1-3, wherein the performance information further includes the minimum number of servers for maintaining system stability corresponding to the message partition; Determining the performance information corresponding to each message partition based on the relevant data includes: Calculating the minimum number of servers for maintaining system stability corresponding to each message partition through the resource requirement formula based on the number of requests arriving at the message partition per second and the total number of requests that the message partition can process per second.

5. The method for adjusting message partitions according to claim 1, wherein determining the consumption capacity information corresponding to each consumer based on the performance information corresponding to each message partition includes: Determining each consumer corresponding to each message partition, and using the performance information corresponding to each message partition as the performance information of each consumer corresponding to each message partition; Determining the consumption capacity information corresponding to each consumer based on the performance information and the corresponding relationship between the performance information and the consumption capacity information.

6. The method for adjusting message partitions according to claim 1, wherein the relevant data further includes the central processing unit performance information of the consumer itself; Determining the performance information corresponding to each message partition based on the relevant data includes: Determining each consumer corresponding to each message partition; Determining the performance information corresponding to each message partition based on each consumer corresponding to each message partition and the central processing unit performance information of each consumer itself.

7. The method for adjusting message partitions according to claim 6, wherein obtaining the central processing unit performance information of each consumer itself includes: Obtaining the request task complexity information and the remaining memory information corresponding to each consumer respectively; Based on the request task complexity information and remaining memory information corresponding to each consumer end, as well as the corresponding relationship among the request task complexity information, remaining memory information, and central processing unit performance information, determine the central processing unit performance information of each consumer end itself.

8. The method for adjusting message partitioning according to claim 1, wherein the determining the message partitioning quantity information corresponding to each consumer end based on the consumption capacity information includes: Perform normalization processing on the consumption capacity information corresponding to each consumer end to obtain the normalized consumption capacity information; Based on the normalized consumption capacity information, determine the percentage of the number of message partitions that each consumer end can be allocated.

9. The method for adjusting message partitioning according to claim 1 further includes: Obtain the consumption timing data of each consumer end; Detect whether each consumer end is abnormal based on the consumption timing data; If an abnormal consumer end is detected, adjust the message partitioning of the abnormal consumer end.

10. An electronic device, comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for adjusting message partitioning according to any one of claims 1-9.

Images