A method and device for automatically expanding message queues

Abstract

The present application relates to a method and device for automatic expansion of a message queue. The method includes: obtaining the load of each candidate partition in the target message queue; judging whether there is a high-load partition in the candidate partition according to the load of each candidate partition and the target load threshold, wherein the load of the high-load partition is higher than the target load threshold; in the case of determining that there is at least one high-load partition, generating a target instruction for partition expansion of the target message queue; according to the target instruction, adding at least one new partition to the target message queue. Through the method in this embodiment, when there is a high-load partition in the target message queue, it is possible to automatically trigger the addition of at least one new partition to the target message queue, thereby improving the load capacity of the target message queue and facilitating the purpose of sharing the load of the high-load partition by adding a new partition in the later stage.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for automatically expanding a message queue. Background Art

[0002] A message queue is a message transmission middleware, which is often used in scenarios such as decoupling between systems, data forwarding, asynchronous communication, and traffic peak shaving. Among them, the topic messaging model that supports publish / subscribe (pub / sub) is a widely used message queue model that can be broadcast. The topic messaging model includes producers, topics (i.e. message queues), partitions, and consumers.

[0003] In the message queue component of the related art, the expansion of partitions (for example, the current number of partitions is 1, and because a larger amount of data needs to be carried, the partition needs to be modified to 2 or 3 or even larger) can only be achieved by manually calling the message queue command or by modifying the configuration file. Therefore, it is necessary to manually monitor the load of the message queue in order to achieve the purpose of timely partition expansion.

[0004] In the related art, manual partition expansion of message queues is adopted, which cannot be adaptively and elastically expanded, and thus has a technical problem of low expansion efficiency.

[0005] There is a technical problem of low expansion efficiency in the message queue of related technologies, and no effective solution has been provided yet. Summary of the invention

[0006] In order to solve the technical problem of low expansion efficiency in the above-mentioned message queue, the present application provides a method and device for automatic expansion of a message queue.

[0007] In a first aspect, an embodiment of the present application provides a method for automatically expanding a message queue, comprising:

[0008] Get the load of each candidate partition in the target message queue;

[0009] According to the load of each candidate partition and the target load threshold, determining whether there is a high-load partition in the candidate partitions, wherein the load of the high-load partition is higher than the target load threshold;

[0010] In the case of determining that there is at least one high-load partition, generating a target instruction for partitioning and expanding the target message queue;

[0011] According to the target instruction, at least one new partition is added to the target message queue.

[0012] Optionally, as in the aforementioned method, judging whether there is a high-load partition in the candidate partitions according to the load of each candidate partition and the target load threshold includes:

[0013] When the load of the candidate partition indicates that the data load of the candidate partition is greater than or equal to the target data load threshold, or the computing resource load of the candidate partition is greater than or equal to the target computing resource load threshold, the candidate partition is determined as the high-load partition.

[0014] Optionally, as in the aforementioned method, before generating a target instruction for partition expansion of the target message queue when determining that there is at least one high-load partition, the method further includes:

[0015] Determine a target low-load partition among all the candidate partitions except the high-load partition;

[0016] A target partition identifier of the target low-load partition is determined.

[0017] Optionally, as in the aforementioned method, adding at least one new partition in the target message queue according to the target instruction includes:

[0018] According to the target partition identifier, sending the target instruction to the target low-load partition;

[0019] After the target low-load partition executes the target instruction, at least one newly added partition is added to the target node where the target low-load partition is located.

[0020] Optionally, as in the aforementioned method, determining a target low-load partition among all the candidate partitions except the high-load partition comprises:

[0021] Determine all the candidate partitions except the high-load partition as low-load partitions;

[0022] determining a load of each of the low-load partitions;

[0023] The target low-load partition having the lowest load is determined among all the low-load partitions.

[0024] Optionally, as in the aforementioned method, obtaining the load of each candidate partition in the target message queue includes:

[0025] Obtaining load information of each candidate message queue in the target cluster, wherein each load information includes the load of each partition in the corresponding candidate message queue;

[0026] The target message queue is determined from among all the candidate message queues according to the load information, and the load of each candidate partition of the target message queue is determined, wherein the load of at least one candidate partition in the target message queue is higher than a target load threshold.

[0027] Optionally, as in the aforementioned method, after adding at least one new partition in the target message queue according to the target instruction, the method further includes:

[0028] Obtaining partition information of the newly added partition;

[0029] The historical metadata information of the target message queue is updated according to the partition information to obtain updated metadata information;

[0030] The data to be written is written into the newly added partition according to the updated metadata information.

[0031] In a second aspect, an embodiment of the present application provides an automatic capacity expansion device for a message queue, comprising:

[0032] An acquisition module is used to obtain the load of each candidate partition in the target message queue;

[0033] A judging module, configured to judge whether there is a high-load partition in the candidate partitions according to the load of each candidate partition and the target load threshold, wherein the load of the high-load partition is higher than the target load threshold;

[0034] A generating module, configured to generate a target instruction for partitioning and expanding the target message queue when it is determined that there is at least one high-load partition;

[0035] The new adding module is used to add at least one new partition in the target message queue according to the target instruction.

[0036] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0037] The memory is used to store computer programs;

[0038] The processor is used to implement any of the methods described above when executing the computer program.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium includes a stored program, wherein the program executes a method as described in any of the preceding items when executed.

[0040] Through the method in this embodiment, when there is a high-load partition in the target message queue, it is possible to automatically trigger the addition of at least one new partition to the target message queue, thereby improving the load capacity of the target message queue and facilitating the subsequent purpose of sharing the load of the high-load partition by adding the new partition. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A flowchart of a method for automatically expanding a message queue provided in an embodiment of the present application;

[0044] Figure 2 A flowchart of a method for automatically expanding a message queue provided in another embodiment of the present application;

[0045] Figure 3 A flowchart of a method for automatically expanding a message queue provided in another embodiment of the present application;

[0046] Figure 4 A block diagram of a message queue load balancing device provided in an embodiment of the present application;

[0047] Figure 5 A message queue architecture diagram provided for an embodiment of the present application;

[0048] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0050] According to one aspect of an embodiment of the present application, a method for automatically expanding a message queue is provided. Optionally, in this embodiment, the above-mentioned method for automatically expanding a message queue can be applied to a hardware environment consisting of one or more servers. The server is connected to the terminal via a network and can be used to provide services for the terminal or a client installed on the terminal, and a database can be set on the server or independently of the server to provide data storage services for the server.

[0051] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal may not be limited to a PC, a mobile phone, a tablet computer, etc.

[0052] Taking the automatic expansion method for a message queue in this embodiment executed by a server as an example, Figure 1 An automatic capacity expansion method for a message queue provided in an embodiment of the present application includes the following steps:

[0053] Step S101, obtaining the load of each candidate partition in the target message queue.

[0054] The automatic expansion method for a message queue in this embodiment can be applied to a scenario where the load of at least one candidate partition (i.e., partition) in a topic (i.e., message queue) in a topic messaging model is too high, and the load of each candidate partition can be reduced by automatically expanding the message queue.

[0055] The target message queue can be one of multiple message queues that need to determine whether partition expansion is required. The target message queue can be deployed in a cluster consisting of multiple nodes. The target message queue can be provided with multiple candidate partitions, each candidate partition is provided in a node, and each node can be provided with multiple candidate partitions. The candidate partitions of each message queue can also be distributed in different nodes. Each candidate partition corresponds to a folder on the physical disk, and all messages and index files of this candidate partition are stored in the folder. For example, there are 3 candidate partitions under a certain topic, and the 3 candidate partitions are respectively in three different nodes. After determining all candidate partitions of the target message queue, the load of each candidate partition can be determined through the historical metadata information of the target message queue.

[0056] The load can be information used to indicate the data processing intensity of the corresponding candidate partition during the data processing process. There is a one-to-one correspondence between the candidate partition and the load. Generally, the load of the candidate partition is the load of the node where the candidate partition is located. For example, when there are 3 candidate partitions, there is a load corresponding to each candidate partition. Furthermore, based on the load, the data processing intensity of each candidate partition can be obtained. Generally, the higher the load of a candidate partition, the higher the data processing intensity of the candidate partition.

[0057] Step S102: According to the load of each candidate partition and the target load threshold, determine whether there is a high-load partition among the candidate partitions, where the load of the high-load partition is higher than the target load threshold.

[0058] After determining the load of each candidate partition, it can be determined whether there is a high-load partition among the candidate partitions through a preset target load threshold.

[0059] The target load threshold can be preset and used to determine whether the load belongs to a high load. For example, when the target load threshold only includes a threshold a of type A, and the load of type A in the candidate partition is a', when a'≥a, it is determined that the candidate partition is a high-load partition; when a'<a, it is determined that the candidate partition is a low-load partition.

[0060] Step S103: In the case of determining that there is at least one high-load partition, generate a target instruction for partitioning and expanding the target message queue.

[0061] When it is determined that there is at least one high-load partition among all candidate partitions. When there is at least one high-load partition in the target message queue, it can be determined that the target message queue needs to be partitioned and expanded, so as to switch part of the traffic to pass through the new partition, thereby reducing the load of each high-load partition. And a target instruction for partitioning and expanding the target message queue can be generated.

[0062] The target instruction can be an instruction that can trigger the addition of a new partition in the target message queue. And as long as the target instruction meets the requirements for triggering partitioning and expansion, including but not limited to: "there is a high-load partition", "needs to be expanded", etc.

[0063] Step S104: According to the target instruction, add at least one new partition to the target message queue.

[0064] After the target instruction is generated, the target message queue can be partitioned and expanded by executing the target instruction to add at least one new partition. Furthermore, the correspondence between the number of new partitions and the overall load of the message queue can be preset, and the overall target load of the target message queue can be obtained based on the load of each high-load partition, and then the number of new partitions required to be added can be obtained based on the correspondence and the overall target load matching.

[0065] Through the method in this embodiment, when there is a high-load partition in the target message queue, it is possible to automatically trigger the addition of at least one new partition to the target message queue, thereby improving the load capacity of the target message queue and facilitating the subsequent purpose of sharing the load of the high-load partition by adding the new partition.

[0066] As an optional real-time method, such as the aforementioned method, the step S102 determines whether there is a high-load partition in the candidate partitions according to the load of each candidate partition and the target load threshold, including:

[0067] When the load of the candidate partition indicates that the data load of the candidate partition is greater than or equal to the target data load threshold, or the computing resource load of the candidate partition is greater than or equal to the target computing resource load threshold, the candidate partition is determined as a high-load partition.

[0068] The data load can be used to indicate the data volume of the corresponding candidate partition to be processed messages; for example, when there are 3 candidate partitions, there is a data load corresponding to each candidate partition, and then the data volume of the to-be-processed messages of each candidate partition can be obtained based on the data load; the data load can be characterized by the number of messages written per second, for example, 1000qps (queries-per-second).

[0069] After determining the data load of each candidate partition, it can be determined whether there is a high-load partition among all candidate partitions through a preset target data load threshold.

[0070] The target data load threshold can be a pre-set threshold used to determine whether the data load is a high-load threshold. For example, when the target data load threshold is b, there is a candidate partition in the candidate partition whose data load is b', and b'≥b, the candidate partition is determined to be a high-load partition. When there is no candidate partition in the target message queue whose load is greater than or equal to a, it is determined that there is no high-load partition in the target message queue, so there is no need to expand the partition. In addition, the data load can include loads of multiple dimensions, so the target data load threshold can also include sub-target data load thresholds corresponding to each dimension; and when the sub-load of one dimension of the data load is greater than or equal to the sub-target data load threshold corresponding to the dimension, the candidate partition corresponding to the data load is determined to be a high-load partition.

[0071] The computing resource load may include the usage of different computing units or storage units. When the computing resource load is for the usage of a computing unit, it may be the ratio of the computing resources occupied by the computing unit to all the computing resources of the computing unit. When the computing resource load is for the usage of a storage unit, it may be the ratio of the storage resources used to store data in the storage unit to all the storage resources of the storage unit. For example, the computing resource load may be characterized by the usage of the CPU load, memory load, and disk storage load of the node where the corresponding candidate partition is located.

[0072] The target computing resource load threshold can be a pre-set threshold for determining whether the computing resource load is a high-load threshold. When the target computing resource load threshold includes multiple types of sub-target computing resource load thresholds (e.g., CPU usage threshold, memory usage threshold, and disk storage usage threshold), the computing resource load of the candidate partition can be obtained, which can also include the current CPU usage, the current disk usage, and the current memory usage; and when the current CPU usage is greater than or equal to the CPU usage threshold, the current disk usage is greater than or equal to the memory usage threshold, and the current memory usage is greater than or equal to the disk storage usage threshold, the candidate partition is determined to be a high-load partition. For example, when there are three partitions in the target message queue, namely, candidate partition 1, candidate partition 2, and candidate partition 3, respectively in three different nodes, the CPU usage threshold is 70%, the disk usage threshold is 90%, and the memory usage threshold is 70%. The current CPU usage, current disk usage and current memory usage of candidate partition 1 are (20%, 30%, 35%), the current CPU usage, current disk usage and current memory usage of candidate partition 2 are (30%, 95%, 35%), and the current CPU usage, current disk usage and current memory usage of candidate partition 3 are (80%, 40%, 85%). The current CPU usage, current disk usage and current memory usage of candidate partition 1 are all less than the corresponding thresholds, so candidate partition 1 is a low-load partition; the current disk usage of candidate partition 2 is greater than the disk usage threshold, so although the current CPU usage of candidate partition 2 is less than the CPU usage threshold and the current memory usage is less than the memory usage threshold, candidate partition 2 is still a high-load partition; the current CPU usage of candidate partition 3 is greater than the CPU usage threshold, and the current memory usage is greater than the memory usage threshold, so although the current disk usage is less than the disk usage threshold, candidate partition 3 is still a high-load partition.

[0073] like Figure 2 As shown, as an optional implementation, as in the aforementioned method, in the step S103, when it is determined that there is at least one high-load partition, before generating a target instruction for partitioning and expanding the target message queue, the method further includes the following steps:

[0074] Step S301: determine a target low-load partition from all candidate partitions except the high-load partition.

[0075] After the high-load partition is determined among all candidate partitions, a target low-load partition can be determined among all candidate partitions except the high-load partition.

[0076] The target low-load partition may be a candidate partition in a target message queue for determining a node where a newly added partition is located.

[0077] After the target low-load partition is determined, since the load of the target low-load partition is consistent with the load of the node where the target low-load partition is located, a partition can be added in the node where the target low-load partition is located later.

[0078] Step S302: determine the target partition identifier of the target low-load partition.

[0079] After the target low-load partition is determined, since each candidate partition is located in a corresponding node and each candidate partition is equivalent to a folder in each node, the target partition identifier of each target low-load partition can be determined.

[0080] The target partition identifier may be identification information uniquely corresponding to the target low-load partition, or may be location information or encoding information of the target low-load partition, etc. Moreover, each candidate partition has a unique corresponding partition identifier.

[0081] Through the method in this embodiment, a target low-load partition with a lower load and a target partition identifier of the target low-load partition can be determined among all candidate partitions. Since the load of the target low-load partition is consistent with the load of the node where the target low-load partition is located, a new partition can be added to the node where the target low-load partition is located later to avoid adding new partitions to nodes with higher loads, which would lead to the problem of further increasing the load of nodes with higher loads.

[0082] As an optional implementation, as in the above method, the step S104 of adding at least one new partition in the target message queue according to the target instruction includes the following steps:

[0083] Step S401: sending a target instruction to a target low-load partition according to the target partition identifier.

[0084] After the target partition identifier is determined, the low-load partition for receiving the target instruction can be determined among all candidate partitions, and the target instruction can be accurately sent to the target low-load partition according to the target partition identifier.

[0085] Step S402: according to the target instruction, at least one new partition is added in the target node where the target low-load partition is located.

[0086] After the target low-load partition receives the target instruction, at least one new partition can be added to the target node where the target low-load partition is located according to the target instruction through the component used for partition expansion.

[0087] The target node may be a node in the node cluster where the target low-load partition is located, and since the target low-load partition is a candidate partition with a lower load, the target node is also a node with a relatively lower load in the node cluster.

[0088] The newly added partition may be a partition newly generated in the target node according to the target instruction.

[0089] Furthermore, the number of newly added partitions can also be determined based on the overall load situation of the target message queue: first determine the number of high-load partitions in the target message queue; according to a preset partition addition scheme, determine the number of newly added partitions based on the number of high-load partitions, wherein the partition addition scheme is used to indicate the correspondence between the number of newly added partitions required to increase the number of different high-load partitions.

[0090] For example, the partitions (i.e., candidate partitions) of a topic (i.e., the target message queue) are distributed in different brokers (i.e., nodes), where different partitions can be divided into two roles, one is the leader (primary partition, only one is set), and the other is the follower (slave partition, all partitions other than the leader are set as followers). The leader and the follower can maintain a heartbeat link through the TCP heartbeat, and the basic information of each partition (including load and partition identifier, etc.) can be synchronized through the heartbeat.

[0091] In addition, the load can be acquired through a partition manager added to the topic layer. This partition manager collects some current loads of each partition, such as the CPU of the broker where the partition is located, the memory resource usage, the partition data write flow size, etc. The follower reports the load to the leader through heartbeats. After receiving the heartbeats of each follower, the leader determines whether there is a high-load partition in the topic based on the preset target load threshold (the user can configure the expansion conditions in the configuration file independently). In the case of a high-load partition, it determines which target broker (i.e., target node) is located to expand the partition.

[0092] Through the method in this embodiment, partition expansion can be quickly implemented in the target node where the target low-load partition is located, and at least one new newly added partition can be obtained.

[0093] As an optional implementation, as in the above method, the step S301 of determining the target low-load partition from all candidate partitions except the high-load partition includes the following steps:

[0094] Step S501, determining all candidate partitions except the high-load partition as low-load partitions;

[0095] Step S502, determining the load of each low-load partition;

[0096] Step S503: determining a target low-load partition with the lowest load among all low-load partitions.

[0097] After determining the high-load partition among all candidate partitions, other candidate partitions except the high-load partition can be determined, and the other candidate partitions except the high-load partition are recorded as low-load partitions. Since the high-load partition is a candidate partition whose load is greater than or equal to the target load threshold, the load of the low-load partition is less than the target load threshold, and therefore, the node where the low-load partition is located can be used to distribute more load.

[0098] After each low-load partition is determined, the load of each low-load partition may be determined according to the method in the above embodiment.

[0099] After the load of each low-load partition is determined, the target low-load partition with the lowest load can be determined among all the low-load partitions by comparing the sizes of the loads.

[0100] For example, when it is determined that there are low-load partitions D1 and D2, the load F1 of the low-load partition D1 and the load F2 of the low-load partition D2 can be obtained. When there is only one dimension of value in the load, the sizes of F1 and F2 can be directly compared to determine the target low-load partition; when there are sub-loads of multiple dimensions in the load (for example, CPU load, memory load, disk storage load, etc.), the corresponding proportions of the usage rates of each dimension can be weighted to obtain weighted values, and then each low-load partition can be judged based on the weighted values ​​to determine the target low-load partition with the smallest weighted value. For example, when the weights of the CPU load, memory load, and disk storage load are 0.3, 0.4, and 0.3 respectively, the CPU load of the low load partition D1 is 50%, the memory load is 40%, and the disk storage load is 40%, and the CPU load of the low load partition D2 is 40%, the memory load is 50%, and the disk storage load is 40%, then the weighted value corresponding to the low load partition D1 is (50%×0.3)+(40%×0.4)+(40%×0.3)=43%, and the weighted value corresponding to the low load partition D2 is (40%×0.3)+(50%×0.4)+(40%×0.3)=48%. Therefore, the weighted value corresponding to the low load partition D1 is less than the weighted value corresponding to the low load partition D2, and the low load partition D1 is used as the target low load partition.

[0101] Through the method in this embodiment, the target low-load partition can be quickly selected, and the partition expansion can be performed faster, thereby effectively improving the efficiency of partition expansion and reducing the load of the high-load partition in the target message queue more quickly.

[0102] As an optional implementation, as in the above method, the step S101 of obtaining the load of each candidate partition in the target message queue includes the following steps:

[0103] Step S601: Obtain load information of each candidate message queue in the target cluster, wherein each load information includes the load of each partition in the corresponding candidate message queue.

[0104] The target cluster may be a computing cluster composed of multiple broker nodes, and each broker node may be provided with one or more candidate message queue partitions.

[0105] After each candidate message queue is determined, the load information of the candidate message queue can be determined.

[0106] The load information includes the load of each partition in the candidate message queue corresponding to the load information. That is, when the load information k corresponds to the candidate message queue k', the load information k includes the load of each partition in the candidate message queue k'.

[0107] Step S602: determine a target message queue from all candidate message queues according to the load information, and determine the load of each candidate partition of the target message queue, wherein the load of at least one candidate partition in the target message queue is higher than the target load threshold.

[0108] After obtaining the load information of each candidate message queue, a target message queue can be determined from all candidate message queues according to the load information, and the target message queue needs to meet the requirement that at least one candidate partition has a load higher than a target load threshold.

[0109] Furthermore, each candidate message queue may correspond to a usage threshold, and the target usage threshold is the usage threshold corresponding to the target message queue. Therefore, the resource usage of each candidate partition may be compared with the target load threshold, and if there is at least one candidate partition whose corresponding resource usage is higher than the target load threshold, it indicates that there is at least one high-load partition in the target message queue. It can then be determined that the target message queue needs to be load balanced, and the load of each candidate partition in the target message queue can be obtained.

[0110] Through the method in this embodiment, when there are multiple candidate message queues, a target message queue that needs to be load balanced can be determined.

[0111] like Figure 3 As shown, as an optional implementation, as in the aforementioned method, after adding at least one new partition to the target message queue according to the target instruction in step S104, the method further includes the following steps:

[0112] Step S701, obtaining partition information of the newly added partition.

[0113] After adding a partition to the target node, you can obtain the new partition and partition information.

[0114] Partition information may include but is not limited to: newly added partition location information, real-time load, etc.

[0115] Step S702: Update the historical metadata information of the target message queue according to the partition information to obtain updated metadata information.

[0116] After the partition information of the newly added partition is determined, the historical metadata information of the target message queue can be updated according to the partition information to obtain updated metadata information.

[0117] The historical metadata information may record the distribution of the target message queue and candidate partitions before the partition expansion (for example, location information corresponding to each candidate partition), and the load of each candidate partition (including real-time load).

[0118] The updated metadata information may record the distribution of each current partition (ie, candidate partition and newly added partition) in the target message queue after the partition expansion, as well as the load of each current partition.

[0119] Step S703: write the data to be written into the newly added partition according to the updated metadata information.

[0120] After obtaining the updated metadata information, the data to be written can be written to the newly added partition of the new expansion.

[0121] The data to be written may be data used to be written into a newly added partition from among all the data that needs to be written into the target message queue after the target message queue is partitioned and expanded.

[0122] Moreover, after obtaining the updated metadata information, a diversion plan for diverting all data can be obtained. The diversion plan can indicate that the data flow allocated to the high-load partition is lower than the data flow before the partition expansion, and part of the data flow is allocated to the newly added partition; then all data is diverted according to the diversion plan, so that the data to be written can be written into the newly added partition, thereby reducing the load of the high-load partition.

[0123] Through the method in this embodiment, the historical metadata information can be updated to obtain updated metadata information, so as to quickly enable the newly added partition and use it to write the data to be written, thereby achieving the purpose of more quickly diverting the load of the high-load partition.

[0124] like Figure 4 As shown, according to an embodiment of another aspect of the present application, there is also provided an automatic expansion device for a message queue, comprising:

[0125] Acquisition module 1, used to obtain the load of each candidate partition in the target message queue;

[0126] A judgment module 2 is used to judge whether there is a high-load partition in the candidate partitions according to the load of each candidate partition and the target load threshold, wherein the load of the high-load partition is higher than the target load threshold;

[0127] A generating module 3, configured to generate a target instruction for partitioning and expanding the target message queue when it is determined that there is at least one high-load partition;

[0128] The new module 4 is used to add at least one new partition in the target message queue according to the target instruction.

[0129] like Figure 5In the message queue architecture shown, the partitions (i.e., candidate partitions) of the topic (i.e., the target message queue) are distributed in different brokers (i.e., nodes), where different partitions can be divided into two roles, one is the leader (the primary partition, only one is set), and the other is the follower (the slave partition, all partitions other than the leader are set as followers). The leader and the follower can maintain a heartbeat link through the TCP heartbeat, and the basic information of each partition (including load and partition identifier, etc.) can be synchronized through the heartbeat. The above-mentioned new module 4 can be set in each partition.

[0130] The automatic expansion device may include a partition manager (manager, provided with the above-mentioned acquisition module 1, judgment module 2 and generation module 3) in the topic, and the partition manager can collect some current loads of each partition (i.e., candidate partition), such as the CPU of the broker where the partition is located, the resource usage of the memory, the data write flow size of the partition, etc. The follower reports the load to the leader through the heartbeat. After receiving the heartbeat of each follower, the leader determines whether there is a high-load partition in the topic according to the preset target load threshold (the user can configure the expansion conditions in the configuration file independently), so as to determine in which target partition (i.e., the target low-load partition) The target broker (i.e., the target node) is located to perform partition expansion.

[0131] If the load of a partition reaches the target load threshold, the partition is a high-load partition, and the topic needs to be expanded. Then the leader will send the expansion target instruction to a follower with a relatively low load through the heartbeat method. After receiving the target instruction, the new module 4 in the follower can automatically create a partition. At the same time, the basic information of the newly generated new partition (i.e., the new partition) is reported to the leader.

[0132] When the number of partitions in the original message queue is 3 and the number of newly added partitions is 1, the number of partitions in the message queue will automatically expand from 3 to 4, and the metadata information of the topic and partition will also be updated, so that data can be written normally to the newly expanded partition.

[0133] According to another embodiment of the present application, there is also provided an electronic device, including: Figure 6 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0134] Memory 1503, used for storing computer programs;

[0135] The processor 1501 is used to implement the steps of the above method embodiment when executing the program stored in the memory 1503.

[0136] The bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0137] The communication interface is used for communication between the above electronic device and other devices.

[0138] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0139] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0140] An embodiment of the present application further provides a computer-readable storage medium, the storage medium including a stored program, wherein the method steps of the above method embodiment are executed when the program is run.

[0141] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0142] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for automatic expansion of a message queue, characterized in that: include: Obtaining the load of each candidate partition in the target message queue, wherein the load of the candidate partition is determined according to the historical metadata information of the target message queue, and there are at least two candidate partitions located on the same node; According to the load of each candidate partition and the target load threshold, determining whether there is a high-load partition in the candidate partitions, wherein the load of the high-load partition is higher than the target load threshold; In the case of determining that there is at least one high-load partition, generating a target instruction for partitioning and expanding the target message queue; According to the target instruction, adding at least one new partition in the target message queue; Wherein, before generating a target instruction for partition expansion of the target message queue in the case where it is determined that there is at least one high-load partition, the method further includes: determining a target low-load partition among all the candidate partitions except the high-load partition; determining a target partition identifier of the target low-load partition, Among them, adding at least one new partition in the target message queue according to the target instruction includes: sending the target instruction to the target low-load partition according to the target partition identifier; after the target low-load partition executes the target instruction, adding at least one of the new partitions in the target node where the target low-load partition is located.

2. The method according to claim 1, characterized in that The determining, according to the load of each candidate partition and the target load threshold, whether there is a high-load partition in the candidate partitions comprises: When the load of the candidate partition indicates that the data load of the candidate partition is greater than or equal to the target data load threshold, or the computing resource load of the candidate partition is greater than or equal to the target computing resource load threshold, the candidate partition is determined as the high-load partition.

3. The method according to claim 1, characterized in that Determining a target low-load partition among all the candidate partitions except the high-load partition comprises: Determine all the candidate partitions except the high-load partition as low-load partitions; determining a load of each of the low-load partitions; The target low-load partition having the lowest load is determined among all the low-load partitions.

4. The method according to claim 1, characterized in that The step of obtaining the load of each candidate partition in the target message queue includes: Obtaining load information of each candidate message queue in the target cluster, wherein each load information includes the load of each partition in the corresponding candidate message queue; The target message queue is determined from among all the candidate message queues according to the load information, and the load of each candidate partition of the target message queue is determined, wherein the load of at least one candidate partition in the target message queue is higher than a target load threshold.

5. The method according to claim 1, characterized in that After adding at least one new partition in the target message queue according to the target instruction, the method further includes: Obtaining partition information of the newly added partition; The historical metadata information of the target message queue is updated according to the partition information to obtain updated metadata information; The data to be written is written into the newly added partition according to the updated metadata information.

6. An automatic expansion device for a message queue, characterized in that: include: An acquisition module, used to acquire the load of each candidate partition in the target message queue, wherein the load of the candidate partition is determined according to the historical metadata information of the target message queue, and there are at least two candidate partitions located on the same node; A judging module, configured to judge whether there is a high-load partition in the candidate partitions according to the load of each candidate partition and the target load threshold, wherein the load of the high-load partition is higher than the target load threshold; A generating module, configured to generate a target instruction for partitioning and expanding the target message queue when it is determined that there is at least one high-load partition; A new adding module, used for adding at least one new partition in the target message queue according to the target instruction; Wherein, before generating a target instruction for partition expansion of the target message queue when it is determined that there is at least one high-load partition, the method further includes: determining a target low-load partition among all the candidate partitions except the high-load partition; determining a target partition identifier of the target low-load partition; Among them, adding at least one new partition in the target message queue according to the target instruction includes: sending the target instruction to the target low-load partition according to the target partition identifier; after the target low-load partition executes the target instruction, adding at least one of the new partitions in the target node where the target low-load partition is located.

7. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 5 when executed.

Images