A disk balancing method and device based on a distributed message subscription platform

Abstract

The application relates to the technical field of data disk allocation, and discloses a disk balancing method and device based on a distributed message subscription platform. The application is based on a broker component in a target server to store basic data of multiple topics and multiple disks, further calculates the number of to-be-migrated copies of target topics stored in a target disk, and migrates the topic copies stored in the target disk with excessively high usage, so that the problem of disk imbalance is avoided, and the global optimization of the broker component is finally ensured.

Description

Technical Field

[0001] This invention relates to the field of data disk allocation technology, and specifically to a disk balancing method and apparatus based on a distributed message subscription platform. Background Technology

[0002] Kafka is a distributed message streaming platform that supports multiple partitions and replicas, enabling the transfer of large amounts of data between multiple producers and consumers. A Kafka cluster consists of multiple servers, where each server is called a broker component. A cluster comprises multiple broker components, each of which can hold multiple topics and mount multiple disks. Producers write data to topics within a broker component, which are then stored on the broker component's disks. Consumers retrieve data from the corresponding topics through the broker component. During the production data writing process, sometimes some disks on a particular broker component may experience high utilization due to excessive data writes, while other disks may experience low utilization due to less data writes. This can lead to disk imbalance within that broker component, potentially causing localized overload and impacting the overall system load. Summary of the Invention

[0003] In view of this, the present invention provides a disk balancing method and apparatus based on a distributed message subscription platform to solve the problem of uneven data writing to broker components.

[0004] In a first aspect, the present invention provides a disk balancing method based on a distributed message subscription platform. The distributed message subscription platform includes multiple servers, and the target server stores multiple topics and multiple disks through a broker component. Each target topic corresponds to a physically partitioned topic partition. Each target topic is one of the multiple topics, and each topic partition contains multiple topic replicas. Each target disk is one of the multiple disks, and each target server is one of the multiple servers. The method is used on the target server and includes:

[0005] Get the total number of disks across multiple disks, the number of current theme copies stored on the target disk, and the current number of disks occupied by the target theme on the target disk;

[0006] Based on the total number of disks across multiple disks and the number of current topic replicas of the target topic stored on the target disk, calculate the average number of topic replicas corresponding to the target disk;

[0007] Based on the current number of theme replicas of the target topic stored on the target disk and the average number of theme replicas corresponding to the target disk, calculate the number of target theme replicas to be migrated stored on the target disk;

[0008] Based on the number of copies of the target topic to be migrated stored on the target disk, the copies to be migrated on the target disk are migrated to the optimal target disk according to the selection principle.

[0009] The disk balancing method based on a distributed message subscription platform in this embodiment calculates the number of target topic replicas to be migrated on the target disk based on the basic data of multiple topics and multiple disks stored in the broker component of the target server, and migrates topic replicas stored on the target disk with excessive usage to avoid disk imbalance and ultimately ensure the global optimization of the broker component.

[0010] In some optional implementations, based on the number of replicas of the target topic stored on the target disk to be migrated, the disk to be migrated is migrated to the optimal target disk according to a selection principle, including:

[0011] Determine if the number of replicas of the target topic to be migrated stored on the target disk is greater than zero;

[0012] If the number of replicas of the target topic to be migrated stored on the target disk is greater than zero, the target disk is determined to be a disk to be migrated;

[0013] If the number of copies of the target topic to be migrated stored on the target disk is less than zero, the target disk is determined to be the target migration object;

[0014] Sort the number of current theme replicas of the target migration object in descending order and output the sorting results;

[0015] Select the top K disks from the sorting results, and then select the target disk with the smallest storage capacity from the K disks according to a preset time as the optimal target disk.

[0016] Migrate the target disk containing the target topic's copy to the optimal target disk.

[0017] This disclosure improves the balance of disk usage in the broker component by dynamically finding the target disk with lower usage in the broker component as the optimal target disk, and migrating the copy of the target disk with higher usage to the optimal target disk.

[0018] In some optional implementations, after migrating the target disk containing a copy of the target topic to be migrated to the optimal target disk, the method further includes:

[0019] Based on the number of replicas of the target topic to be migrated on the target disk storage, update the current number of replicas of the target topic on the optimal target disk storage.

[0020] In some optional implementations, the average number of topic replicas corresponding to the target disk is calculated based on the total number of disks across multiple disks and the number of current topic replicas of the target topic stored on the target disk, using the following formula:

[0021]

[0022] Where A is the average number of subject replicas corresponding to the target disk, and D... i The number of current theme copies of the target topic stored on the target disk, where M is the total number of disks across multiple disks, and n is the current number of disks occupied by the target topic on the target disk.

[0023] In some optional implementations, the disk balancing method based on a distributed message subscription platform in this disclosure further includes: subtracting the current number of disks occupied by the target topic from the total number of disks of multiple disks to obtain the remaining number of disks not occupied by the target topic.

[0024] In some optional implementations, the disk balancing method based on a distributed message subscription platform in this disclosure further includes: obtaining the total number of topic partitions of the target topic and the total number of topic replicas for each topic partition.

[0025] Secondly, the present invention provides a disk balancing device based on a distributed message subscription platform. The distributed message subscription platform includes multiple servers. The target server stores multiple topics and multiple disks through a broker component. Each target topic corresponds to a physically partitioned topic partition. Each target topic is one of the multiple topics, and each topic partition contains multiple topic replicas. Each target disk is one of the multiple disks. Each target server is one of the multiple servers. The method is used on the target server and includes:

[0026] The data acquisition module is used to obtain the total number of disks on multiple disks, the number of current theme copies of the target theme stored on the target disk, and the number of disks currently occupied by the target theme on the target disk.

[0027] The first calculation module is used to calculate the average number of topic replicas corresponding to the target disk based on the total number of disks and the number of current topic replicas of the target topic stored on the target disk.

[0028] The second calculation module is used to calculate the number of target theme replicas to be migrated on the target disk based on the current number of target theme replicas stored on the target disk and the average number of target theme replicas corresponding to the target disk.

[0029] The data migration module is used to migrate disks to the optimal target disk based on the number of copies of the target topic stored on the target disk.

[0030] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the disk balancing method based on a distributed message subscription platform as described in the first aspect or any corresponding embodiment thereof.

[0031] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the disk balancing method based on a distributed message subscription platform according to the first aspect or any corresponding embodiment described above.

[0032] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the disk balancing method based on a distributed message subscription platform as described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0033] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating a disk balancing method for a distributed message subscription platform according to an embodiment of the present invention.

[0035] Figure 2 This is a schematic flowchart of a disk balancing method for another distributed message subscription platform according to an embodiment of the present invention;

[0036] Figure 3 This is a schematic flowchart of a disk balancing method for another distributed message subscription platform according to an embodiment of the present invention;

[0037] Figure 4 This is a simplified flowchart illustrating a disk balancing method for a distributed message subscription platform according to an embodiment of the present invention.

[0038] Figure 5 This is a structural block diagram of a disk balancing device for another distributed message subscription platform according to an embodiment of the present invention.

[0039] Figure 6 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] The disk balancing method based on the distributed message subscription platform is applied to Kafka. In practical applications, Kafka is a high-real-time data processing tool, and data latency and loss are likely to occur if it fails. For example, if the utilization of a disk on some broker is too high, it can easily lead to disk failure and consequently cause Kafka failure.

[0042] In view of this, this disclosure provides a disk balancing method based on a distributed message subscription platform. The distributed message subscription platform includes multiple servers. The target server stores multiple topics and multiple disks through a broker component. The target topic corresponds to a physically partitioned topic partition. The target topic is each of the multiple topics, and each topic partition contains multiple topic replicas. The target disk is each of the multiple disks. The target server is each of the multiple servers. The method is used on the target server.

[0043] Specifically, each topic is called a Topic, and each message sent to the Kafka cluster has a category called a Topic. Messages are categorized by topic; a topic is essentially a directory, grouping messages on the same topic into the same directory. Topic partitions are called Topic partitions, and each message sent to the Kafka cluster has a category called a Topic. Messages are categorized by topic; a topic is essentially a directory, grouping messages on the same topic into the same directory. Kafka is a distributed message streaming platform developed by the Apache Software Foundation that supports multiple partitions and replicas. It can transfer large amounts of data between multiple producers and consumers. Producers are responsible for sending messages to Kafka cluster topics, and consumers are responsible for consuming data from Kafka cluster topics.

[0044] According to an embodiment of the present invention, a disk balancing method based on a distributed message subscription platform is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0045] This embodiment provides a disk balancing method based on a distributed message subscription platform, which can be used in computer devices such as mobile phones, tablets, desktop computers, laptops, servers, etc. Figure 1 This is a flowchart of a disk balancing method based on a distributed message subscription platform according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0046] Step S101: Obtain the total number of disks across multiple disks, the number of current theme copies of the target theme stored on the target disk, and the number of current disks occupied by the target theme on the target disk.

[0047] Specifically, "multiple disks" refers to the total number of disks contained in the broker component of the target server, denoted by M. "Target topic" refers to the target topic, and "target topic" represents the topics corresponding to the multiple topic partitions. The number of current topic replicas of the target topic stored on the target disks can be represented by D. i This indicates the number of disk replicas of the target topic under the broker component. The current number of disks occupied by the target topic is represented by n, which also represents the data of the target topic stored on n disks of this broker component. Step S101 is equivalent to retrieving basic data from each broker component of the Kafka cluster.

[0048] Step S102: Based on the total number of disks across multiple disks and the number of current topic replicas of the target topic stored on the target disk, calculate the average number of topic replicas corresponding to the target disk.

[0049] In a specific example, based on the total number of disks across multiple disks and the number of current topic replicas stored on the target disk, the average number of topic replicas corresponding to the target disk is calculated using the following formula:

[0050]

[0051] Where A is the average number of subject replicas corresponding to the target disk, and D... i The target disk stores the current number of copies of the target topic, where M is the total number of disks across multiple disks, and n is the current number of disks occupied by the target topic on the target disk. The target disk is each of the multiple disks.

[0052] For example, a broker component has 9 disks storing Kafka cluster data. If the total number of disks M = 9, and a target topic stores its data on 3 disks in this broker component, then the target topic occupies the current number of disks n = 3. Among the 3 disks, target disk D1 stores the replica data of the first, third, and fifth partitions of the target topic, so target disk D1 = 3; target disk D2 stores the replica data of the second and fourth partitions of the target topic, so D2 = 2; and target disk D3 stores the data of the sixth partition of the target topic, so D1 = 1. It can be calculated that A = ceil((D1 + D2 + D3) / M) = ceil((3 + 2 + 1) / 9) = 1, where ceil represents rounding up.

[0053] Step S103: Calculate the number of target theme replicas to be migrated on the target disk based on the current number of target theme replicas stored on the target disk and the average number of target theme replicas corresponding to the target disk.

[0054] Step S104: Based on the number of copies of the target topic to be migrated stored on the target disk, migrate the copies of the target disk to the optimal target disk according to the selection principle.

[0055] Specifically, in the broker component, identify the target disk where the number of theme replicas of the target theme is greater than the average number of theme replicas, and migrate the number of replicas of the target theme to be migrated stored on the target disk.

[0056] For example, let B represent the number of replicas to be migrated from the target disk, let A represent the average number of topic replicas corresponding to the target disk, and let D represent the number of current topic replicas of the target topic stored on the target disk. Then B = DA. B > 0 indicates that the target topic needs disk balancing in this broker component, and the number of replicas to be migrated from the target disk needs to be migrated to the target disk where B < 0.

[0057] The disk balancing method based on a distributed message subscription platform in this embodiment calculates the number of target topic replicas to be migrated on the target disk based on the basic data of multiple topics and multiple disks stored in the broker component of the target server, and migrates topic replicas stored on the target disk with excessive usage to avoid disk imbalance and ultimately ensure the global optimization of the broker component.

[0058] This embodiment provides a disk balancing method based on a distributed message subscription platform, which can be used in computer devices such as mobile phones, tablets, desktop computers, laptops, servers, etc. Figure 2This is a flowchart of a disk balancing method based on a distributed message subscription platform according to an embodiment of the present invention. In step S103 above, based on the current number of topic replicas of the target topic stored on the target disk and the average number of topic replicas corresponding to the target disk, the number of replicas of the target topic to be migrated stored on the target disk is calculated. Figure 2 As shown, the process includes the following steps:

[0059] Step S1031: Determine whether the number of replicas of the target topic to be migrated stored on the target disk is greater than zero.

[0060] Specifically, the number of replicas of the target topic to be migrated stored on the target disk is B. B>0 indicates that the target topic needs disk balancing in this broker component.

[0061] Step S1032: If the number of copies of the target topic to be migrated stored on the target disk is greater than zero, the target disk is determined to be the disk to be migrated.

[0062] For example, if the broker component contains multiple disks, namely target disks E1, E2, E3, and E4, and the number of replicas to be migrated for target disks E2 and E4 is greater than zero, then target disks E2 and E4 are determined to be disks to be migrated.

[0063] Step S1033: If the number of copies of the target topic to be migrated stored on the target disk is less than zero, the target disk is determined as the target migration object.

[0064] For example, in the above example, the target disks E1, E2, and E3 are all less than zero, so target disks E1, E2, and E3 are used as target migration objects.

[0065] Step S1034: Sort the number of current theme replicas of the target migration object storage target theme in descending order, and output the sorting result.

[0066] For example, the current number of theme copies of the target migration objects E1, E2, and E3 mentioned above are 5, 6, and 8 respectively. Sorted in descending order, the corresponding output target migration objects are E3, E2, and E1 respectively.

[0067] Step S1035: Select the top K disks from the sorting results, and select the target disk with the smallest storage capacity from the K disks according to a preset time as the optimal target disk.

[0068] For example, if K=2, the first two target disks E3 and E2 are selected from the target migration objects E3, E2 and E1 in the above example as the optimal target disks. The optimal target disk can be represented by K.

[0069] If the number of copies of the target topic to be migrated stored on the target disk is zero, it means that the utilization rate of the target disk just meets the requirements, and there is no need to migrate the copies of the target disk to be migrated.

[0070] This embodiment of the disclosure, by executing the above steps S1031-S1035, dynamically finds the target disk with a lower usage rate in the broker component as the optimal target disk, and migrates the copy to be migrated from the target disk with a higher usage rate to the optimal target disk, thereby improving the disk usage balance of the broker component.

[0071] In some alternative implementations, such as Figure 3 As shown, after migrating the target disk containing the target topic's copy to the optimal target disk, the process also includes:

[0072] Step S105: Based on the number of replicas of the target topic to be migrated in the target disk storage, update the current number of replicas of the target topic in the optimal target disk storage.

[0073] For example, target disks with B<0 on the broker component for the target topic are sorted in ascending order of the number D of replicas of that target topic on that target disk. From the k smallest disks, the target disk with the smallest storage increase over a recent time period is selected as the optimal target migration object W. The value of k is configurable, for example, k=5. One replica of the disk with B>0 on the broker component is migrated to the target disk W. For each migrated replica, the number of replicas of that target topic on the target disk W is updated to D=D+1. Following steps S1031-S1035, target disks with higher usage are continuously migrated until all target topics with B>0 become B<=0.

[0074] In some optional implementations, the disk balancing method based on a distributed message subscription platform in this disclosure further includes: subtracting the current number of disks occupied by the target topic from the total number of disks of multiple disks to obtain the remaining number of disks not occupied by the target topic.

[0075] For example, if the target server's broker component has M disks, and a target topic stores 3 copies of its partitions on the first disk, then the target topic has 3 copies on the first disk. If it stores 2 copies of its partitions on the second disk, then the target topic has 2 copies on the second disk. The remaining M-2 disks do not store data for the target topic, so the target topic has 0 copies on these remaining M-2 disks. In other words, the remaining number of disks not occupied by the target topic is 0.

[0076] In some optional implementations, the disk balancing method based on a distributed message subscription platform in this disclosure further includes: obtaining the total number of topic partitions of the target topic and the total number of topic replicas for each topic partition.

[0077] For example, the broker component of the target server contains target topic1, target topic2, target topic3, target topic4, and target topic5. Therefore, the total number of topic partitions for the corresponding target topic is 5, and each topic partition contains multiple topic replicas, for example, the total number of topic replicas is 8.

[0078] like Figure 4 The diagram shown is a simplified flowchart of a disk balancing method based on a distributed message subscription platform in an embodiment of this disclosure.

[0079] This embodiment also provides a disk balancing device based on a distributed message subscription platform. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0080] This embodiment provides a disk balancing device based on a distributed message subscription platform. The distributed message subscription platform includes multiple servers. The target server stores multiple topics and multiple disks through a broker component. Each target topic corresponds to a physically partitioned topic partition. The target topic is each of the multiple topics, and each topic partition contains multiple topic replicas. The target disk is each of the multiple disks. The target server is each of the multiple servers. The method is used on the target server, such as... Figure 5 As shown, it includes:

[0081] The first acquisition module 51 is used to acquire the total number of disks of multiple disks, the number of current theme copies of the target theme stored on the target disk, and the number of disks currently occupied by the target theme on the target disk;

[0082] The first calculation module 52 is used to calculate the average number of topic replicas corresponding to the target disk based on the total number of disks of multiple disks and the number of current topic replicas of the target topic stored on the target disk;

[0083] The second calculation module 53 is used to calculate the number of target theme replicas to be migrated on the target disk based on the current number of target theme replicas stored on the target disk and the average number of target theme replicas corresponding to the target disk.

[0084] The data migration module 54 is used to migrate the disks to be migrated to the optimal target disk based on the number of copies of the target topic stored on the target disk.

[0085] In some alternative implementations, the data migration module 54 includes:

[0086] The data judgment submodule is used to determine whether the number of replicas of the target topic to be migrated stored on the target disk is greater than zero;

[0087] The first determination submodule is used to determine the target disk as the disk to be migrated if the number of copies of the target topic stored on the target disk is greater than zero.

[0088] The second determination submodule is used to determine the target disk as the target migration object if the number of copies of the target topic to be migrated stored on the target disk is less than zero.

[0089] The data sorting submodule is used to sort the number of current theme replicas of the target migration object storage target theme in descending order and output the sorting results;

[0090] The data selection submodule is used to select the top K disks from the sorting results, and select the target disk with the smallest storage capacity from the K disks according to a preset time as the optimal target disk.

[0091] The data migration submodule is used to migrate copies of the target topic stored on the target disk to the optimal target disk.

[0092] In some alternative implementations, a disk balancing device based on a distributed message subscription platform, in Figure 5 In addition, it also includes:

[0093] The data update module 55 is used to update the current number of target theme replicas of the target theme on the target disk storage based on the number of replicas to be migrated of the target theme.

[0094] In some optional implementations, the average number of topic replicas corresponding to the target disk is calculated based on the total number of disks across multiple disks and the number of current topic replicas of the target topic stored on the target disk, using the following formula:

[0095]

[0096] Where A is the average number of subject replicas corresponding to the target disk, and D... i The number of current theme copies of the target topic stored on the target disk, where M is the total number of disks across multiple disks, and n is the current number of disks occupied by the target topic on the target disk.

[0097] In some alternative embodiments, it further includes: a second acquisition module, configured to subtract the current disk quantity occupied by the target topic on the target disk from the total disk quantity of multiple disks, so as to obtain the remaining quantity of the target disk not occupied by the target topic.

[0098] In some alternative embodiments, it further includes: a second acquisition module, configured to acquire the total number of topic partitions of the target topic and the total number of topic replicas of each topic partition.

[0099] The further function descriptions of each of the above modules and units are the same as those in the corresponding foregoing embodiments, and will not be elaborated herein.

[0100] The disk balancing device based on the distributed message subscription platform in this embodiment is presented in the form of functional units. Here, the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0101] An embodiment of the present invention further provides a computer device having the above-mentioned disk balancing device based on the distributed message subscription platform.

[0102] Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of a computer device provided by an alternative embodiment of the present invention. As shown in Figure 6 , the computer device includes: one or more processors 10, a memory 20, and an interface for connecting each component, including a high-speed interface and a low-speed interface. Each component communicates with each other through different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphic information of a GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a set of blade servers, or a multi-processor system). Figure 6 Taking one processor 10 as an example in

[0103] The processor 10 can be a central processor, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The above programmable logic device can be a complex programmable logic device, a field programmable gate array, a general array logic, or any combination thereof.

[0104] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0105] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0106] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0107] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0108] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0109] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0110] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A disk balancing method based on a distributed message subscription platform, characterized in that, The distributed message subscription platform includes multiple servers. The target server stores multiple topics and multiple disks through a broker component. Each target topic corresponds to a physically partitioned topic partition. Each topic partition contains multiple topic replicas. Each target disk is one of the multiple disks. The target server is each of the multiple servers. The method is used on the target server and includes: Obtain the total number of disks in the plurality of disks, the number of current theme copies of the target theme stored on the target disk, and the current number of disks occupied by the target theme on the target disk; Based on the total number of disks of the plurality of disks and the number of current theme replicas of the target theme stored on the target disk, calculate the average number of theme replicas corresponding to the target disk; Based on the number of current theme replicas of the target topic stored on the target disk and the average number of theme replicas corresponding to the target disk, calculate the number of target theme replicas to be migrated stored on the target disk; Based on the number of copies of the target topic to be migrated stored on the target disk, the copies to be migrated on the target disk are migrated to the optimal target disk according to the selection principle.

2. The method according to claim 1, characterized in that, Based on the number of replicas of the target topic stored on the target disk to be migrated, the disks to be migrated are migrated to the optimal target disk according to the selection principle, including: Determine whether the number of replicas of the target topic to be migrated stored on the target disk is greater than zero; If the number of copies of the target topic to be migrated stored on the target disk is greater than zero, the target disk is determined to be a disk to be migrated; If the number of copies of the target topic to be migrated stored on the target disk is less than zero, the target disk is determined to be the target migration object; Sort the number of current topic replicas of the target topic stored in the target migration object in descending order, and output the sorting result; Select the top K disks from the sorting results, and select the target disk with the smallest storage capacity from the K disks according to a preset time as the optimal target disk; The target disk stores the copy of the target topic to be migrated to the optimal target disk.

3. The method according to claim 1, characterized in that, After migrating the target subject's copy to the optimal target disk, the process further includes: Based on the number of target theme replicas to be migrated stored on the target disk, update the current number of target theme replicas stored on the optimal target disk.

4. The method according to claim 1, characterized in that, Based on the total number of disks and the number of current theme replicas of the target topic stored on the target disk, the average number of theme replicas corresponding to the target disk is calculated using the following formula: Where A is the average number of subject replicas corresponding to the target disk, and D i The target disk stores the current number of copies of the target topic, where M is the total number of disks and n is the current number of disks occupied by the target topic on the target disk.

5. The method according to claim 1, characterized in that, Also includes: The remaining number of disks not occupied by the target topic is obtained by subtracting the total number of disks on the target disk from the total number of disks on the target disk.

6. The method according to claim 1, characterized in that, Also includes: Obtain the total number of topic partitions for the target topic and the total number of topic copies for each topic partition.

7. A disk load balancing device based on a distributed message subscription platform, characterized in that, The distributed message subscription platform includes multiple servers. The target server stores multiple topics and multiple disks through a broker component. Each target topic corresponds to a physically partitioned topic partition. Each target topic is one of the multiple topics, and each topic partition contains multiple topic replicas. Each target disk is one of the multiple disks. The target server is one of the multiple servers. The device is used for the target server and includes: The data acquisition module is used to acquire the total number of disks of the multiple disks, the number of current theme copies of the target theme stored on the target disk, and the current number of disks occupied by the target theme on the target disk; The first calculation module is used to calculate the average number of topic replicas corresponding to the target disk based on the total number of disks of the plurality of disks and the number of current topic replicas of the target topic stored on the target disk; The second calculation module is used to calculate the number of target theme replicas to be migrated stored on the target disk based on the current number of target theme replicas stored on the target disk and the average number of target theme replicas corresponding to the target disk. The data migration module is used to migrate the target disk to the optimal target disk according to the number of copies of the target topic to be migrated stored on the target disk.

8. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory stores computer instructions, and the processor executes the computer instructions to perform the disk balancing method based on a distributed message subscription platform as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the disk balancing method based on a distributed message subscription platform as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the disk balancing method based on a distributed message subscription platform as described in any one of claims 1 to 6.

Images