Disk balancing method and device based on distributed message subscription platform

Abstract

The invention relates to the technical field of data disk distribution, and discloses a disk balancing method and device based on a distributed message subscription platform, and the method comprises the steps: storing a plurality of themes and basic data of a plurality of disks based on a broker component in a target server, and further calculating the number of to-be-migrated copies of a target theme stored in a target disk; and the theme copy stored in the target disk with the overhigh utilization rate is migrated, so that the problem of disk imbalance is avoided, and finally, the global optimum of the broker component is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data disk allocation, and in particular to a disk balancing method and device based on a distributed message subscription platform. Background Art

[0002] Kafka is a distributed message flow platform that supports multiple partitions and multiple copies. It can transfer large amounts of data between multiple producers and consumers. A kafka cluster consists of multiple servers, where a server is called a broker component. A cluster consists of multiple broker components, and each broker component can accommodate multiple topics called topics and mount multiple disks. The producer writes data to the topic in the broker component and stores it on the disk of the broker component. The consumer retrieves the corresponding topic data through the broker component. In the process of producing and writing data, sometimes there will be too much data written to individual disks under a broker component, resulting in high utilization, and some disks will write less data, resulting in low utilization. This will cause the broker component to have disk imbalance, which will eventually cause excessive local load and affect the overall situation. Summary of the invention

[0003] In view of this, the present invention provides a disk balancing method and device based on a distributed message subscription platform to solve the problem of uneven data writing by broker components.

[0004] In a first aspect, the present invention provides a disk balancing method based on a distributed message subscription platform, wherein the distributed message subscription platform includes multiple servers, and the target server stores multiple topics and multiple disks through a broker component, wherein the target topic corresponds to a physically divided topic partition, the target topic is each topic in the multiple topics, and each topic partition includes multiple topic copies, the target disk is each disk in the multiple disks, and the target server is each server in the multiple servers. The method is used for the target server, including:

[0005] Get the total number of disks of multiple disks, the number of current topic copies of the target topic stored on the target disk, and the current number of disks occupied by the target topic on the target disk;

[0006] Based on the total number of disks of multiple disks and the number of current topic copies of the target topic stored on the target disk, the average number of topic copies corresponding to the target disk is calculated;

[0007] Calculate the number of copies of the target topic to be migrated stored on the target disk based on the current number of copies of the target topic stored on the target disk and the average number of copies of the target topic corresponding to the target disk;

[0008] Based on the number of to-be-migrated copies of the target subject stored on the target disk, the to-be-migrated copies of the target disk are migrated to the optimal target disk according to the principle of priority.

[0009] The disk balancing method based on the distributed message subscription platform in the disclosed embodiment further calculates the number of copies to be migrated of the target topic stored on the target disk based on the basic data of multiple topics and multiple disks stored in the broker component of the target server, and migrates the topic copies stored on the target disk with excessive utilization rate, so as to avoid the problem of disk imbalance and ultimately ensure the global optimization of the broker component.

[0010] In some optional implementations, based on the number of to-be-migrated copies of the target subject stored on the target disk, the disk to be migrated is migrated to the optimal target disk according to the principle of priority, including:

[0011] Determine whether the number of copies to be migrated of the target topic stored on the target disk is greater than zero;

[0012] If the number of to-be-migrated copies of the target subject stored on the target disk is greater than zero, the target disk is determined to be a to-be-migrated disk;

[0013] If the number of copies to be migrated of the target subject stored on the target disk is less than zero, the target disk is determined to be the target migration object;

[0014] Sort the number of current topic copies of the target migration object storage target topic in descending order, and output the sorting result;

[0015] Select the top K disks from the sorting results, and select the target disk with the smallest storage capacity from the K disks as the optimal target disk according to the preset time;

[0016] Migrate the target disk storage target topic's replica to be migrated to the optimal target disk.

[0017] The disclosed embodiment improves the balance of disk usage of the broker component by dynamically searching for a target disk with a lower usage rate as the optimal target disk in the broker component and migrating the to-be-migrated copies of the target disk with a higher usage rate to the optimal target disk.

[0018] In some optional implementations, after migrating the to-be-migrated copy of the target subject stored on the target disk to the optimal target disk, the method further includes:

[0019] Based on the number of to-be-migrated copies of the target topic stored on the target disk, the optimal number of current topic copies of the target topic stored on the target disk is updated.

[0020] In some optional implementations, based on the total number of disks of the plurality of disks and the number of current theme copies of the target theme stored on the target disk, the average theme copy number corresponding to the target disk is calculated, which is performed by the following formula:

[0021]

[0022] Among them, A is the average number of topic copies corresponding to the target disk, D i The target disk stores the current number of topic copies of the target topic, M is the total number of disks of the multiple disks, and n is the current number of disks occupied by the target topic.

[0023] In some optional implementations, the disk balancing method based on a distributed message subscription platform in the embodiments of the present disclosure further includes: subtracting the current number of disks occupied by the target topic from the total number of disks of multiple disks to obtain the remaining number of target disks not occupied by the target topic.

[0024] In some optional implementations, the disk balancing method based on a distributed message subscription platform in the embodiment of the present disclosure further includes: obtaining the total number of topic partitions of the target topic and the total number of topic replicas of each topic partition.

[0025] In a second aspect, the present invention provides a disk balancing device based on a distributed message subscription platform, wherein the distributed message subscription platform includes multiple servers, and the target server stores multiple topics and multiple disks through a broker component, wherein the target topic corresponds to a physically divided topic partition, the target topic is each topic in the multiple topics, and each topic partition includes multiple topic copies, the target disk is each disk in the multiple disks, and the target server is each server in the multiple servers. The method is used for the target server, including:

[0026] A data acquisition module is used to obtain the total number of disks of multiple disks, the number of current topic copies of the target topic stored on the target disk, and the current number of disks occupied by the target topic on the target disk;

[0027] The first calculation module is used to calculate the average number of topic copies corresponding to the target disk based on the total number of disks of the multiple disks and the number of current topic copies storing the target topic on the target disk;

[0028] The second calculation module is used to calculate the number of copies to be migrated of the target subject stored on the target disk based on the current number of subject copies of the target subject stored on the target disk and the average number of subject copies corresponding to the target disk;

[0029] The data migration module is used to migrate the disk to the optimal target disk based on the number of copies of the target subject to be migrated stored in the target disk.

[0030] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the disk balancing method based on a distributed message subscription platform of the above-mentioned first aspect or any corresponding embodiment thereof.

[0031] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the disk balancing method based on a distributed message subscription platform of the above-mentioned first aspect or any corresponding embodiment thereof.

[0032] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the disk balancing method based on a distributed message subscription platform of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 It is a flowchart of a disk balancing method of a distributed message subscription platform according to an embodiment of the present invention;

[0035] Figure 2 It is a flowchart of a disk balancing method of another distributed message subscription platform according to an embodiment of the present invention;

[0036] Figure 3 It is a flowchart of a disk balancing method of another distributed message subscription platform according to an embodiment of the present invention;

[0037] Figure 4 It is a simplified flow chart of a disk balancing method of a distributed message subscription platform according to an embodiment of the present invention;

[0038] Figure 5 is a structural block diagram of a disk balancing device of another distributed message subscription platform according to an embodiment of the present invention;

[0039] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0041] The disk balancing method based on the distributed message subscription platform is applied to the distributed message subscription platform Kafka. In practical applications, Kafka is a high-real-time data processing tool. Once a failure occurs, it is easy to cause problems such as data delay and loss. For example, if the usage rate of a disk under some brokers is too high, it is easy to cause disk failure and then cause the risk of Kafka failure.

[0042] In view of this, an embodiment of the present disclosure provides a disk balancing method based on a distributed message subscription platform, and the distributed message subscription platform includes: the distributed message subscription platform includes multiple servers, and the target server stores multiple topics and multiple disks through a broker component, wherein the target topic corresponds to a physically divided topic partition, the target topic is each topic in the multiple topics, and each topic partition contains multiple topic copies, the target disk is each disk in the multiple disks, and the target server is each server in the multiple servers, and the method is used for the target server.

[0043] Specifically, each topic is a Topic topic, and each message sent to the Kafka cluster has a category, which is called a Topic. Messages are classified according to topics. Topic is essentially a directory, that is, messages on the same topic are classified into the same directory. Topic partitions are Topic partitions, and each message sent to the Kafka cluster has a category, which is called a Topic. Messages are classified according to topics. Topic is essentially a directory, that is, messages on the same topic are classified into the same directory. Kafka: A distributed message streaming platform developed by the Apache Software Foundation that supports multiple partitions and multiple copies. It can transfer large amounts of data between multiple producers and consumers. Producers are responsible for sending messages to the Kafka cluster topic, and consumers are responsible for consuming data from the Kafka cluster topic.

[0044] According to an embodiment of the present invention, a disk balancing method embodiment based on a distributed message subscription platform is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] In this embodiment, a disk balancing method based on a distributed message subscription platform is provided, which can be used for computer devices such as mobile phones, tablet computers, desktop computers, portable notebooks, servers, etc. Figure 1 is a flow chart of a disk balancing method based on a distributed message subscription platform according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0046] Step S101, obtain the total number of disks of multiple disks, the number of current theme copies of the target theme stored on the target disk, and obtain the current number of disks occupied by the target theme on the target disk.

[0047] Specifically, the number of disks is the total number of disks that the broker component of the target server can accommodate, and the total number of disks is represented by M. The target topic is the target topic, and the target topic represents the topic corresponding to the multiple topic partitions. The number of current topic copies stored on the target disk can be represented by D i It represents the number of disk copies corresponding to the target topic under the broker component. The current number of disks occupied by the target topic is represented by n, which also represents that the data of the target topic is stored in n disks of the broker component. Step S101 is equivalent to obtaining basic data in each broker component of the Kafka cluster.

[0048] Step S102, based on the total number of disks of the plurality of disks and the number of current theme copies of the target theme stored on the target disk, calculate the average number of theme copies corresponding to the target disk.

[0049] In a specific example, based on the total number of disks of multiple disks and the number of current topic copies of the target topic stored on the target disk, the average number of topic copies corresponding to the target disk is calculated, which is performed by the following formula:

[0050]

[0051] Among them, A is the average number of topic copies corresponding to the target disk, D i The target disk stores the current number of copies of the target topic, M is the total number of disks in the multiple disks, and n is the current number of disks occupied by the target topic. The target disk is each disk in the multiple disks.

[0052] Exemplarily, for example, a broker component has 9 disks storing kafka cluster data. If the total number of disks of multiple disks is M=9; a target topic has 3 disks in the broker component storing the data of the topic, then the target topic occupies the current number of target disks n=3; among the 3 disks, the target disk D1 stores the replica data of the first partition, the third partition, and the fifth partition of the target topic, then the target disk D1=3, the target disk D2 stores the replica data of the second partition and the fourth partition of the target topic, then D2=2, the target disk D3 stores the sixth partition data of the target topic, then D1=1; it can be calculated that A=ceil((D1+D2+D3) / M)=ceil((3+2+1) / 9)=1, ceil means rounding up.

[0053] Step S103, based on the current number of subject copies of the target subject stored on the target disk and the average number of subject copies corresponding to the target disk, calculate the number of copies to be migrated of the target subject stored on the target disk.

[0054] Step S104 , based on the number of to-be-migrated copies of the target subject stored on the target disk, the to-be-migrated copies of the target disk are migrated to the optimal target disk according to the principle of priority.

[0055] Specifically, a target disk whose number of subject copies of the target subject is greater than the average number of subject copies is found in the broker component, and the number of copies to be migrated of the target subject stored on the target disk is migrated.

[0056] For example, the number of copies to be migrated on the target disk is represented by B, the average number of topic copies corresponding to the target disk is represented by A, and the number of current topic copies of the target topic stored on the target disk is represented by D, then B = DA. B>0 indicates that the target topic needs to be disk balanced in the broker component, and the number of copies to be migrated on the target disk needs to be migrated to the target disk with B<0.

[0057] The disk balancing method based on the distributed message subscription platform in the disclosed embodiment further calculates the number of copies to be migrated of the target topic stored on the target disk based on the basic data of multiple topics and multiple disks stored in the broker component of the target server, and migrates the topic copies stored on the target disk with excessive utilization rate, so as to avoid the problem of disk imbalance and ultimately ensure the global optimization of the broker component.

[0058] In this embodiment, a disk balancing method based on a distributed message subscription platform is provided, which can be used for computer devices such as mobile phones, tablet computers, desktop computers, portable notebooks, servers, etc. Figure 2is a flow chart of a disk balancing method based on a distributed message subscription platform according to an embodiment of the present invention. In the above step S103, based on the current number of subject copies of the target subject stored on the target disk and the average number of subject copies corresponding to the target disk, the number of copies to be migrated of the target subject stored on the target disk is calculated, such as Figure 2 As shown, the process includes the following steps:

[0059] Step S1031, determining whether the number of copies to be migrated of the target subject stored on the target disk is greater than zero.

[0060] Specifically, the number of copies to be migrated of the target topic stored on the target disk is B, and B>0 indicates that the target topic needs to be disk balanced in the broker component.

[0061] Step S1032: If the number of to-be-migrated copies of the target subject stored on the target disk is greater than zero, the target disk is determined to be a to-be-migrated disk.

[0062] For example, the multiple disks contained in the broker component are target disks E1, E2, E3, and E4, respectively. If the number of copies to be migrated of the target disks E2 and E4 is greater than zero, the target disks E2 and E4 are determined to be disks to be migrated.

[0063] Step S1033: If the number of copies to be migrated of the target subject stored on the target disk is less than zero, the target disk is determined to be the target migration object.

[0064] For example, in the above example, the target disks E1, E2, and E3 are all less than zero, and the target disks E1, E2, and E3 are used as target migration objects.

[0065] Step S1034, sorting the current topic replica quantities of the target migration object storage target topic in descending order, and outputting the sorting result.

[0066] For example, the current subject replica numbers of the target migration objects E1, E2, and E3 in the above are 5, 6, and 8, respectively. Sorted in descending order, the corresponding output target migration objects are E3, E2, and E1, respectively.

[0067] Step S1035 , selecting the top K disks from the sorting results, and selecting the target disk with the smallest storage capacity from the K disks as the optimal target disk according to a preset time.

[0068] For example, if K=2, the first two target disks E3 and E2 are selected from the target migration objects E3, E2, and E1 in the above example as the optimal target disks. The optimal target disk can be represented by K.

[0069] If the number of copies to be migrated of the target subject stored on the target disk is zero, it means that the usage rate of the target disk just meets the requirement and there is no need to migrate the copies to be migrated of the target disk.

[0070] The disclosed embodiment executes the above steps S1031-S1035 to dynamically search for a target disk with a lower usage rate as the optimal target disk in the broker component, and migrates the to-be-migrated copies of the target disk with a higher usage rate to the optimal target disk, thereby improving the disk usage balance of the broker component.

[0071] In some optional embodiments, such as Figure 3 As shown, after the target disk stores the target subject's to-be-migrated copy to the optimal target disk, the following steps are also included:

[0072] Step, S105, based on the number of to-be-migrated copies of the target subject stored on the target disk, update the optimal number of current subject copies of the target subject stored on the target disk.

[0073] For example, the target disks with target topic B<0 on the broker component are sorted from small to large according to the number of copies D of the target topic on the target disk, and the target disk with the smallest storage increment in the recent time range is selected from the first k disks with the smallest size as the optimal target migration object W, where the k value is configurable, for example, k=5. Migrate a copy of the disk with target topic B>0 on the broker component to the target disk W. Each time a copy is migrated, the number of copies of the target topic on the target disk W is updated to D=D+1, and the target disks with larger usage rates are continuously migrated according to the above steps S1031-S1035 until all target topics with B>0 are changed to B<=0.

[0074] In some optional implementations, the disk balancing method based on a distributed message subscription platform in the embodiments of the present disclosure further includes: subtracting the current number of disks occupied by the target topic from the total number of disks of multiple disks to obtain the remaining number of target disks not occupied by the target topic.

[0075] For example, the broker component of the target server has M disks, and a target topic stores 3 partition copies on the first disk. Then the number of copies of the target topic on the first disk is 3, and the number of copies of 2 partitions is stored on the second disk. Then the number of copies of the target topic on the second disk is 2. The remaining M-2 disks do not store the data of the target topic. Then the number of copies of the target topic on the remaining M-2 disks is 0, that is, the remaining number of target disks not occupied by the target topic is 0.

[0076] In some optional implementations, the disk balancing method based on a distributed message subscription platform in the embodiment of the present disclosure further includes: obtaining the total number of topic partitions of the target topic and the total number of topic replicas of each topic partition.

[0077] For example, the broker component of the target server includes target topic1, target topic2, target topic3, target topic4, and target topic5. Therefore, the total number of topic partitions of the corresponding target topics is 5, and each topic partition contains multiple topic replicas. For example, the total number of topic replicas is 8.

[0078] like Figure 4 As shown, it is a simplified flow chart of the disk balancing method based on the distributed message subscription platform in the embodiment of the present disclosure.

[0079] In this embodiment, a disk balancing device based on a distributed message subscription platform is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0080] This embodiment provides a disk balancing device based on a distributed message subscription platform, the distributed message subscription platform includes multiple servers, the target server stores multiple topics and multiple disks through a broker component, wherein the target topic corresponds to a physically divided topic partition, the target topic is each topic in the multiple topics, and each topic partition includes multiple topic copies, the target disk is each disk in the multiple disks, the target server is each server in the multiple servers, and the method is used for the target server, such as Figure 5 As shown, including:

[0081] The first acquisition module 51 is used to acquire the total number of disks of the multiple disks, the number of current theme copies of the target theme stored in the target disk, and the current number of disks occupied by the target theme on the target disk;

[0082] A first calculation module 52 is used to calculate the average number of topic copies corresponding to the target disk based on the total number of disks of the multiple disks and the number of current topic copies storing the target topic on the target disk;

[0083] The second calculation module 53 is used to calculate the number of copies to be migrated of the target subject stored on the target disk based on the current number of subject copies of the target subject stored on the target disk and the average number of subject copies corresponding to the target disk;

[0084] The data migration module 54 is used to migrate the disk to be migrated to the optimal target disk according to the principle of priority based on the number of copies to be migrated of the target subject stored in the target disk.

[0085] In some optional implementations, the data migration module 54 includes:

[0086] The data judgment submodule is used to judge whether the number of copies to be migrated of the target subject stored on the target disk is greater than zero;

[0087] A first determination submodule is used to determine that the target disk is a disk to be migrated if the number of copies to be migrated of the target subject stored in the target disk is greater than zero;

[0088] The second determination submodule is used to determine the target disk as a target migration object if the number of to-be-migrated copies of the target subject stored on the target disk is less than zero;

[0089] The data sorting submodule is used to sort the number of current topic copies of the target migration object storage target topic in descending order and output the sorting result;

[0090] The data selection submodule is used to select the top K disks from the sorting results, and select the target disk with the smallest storage capacity from the K disks as the optimal target disk according to a preset time;

[0091] The data migration submodule is used to migrate the to-be-migrated copy of the target subject stored on the target disk to the optimal target disk.

[0092] In some optional implementations, a disk balancing device based on a distributed message subscription platform is used. Figure 5 Also included:

[0093] The data updating module 55 is used to update the optimal number of current theme copies of the target disk storing the target theme based on the number of to-be-migrated copies of the target theme stored on the target disk.

[0094] In some optional implementations, based on the total number of disks of the plurality of disks and the number of current theme copies of the target theme stored on the target disk, the average theme copy number corresponding to the target disk is calculated, which is performed by the following formula:

[0095]

[0096] Among them, A is the average number of topic copies corresponding to the target disk, D i The target disk stores the current number of topic copies of the target topic, M is the total number of disks of the multiple disks, and n is the current number of disks occupied by the target topic.

[0097] In some optional implementations, it also includes: a second acquisition module, which is used to subtract the current number of disks occupied by the target subject from the total number of disks of multiple disks to obtain the remaining number of target disks not occupied by the target subject.

[0098] In some optional implementations, the method further includes: a second acquisition module, configured to acquire the total number of topic partitions of the target topic and the total number of topic replicas of each topic partition.

[0099] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0100] The disk balancing device based on the distributed message subscription platform in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0101] An embodiment of the present invention further provides a computer device having the above-mentioned disk balancing device based on the distributed message subscription platform.

[0102] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0103] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0104] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0105] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0106] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0107] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0108] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0109] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0110] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A disk balancing method based on a distributed message subscription platform, characterized in that: The distributed message subscription platform includes multiple servers, and the target server stores multiple topics and multiple disks through a broker component, wherein the target topic corresponds to a physically divided topic partition, the target topic is each topic in the multiple topics, and each topic partition contains multiple topic copies, the target disk is each disk in the multiple disks, and the target server is each server in the multiple servers. The method is used for the target server, including: Obtain the total number of disks of the multiple disks, the number of current theme copies of the target theme stored on the target disk, and the current number of disks occupied by the target theme on the target disk; Based on the total number of disks of the multiple disks and the number of current theme copies of the target theme stored on the target disk, calculate the average number of theme copies corresponding to the target disk; Calculate the number of to-be-migrated copies of the target subject stored on the target disk based on the current number of subject copies stored on the target disk and the average number of subject copies corresponding to the target disk; Based on the number of to-be-migrated copies of the target subject stored on the target disk, the to-be-migrated copies of the target disk are migrated to the optimal target disk according to the principle of priority.

2. The method according to claim 1, characterized in that Based on the number of to-be-migrated copies of the target subject stored on the target disk, the disk to be migrated is migrated to the optimal target disk according to the principle of priority, including: Determine whether the number of copies to be migrated of the target subject stored on the target disk is greater than zero; If the number of to-be-migrated copies of the target subject stored in the target disk is greater than zero, determining that the target disk is a to-be-migrated disk; If the number of to-be-migrated copies of the target subject stored on the target disk is less than zero, determining the target disk as a target migration object; Sorting the number of current topic copies of the target topic stored in the target migration object in descending order, and outputting the sorting result; Selecting K disks with the highest ranking from the sorting results, and selecting a target disk with the smallest storage capacity from the K disks as the optimal target disk according to a preset time; The target disk stores the to-be-migrated copy of the target subject to the optimal target disk.

3. The method according to claim 1, characterized in that After migrating the to-be-migrated copy of the target subject stored on the target disk to the optimal target disk, the method further includes: Based on the number of to-be-migrated copies of the target subject stored on the target disk, the number of current subject copies of the target subject stored on the optimal target disk is updated.

4. The method according to claim 1, characterized in that: Based on the total number of disks of the multiple disks and the number of current theme copies of the target theme stored on the target disk, the average number of theme copies corresponding to the target disk is calculated, which is performed by the following formula: Where A is the average number of subject copies corresponding to the target disk, D i The target disk stores the current number of theme copies of the target theme, M is the total number of disks of the multiple disks, and n is the current number of disks occupied by the target theme.

5. The method according to claim 1, characterized in that Also includes: The total number of disks of the plurality of disks minus the current number of disks occupied by the target subject can be used to obtain the remaining number of disks not occupied by the target subject.

6. The method according to claim 1, characterized in that Also includes: Get the total number of topic partitions and the total number of topic replicas for each topic partition of the target topic.

7. A disk balancing device based on a distributed message subscription platform, characterized in that: The distributed message subscription platform includes multiple servers, and the target server stores multiple topics and multiple disks through a broker component, wherein the target topic corresponds to a physically divided topic partition, the target topic is each topic in the multiple topics, and each topic partition includes multiple topic copies, the target disk is each disk in the multiple disks, and the target server is each server in the multiple servers. The device is used for the target server, including: A data acquisition module, used to acquire the total number of disks of the multiple disks, the number of current theme copies of the target theme stored on the target disk, and the current number of disks occupied by the target theme on the target disk; A first calculation module is used to calculate the average number of theme copies corresponding to the target disk based on the total number of disks of the multiple disks and the number of current theme copies of the target theme stored on the target disk; A second calculation module is used to calculate the number of copies to be migrated that store the target topic on the target disk based on the current number of topic copies that store the target topic on the target disk and the average number of topic copies corresponding to the target disk; The data migration module is used to migrate the disks to be migrated to the optimal target disk according to the principle of priority based on the number of copies to be migrated of the target subject stored on the target disk.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the disk balancing method based on a distributed message subscription platform as described in any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the disk balancing method based on a distributed message subscription platform according to any one of claims 1 to 6.

10. A computer program product, characterized in that It comprises computer instructions, and the computer instructions are used to enable a computer to execute the disk balancing method based on a distributed message subscription platform as described in any one of claims 1 to 6.

Images