Resource management method, device, electronic device and storage medium

Through automated replica data migration strategies, the problem of unbalanced resource utilization in distributed message systems is solved, system stability and resource allocation efficiency are improved, and high costs and low efficiency brought about by human allocation are avoided.

CN115277710BActive Publication Date: 2024-11-15MASHANG CONSUMER FINANCE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210614254.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-31
Publication Date
2024-11-15
Estimated Expiration
2042-05-31

AI Technical Summary

Technical Problem

The prior art uses broker's resource usage in a distributed message system, resulting in low resource allocation efficiency, unable to effectively balance the resource utilization rate of service nodes, affecting system stability.

Method used

By obtaining the resource allocation situation and resource floating parameters of the distributed message system, the resource utilization range is automatically determined, the service nodes and target nodes that need to be migrated, the replica migration strategy is generated, the automatic replica data migration is realized, and the resource usage of service nodes is balanced. Rate.

Benefits of technology

The stability and resource allocation efficiency of the distributed message system are improved, high cost and inefficiency problems caused by human intervention are avoided, and automatic balance of resource utilization of service nodes is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115277710B_ABST
    Figure CN115277710B_ABST
Patent Text Reader

Abstract

The present application discloses a resource management method, device, electronic device and storage medium, the method comprising: obtaining the resource allocation of a distributed message system, and obtaining resource floating parameters between multiple service nodes; determining the resource utilization range allowed by the distributed message system according to the resource allocation and resource floating parameters; determining the service nodes to be migrated in the distributed message system that need to perform replica data migration based on the resource utilization range, and determining the target service nodes corresponding to the service nodes to be migrated based on the resource utilization range; generating a replica migration strategy based on the service nodes to be migrated and the target service nodes, and performing replica data migration between the service nodes to be migrated and the target service nodes based on the replica migration strategy. In the above manner, the present application can quickly balance the resource utilization of service nodes in the distributed message system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of resource management, and in particular to a resource management method, device, electronic device and storage medium. Background Art

[0002] Nowadays, with the rapid development of the Internet, various terminal applications are constantly being updated and iterated. In these numerous applications, in order to achieve communication between application systems, it is inevitable to use Kafka, a message queue middleware for distributed messaging, to achieve asynchronous messaging between our applications. In this context, especially in the current era of big data, we rely more on Kafka, and with the continuous increase in application system traffic, the stability of our message middleware is also constantly improving.

[0003] A distributed messaging system usually includes multiple service nodes. For example, Kafka can include multiple brokers. A broker can be regarded as a service node, and each service node can store a copy of the target data. In order to improve the stability of Kafka, the existing practice is to manually allocate the resources used by the broker in Kafka (which can be regarded as a service node in the distributed messaging system), that is, to manually determine whether it is necessary to perform copy migration of the copy data stored in each broker based on the resource usage of the broker. However, manual judgment and allocation leads to low efficiency in allocating resources. Summary of the invention

[0004] The present application provides a resource management method, device, electronic device and storage medium, which can quickly balance the resource utilization rate of service nodes in a distributed message system.

[0005] On the one hand, the present application provides a resource management method, including: obtaining the resource allocation situation of a distributed message system, and obtaining the resource floating parameters between multiple service nodes; determining the resource utilization range allowed by the distributed message system based on the resource allocation situation and the resource floating parameters; determining the service nodes to be migrated in the distributed message system that need to perform replica data migration based on the resource utilization range, and determining the target service nodes corresponding to the service nodes to be migrated based on the resource utilization range; generating a replica migration strategy based on the service nodes to be migrated and the target service nodes, and performing replica data migration between the service nodes to be migrated and the target service nodes based on the replica migration strategy.

[0006] On the one hand, another technical solution adopted by the present application is: to provide a resource management device, the device comprising:

[0007] The acquisition module is used to obtain the resource allocation status of the distributed message system and the resource floating parameters between multiple service nodes;

[0008] A first determination module is used to determine the resource usage rate range allowed by the distributed messaging system according to the resource allocation situation and the resource floating parameter;

[0009] A second determination module is used to determine a service node to be migrated in the distributed messaging system that needs to perform replica data migration based on a resource usage rate range, and to determine a target service node corresponding to the service node to be migrated based on the resource usage rate range;

[0010] The processing module is used to generate a replica migration strategy according to the service node to be migrated and the target service node, and perform replica data migration between the service node to be migrated and the target service node based on the replica migration strategy.

[0011] On the one hand, another technical solution adopted by the present application is: to provide an electronic device, the electronic device includes a memory and a processor, the memory is used to store program data, and the processor is used to execute the program data to implement the resource management method as described above.

[0012] On the one hand, another technical solution adopted by the present application is: to provide a computer-readable storage medium, which stores program data, and the program data is used to implement the resource management method as described above when executed by a processor.

[0013] The beneficial effect of the present application is that, unlike the prior art, the present application does not require manual deployment of replica data, but is executed automatically. Specifically, the present application first obtains the resource allocation and resource floating parameters of the distributed message system; determines the resource utilization range allowed by the distributed message system according to the resource allocation and resource floating parameters; determines the service nodes to be migrated in the distributed message system that need to migrate replica data based on the resource utilization range, and determines the target service nodes corresponding to the service nodes to be migrated based on the resource utilization range; generates a replica migration strategy based on the service nodes to be migrated and the target service nodes, and performs replica data migration between the service nodes to be migrated and the target service nodes based on the replica migration strategy.

[0014] That is to say, the present application uses the resource utilization range to quickly discover and detect service nodes with skewed resource utilization in the distributed message system, and quickly migrates copies based on the relationship between the resource utilization of the service node and the resource utilization range, so that the resource utilization of all service nodes in the distributed message system is balanced. This not only improves the stability of the distributed message system, but also improves the efficiency of resource allocation, avoiding the high cost and low efficiency caused by the prior art of adjusting the resource utilization of service nodes through human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0016] Figure 1 It is a schematic diagram of the structure of the resource management system provided by this application;

[0017] Figure 2 It is a flowchart of an embodiment of a resource management method provided by the present application;

[0018] Figure 3 is a flow chart of a method for determining a resource usage rate range in an optional embodiment;

[0019] Figure 4 It is a flow chart of the migration strategy in the resource management method provided by this application;

[0020] Figure 5 is another flow chart of the migration strategy in the resource management method provided by this application;

[0021] Figure 6 It is a schematic diagram of the structure of the resource management device provided by this application;

[0022] Figure 7 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The following is an explanation of the professional terms and technical abbreviations involved in this application.

[0025] Kafka: is a distributed publish-subscribe messaging system that can process all action stream data of consumers on the website. Broker: is essentially a commit log that can only append messages. Generally speaking, a Kafka cluster contains one or more brokers. A broker can be regarded as a service node.

[0026] The resource management method provided in this application is applied to distributed messaging systems. Figure 1 , is a schematic diagram of the structure of a distributed messaging system provided in an embodiment of the present application. Figure 1 The distributed messaging system shown includes multiple service nodes 101 and a management node 102 . The management node 102 may be a node completely different from the service node 101 , and the management node 102 may also be any service node among the service nodes 101 .

[0027] The service nodes 101 may be connected to each other, and each service node 101 is connected to the management node 102. Each service node 101 may store copy data of the target data, the copy data may be represented as copy, and the number of target data is at least one. The management node 102 is used to manage the service nodes 101, such as managing the migration of the copy data in each service node 101.

[0028] Optionally, the service node 101 and the management node 102 may be devices of the same type, such as both terminal devices or both servers; or, the service node 101 and the management node 102 may be different devices in the terminal device and the server respectively. The terminal devices mentioned above may include laptops, intelligent interactive devices, vehicle-mounted devices, etc.; the server may include an independent physical server, a server cluster, or a cloud server capable of cloud computing.

[0029] Based on the above-mentioned distributed system, the present application provides a resource management method, which can be applied to the above-mentioned distributed messaging system. The distributed messaging system includes multiple service nodes and management nodes, each service node is used to store copy data of the target data, and the resource management method is executed by the management node.

[0030] Please refer to Figure 2 , Figure 2 : is a flowchart of the first embodiment of the resource management method provided by the present application. The method may include the following steps:

[0031] Step S110: Obtain resource allocation status and resource floating parameters of the distributed message system.

[0032] Among them, the distributed messaging system can be a Kafka cluster, and the service node in the distributed messaging system can be a broker. For ease of understanding, the embodiments of the present application are explained by taking the distributed messaging system as a Kafka cluster and the service node as a broker as an example.

[0033] For example, resource allocation can be the number of replicas (also referred to as replicas in the following description) allocated to each broker in the Kafka cluster. For example, assuming that a Kafka cluster contains 3 brokers, the resource allocation of the Kafka cluster can be that broker 1 is allocated 10 replicas, broker 2 is allocated 12 replicas, and broker 3 is allocated 8 replicas.

[0034] The resource floating parameter is pre-set and is relatively flexible. For example, different resource floating parameters can be set for each Kafka cluster, or multiple Kafka clusters can share one resource floating parameter.

[0035] After the resource allocation and resource floating parameters of the distributed messaging system are obtained, the resource usage range is further obtained, that is, the resource usage range allowed by the distributed messaging system is determined according to the resource allocation and resource floating parameters in the following step S120.

[0036] Step S120: Determine the resource usage rate range allowed by the distributed messaging system according to the resource allocation situation and the resource floating parameter.

[0037] The resource usage range allowed by the distributed messaging system can be said to be a relatively balanced usage range set for each service node or server in the distributed messaging system. Within this range, the comprehensive resource utilization of the distributed messaging system can reach a more reasonable level. For example, when it is found that the resource utilization of a broker in the Kafka cluster is not within this range, the resources of the broker can be managed in a timely manner.

[0038] Exemplarily, the average resource utilization of the broker can be calculated based on the resource allocation situation, and the resource utilization range can be calculated in combination with the resource floating parameters; or the average resource utilization of the broker can be calculated based on the resource allocation situation, the allowable floating rate can be deduced based on the resource floating parameters, and the resource utilization range can be calculated. There can be many ways to calculate the resource utilization range, and this application does not limit it here.

[0039] After determining the resource utilization range according to the above steps, it is necessary to further determine the service node to be migrated and the target service node, that is, through the following step S130, determine the service node to be migrated in the distributed message system that needs to perform copy data migration based on the resource utilization range, and determine the target service node corresponding to the service node to be migrated based on the resource utilization range.

[0040] Step S130: determining a service node to be migrated that needs to perform replica data migration in the distributed messaging system based on the resource usage rate range, and determining a target service node corresponding to the service node to be migrated based on the resource usage rate range.

[0041] In a specific implementation, the resource usage of the brokers in the Kafka cluster can be compared with the resource usage range to determine the service nodes to be migrated that require replica data migration in the distributed messaging system.

[0042] Specifically, if the broker's resource utilization is within the resource utilization range, no migration is required; if the broker's resource utilization is outside the resource utilization range, it is determined to be a service node to be migrated that requires replica data migration; and the target service node is the service node whose resource utilization can be maintained within the resource utilization range after replica migration (either in or out) with the service node to be migrated.

[0043] Among them, the number of target service nodes can be one or more, but under normal circumstances, after the computer calculates a target service node corresponding to the current service node to be migrated, in order not to affect the replica migration of other service nodes to be migrated, it will generally stop calculating other target service nodes that meet the replica migration of the current service node to be migrated, and continue to screen target service nodes that can be migrated for replica migration for other service nodes to be migrated.

[0044] Step S140: Generate a replica migration strategy according to the service node to be migrated and the target service node, and perform replica data migration between the service node to be migrated and the target service node based on the replica migration strategy.

[0045] After selecting the appropriate target service node, further replica data migration is performed between the service node to be migrated and the target service node. For example, if the resource usage rate of the service node to be migrated is smaller than the minimum value of the resource usage rate range, then according to the difference between the resource usage rate ranges of the service node to be migrated, a target service node capable of replica migration is selected from the service nodes with a relatively large resource usage rate, and a replica migration strategy is generated, such as appropriately migrating some qualified replicas in the target service node to the service node to be migrated, that is, completing the replica data migration between the service node to be migrated and the target service node according to the replica migration strategy; on the contrary, if the resource usage rate of the broker is larger than the maximum value of the resource usage rate range, then according to the difference between the resource usage rate ranges of the service node to be migrated, a target service node capable of replica migration is selected from the service nodes with a relatively small resource usage rate, and a replica migration strategy is generated, such as appropriately migrating some qualified replicas in the service node to be migrated to the target service node, that is, completing the replica data migration between the service node to be migrated and the target service node according to the replica migration strategy.

[0046] The beneficial effect of the first embodiment of the present invention is as follows: different from the prior art, the present application first obtains the resource allocation situation and resource floating parameters of the distributed message system; determines the resource utilization rate range allowed by the distributed message system according to the resource allocation situation and the resource floating parameters; determines the service nodes to be migrated in the distributed message system that need to perform replica data migration based on the resource utilization rate range, and determines the target service nodes corresponding to the service nodes to be migrated based on the resource utilization rate range; generates a replica migration strategy according to the service nodes to be migrated and the target service nodes, and performs replica data migration between the service nodes to be migrated and the target service nodes based on the replica migration strategy, so that the resource utilization rate of each service node in the distributed message system is maintained within the set allowable resource utilization rate range, which can avoid the situation where the distributed message system is unavailable due to excessive resource utilization of some service nodes (or excessive load in some servers), thereby improving the stability of the distributed message system.

[0047] In addition, the present application can automatically obtain the resource usage of each service node in the distributed message system through monitoring, quickly discover and detect the service nodes with skewed resource usage in the distributed message system, and promptly perform replica migration based on the relationship between the resource usage of the service node and the resource usage range, thereby avoiding the high cost and low efficiency problems caused by the prior art of allocating the resource usage of the service nodes through human intervention in order to achieve a balanced resource usage of all service nodes in the distributed message system.

[0048] In some implementations, the method for determining the resource usage range may include the following steps, which may be specifically referred to in Figure 3 , Figure 3 This is a flow chart of a method for determining a resource usage range in an optional embodiment, which specifically includes:

[0049] Step S210: Obtain a preset balance threshold of the distributed messaging system, and obtain a resource usage rate of each service node in a plurality of service nodes in the distributed messaging system.

[0050] Step S220: Determine a resource utilization range according to a preset balance threshold and resource utilization.

[0051] In step S210 and step S220, the preset balance threshold may be a reference value of the system by default, or may be modified to an arbitrary value by receiving a user instruction, including but not limited to 1.1, 1.05, 1.15, etc. The resource utilization rate of all service nodes in the distributed messaging system, i.e., the resource utilization rate of each broker in the Kafka cluster, is obtained through monitoring.

[0052] For example, the corresponding indicator data can be obtained through a REST API interface, that is, the current resource usage of each broker in the Kafka cluster can be obtained, and the resource utilization rate of the broker is the ratio of the current resource usage to the allocated resource of each broker.

[0053] For example, a Kafka cluster contains 3 brokers. The resource allocation of the Kafka cluster can be that broker1 is allocated 10 copies, broker2 is allocated 12 copies, and broker3 is allocated 8 copies. Through monitoring, it can be obtained that broker1 has used 5 copies, broker2 has used 8 copies, and broker3 has used 6 copies. The corresponding resource utilization rate of broker1 is 50%, the corresponding resource utilization rate of broker2 is 66%, and the corresponding resource utilization rate of broker3 is 75%.

[0054] In addition, it should be noted that after obtaining the resource utilization rate of each broker, the average resource utilization rate of the brokers in the Kafka cluster can be calculated. Combined with the preset balance threshold obtained, the resource utilization rate range can be calculated.

[0055] The resource usage rate range includes a minimum resource usage rate b1 and a maximum resource usage rate b2, which can also be said to be a minimum value b1 of the resource usage rate range and a maximum value b2 of the resource usage rate range.

[0056] Exemplarily, the resource usage range {b1, b2} may be calculated using the following formula:

[0057]

[0058] Wherein, m is the average resource utilization rate, and x is the preset balance threshold.

[0059] For example, if the preset balance threshold x is 1.1, and the monitoring and detection calculation shows that broker1=60%, broker2=81%, and broker3=60% in the current Kafka cluster, then the average resource utilization rate m of these three brokers is 67%, which can be calculated according to the formula:

[0060] lowThresholds: b1 = 67% - (1.1-1) * 67% = 60.3%;

[0061] upperThresholds: that is, b2=67%+(1.1-1)*67%=73.3%;

[0062] Among them, 1 is a fixed value.

[0063] The resource utilization rate range finally determined by calculation is {60.3%, 73.3%}.

[0064] Alternatively, assuming that the preset balance threshold x is 1.05, and through monitoring and detection, broker1=60%, broker2=81%, and broker3=60% in the Kafka cluster are calculated, then the average resource utilization rate m of these three brokers is 67%, calculated according to the formula:

[0065] lowThresholds: b1 = 67% - (1.05-1) * 67% = 63.65%;

[0066] upperThresholds: that is, b2=67%+(1.05-1)*67%=70.35%;

[0067] Among them, 1 is a fixed value.

[0068] The resource utilization range finally determined by calculation is {63.65%, 70.35%}.

[0069] Alternatively, in some embodiments, the resource usage range may be determined based on the average resource usage and the allowable floating rate. The specific calculation process may include the following two steps:

[0070] 1) The floating rate threshold is subtracted from the allowed floating rate, and the average resource utilization rate is multiplied by the subtraction result to obtain the minimum resource utilization rate.

[0071] The floating rate threshold and the allowed floating rate are summed, and the average resource utilization rate and the summed result are multiplied to obtain the maximum resource utilization rate.

[0072] The allowable floating rate may be first obtained through a preset balance threshold, and the floating rate threshold may be a fixed value 1 set by the system.

[0073] For ease of understanding, the above two steps can be directly converted into the following formula:

[0074]

[0075] Among them, m is the average resource utilization rate, f is the allowed floating rate, and the resource utilization rate range is {b1, b2}.

[0076] For example, assuming that the preset balance threshold x is 1.1, it means that the resource usage rate of the broker after balancing can fluctuate up and down by 0.1, that is, the allowed floating rate f is 0.1.

[0077] If monitoring detects that broker1=60%, broker2=81%, and broker3=60% in the Kafka cluster, then the average resource utilization rate m of these three brokers is 67%, calculated according to the formula:

[0078] lowThresholds: b1 = 67% * (1-0.1) = 60.3%;

[0079] upperThresholds: that is, b2=67%*(1+0.1)=73.3%;

[0080] That is, the calculated resource utilization range is {60.3%, 73.3%}.

[0081] Alternatively, assuming that the preset balance threshold x is 1.05, this means that the resource usage of the broker after balancing can fluctuate up and down by 0.05, that is, the allowed floating rate f is 0.05.

[0082] If monitoring detects that broker1=60%, broker2=81%, and broker3=60% in the Kafka cluster, then the average resource utilization rate m of these three brokers is 67%, calculated according to the formula:

[0083] lowThresholds: b1 = 67%*(1-0.05) = 63.65%;

[0084] upperThresholds: that is, b2=67%*(1+0.05)=70.35%;

[0085] That is to say, the resource utilization rate range finally calculated is {63.65%, 70.35%}.

[0086] It should be noted that since the brokers in a Kafka cluster are not limited and can include multiple brokers or only one broker, there are two situations for calculating the average resource usage.

[0087] For example, a Kafka cluster contains 3 brokers. The monitoring and detection calculations show that broker1 = 60%, broker2 = 81%, and broker3 = 60%. The average resource utilization rate m of these three brokers is 67%;

[0088] When a Kafka cluster contains only one broker, assuming that the resource utilization rate of the broker = 80%, then the average resource utilization rate is equal to 80%.

[0089] The above embodiment obtains the resource allocation and usage of each broker (such as CPU, DISK, MEMORY) in the Kafka cluster through monitoring, and quickly detects whether the broker needs to perform replica migration based on the determined resource usage range, so that the Kafka cluster can achieve replica rebalancing and ensure that the Kafka cluster will not be unavailable due to uneven allocation of brokers.

[0090] In addition, it should be noted that, in some embodiments, in order to ensure that both parties performing replica migration (i.e., the service node to be migrated and the target service node) are within the resource utilization range after migration, after determining that the resource utilization of the current broker is not within the resource utilization range calculated in the above steps, the next step of operation, referred to as pre-check, can be performed, and the final target service node can be determined based on the results of the pre-check.

[0091] The pre-checking process may include the following specific steps: Figure 4 , Figure 4 It is a flowchart of pre-checking, which may include:

[0092] Step S310: Based on the resource utilization range, the resource utilization of the remaining service nodes, and the relationship between the resource utilization of the service node to be migrated and the resource utilization range, determine at least one candidate service node from the remaining service nodes, where the remaining service nodes refer to other service nodes among the multiple service nodes except the service node to be migrated.

[0093] Step S320: determining a target service node from the at least one candidate service node according to the resource usage rate of each candidate service node and the resource occupancy rate of the replica data to be migrated.

[0094] Step S330: If the relationship between the resource utilization rate of the service node to be migrated and the resource utilization rate range is: the resource utilization rate range of the service node to be migrated is less than the minimum resource utilization rate, then the service node whose resource utilization rate among the remaining service nodes is greater than the minimum resource utilization rate is taken as at least one candidate service node.

[0095] Step S340: Determine the result of adding the resource usage rate of the service node to be migrated and the resource occupancy rate of the replica data to be migrated in each candidate service node;

[0096] Step S350: Determine the candidate service node corresponding to the addition operation result within the resource usage rate range as the target service node.

[0097] Step S360: Generate a migration strategy; embed the replica data in the target service node into the service node to be migrated based on the migration strategy.

[0098] For ease of understanding, a service node whose resource utilization rate is greater than the minimum resource utilization rate may be taken as a first candidate service node, and a service node whose resource utilization rate is less than the maximum resource utilization rate may be taken as a second candidate service node. In this way, the candidate service nodes include the first candidate service node and the second candidate service node.

[0099] In step S310-step S360, illustratively, if the resource usage of the current service node to be migrated (i.e., the current broker to be migrated) is less than the minimum value of the resource usage range, it is necessary to traverse the first candidate brokers and select a suitable replica to migrate to the current broker to be migrated. If the resource usage of one of the first candidate brokers is within the resource usage range after migrating out the replica, then the first candidate broker is the target broker for the migration of the replica. It should be noted that if the resource usage of the first candidate broker is not within the resource usage range after migrating out the replica, it is necessary to repeatedly select a suitable replica (the replica here can be another replica in the same first candidate broker or a replica in another first candidate broker) to migrate to the current broker to be migrated until the resource usage of the first candidate broker is within the resource usage range.

[0100] For example, assuming that the preset balance threshold x is 1.05, this means that the usage rate after balance can fluctuate up and down by 0.05, that is, the allowed floating rate f is 0.05.

[0101] If monitoring detects that broker1=60%, broker2=81%, and broker3=60% in the Kafka cluster, then the average resource utilization rate m of these three brokers is 67%, calculated according to the formula:

[0102] lowThresholds: b1 = 67%*(1-0.05) = 63.65%;

[0103] upperThresholds: that is, b2=67%*(1+0.05)=70.35%;

[0104] That is, the resource usage range is {63.65%, 70.35%}.

[0105] At this time, the three brokers in the Kafka cluster: broker1=60%, broker2=81%, broker3=60% are all outside the resource usage range of {63.65%, 70.35%}, so they all need to migrate their replicas.

[0106] Assuming that the current broker to be migrated is broker1=60%, which is less than the minimum value (lowThresholds) of the resource utilization range b1=63.65%, it is necessary to screen out the service nodes whose resource utilization is greater than the minimum value of the resource utilization range among all service nodes as the first candidate service nodes, that is, all brokers whose resource utilization is greater than 63.65% are the first candidate brokers. By calculating that the difference between broker1=60% and b1=63.65% is 3.65%, all replica sets of the first candidate broker are traversed, and the replicas whose resource utilization accounts for more than 3.65% and less than 10.35% of the total resources of the current broker to be migrated are screened out, and the replicas are determined to be the target replicas. If the resource utilization of the first candidate broker itself meets the resource utilization range after the replica that meets the above conditions is migrated, the first candidate broker currently traversed is determined to be the target broker.

[0107] That is to say, after the candidate broker migrates a replica whose resource utilization accounts for more than 3.65% and less than 10.35% of the total resources of the current broker to be migrated, the candidate broker meets the resource utilization range and does not cause the overall replica distribution of the cluster to violate the restrictions of other algorithms (because generally speaking, more than one algorithm is used for balancing). In this case, the candidate broker becomes the target broker and the migrated replica becomes the target replica.

[0108] Alternatively, the pre-checking approach may also include the following specific steps, see Figure 5 , Figure 5 This is another flowchart of the pre-check, which may include the following steps:

[0109] Step S410: Based on the resource utilization range, the resource utilization of the remaining service nodes, and the relationship between the resource utilization of the service node to be migrated and the resource utilization range, determine at least one candidate service node from the remaining service nodes, where the remaining service nodes refer to other service nodes among the multiple service nodes except the service node to be migrated.

[0110] Step S420: determining a target service node from the at least one candidate service node according to the resource usage rate of each candidate service node and the resource occupancy rate of the replica data to be migrated.

[0111] Step S430: If the relationship between the resource utilization rate of the service node to be migrated and the resource utilization rate range is: the resource utilization rate range of the service node to be migrated is greater than the maximum resource utilization rate, then the service node whose resource utilization rate is less than the maximum resource utilization rate among the remaining service nodes is taken as at least one candidate service node.

[0112] Step S440: Determine the result of the addition operation between the resource usage rate of each candidate service node and the resource occupancy rate of the replica data to be migrated.

[0113] Step S450: Determine the candidate service node corresponding to the addition operation result within the resource usage rate range as the target service node.

[0114] Step S460: Generate a migration strategy, and migrate the replica data in the service node to be migrated to the target service node based on the migration strategy.

[0115] In step S410-step S460, illustratively, if the resource usage of the current service node to be migrated (i.e., the current broker to be migrated) is greater than the maximum value of the resource usage range, it is necessary to traverse the copies of the current broker to be migrated, select a suitable copy to migrate to the second candidate broker, and if the resource usage of a second candidate broker is within the resource usage range after migrating to the copy, then the second candidate broker is the target broker for the copy migration. It should be noted that if the resource usage of the second candidate broker is not within the resource usage range after migrating to the copy, it is necessary to repeatedly select a suitable copy to migrate to the second candidate broker (here, the second candidate broker can be kept unchanged and another copy in the current broker to be migrated can be selected, or the copy can be kept unchanged and another second candidate broker can be selected for trial) until the resource usage of the second candidate broker is within the resource usage range.

[0116] For example, assuming that the preset balance threshold x is 1.05, this means that the usage rate after balance can fluctuate up and down by 0.05, that is, the allowed floating rate f is 0.05.

[0117] If monitoring detects that broker1=60%, broker2=81%, and broker3=60% in the Kafka cluster, then the average resource utilization rate m of these three brokers is 67%, calculated according to the formula:

[0118] lowThresholds: b1 = 67%*(1-0.05) = 63.65%;

[0119] upperThresholds: that is, b2=67%*(1+0.05)=70.35%;

[0120] That is, the resource usage range is {63.65%, 70.35%}.

[0121] At this time, the three brokers in the Kafka cluster: broker1=60%, broker2=81%, broker3=60% are all outside the resource usage range of {63.65%, 70.35%}, so they all need to migrate their replicas.

[0122] Assuming that the current broker to be migrated is broker2=81%, which is greater than the maximum value (lowThresholds) b2=70.35% of the resource utilization range, it is necessary to screen out service nodes whose resource utilization is less than the maximum value of the resource utilization range among all service nodes as the first candidate service nodes, that is, all brokers whose resource utilization is less than 70.35% are the second candidate brokers. By calculating that the difference between broker2=81% and b2=70.35% is 10.65%, all replica sets of the second candidate broker are traversed, and replicas whose resource utilization accounts for more than 10.65% and less than 17.35% of the total resources of the current broker to be migrated are screened out, and the replica is determined to be the target replica. If the resource utilization of the second candidate broker itself meets the resource utilization range after the replica that meets the above conditions is migrated to the currently traversed second candidate broker, the currently traversed second candidate broker is determined to be the target broker.

[0123] That is to say, after the current broker to be migrated has migrated more than 10.65% and less than 17.35% of its total resources to the candidate broker, the candidate broker meets the resource utilization range and does not cause the overall copy distribution of the cluster to violate the restrictions of other algorithms (because generally speaking, the system is balanced by more than one algorithm), then the candidate broker is the target broker, and the migrated copy is the target copy.

[0124] It should be noted that the core of the generated replica migration strategy is composed of two parts: "current replica distribution" and "expected replica distribution", that is, the replica distribution in the two states before and after migration. The detailed composition model is as follows:

[0125] For example,

[0126]

[0127] This includes the current replica distribution, which is the content of replicas under oldReplicaAssignment, and the expected distribution, which is the content of replicas under newReplicaAssignment. The data under replicas is the unique ID of the broker in the Kafka cluster.

[0128] The specific migration is directly performed by relying on the AdminClient.alterPartitionReassignments method provided in the official software development kit of the Kafka cluster.

[0129] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a resource management device provided in the present application. The device 50 includes an acquisition module 51, a first determination module 52, a second determination module 53 and a processing module 54.

[0130] The acquisition module 51 is used to obtain the resource allocation status of the distributed message system and the resource floating parameters between multiple service nodes;

[0131] A first determination module 52, used to determine the resource usage rate range allowed by the distributed messaging system according to the resource allocation situation and the resource floating parameter;

[0132] A second determination module 53 is used to determine a service node to be migrated in the distributed messaging system that needs to perform replica data migration based on a resource usage rate range, and to determine a target service node corresponding to the service node to be migrated based on the resource usage rate range;

[0133] The processing module 54 is used to generate a replica migration strategy according to the service node to be migrated and the target service node, and perform replica data migration between the service node to be migrated and the target service node based on the replica migration strategy.

[0134] Optionally, the acquisition module 51 is further used to acquire a preset balance threshold of the distributed messaging system, and to acquire a resource usage rate of each service node in a plurality of service nodes in the distributed messaging system, or to acquire an average resource usage rate and an allowable floating rate.

[0135] The first determination module 52 can be used to determine the resource utilization range according to the preset balance threshold and the resource utilization rate, or to determine the resource utilization range according to the average resource utilization rate and the allowed floating rate. For example, the floating rate threshold and the allowed floating rate are subjected to a difference operation, and the average resource utilization rate and the difference operation result are multiplied to obtain the minimum resource utilization rate; and the floating rate threshold and the allowed floating rate are subjected to a sum operation, and the average resource utilization rate and the sum operation result are multiplied to obtain the maximum resource utilization rate.

[0136] The second determination module 53 may be to compare the resource usage rate of the service node with the resource usage rate range to determine the service node to be migrated in the distributed message system that needs to perform replica data migration. If the resource usage rate of the broker is within the resource usage rate range, no migration is required; if the resource usage rate of the broker is outside the resource usage rate range, it is determined to be the service node to be migrated that needs to perform replica data migration; and the target service node is the service node whose resource usage rate can be kept within the resource usage rate range after the migration between replicas (which can be migration in or migration out) with the service node to be migrated, and the target service node is the service node.

[0137] The processing module 54 is used to generate a replica migration strategy for the service node to be migrated and the target service node determined by the second determination module 53. For example, if the resource usage rate of the service node to be migrated is smaller than the minimum value of the resource usage rate range, the processing module 54 selects a target service node that can perform replica migration from the service nodes with larger resource usage rates according to the difference between the resource usage rate of the service node to be migrated and the resource usage rate range, and generates a replica migration strategy. For example, the processing module 54 appropriately migrates some qualified replicas in the target service node to the service node to be migrated, that is, completes the migration of the target service node to the service node to be migrated according to the replica migration strategy. replica data migration is performed between the service node to be migrated and the target service node; on the contrary, if the resource utilization of the broker is greater than the maximum value of the resource utilization range, the processing module 54 can select a target service node that can perform replica migration from the service nodes with smaller resource utilization according to the difference between the resource utilization range of the service node to be migrated, and generate a replica migration strategy. For example, the processing module 54 appropriately migrates some qualified replicas in the service node to be migrated to the target service node, that is, completes the replica data migration between the service node to be migrated and the target service node according to the replica migration strategy.

[0138] See also Figure 7 , Figure 7 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The electronic device 60 includes a memory 61 and a processor 62. The memory 61 is used to store program data, and the processor 62 is used to execute the program data to implement the following method:

[0139] Obtain resource allocation of a distributed messaging system, and obtain resource floating parameters between multiple service nodes; determine the resource usage range allowed by the distributed messaging system based on the resource allocation and resource floating parameters; determine the service nodes to be migrated in the distributed messaging system that require replica data migration based on the resource usage range, and determine the target service nodes corresponding to the service nodes to be migrated based on the resource usage range; generate a replica migration strategy based on the service nodes to be migrated and the target service nodes, and perform replica data migration between the service nodes to be migrated and the target service nodes based on the replica migration strategy.

[0140] Optionally, in one embodiment, the electronic device 60 can be a chip, a field programmable gate array (FPGA), a single chip microcomputer, etc., wherein the chip can be a processing chip such as a CPU, a GPU, an MCU, etc., or a storage chip such as a DRAM, a SRAM, etc.

[0141] A schematic diagram of a computer-readable storage medium according to an embodiment of the present application is provided. The computer-readable storage medium stores program data. When the program data is executed by a processor, it is used to implement the following method:

[0142] Obtain resource allocation of a distributed messaging system, and obtain resource floating parameters between multiple service nodes; determine the resource usage range allowed by the distributed messaging system based on the resource allocation and resource floating parameters; determine the service nodes to be migrated in the distributed messaging system that require replica data migration based on the resource usage range, and determine the target service nodes corresponding to the service nodes to be migrated based on the resource usage range; generate a replica migration strategy based on the service nodes to be migrated and the target service nodes, and perform replica data migration between the service nodes to be migrated and the target service nodes based on the replica migration strategy.

[0143] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0144] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A resource management method, characterized in that: Applied to a distributed messaging system, the distributed messaging system includes a plurality of service nodes and a management node, each service node is used to store replica data of target data, and the resource management method is executed by the management node, including: Obtaining resource allocation status and resource floating parameters of the distributed messaging system, wherein the resource allocation status is the number of replica data allocated to each service node in the distributed messaging system; Determine, according to the resource allocation situation, a resource utilization rate of each service node among a plurality of service nodes in the distributed messaging system, wherein, for each service node, the resource utilization rate of the service node is a ratio of the number of replica data used by the service node to the number of replica data allocated to the service node; Determining a resource utilization range allowed by the distributed messaging system according to the resource utilization and resource floating parameters; Determine a service node to be migrated in the distributed messaging system that needs to perform replica data migration based on the resource usage rate range, and determine a target service node corresponding to the service node to be migrated based on the resource usage rate range; A replica migration strategy is generated according to the service node to be migrated and the target service node, and replica data migration is performed between the service node to be migrated and the target service node based on the replica migration strategy, wherein the replica migration strategy is to migrate the replica data in the service node with high resource utilization to the service node with low resource utilization.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining a preset balance threshold of the distributed messaging system; and Obtaining a resource usage rate of each service node among a plurality of service nodes in the distributed messaging system; The resource usage rate range is determined according to the preset balance threshold and the resource usage rate.

3. The method according to claim 2, characterized in that The determining the resource usage rate range according to the preset balance threshold and the resource usage rate includes: Determining an average resource utilization rate according to a resource utilization rate of each service node in a plurality of service nodes in the distributed messaging system; and Determining an allowable floating rate according to the preset balance threshold; The resource utilization rate range is determined according to the average resource utilization rate and the allowable floating rate.

4. The method according to claim 3, characterized in that The resource usage rate range includes a minimum resource usage rate and a maximum resource usage rate, and determining the resource usage rate range according to the average resource usage rate and the allowable floating rate includes: Performing a difference operation on the floating rate threshold and the allowed floating rate, and multiplying the average resource utilization rate by the difference operation result to obtain a minimum resource utilization rate; The floating rate threshold value and the allowed floating rate are summed, and the average resource utilization rate and the summed result are multiplied to obtain the maximum resource utilization rate.

5. The method according to claim 4, characterized in that The determining, based on the resource usage rate range, a target service node corresponding to the service node to be migrated includes: Determine at least one candidate service node from the remaining service nodes based on the resource usage range, the resource usage of the remaining service nodes, and the relationship between the resource usage of the service node to be migrated and the resource usage range, wherein the remaining service nodes refer to other service nodes among the multiple service nodes except the service node to be migrated; A target service node is determined from the at least one candidate service node according to the resource usage rate of each candidate service node and the resource occupancy rate of the replica data to be migrated.

6. The method according to claim 5, characterized in that The determining, based on the resource usage rate range, of a service node to be migrated that requires replica data migration in the distributed message system includes: It is determined that, among the multiple service nodes of the distributed message system, a service node whose resource usage rate is outside the resource usage rate range is a service node to be migrated.

7. The method according to claim 6, characterized in that The determining at least one candidate service node from the remaining service nodes based on the resource usage rate range, the resource usage rates of the remaining service nodes, and the relationship between the resource usage rate of the service node to be migrated and the resource usage rate range includes: If the relationship between the resource utilization rate of the service node to be migrated and the resource utilization rate range is: the resource utilization rate of the service node to be migrated is less than the minimum resource utilization rate, then the service node among the remaining service nodes whose resource utilization rate is greater than the minimum resource utilization rate is taken as at least one candidate service node.

8. The method according to claim 6, characterized in that The determining at least one candidate service node from the remaining service nodes based on the resource usage rate range, the resource usage rates of the remaining service nodes, and the relationship between the resource usage rate of the service node to be migrated and the resource usage rate range includes: If the relationship between the resource utilization rate of the service node to be migrated and the resource utilization rate range is: the resource utilization rate of the service node to be migrated is greater than the maximum resource utilization rate, then the service node whose resource utilization rate is less than the maximum resource utilization rate among the remaining service nodes is taken as at least one candidate service node.

9. The method according to claim 7, characterized in that: The replica data to be migrated refers to the replica data to be migrated in each candidate service node, and determining the target service node from the at least one candidate service node according to the resource usage rate of each candidate service node and the resource occupancy rate of the replica data to be migrated includes: Determine the resource utilization rate of the service node to be migrated, and add the resource utilization rate of the replica data to be migrated in each candidate service node to the resource utilization rate; The candidate service node corresponding to the addition operation result within the resource usage rate range is determined as the target service node.

10. The method according to claim 8, characterized in that The replica data to be migrated includes the replica data in the service node to be migrated, and determining the target service node from the at least one candidate service node according to the resource usage rate of each candidate service node and the resource occupancy rate of the replica data to be migrated includes: Determine the result of adding the resource usage rate of each candidate service node and the resource occupancy rate of the replica data to be migrated; The candidate service node corresponding to the addition operation result within the resource usage rate range is determined as the target service node.

11. The method according to claim 9, characterized in that The replica migration strategy includes a migration strategy. The replica migration strategy is generated according to the service node to be migrated and the target service node, including: Generate migration strategy; The replica data in the target service node is migrated to the service node to be migrated based on the migration strategy.

12. The method according to claim 10, characterized in that The replica migration strategy includes a migration-out strategy. The replica migration strategy is generated according to the service node to be migrated and the target service node, including: Generate an eviction strategy; The replica data in the service node to be migrated is migrated to the target service node based on the migration strategy.

13. A resource management device, characterized in that: The device includes: An acquisition module, used to acquire resource allocation of a distributed messaging system and resource floating parameters between multiple service nodes, wherein the resource allocation is the number of replica data allocated to each service node in the distributed messaging system; A first determination module is used to determine the resource utilization rate of each service node among the multiple service nodes in the distributed messaging system according to the resource allocation situation, wherein, for each service node, the resource utilization rate of the service node is the ratio of the number of replica data used by the service node to the number of replica data allocated to the service node; and determine the resource utilization rate range allowed by the distributed messaging system according to the resource utilization rate and the resource floating parameter; A second determination module is used to determine a service node to be migrated in the distributed messaging system that needs to perform replica data migration based on a resource usage rate range, and to determine a target service node corresponding to the service node to be migrated based on the resource usage rate range; A processing module is used to generate a replica migration strategy according to the service node to be migrated and the target service node, and perform replica data migration between the service node to be migrated and the target service node based on the replica migration strategy, wherein the replica migration strategy is to migrate the replica data in the service node with high resource utilization to the service node with low resource utilization.

14. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory is used to store program data, and the processor is used to execute the program data to implement the resource management method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program data, and when the program data is executed by a processor, it is used to implement the resource management method according to any one of claims 1 to 12.