Server cluster resource scheduling method and system based on Harris eagle algorithm

Abstract

The invention relates to a server cluster resource scheduling method and system based on a Harris eagle algorithm. The method comprises the following steps: S1, collecting node resource data in real time; s2, determining a current resource scheduling strategy according to a resource allocation algorithm and the node resource data; s3, deploying Pod according to the current resource scheduling strategy; and S4, monitoring load data of each working node, and determining a dynamically adjusted resource scheduling strategy according to the load data. According to the method, after the application program is deployed, the load data of the working node is monitored, and if the load of the working node is too high, the step S1 is executed again to dynamically adjust the resource scheduling strategy. Meanwhile, the resource allocation algorithm provided by the invention fully considers the utilization rates of a CPU, a memory, a disk and a network, and an improved Harris eagle algorithm is used for determining a resource scheduling strategy of each adjustment.

Description

Technical Field

[0001] The present application relates to the technical field of server clusters, and in particular to a server cluster resource scheduling method and system based on the Harris Hawk algorithm. Background Art

[0002] In a server cluster, different dependencies of an application are usually deployed in multiple containers, and the master node of the server cluster allocates a working node to each container for computing.

[0003] For example, in the K8s (Kubernetes) container management platform, the default scheduler divides the application Pod deployment process into two stages: filtering and scoring. In the filtering stage, the working nodes that meet the computing indicators are filtered out, and in the scoring stage, the default scheduler uses the optimization strategy to score the remaining nodes and selects the working node with the highest score as the running location of the application Pod.

[0004] However, the default scheduler only allocates resources and schedules Pods during initial deployment. After the application business is running, as the business volume on the node changes, the use of resources will also change accordingly, and the resources allocated during deployment are not enough to meet the needs of Pod growth.

[0005] In order to dynamically adjust the node resources in the server cluster as the Pod changes, the present application provides a server cluster resource scheduling method and system based on the Harris Eagle algorithm. Summary of the invention

[0006] In order to overcome the problems existing in the related art, the first aspect of the present application provides a server cluster resource scheduling method based on the Harris Hawk algorithm, comprising:

[0007] S1. Real-time collection of node resource data; the node resource data includes CPU usage, memory usage, disk usage and network usage;

[0008] S2. Determine the current resource scheduling strategy according to the resource allocation algorithm and the node resource data;

[0009] S3. Deploy pods according to the current resource scheduling strategy;

[0010] S4. Monitor the load data of each working node, and determine a dynamically adjusted resource scheduling strategy according to the load data.

[0011] In one embodiment, the node resource data is represented by a matrix, where

[0012] The resource matrix is:

[0013]

[0014] Among them, u i,j is the utilization rate of the jth resource on worker node i.

[0015] In one implementation, S2 specifically includes: the resource allocation algorithm is improved based on the Harris Hawk algorithm.

[0016] In one implementation, S2 specifically includes:

[0017] S201, initializing the eagle group; wherein the position of the eagle individual is defined by the node resource matrix, and the eagle individual is used to simulate the working node;

[0018] S202, iterative optimization of eagle group;

[0019] S203, update the position of the eagle individual:

[0020] S204: Determine whether to end the iteration according to a preset number of iterations or a convergence condition.

[0021] In one implementation, in S203, Cauchy perturbation is used to optimize individuals to enhance global search capability and convergence speed.

[0022] In one implementation, step S202 specifically includes:

[0023] The fitness value of each eagle individual is calculated through the fitness calculation function, and the eagle group is iteratively optimized according to the fitness value;

[0024] The fitness calculation function is:

[0025]

[0026] Among them, X i is the fitness value of the eagle individual, w j is the weight parameter of the j-th resource, is the variance of resource utilization on node i, and α is the adjustment coefficient.

[0027] In one embodiment, the variance calculation formula of each resource utilization rate on the node i is

[0028] The formula is:

[0029]

[0030] Among them, M is the number of resource types, u i,j is the usage of the jth resource on node i, is the average utilization rate of all resources of node i.

[0031] In one embodiment, The calculation formula is as follows:

[0032]

[0033] Among them, M is the number of resource types, u i,j is the usage of the jth resource on node i, is the average utilization rate of all resources of node i.

[0034] In one implementation, a group of eagles are initialized in the search space, each eagle represents a possible node resource allocation scheme, and the dimension of each eagle corresponds to a different resource indicator. The population is initialized using the Tent chaotic sequence to ensure population diversity.

[0035] A second aspect of the present application provides a server cluster resource scheduling system based on the Harris Hawk algorithm, which is used to execute the resource scheduling method and update the resource scheduling strategy of the server cluster.

[0036] The technical solution provided by this application may have the following beneficial effects:

[0037] When deploying an application to a server cluster, this application obtains the resource data of each working node and allocates the working node to the Pod of the application through a resource allocation algorithm. After the deployment is completed, the load data of the working node is monitored. If the load of the working node is too high, step S1 is re-executed to adjust the resource scheduling strategy; if the load of the working node is normal, the current resource scheduling strategy is maintained.

[0038] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0040] Figure 1 A flow chart of a server cluster resource scheduling method shown in an embodiment of the present application;

[0041] Figure 2 Another flowchart of the server cluster resource scheduling method shown in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0043] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0044] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0045] Embodiment 1

[0046] In the existing resource scheduling and allocation algorithm of server clusters, resources are allocated and scheduled for Pods only during initial deployment. After the application business is running, as the business volume on the node changes, the resources allocated during deployment are not enough to meet the needs of Pod growth.

[0047] In order to dynamically adjust the node resources in the server cluster as the Pod changes, this application provides a server cluster resource scheduling method based on the Harris Eagle algorithm, such as Figure 1 As shown, the following steps are included:

[0048] S1, real-time collection of node resource data;

[0049] S2. Determine the current resource scheduling strategy according to the resource allocation algorithm and the node resource data;

[0050] S3. Deploy pods according to the current resource scheduling strategy;

[0051] S4. Monitor the load data of each working node, and determine the dynamically adjusted resource scheduling strategy according to the load data.

[0052] Specifically, the resource data includes CPU usage, memory usage, disk usage, and network usage.

[0053] Specifically, the resource scheduling strategy is the working node address allocated to each Pod and the computing resources of the working node.

[0054] In the embodiment of the present application, when deploying the application to the server cluster, the resource data of each working node is obtained, and the working node is allocated to the Pod of the application through the resource allocation algorithm. After the deployment is completed, the load data of the working node is monitored. If the load of the working node is too high, step S1 is re-executed to adjust the resource scheduling strategy; if the load of the working node is normal, the current resource scheduling strategy is maintained.

[0055] Embodiment 2

[0056] Based on the server cluster resource scheduling method of embodiment 1, in order to clearly illustrate the specific implementation of the resource allocation algorithm, the embodiment of the present application also provides a server cluster resource scheduling method based on the Harris Hawk algorithm, including steps S1 to S4 described in embodiment 1, and the process is as follows: Figure 1 shown.

[0057] Furthermore, in step S1, the application is deployed in the Kubernetes cluster, and the node resource data is collected through the Prometheus monitoring system.

[0058] In the embodiment of the present application, the collected node resource data is encoded as a node resource matrix. The node resource matrix is:

[0059]

[0060] Among them, u i,j is the utilization rate of the jth resource on node i.

[0061] In an embodiment of the present application, each column of the node resource matrix represents the resource utilization of a working node, and each row in the mth column represents the utilization of a resource type.

[0062] The Kubernetes cluster monitors resource usage by deploying exporters on each node, such as CPU usage, memory usage, disk IO usage, and network IO usage.

[0063] Prometheus connects to the metrics data collection interface in RPC mode, and uses promql query statements to query node resource data from the Prometheus database.

[0064] Furthermore, the resource allocation algorithm in step S2 is improved based on the Harris Hawk algorithm. Specifically, each hawk individual has multiple dimensions.

[0065] Specifically, the dimension of each eagle individual i is expressed as:

[0066] D i =M

[0067] Here, M is the number of resource types (such as CPU, memory, disk I / O, network I / O).

[0068] The Harris Hawk algorithm is a group optimization algorithm that simulates the hunting behavior of Harris Hawks in nature. When solving the group optimization problem through the Harris Hawk algorithm, the position of the hawk individual, the position of the prey, and the energy of the prey are first defined.

[0069] In an embodiment of the present application, the position of the eagle individual is the mth column of the node resource matrix, that is, the resource utilization rate of the i-th working node.

[0070] When performing group optimization, the time required to capture prey is calculated by continuously replacing the positions of individual eagles, and finally the optimal allocation plan for the positions of individual eagles is determined. In the embodiment of the present application, replacing the positions of individual eagles is to change the resource utilization rate of each column in the node resource matrix. When the optimal allocation plan is determined, the optimal node resource matrix can also be determined. Specifically, step S2 is as follows: Figure 2 As shown, including:

[0071] S201, initializing the eagle group;

[0072] Specifically, the tent chaotic sequence is used to initialize the eagle group to ensure the diversity of the eagle group.

[0073] S202, iterative optimization of eagle group;

[0074] Specifically, the fitness value of each eagle is calculated based on the node resource data. The fitness function comprehensively considers the weights and actual data of each indicator and dynamically adjusts the resource allocation plan.

[0075] S203, update the position of the eagle individual:

[0076] Specifically, the eagle updates its position based on its historical optimal position and the global optimal position in order to move in the search space and find the optimal solution.

[0077] In the present embodiment, the eagle individual x i The position of x i =[u i,1 ,...u i,j ].

[0078] Where i is the node currently being considered, u i,j is the utilization rate of the jth resource on worker node i.

[0079] In the embodiment of the present application, the energy formula of the prey is:

[0080]

[0081] Among them, E i represents the prey energy of worker node i, M is the number of resource types (e.g., CPU, memory, disk I / O, network I / O), u i,j is the utilization rate of the jth resource on worker node i, w j is the weight of the jth resource, reflecting the importance of this resource in the energy calculation of the prey.

[0082] S204: Determine whether to end the iteration according to a preset number of iterations or a convergence condition.

[0083] In S203, the individuals are optimized using Cauchy perturbation to enhance the global search capability and convergence speed.

[0084] In step S202, the fitness value of each eagle individual is calculated by a fitness calculation function. The fitness calculation function is:

[0085]

[0086] Among them, X i is the fitness value of the eagle individual, w j is the weight parameter of the j-th resource, is the variance of resource utilization on node i, and α is the adjustment coefficient.

[0087] In the embodiment of the present application, the fitness value of the eagle individual represents the resource utilization score of node i. The higher the fitness value, the higher the resource utilization score, indicating that the working node is more suitable for scheduling new Pods. j It reflects the importance of the resource in scheduling decisions. It reflects the uniformity of resource use. The smaller the variance, the more uniform the resource use. α is used to adjust the effect of variance on fitness value.

[0088] Furthermore, The calculation formula is

[0089]

[0090] The calculation formula is:

[0091]

[0092] Among them, M is the number of resource types, u i,j is the usage of the jth resource on node i, is the average utilization rate of all resources of node i.

[0093] In step S4, the performance data after resource adjustment is collected regularly, including indicators such as node resource utilization and response time, and then the Prometheus monitoring system is used to analyze these performance data in real time to monitor the effect of resource balancing and performance improvement. If insufficient resource balance or poor performance is found, the system will review historical performance data and make adjustments in combination with the improved Harris Eagle algorithm.

[0094] During the iteration process, the eagle updates its position according to its historical optimal position and the global optimal position, and optimizes it using Cauchy perturbation and Tent chaotic sequence. The embodiment of the present application comprehensively considers the actual usage data of CPU, memory, disk and network, and can dynamically adjust the resource allocation scheme. At the same time, according to the analysis results, the weight parameters in the resource allocation algorithm are adjusted to optimize the overall performance, and the resource evaluation and node selection are re-performed.

[0095] In the embodiment of the present application, it is proposed to optimize the population by using Cauchy perturbation and Tent chaotic sequence, which can enhance the global search capability and convergence speed. Tent chaotic sequence can improve the search capability of the algorithm and avoid premature convergence.

[0096] The resource allocation method shown in the embodiment of the present application utilizes the Harris Hawk algorithm to represent the resource utilization of i nodes as the dimensions of the Hawk individuals.

[0097] During the node selection process, the resource allocation algorithm fully considers the utilization of CPU, memory, disk and network, and designs the fitness calculation function.

[0098] When the fitness of the swarm meets the preset requirements or reaches the preset number of iterations, a new resource allocation strategy is obtained and fed back to the Kubernetes cluster for actual resource allocation and scheduling.

[0099] Furthermore, the embodiment of the present application also monitors the resource utilization of each working node by combining with the Prometheus monitoring system. When the computing requirements of the Pod change, step S1 is re-executed to perform Eagle Group iteration, thereby realizing dynamic resource balancing and optimal scheduling of the Kubernetes cluster, improving resource utilization efficiency, and ensuring the performance and stability of the application.

[0100] Embodiment 3

[0101] A server cluster resource scheduling system based on the Harris Hawk algorithm is used in the resource scheduling method in Embodiments 1 to 2 to update the resource scheduling strategy of the server cluster.

[0102] The scheme of the present application has been described in detail above with reference to the accompanying drawings.

[0103] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art should also be aware that the actions and modules involved in the description are not necessarily required for this application.

[0104] In addition, it can be understood that the steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the modules in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs.

[0105] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0106] Alternatively, the present application can also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or electronic device, server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0107] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the application herein may be implemented as electronic hardware, computer software, or a combination of both.

[0108] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, a program segment or a code, and the part of the module, the program segment or the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or the flow chart, and the combination of the square boxes in the block diagram and / or the flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A server cluster resource scheduling method based on Harris Hawk algorithm, characterized in that: include: S1, real-time collection of node resource data; The node resource data includes CPU usage, memory usage, disk usage and network usage; S2. Determine the current resource scheduling strategy according to the resource allocation algorithm and the node resource data; S3. Deploy Pod according to the current resource scheduling strategy; S4. Monitor the load data of each working node, and determine the dynamically adjusted resource scheduling strategy according to the load data.

2. A server cluster resource scheduling method based on Harris Hawk algorithm according to claim 1, characterized in that: The node resource data is represented by a matrix, and the node resource matrix is: Among them, u i,j is the utilization rate of the jth resource on worker node i.

3. A server cluster resource scheduling method based on Harris Hawk algorithm according to claim 1, characterized in that: S2 specifically includes: The resource allocation algorithm is improved based on the Harris Hawk algorithm.

4. A server cluster resource scheduling method based on Harris Hawk algorithm according to claim 3, characterized in that: S2 specifically includes: S201, initializing the eagle group; wherein the position of the eagle individual is defined by the node resource matrix, and the eagle individual is used to simulate the working node; S202, iterative optimization of eagle group; S203, update the position of the eagle individual: S204: Determine whether to end the iteration according to a preset number of iterations or a convergence condition.

5. A server cluster resource scheduling method based on Harris Hawk algorithm according to claim 4, characterized in that: In S203, the individuals are optimized using Cauchy perturbation to enhance the global search capability and convergence speed.

6. A server cluster resource scheduling method based on Harris Hawk algorithm according to claim 4, characterized in that: Step S202 specifically includes: The fitness value of each eagle individual is calculated through the fitness calculation function, and the eagle group is iteratively optimized according to the fitness value; The fitness calculation function is: Among them, X i is the fitness value of the eagle individual, w j is the weight parameter of the j-th resource, is the variance of resource utilization on node i, and α is the adjustment coefficient.

7. A server cluster resource scheduling method based on Harris Hawk algorithm according to claim 6, characterized in that: The variance of resource usage on the working node i The calculation formula is: Among them, M is the number of resource types, u i,j is the usage of the jth resource on worker node i, is the average utilization rate of all resources of worker node i.

8. A server cluster resource scheduling method based on Harris Hawk algorithm according to claim 7, The calculation formula is as follows: in, M is the number of resource types, u i,j is the usage of the jth resource on worker node i, is the average utilization rate of all resources of node i.

9. The server cluster resource scheduling method based on Harris Hawk algorithm according to claim 1, characterized in that: A group of eagles are initialized in the search space. Each eagle represents a possible node resource allocation scheme. The dimension of each eagle corresponds to different resource indicators. The population is initialized using the Tent chaotic sequence to ensure population diversity.

10. A server cluster resource scheduling system based on Harris Hawk algorithm, characterized in that: Used to execute the resource scheduling method described in claims 1 to 9 to update the resource scheduling strategy of the server cluster.

Images