Mesos resource scheduling method based on resource matching degree

Abstract

A kind of Mesos resource scheduling method based on resource matching degree, the matching degree of physical machine resource occupation and application framework resource demand is calculated, and application framework resource demand response and resource allocation are carried out according to the matching degree, and finally resource demand information and resource receiving logic are sent according to allocation.The present application can achieve about 4% improvement in resource average utilization rate, reduce the number of allocation failures by up to 19%, reduce the resource utilization gap by up to about 6%, and reduce idle energy consumption by about 23%.The resource scheduling method based on resource matching degree proposed in the present application can effectively reduce resource fragmentation and improve the resource utilization of the cluster through reasonable resource scheduling.The centralized scheduling strategy proposed in the present application can further improve the energy efficiency of the cluster by reducing idle energy consumption.

Description

Technical Field

[0001] This invention relates to a technology in the field of distributed system resource allocation, specifically a Mesos (open-source cluster management system) based on resource matching degree. https: / / mesos.apache.org / Resource scheduling methods. Background Technology

[0002] Cluster management systems are a crucial component of modern data centers, and resource scheduling technology is one of the most important technologies within them. Apache Mesos is a widely used cluster management system. Mesos employs a two-tier resource scheduling architecture, using a hierarchical Dominant Resource Fairness (DRF) scheduling strategy to manage various types of applications. The first-tier scheduler is the master daemon, which manages the slave daemons running on all nodes in the Mesos cluster. The cluster consists of physical or virtual servers used to run application tasks. The second-tier scheduler consists of application frameworks, each including a scheduler and an executor process, with the executor running on each node. Cluster resource utilization and energy efficiency are important metrics for evaluating cluster management systems; however, Mesos still falls short of meeting current industry demands in terms of both. Summary of the Invention

[0003] This invention addresses the aforementioned shortcomings of existing technologies by proposing a Mesos resource scheduling method based on resource matching degree. This method achieves a maximum improvement of approximately 4% in average resource utilization; reduces the number of allocation failures by up to 19%; reduces resource utilization gaps by up to approximately 6%; and lowers idle energy consumption by approximately 23%. The resource matching degree-based scheduling method proposed in this invention effectively reduces resource fragmentation and improves cluster resource utilization through reasonable resource scheduling. The proposed centralized scheduling strategy further enhances cluster energy efficiency by reducing idle energy consumption.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a Mesos resource scheduling method based on resource matching degree. It calculates the matching degree between physical machine resource usage and application framework resource requirements, responds to application framework resource requirements and allocates resources according to the matching degree, and finally sends resource requirement information and resource receiving logic according to the allocation.

[0006] The matching degree between the physical machine resource consumption and the application framework resource requirements is obtained in the following way:

[0007] Step 1) Pre-allocate resources for the physical machine and the application framework. Add the resource requirements of the application framework to the current resource usage of the physical machine to obtain the utilization rate of each resource and the overall resource utilization rate after pre-allocation. Calculate the overall average resource utilization rate. Where there are r types of resources, the utilization rate of each resource is represented by res_usage. k Let k = 1, 2, ..., r.

[0008] Step 2) Calculate the resource utilization gap to represent the difference between the utilization rate of each resource and the overall resource utilization rate, including:

[0009] ① Euclidean distance, used to calculate the Euclidean distance between each resource utilization rate and the overall average resource utilization rate. or

[0010] ② Chebyshev distance, used to calculate the maximum distance between an individual resource utilization rate and the overall average resource utilization rate: usage_distance = MAX(|res_usage) k -avg_usage|)k=1,2,…,r.

[0011] Step 3) Introduce the overall average utilization rate of the original resources before the pre-allocation of the physical machine as an optional additional variable, specifically: resource matching degree. Wherein: the coefficient α∈(0,1) is used to adjust the importance of the difference between the original overall average utilization rate of resources and the pre-allocated resource utilization rate in the formula, and the coefficient β∈{-1,1} is used to adjust the algorithm's strategy of using the original overall average utilization rate of resources. If β=1 is selected, the scheduling module will adopt centralized allocation; if β=-1 is selected, the scheduling module will adopt decentralized allocation.

[0012] The calculation methods for the adjustment coefficient and the resource utilization gap can be further combined to obtain four allocation strategies: a centralized allocation strategy using Euclidean distance, a decentralized allocation strategy using Euclidean distance, a centralized allocation strategy using Chebyshev distance, and a decentralized allocation strategy using Chebyshev distance.

[0013] The application framework resource request response implements the Mesos requestResources() method, which is used to respond to resource requests issued by the application framework's scheduler and record resource request information using an appropriate data structure.

[0014] The resource allocation mentioned above refers to resource matching using either the Default Max algorithm or the Random Select algorithm, which iteratively matches resource matching with varying resource matching degrees. The Default Max algorithm achieves the best scheduling effect, while the Random Select algorithm improves the time efficiency of the algorithm by reducing certain scheduling accuracy.

[0015] The maximum value selection algorithm includes:

[0016] Step i) Iterate through all the Frameworks to which a Role belongs, and find the physical machine with the best resource matching for each Framework. Specifically, this includes:

[0017] Step i.1) ​​Traverse all Slaves, calculate the resource matching degree res_match in turn, and find the Slave with the largest res_match.

[0018] Step i.2) Allocate the resources on the Slave to the Framework.

[0019] Step ii) Repeat step i until all Frameworks have been allocated resources.

[0020] The random selection algorithm includes:

[0021] Step a) Iterate through all the Frameworks to which a Role belongs, and find the physical machine with the best resource matching for each Framework. Specifically, this includes:

[0022] Step a.1) Iterate through all Slaves, calculating the resource matching degree res_match sequentially, and finding the Slave with the largest res_match. Specifically: for the first γ% of Slaves, only the resource matching degree is calculated, and the maximum value of the resource matching degree in this group is recorded as temp_max; for the last γ% of Slaves, the selection phase begins. After calculating the resource matching degree res_match between the Slave and the target Framework, the first Slave physical machine that satisfies res_match ≥ temp_max·(1-offset) is selected as the Slave physical machine with the qualified resource matching degree. The variable γ is a variable parameter, γ∈(0,1). The larger its value, the more accurate the obtained resource matching degree, and the more iterations are required; the smaller the γ value, the less accurate the obtained resource matching degree, and the fewer iterations are required. offset∈(0,1) is a decimal between 0 and 1, used to adjust the acceptable range of resource matching degree. When offset is 0, the algorithm will try its best to search for the optimal solution; offset > 0 indicates an allowable error.

[0023] Step a.2) Allocate the resources on the Slave to the Framework.

[0024] Step b) Repeat step i until all Frameworks have been allocated resources.

[0025] Sending resource request information refers to sending resource request information at two different points in time using the interface provided by ProtoBuf for serialization information classes. Specifically:

[0026] ① During registration, the scheduler needs to register with the Master before it can run, at which point it can send resource request information.

[0027] ② When a resource offer is received, the Master will send resources to the Scheduler at a fixed frequency. At this time, the Scheduler can send the resource requirement information of the tasks to be run to the Master.

[0028] Regarding the resource receiving logic, for resources from the same physical machine, the Scheduler should choose to "receive all" or "reject all".

[0029] Technical effect

[0030] Compared with existing technologies, this invention significantly optimizes the overall utilization rate of cluster resources, energy efficiency, and time efficiency of resource allocation algorithms by introducing two methods for calculating the resource utilization gap (usage_distance), two methods for using the original physical machine resource utilization parameter (pre_avg_usage), and adjusting the iterative structure of the resource allocation algorithm. This includes optimizing the proportion of idle energy consumption in the overall energy consumption and the number of iterations in the resource allocation algorithm iterative structure. This invention also adds consideration to the resource matching degree indicator during resource scheduling and allows energy consumption to be managed by selecting centralized or decentralized resource scheduling sub-strategies.

[0031] Through simulation testing, the resource scheduling strategy proposed in this invention has improved some indicators, including: 1. Achieving a maximum improvement of 4% in average resource utilization; 2. In terms of idle energy consumption, the centralized strategy can reduce idle energy consumption by about 23%. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0033] Figure 2 A flowchart of the random selection algorithm used to optimize the time efficiency of FineGrainedAllocator;

[0034] Figure 3This is a schematic diagram illustrating the combination of resource scheduling strategies proposed in this invention. Detailed Implementation

[0035] To better illustrate the objectives, features, and implementation methods of this invention, the resource allocation process of the Mesos cluster management system is first explained: The Mesos resource allocation process is mainly reflected in the Mesos Master (hereinafter referred to as Master) and the FrameworkScheduler (hereinafter referred to as Scheduler). Furthermore, in the Master, the specific resource allocation logic is implemented in the Allocation Module. Mesos adopts HierarchicalDRFAllocator as the default resource allocation strategy.

[0036] like Figure 1 As shown in this embodiment, a Mesos resource scheduling system based on resource matching degree is included, comprising: a Mesos cluster management system, a Mesos resource scheduling module (FineGrainedAllocator) based on resource matching degree, and a Framework based on resource requirements. The Framework's Scheduler, based on its own resource requirements, transmits resource requirement information to the Mesos Master through resource request instructions (requestResources) via the Mesos system's scheduling driver unit (SchedulerDriver). The resource scheduling module receives the resource requirement information and, during periodic calls to the resource allocation algorithm (generateOffers), calculates the resource matching degree between the physical machine Slave and the Framework based on the resource requirement information. After obtaining a resource allocation scheme based on this resource matching degree, the scheme is implemented through the Mesos Master's Executor.

[0037] This embodiment relates to a Mesos resource scheduling method based on the above system, specifically including:

[0038] Step 1) Respond to and record the resource requirements proposed by the application framework.

[0039] Mesos' Allocation Module provides numerous methods to be implemented, including the `requestResources()` method, which is used to respond to resource requests issued by the Framework Scheduler. However, the default resource scheduling module in Mesos, HierarchicalDRFAllocator, does not effectively utilize resource request information through `requestResources()`.

[0040] Since the calculation of resource matching degree requires the resource requirement information provided by the application framework, FineGrainedAllocator needs to respond in requestResources() and record the resource requirement information with an appropriate data structure.

[0041] Step 2) Implement a method for calculating the matching degree between the physical machine's resource usage and the application framework's resource requirements, referred to as the resource matching degree. The resource matching degree represents the approximate level of utilization of various resources of the physical machine after allocating resources to the application framework. That is, the higher the resource matching degree, the more similar the utilization rates of various resources of the physical machine will be after the resource allocation to the application framework is completed.

[0042] The higher the resource matching degree between the physical machine and the application framework, the more the resource allocation algorithm will tend to allocate the physical machine resources to the application framework.

[0043] The definition of resource matching degree is based on the following considerations: if the utilization rates of various resources of physical machines are similar after resource allocation, it means that the generation of resource fragmentation can be avoided, thereby improving resource utilization.

[0044] Resource matching degree calculation is performed on a specific physical machine and application framework. The types of resources involved must be quantifiable using scalar values. In Mesos, CPU, memory, and disk usage can be represented by scalar values, where CPU usage is measured in cores (decimals are allowed), and memory and disk usage are measured in MB. The specific calculation method for resource matching degree includes the following steps:

[0045] Step 2.1) Pre-allocate resources for the physical machine and the application framework. Add the resource requirements of the application framework to the current resource usage of the physical machine to obtain the utilization rate of each resource and the overall resource utilization rate after pre-allocation. Assume there are r types of resources, then the utilization rate of each resource is represented by res_usage. k Let k = 1, 2, ..., r, and let avg_usage represent the overall average resource utilization rate.

[0046] Step 2.2) Define the resource utilization gap (usage_distance) to represent the difference between the utilization rate of each resource and the overall resource utilization rate. In this embodiment, two methods for calculating the resource utilization gap are designed: the Euclidean distance algorithm and the Chebyshev distance algorithm.

[0047] This embodiment defaults to using the Euclidean distance algorithm to calculate resource utilization differences, which can handle most scenarios. However, using the Chebyshev distance algorithm can more effectively exclude situations where the utilization rate of a single resource far exceeds that of other resources.

[0048] Step 2.3) The greater the gap in resource utilization, the more mismatched the physical machine resource usage is with the application framework resource requirements. Therefore, the reciprocal of usage_distance is taken as the core component of resource matching degree.

[0049] Step 2.4) In this embodiment, the original average resource utilization rate (pre_avg_usage) before the physical machine's pre-allocation is introduced as an optional additional variable. The purpose of introducing these two variables is to take the original utilization level of the physical machine into account. In the resource allocation algorithm, it can be chosen to either concentrate tasks on fewer physical machines or distribute tasks across more machines. The former can potentially save costs by reducing the number of physical machines running in the cluster, while the latter can reduce performance interference caused by resource contention. Resource matching degree Among them, the coefficient α∈(0,1) is used to adjust the importance of the difference between the original overall average utilization rate of resources and the pre-allocated resource utilization rate in the formula. The coefficient β∈{-1,1} is more important, used to adjust the algorithm's strategy of using the original overall average utilization rate of resources. Different choices of coefficient β will further divide the resource allocation strategy into two strategies: centralized allocation and decentralized allocation. That is, if β=1 is chosen, then when the resource utilization rate of the physical machine is higher, the matching degree between the physical machine and the resource requirements of the application framework is higher, and the resource scheduling module tends to deploy tasks on fewer physical machines; if β=-1 is chosen, the tasks will be distributed and deployed on more physical machines.

[0050] In summary, the resource matching degree `res_match` consists of the average utilization rate of the original physical machine's resources and the difference in resource utilization rates. By adjusting the calculation methods of these two factors, four resource matching degree algorithms can be obtained, leading to four allocation strategies: Euclidean Centralization (using Euclidean distance), Euclidean Decentralization (using Euclidean distance), Chebyshev Centralization (using Chebyshev distance), and Chebyshev Decentralization (using Chebyshev distance). In practical applications, the optimal performance can be achieved by adjusting the α coefficient and selecting different strategies.

[0051] Step 3) Use the resource allocation algorithm iterative structure to allocate resources.

[0052] The resource matching degree calculation calls the resource allocation algorithm generateOffers() in the resource scheduling module based on resource matching degree. The resource allocation algorithm needs to solve three problems in resource allocation: the object of resource allocation, the amount of resource allocation, and the source of allocable resources. FineGrainedAllocator focuses on optimizing the latter two, that is, allocating a resource amount that is appropriate to the needs of each application framework, and selecting a suitable physical machine as the resource source.

[0053] Mesos' default resource scheduling module uses generateOffers() and a series of methods to handle the above problems. The following will first explain the implementation of the default resource scheduling module. The core logic of generateOffers() is implemented using a three-level iterative structure.

[0054] This embodiment uses the default maximum selection algorithm for resource matching degree (Default Max). This structure can be divided into two steps:

[0055] Step 3.1) Iterate through all the Frameworks to which a Role belongs, and find the physical machine with the best resource matching degree for each Framework. The default method is to iterate through all Slaves, calculate the resource matching degree res_match in turn, select the Slave with the largest res_match, and allocate the resources on that Slave to the Framework. The calculation method of res_match is as shown in Step 2.

[0056] Step 3.2) Repeat step 3.1 until all Frameworks have been allocated resources.

[0057] By reversing this iterative structure, the goal of the resource allocation algorithm becomes finding a suitable Slave for each Framework. It's important to note that the time complexity of the above method is not increased compared to Mesos' default resource allocation algorithm; the core iterative part still has a time complexity of O(n*m). Furthermore, it should be noted that there are multiple ways to calculate `res_match`, and choosing different calculation methods will result in different allocation strategies.

[0058] However, in large-scale cluster applications, the significant increase in the number of physical machines will affect the efficiency of resource allocation algorithms. To address this issue, this embodiment provides a RandomSelect algorithm based on resource matching degree as an optimization scheme. At the cost of appropriately reducing the accuracy of resource matching, it can effectively improve the efficiency of the algorithm.

[0059] like Figure 1 As shown, the random selection algorithm includes:

[0060] Step 3.a) Iterate through all the Frameworks to which a Role belongs, and find the physical machine with the best resource matching degree for each Framework. The default search method is to iterate through all Slaves, calculate the resource matching degree res_match in turn, select the Slave with the largest res_match, and allocate the resources on that Slave to that Framework. For the first γ% of Slaves, only the resource matching degree is calculated, and the maximum value of the resource matching degree in this part is recorded as temp_max; for the last γ% of Slaves, the selection phase begins. After calculating the resource matching degree res_match between the Slave and the target Framework, the first physical machine that satisfies res_match ≥ temp_max·(1-offset) can be selected as the physical machine of the Slave with the qualified resource matching degree.

[0061] The variable γ is a variable parameter, γ∈(0,1). A larger value of γ results in a more accurate resource matching, but increases the number of iterations. Conversely, a smaller value of γ results in a less accurate resource matching, but fewer iterations. offset∈(0,1) is a decimal between 0 and 1, affecting the acceptable range of resource matching. When offset is 0, the algorithm will try its best to search for the optimal solution; offset > 0 indicates that an error is acceptable. The size of offset also affects the accuracy of resource matching and the number of iterations; a smaller offset results in a more accurate resource matching and more iterations, while a larger offset has the opposite effect.

[0062] Step 3.b) Repeat step 3.a until all Frameworks have been allocated resources.

[0063] The resource scheduling strategy described in this invention is composed of the above-mentioned parts, and its composition is as follows: Figure 2 As shown. Each part can be extended into two sub-strategies, resulting in a total of 8 strategies. Among them, the usage method and iteration structure of the original resource utilization parameter have a significant impact on the final scheduling result. The sub-strategies derived from these two aspects will have significant differences in resource utilization and energy efficiency.

[0064] After implementing the requestResources() and generateOffers() methods, you need to define the necessary module information at the end of the module, including the module name, module version information, module implementer information, and module initialization method name. This information will serve as the basis for the Mesos system to search for and call the module.

[0065] Step 4) Application Framework Sends Resource Request Information: In Mesos, the Framework Scheduler communicates with the Mesos Master through the SchedulerDriver module. The SchedulerDriver provides an interface called requestResources(), which is used by the Framework Scheduler to send specific information about resource requests to the Mesos Master.

[0066] Mesos uses Protocol Buffers (ProtoBuf) to serialize information and uses this information for communication between different layers and physical machines. Resource requirement information is also defined by ProtoBuf. Resource requirement information is defined by Request, which includes mandatory Resource information. The Resource information defines the type and quantity of resources. Request can contain multiple Resource information entries to express requests for various resources. The interface provided by ProtoBuf for serialized information classes allows for writing and reading information. In the Scheduler design and implementation of this embodiment, resource requirement information needs to be sent at two different times: during registration (when the Scheduler needs to register with the Master before running, at which time it can send resource requirement information), and when a resource offer is received, the Master sends resources to the Scheduler at a fixed frequency, at which time the Scheduler can send resource requirement information for tasks to be run to the Master.

[0067] Step 5) Apply the Scheduler framework's resource receiving logic. This resource receiving logic is located in an interface named resourceOffers(). In this embodiment, the resource offers received by the Scheduler will contain information such as the resource source slave, resource type, and resource quantity. For resources from the same physical machine, the Scheduler should select "accept all" or "reject all" in resourceOffers().

[0068] After implementing the FineGrainedAllocator module and Framework Scheduler, they can be run in the Mesos system. The resource scheduling module FineGrainedAllocator needs to be compiled into a .so file first. After the Mesos system starts running, the `--module` command is used to load third-party modules. The command format is `--modules='{"libraries":[{"file":" / X / module_name.so","modules":[{"name":"module_name"}]}]}'`. Here, the `file` parameter specifies the address of the .so file, and `name` specifies the module name; this parameter will be used as the basis for specifying the scheduling module. Alternatively, the above information can be written into a .json file and called directly through the .json file using `--module`. Next, the `--allocator` command is used to specify the resource scheduling module implemented in this invention. The command format is `--allocator=module_name`, where `module_name` is consistent with the name in the `--module` command. At this point, the resource scheduling module is loaded. The Framework Scheduler of this invention is written in Java. Before running the Framework, it also needs to be compiled to obtain .class files. It is run using the java command, and the IP address and port of the Mesos are appended to the command.

[0069] Through specific practical experiments, a simulated experimental environment was built to test several key indicators. The following section will describe the design, construction, and optimization of the simulated experimental environment.

[0070] The experiment simulates the resource scheduling process of a Mesos cluster, which mainly involves the Slave physical machine, the Framework application framework, and the Allocator resource scheduling module.

[0071] The physical slave machines in the experiment contain two types of resources: CPU and memory, expressed in units of processing units (CPU) and MB, respectively. Each framework contains a resource requirement, including both CPU and memory requirements. To simulate the differences in resource requirements among different frameworks, the resource requirements in the experiment are divided into Balanced, CPU Heavy (preferring computational resources), and Mem Heavy (preferring memory resources). CPU Heavy has a higher CPU requirement, while Mem Heavy has a higher memory requirement. Furthermore, to simulate the differences in resource requirements among different frameworks, the above resource requirement types are categorized into large tasks and small tasks based on the size of the resource requirement. In the experiment, the framework generates the resource requirement type based on probability, and then randomizes the specific resource requirement, randomly increasing or decreasing the resource requirement proportionally.

[0072] The resource scheduling process in the experiment includes multiple rounds of resource allocation, used to simulate the Mesos resource scheduling module initiating resource allocation at a fixed frequency. The Allocator simulates and implements both the default Mesos resource allocation algorithm and the resource allocation algorithm proposed in this embodiment.

[0073] The simulation experiment will involve multiple rounds of resource allocation for testing, and includes the following key steps:

[0074] Step 1) Generate a Slave queue and a Framework queue. When generating the Framework queue, resource requirements are randomly generated for each Framework. Based on this model, the resource requirements are then randomly adjusted.

[0075] Step 2) The Allocator performs several rounds of resource allocation.

[0076] Step 2-1) Allocate resources to all Frameworks and select the appropriate physical machine according to the resource allocation algorithm.

[0077] Step 2-2) Update the Slave resource usage and collect necessary parameters. It should be noted that, unlike the actual Mesos operation, the simulation simplifies the series of processes of communication between the Master and the Framework for resource allocation. Instead, resource allocation is considered complete when the Allocator completes the current round of resource allocation, and the resource usage of the Slave queue is updated.

[0078] Step 3) Using different strategies and algorithms, repeat steps 1 and 2 to conduct multiple tests to obtain the average value of the results.

[0079] Mesos Default Resource Allocation Algorithm: The Mesos default resource allocation algorithm (referred to as Mesos default) was simulated in the experimental environment. The performance of this algorithm was used as the benchmark for testing.

[0080] In this embodiment, the resource scheduling module based on resource matching degree consists of two main parts: a resource matching degree calculation method and a resource allocation algorithm. The resource matching degree calculation method can be combined to obtain four allocation strategies: Euclidean centralization, Euclidean decentralization, Chebyshev centralization, and Chebyshev decentralization. The resource allocation algorithm is divided into two types: the default maximum selection algorithm and a random selection algorithm. Through permutations and combinations, eight resource scheduling strategies can be obtained. In this experiment, the four scheduling strategies using the Euclidean distance calculation method are mainly tested: maximum selection Euclidean centralization, maximum selection Euclidean decentralization, random selection Euclidean centralization, and random selection Euclidean decentralization. For example, maximum selection Euclidean centralization means that the resource allocation algorithm uses the maximum selection algorithm, while the resource matching degree calculation uses a centralized resource scheduling strategy based on Euclidean distance.

[0081] To evaluate the optimization performance of this embodiment from multiple dimensions, four types of indicators were designed for the experiment: resource utilization rate, resource fragmentation, time efficiency, and energy consumption.

[0082] Resource utilization metrics: Average cluster resource utilization: This metric is used to intuitively reflect the performance of this embodiment in improving cluster resource utilization. It is calculated by summing the utilization rates of all types of resources (CPU and memory in this experiment) of all physical machines in the cluster and then taking the average.

[0083] Resource fragmentation metrics: Framework resource allocation failure count: Resource allocation failures occur when there are insufficient physical machine resources in the cluster; Cluster resource utilization gap: Its definition is the same as that in the resource matching degree calculation method, and the Euclidean distance calculation method is used in the experiment.

[0084] Time efficiency metric: Number of Slave iterations in the resource allocation algorithm iteration structure: In the resource allocation algorithm, the Slave queue needs to be iterated to find the Slave with the best resource matching degree. This metric is used to compare the time efficiency difference between the default maximum selection algorithm and the random selection algorithm.

[0085] Energy Consumption Metric: Idle Energy Consumption: Idle power is the power consumed when a physical machine is running normally but under zero load. Idle energy consumption is the integral of idle power over time. Idle power is independent of resource utilization levels and depends only on the number of running physical machines. Idle energy consumption is directly proportional to the number of currently running physical machines. In this experiment, resource allocation rounds can be used for time periods; therefore, idle energy consumption is calculated by accumulating the number of running physical machines in each round of resource allocation. This metric is primarily used to compare the energy consumption differences between centralized and decentralized resource scheduling strategies.

[0086] The resource allocation rounds were conducted 25 times, with a total of 500 tests. There were 100 slave physical machines, configured with 32 CPUs and 256GB of memory. There were 50 application frameworks. The resource requirements, generation probabilities, and randomization ranges for different resource requirement types are shown in the table below.

[0087]

[0088] In the resource matching degree calculation, the parameter α is 0.8, and the relevant parameters γ and offset of the Random Select algorithm are 0.2 and 0.5, respectively.

[0089] Table 1 shows the results of the test experiments. The experiments were conducted in multiple rounds, and the algorithm performed relatively stably. The relevant parameters were adjusted to achieve good results in this embodiment.

[0090]

[0091] According to the test results, compared with the benchmark, the algorithms tested in this embodiment achieved an improvement of at least 2.2% and at most 4% in average resource utilization; reduced the number of allocation failures by at least 5% and at most 19%; reduced the resource utilization gap by at least 3.5% and at most 6.4%; and in terms of idle energy consumption, the centralized scheduling strategy using the random selection algorithm was the most effective, reducing idle energy consumption by about 23%.

[0092] In summary, this method effectively reduces resource fragmentation and improves cluster resource utilization through reasonable resource scheduling. The proposed centralized scheduling strategy further improves cluster energy efficiency by reducing idle energy consumption.

[0093] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A Mesos resource scheduling method based on resource matching degree, characterized in that, The system calculates the matching degree between physical machine resource usage and application framework resource requirements, and responds to application framework resource requirements and allocates resources based on the matching degree. Finally, it sends resource requirement information and resource receiving logic based on the allocation. The matching degree between the physical machine resource consumption and the application framework resource requirements is obtained in the following way: Step 1) Pre-allocate resources for the physical machine and the application framework. Add the resource requirements of the application framework to the current resource usage of the physical machine to obtain the utilization rate of each resource and the overall resource utilization rate after pre-allocation. Calculate the overall average resource utilization rate. Where there are r types of resources, the utilization rate of each resource is denoted by r. express ; Step 2) Calculate the resource utilization gap This indicates the difference between the utilization rate of each resource and the overall resource utilization rate, including: ① Euclidean distance, used to calculate the Euclidean distance between each resource utilization rate and the overall average resource utilization rate. ,or ② Chebyshev distance, used to calculate the maximum distance between an individual resource utilization rate and the overall average resource utilization rate. ; Step 3) Introduce the overall average utilization rate of the original resources before the pre-allocation of the physical machine as an optional additional variable, specifically: resource matching degree. , where: coefficient The coefficient is used to adjust for the importance of the difference between the original overall average utilization rate of resources and the pre-allocated resource utilization rate in the formula. This is a strategy used to adjust the algorithm's overall average utilization rate of the original resources. The scheduling module will then use centralized allocation; select In this case, the scheduling module will adopt decentralized allocation.

2. The Mesos resource scheduling method based on resource matching degree according to claim 1, characterized in that, The allocation strategies obtained by combining the adjustment coefficient and the calculation method of the resource utilization gap include: a centralized allocation strategy using Euclidean distance, a decentralized allocation strategy using Euclidean distance, a centralized allocation strategy using Chebyshev distance, and a decentralized allocation strategy using Chebyshev distance.

3. The Mesos resource scheduling method based on resource matching degree according to claim 1, characterized in that, The application framework resource request response implements the Mesos requestResources method, which is used to respond to resource requests issued by the application framework's scheduler and record resource request information using an appropriate data structure.

4. The Mesos resource scheduling method based on resource matching degree according to claim 1, characterized in that, The resource allocation mentioned above refers to the resource matching of the maximum value selection algorithm (Default Max) or the random selection algorithm (Random Select) performed iteratively based on the resource matching degree. The maximum value selection algorithm can achieve the best scheduling effect, while the random selection algorithm improves the time efficiency of the algorithm by reducing the scheduling accuracy. The maximum value selection algorithm includes: Step i) Iterate through all the Frameworks to which a Role belongs, and find the physical machine with the best resource matching for each Framework. Specifically, this includes: Step i.1) ​​Traverse all Slaves, calculate the resource matching degree res_match in turn, and find the Slave with the largest res_match; Step i.2) Allocate the resources on the Slave to the Framework; Step ii) Repeat step i until all Frameworks have received resource allocation; The random selection algorithm includes: Step a) Iterate through all the Frameworks to which a Role belongs, and find the physical machine with the best resource matching for each Framework. Specifically, this includes: Step a.1) Iterate through all Slaves, calculate the resource matching score res_match in turn, and find the Slave with the largest res_match, where: for the first For the slave, only the resource matching degree is calculated, and the maximum value of the resource matching degree in this part is recorded, denoted as temp_max; for the later... The Slave enters the selection phase. After calculating the resource matching degree res_match between the Slave and the target Framework, the first one that meets the requirements is selected. You can then select a Slave physical machine that meets the resource matching criteria; variables It is a variable parameter. The larger the value, the more accurate the resource matching, and the more iterations are required. The smaller the value, the less accurate the resource matching, and the fewer the number of iterations. A decimal number between 0 and 1, used to adjust the range of acceptable resource matching; when the offset is 0, the algorithm will try its best to search for the optimal solution. Indicates the allowable error; Step a.2) Allocate the resources on the Slave to the Framework; Step b) Repeat step i until all Frameworks have been allocated resources.

5. The Mesos resource scheduling method based on resource matching degree according to claim 1, characterized in that, Sending resource request information refers to sending resource request information at two different points in time using the interface provided by ProtoBuf for serialization information classes. Specifically: ① During registration, the Scheduler needs to register with the Master before it runs, and at this time it can send resource request information; ② When a resource offer is received, the Master will send resources to the Scheduler at a fixed frequency. At this time, the Scheduler sends the resource requirement information of the task to be run to the Master.

6. A Mesos resource scheduling system based on resource matching degree that implements the method of any one of claims 1 to 5, characterized in that, include: The Mesos cluster management system comprises a resource matching degree-based Mesos resource scheduling module (FineGrainedAllocator) and a resource requirement-based Framework. The Framework's Scheduler, based on its own resource requirements, transmits resource request information to the Mesos Master via the Mesos system's SchedulerDriver using the `requestResources` command. The resource scheduling module receives this information and, during periodic calls to the resource allocation algorithm (`generateOffers`), calculates the resource matching degree between the physical machine (Slave) and the Framework. Based on this matching degree, it obtains a resource allocation scheme and implements it through the Mesos Master's Executor.

Images