A method for evaluating resource consumption of a computing power cluster for a computing job

Abstract

The present application relates to the field of cluster computing, and is used to solve the problem that the computing power cluster cannot adapt to the dynamically changing computing load when performing a computing job, ignores the comprehensive indicators such as network bandwidth and storage I / O, and causes the mismatch between resource allocation and actual demand, specifically a computing power cluster resource consumption evaluation method for a computing job; in the present application, the idle computing power resources are allocated according to the job task size and the idle computing power resources, and the timeliness requirement of the job, and the resource allocation adjustment is performed multiple times through pre-allocation, one-time allocation and final allocation in the allocation process, fully considering the data transmission and throughput time consumption limitation caused by the interference resource parameters, so that the limitation of storage resources and network resources can be considered when allocating computing power resources, the precision of computing power resource allocation is improved, and through the dynamic comparison between the computing power resource energy consumption and the completion degree, the overall energy consumption of the computing power cluster is evaluated, and the abnormal behavior of computing power energy consumption is found in time.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cluster computing, in particular to a method for evaluating resource consumption of computing power clusters for computing jobs. BACKGROUND

[0002] Computing power refers to data processing capability, which integrates information computing power, network carrying capacity and data storage power. It is mainly provided to society through computing power infrastructure such as computing power centers. It exists in various intelligent hardware devices such as mobile phones, laptops and supercomputers. Its original meaning is to represent the computing performance of a device or system. With the advent of the intelligent era, the three elements of intelligent computing - computing power, algorithm and data - have gradually become an important part of the social information infrastructure. The connotation of "computing power" has further expanded to the computing performance that users can obtain and actually use.

[0003] Computing power is the new productivity in the digital economy era, and has become the core force driving the development of digital economy and the solid foundation supporting the development of digital economy. It plays an important role in promoting technological progress, promoting industry digitalization and supporting economic and social development. It is like water in the agricultural era and electricity in the industrial era, and has become the core productivity of digital economy development.

[0004] With the explosive growth of computing power demand, computing power clusters such as GPU clusters and CPU clusters have become the core infrastructure supporting large-scale computing jobs. However, the resource consumption evaluation method in the prior art has the following problems:

[0005] The traditional method relies on static resource allocation strategy and cannot adapt to dynamic changes in computing load, resulting in low resource utilization or job performance bottleneck. At the same time, most methods only focus on CPU / GPU utilization or memory occupancy, ignoring network bandwidth, storage I / O, energy consumption and other comprehensive indicators, resulting in one-sided evaluation results, which leads to mismatch between resource allocation and actual demand, making it difficult to balance energy consumption and computing efficiency, and unable to meet the demand of green computing;

[0006] In view of the above technical problems, the present application provides a solution. SUMMARY

[0007] In the application, the idle computing power resources are allocated according to the task size and idle computing power resources, and the time efficiency requirement of the task, and the resource allocation adjustment is adjusted through pre-allocation, one-time allocation and final allocation multiple times in the allocation process, and the data transmission and throughput time consumption limit caused by the interference resource parameter is fully considered, so that the limitation of storage resources and network resources can be considered when allocating computing power resources, the precision of computing power resource allocation is improved, and the problems that the computing load cannot be adapted to the dynamic change when the computing task is performed through the computing power cluster, the network bandwidth, storage I / O, energy consumption and other comprehensive indexes are ignored, the resource allocation cannot match the actual demand, it is difficult to balance the energy consumption and computing efficiency, and the resource utilization is low are solved, and the computing power cluster resource consumption evaluation method for computing task is provided.

[0008] The object of the application can be achieved by the following technical solutions:

[0009] A computing power cluster resource consumption evaluation method for computing task, comprising the following steps:

[0010] Step one: obtain the computing power cluster size to obtain the total computing power resources in the idle state;

[0011] Step two: obtain the computing task to obtain the task size, if there are multiple computing task groups, each computing task is numbered, the task size is calculated according to the numbering result, and the task time efficiency is generated according to the computing task completion time requirement;

[0012] Step three: compare the total computing power resources and the task size, allocate the computing power resources according to the task time efficiency, and generate the predicted computing time consumption through one-time computing power resource allocation;

[0013] Step four: obtain the task size and interference resource, obtain the interference limit time consumption according to the interference resource and the task size, compare the predicted computing time consumption and the interference limit time consumption, reserve or correct the one-time computing power resource allocation according to the comparison result, and record as the final computing power resource;

[0014] Step five: calculate the energy consumption of the allocated final computing power resources, and obtain the task completion progress;

[0015] Step six: compare the computing power resource energy consumption and the task completion progress to obtain the energy consumption efficiency, compare the energy consumption evaluation with the set standard to generate the energy consumption evaluation result.

[0016] As a preferred embodiment of the application, the task time efficiency generated in step two is the latest time point for completing the computing task;

[0017] In the third step, when the total computing resources are allocated, first, a standby computing resource part is reserved from the total computing resources, the remaining total computing resources are recorded as available computing resources, then the sizes of all job tasks are counted to obtain a total size of job tasks, the available computing resources are evenly distributed to the total size of job tasks, and the pre-allocation of computing resources is completed.

[0018] According to the pre-allocation result of computing resources, the estimated completion time of the job task is obtained, and the estimated completion time is compared with the job timeliness to determine whether the estimated completion time meets the job timeliness, the job task is calculated and allocated as a computing resource insufficient task and a computing resource redundant task, and the computing resource insufficient task and the computing resource redundant task are adjusted with each other, and finally the standby computing resource is involved in the allocation to complete one computing resource allocation.

[0019] As a preferred embodiment of the present application, the interference resource in the fourth step includes storage resources and network resources, the data throughput time is obtained through the resource throughput speed of the storage resources and the size of the computing task, the data transmission time is obtained through the resource transmission speed of the network resources and the size of the computing task, and the minimum value of the data throughput time and the data transmission time is recorded as the interference limiting time consumption;

[0020] The interference limiting time consumption and the estimated computing time consumption are compared, if the estimated computing time consumption is less than the interference limiting time consumption, the one computing resource is modified, and if the estimated computing time consumption is greater than or equal to the interference limiting time consumption, the one computing resource is reserved.

[0021] A computing job computing cluster resource consumption evaluation system for implementing a computing job computing cluster resource consumption evaluation method, comprising a computing cluster supervision unit, a computing job task interface, a computing resource allocation unit, an interference resource intervention unit, a computing resource modification unit, and an energy efficiency evaluation unit.

[0022] The computing cluster supervision unit is used to connect the computing cluster and obtain the total computing resources of the computing cluster;

[0023] The computing job task interface is used to receive job tasks and computing job completion time requirements, analyze the size of the computing job task, and generate job timeliness according to the computing job completion time requirements;

[0024] The computing resource allocation unit obtains the total computing resources through the computing cluster supervision unit, obtains the size of the computing job task through the computing job task interface, and comprehensively analyzes the total computing resources, the size of the computing job task, and the job timeliness to generate an estimated computing time consumption and a one computing resource allocation result;

[0025] The interference resource intervention unit can obtain interference resources, including network resources and storage resources, and analyze the interference resources and the task scale to obtain interference limiting time consumption;

[0026] The computing power resource correction unit can compare the interference limiting time consumption and the predicted computing time consumption, and reserve or correct the initial computing power resource allocation according to the comparison result to obtain the final computing power resource.

[0027] The energy consumption efficiency evaluation unit analyzes the job completion progress and the computing power resource energy consumption to obtain the energy consumption efficiency, and generates an energy consumption evaluation result according to the energy consumption efficiency.

[0028] As a preferred embodiment of the present application, the computing power resource allocation unit allocates the available computing power resources to the total scale of the computing task, and then integrates the available computing power resources according to different computing task scales to obtain the pre-allocation of the computing power resources allocated to each computing task.

[0029] The computing power resource allocation unit records the computing task whose predicted computing time consumption does not meet the job timeliness as a computing power insufficient task, records the computing task whose predicted computing time consumption meets the job timeliness as a computing power redundant task, and concentrates the redundant computing power of the computing power redundant task and allocates it to the computing power insufficient task. If there are still computing power insufficient tasks after allocation, the standby computing power resources are used for supplementation.

[0030] As a preferred embodiment of the present application, if the predicted computing time consumption is less than the interference limiting time consumption, the computing power allocation unit reduces the computing power allocated to the computing task, so that the predicted computing time consumption is equal to the interference limiting time consumption, and concentrates the reduced computing power in the standby computing power resources.

[0031] As a preferred embodiment of the present application, the energy consumption efficiency evaluation unit obtains the job task completion progress and the duration of the computing task, calculates the computing power resource energy consumption according to the final computing power resource allocated to the computing task and the duration of the computing task, calculates the total energy consumption when all the computing tasks are completed according to the computing power resource energy consumption and the job task completion progress, and calculates the unit energy consumption of the computing task by ratio calculation of the total energy consumption and the computing task scale.

[0032] The energy consumption efficiency evaluation unit standardizes the unit energy consumption of the computing task to obtain high energy consumption tasks and qualified energy consumption tasks.

[0033] As a preferred embodiment of the present application, the energy consumption efficiency evaluation unit sums up the unit energy consumption of each computing job task to obtain the overall unit energy consumption, and judges the overall unit energy consumption and the set threshold, and generates an energy consumption exceeding signal or an energy consumption qualified signal according to the judgment result

[0034] Compared with the prior art, the present application has the following advantages:

[0035] 1、In the present application, the idle computing power resources are allocated according to the job task size and the idle computing power resources, and the time effectiveness requirement of the job, and the precision of the computing power resource allocation is improved through multiple resource allocation adjustments of pre-allocation, one-time allocation and final allocation in the allocation process, avoiding the waste of idle resources caused by resource allocation redundancy.

[0036] 2、In the present application, by considering the data transmission and throughput time consumption limit caused by the interference resource parameter, the limitation of storage resources and network resources can be considered when allocating computing power resources, and the allocation of computing power resources is further checked and corrected based on the limitation of storage resources and network resources, so as to avoid excessive allocation of computing power resources, and to avoid the situation that a large amount of computing power resources are allocated redundantly due to the short board of storage resources and network resources, and to improve the comprehensive consideration when allocating computing power resources.

[0037] 3、In the present application, the allocation result of the computing power resources and the job completion degree are calculated in real time, the energy consumption required for completing the computing task is evaluated through dynamic comparison between energy consumption and completion degree, and the computing task is divided into energy consumption levels according to the energy consumption result, and the overall energy consumption of the computing power cluster is evaluated, so that the abnormal behavior of computing power energy consumption can be found in time, and data support is provided for the optimization of subsequent computing power resource scheduling strategy. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to facilitate the understanding of those skilled in the art, the present application will be further described below with reference to the accompanying drawings.

[0039] Figure 1 The system block diagram of the present application is shown in the figure.

[0040] Figure 2 The system flowchart of the present application is shown in the figure. DETAILED DESCRIPTION

[0041] The technical solutions of the present application will be described below in conjunction with the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0042] Embodiment one:

[0043] Referring to Figure 1 Figure 2 As shown in the figure, a method for evaluating the resource consumption of a computing power cluster for computing jobs includes the following steps:

[0044] Step 1: Obtain the size of the computing power cluster to obtain the total computing power resources in an idle state;

[0045] Step 2: Obtain the computing job tasks to obtain the job task size. If there are multiple sets of computing job tasks, each computing job task is numbered, the job task size is calculated according to the numbering result, and the job timeliness is generated according to the computing job completion time requirement, wherein the generated job timeliness is the latest time point for completing the computing job;

[0046] Step 3: Compare the total computing power resources and the job task size, allocate computing power resources according to the job timeliness, and generate the predicted computing time consumption through the first computing power resource allocation. When allocating the total computing power resources, first reserve a standby computing power resource part from the total computing power resources, record the remaining total computing power resources as available computing power resources, then count all the job task sizes to obtain the total job task size, and then allocate the available computing power resources to the total job task size, and then integrate the computing power resources allocated to a computing job task to complete the computing power resource pre-allocation;

[0047] According to the computing power resource pre-allocation result and the computing job task size, the completion time of the job task is obtained, and the predicted completion time and the job timeliness are compared to determine whether the predicted completion time meets the job timeliness. If the predicted completion time meets the job timeliness, the computing job task is allocated as a computing power redundant task. If the predicted completion time does not meet the job timeliness, the computing job task is allocated as a computing power insufficient task, and the computing power insufficient task and the computing power redundant task are adjusted to each other, so that the computing power of the computing power redundant task is allocated to the computing power insufficient task, thereby avoiding the existence of the computing power insufficient task. If the computing power of the computing power redundant task is allocated after the computing power of the computing power redundant task is allocated, there is still a computing power insufficient task, then the standby computing power resource is allocated to complete the first computing power resource allocation. After the first computing power resource allocation is completed, the allocated computing power resources and the computing task size are predicted to obtain the predicted computing time consumption;

[0048] ​Step four: obtain the task size and interference resources, including storage resources and network resources, obtain the data throughput time through the resource throughput speed of the storage resources and the computing task size, obtain the data transmission time through the resource transmission speed of the network resources and the computing task size, and record the minimum value of the data throughput time and the data transmission time as the interference limit time consumption, compare the expected computing time consumption with the interference limit time consumption, if the expected computing time consumption is less than the interference limit time consumption, modify the one-time computing power resource, if the expected computing time consumption is greater than or equal to the interference limit time consumption, retain the one-time computing power resource, when modifying the computing power resource, reduce the computing power allocated to the computing task to make the expected computing time consumption equal to the interference limit time consumption, and concentrate the reduced computing power in the standby computing power resource, and finally record the computing power allocation after modification or retention as the final computing power resource;

[0049] Step five: calculate the energy consumption of the allocated final computing power resource, obtain the duration of the computing task, calculate the computing power resource energy consumption according to the final computing power resource allocated to the computing task and the duration of the computing task, and obtain the progress of the computing task;

[0050] Step six: calculate the computing power resource energy consumption and the progress of the computing task to obtain the total energy consumption when all computing tasks are completed, sum the total energy consumption of each computing task, and finally calculate the ratio of the sum of the computing task size to obtain the energy consumption efficiency, compare the energy consumption evaluation with the set standard to generate the energy consumption evaluation result, including generating an energy consumption exceeding signal or an energy consumption qualified signal.

[0051] Embodiment two:

[0052] Please refer to Figure 1 - Figure 2 As shown in FIG. 1, a computing power cluster resource consumption evaluation system for computing tasks includes a computing power cluster supervision unit, a computing task interface, a computing power resource allocation unit, an interference resource intervention unit, a computing power resource modification unit, and an energy efficiency evaluation unit.

[0053] The computing power cluster supervision unit is used to connect the computing power cluster and obtain the total computing power resource available to the computing power cluster.

[0054] The computing task interface is used to receive the computing task and the computing task completion time requirement, analyze the size of the computing task, and generate the task timeliness according to the computing task completion time requirement.

[0055] The computing power resource allocation unit obtains total computing power resources through the computing power cluster supervision unit, obtains the computing job task scale through the computing job task interface, and performs comprehensive analysis on the total computing power resources, the computing job task scale and the job timeliness, and evenly distributes the available computing power resources to the total scale of the job tasks to complete the pre-allocation of computing power resources;

[0056] After the computing power resource allocation unit evenly distributes the available computing power resources to the total scale of the job tasks, the available computing power resources are integrated according to different computing task scales to obtain the pre-allocation of computing power resources allocated to each computing job task;

[0057] The computing power resource allocation unit records the pre-estimated computing time that does not meet the job timeliness as a power insufficient task, records the pre-estimated computing time that meets the job timeliness as a power redundant task, and concentrates the redundant power of the power redundant task and allocates it to the power insufficient task. If there are still power insufficient tasks after allocation, the standby computing power resources are supplemented to obtain a one-time computing power resource allocation result, and the pre-estimated computing time is generated according to the one-time computing power resource allocation result;

[0058] The interference resource intervention unit can obtain interference resources, including network resources and storage resources, and analyze the interference resources and the job task scale to obtain the interference limitation time;

[0059] The computing power resource correction unit can compare the interference limitation time and the pre-estimated computing time, and correct or retain the one-time computing power resource allocation according to the comparison result to obtain the final computing power resource. If the pre-estimated computing time obtained by the computing power resource correction unit is less than the interference limitation time, the allocated computing power of the computing task is reduced so that the pre-estimated computing time is equal to the interference limitation time, and the reduced computing power is concentrated in the standby computing power resource;

[0060] The energy consumption efficiency evaluation unit analyzes the job completion progress and the computing power resource energy consumption to obtain the energy consumption efficiency, and generates an energy consumption evaluation result according to the energy consumption efficiency;

[0061] The specific evaluation method is:

[0062] The energy consumption efficiency evaluation unit obtains the job task completion progress and the duration of the computing job task, calculates the computing power resource energy consumption according to the final computing power resource allocated to the computing job task and the duration of the computing job task, and calculates the total energy consumption when all the computing job tasks are completed by the energy consumption efficiency evaluation unit. The computing power resource energy consumption and the job task completion progress are calculated to obtain the total energy consumption when all the computing job tasks are completed, and the unit energy consumption of the computing job task is obtained by ratio calculation of the total energy consumption and the computing job task scale;

[0063] The energy consumption efficiency evaluation unit makes a standard judgment on the unit energy consumption of the computing job task, obtains a high energy consumption task and a qualified energy consumption task, and sends the high energy consumption task and the qualified energy consumption task through a network, so as to realize visual output of energy consumption of different computing tasks.

[0064] The energy consumption efficiency evaluation unit performs summation calculation on the unit energy consumption of each computing job task to obtain overall unit energy consumption, and judges the overall unit energy consumption and a set threshold value, if the overall unit energy consumption is greater than the set threshold value, an energy consumption over-standard signal is generated, and if the overall unit energy consumption is not greater than the set threshold value, an energy consumption qualified signal is generated.

[0065] The preferred embodiments disclosed above are only used to help explain the present application. The preferred embodiments do not describe all the details and limit the present application to the specific embodiments. Obviously, many modifications and variations can be made according to the content of the present application. The present application is selected and described in detail to better explain the principles and practical application of the present application, so that those skilled in the art can well understand and utilize the present application. The present application is limited by the claims and their full scope and equivalents.

Claims

1. A method for evaluating resource consumption of a computing power cluster for a computing job, characterized in that, Comprising the following steps: Step one: obtain the scale of the computing cluster, get the total computing power resources in idle state; Step two: obtain the computing job task, get the job task scale, if there are multiple computing job tasks, number each computing job task, calculate the job task scale according to the numbering result, and generate the job timeliness according to the computing job completion time requirement; The generated job timeliness is the latest time point to complete the computing job; Step three: compare the total computing power resources and the job task scale, allocate computing power resources according to the job timeliness, and generate the expected computing time through the first computing power resource allocation; In step three, when allocating the total computing power resources, first reserve a part of the standby computing power resources from the total computing power resources, record the remaining total computing power resources as available computing power resources, then count all the job task scales to get the total job task scale, and allocate the available computing power resources to the total job task scale, complete the pre allocation of computing power resources; According to the pre allocation result of computing power resources, the expected completion time of the job task is obtained, and the expected completion time and the job timeliness are compared to determine whether the expected completion time meets the job timeliness, the expected computing time that does not meet the job timeliness is recorded as the insufficient computing task, the expected computing time that meets the job timeliness is recorded as the redundant computing task, and the redundant computing power of the redundant computing task is concentrated and allocated to the insufficient computing task, if there are still insufficient computing tasks after allocation, the standby computing power resources are supplemented to complete the first computing power resource allocation; Step four: obtain the job task scale and interference resources, get the interference limit time according to the interference resources and the task job scale, compare the expected computing time and the interference limit time, and according to the comparison result, retain or modify the first computing power resource allocation, and record it as the final computing power resource; The interference resources include storage resources and network resources, the data throughput time is obtained by the resource throughput speed of the storage resources and the computing task scale, the data transmission time is obtained by the resource transmission speed of the network resources and the computing task scale, and the minimum value of the data throughput time and the data transmission time is recorded as the interference limit time; Then compare the interference limit time and the expected computing time, if the expected computing time is less than the interference limit time, modify the first computing power resource, if the expected computing time is greater than or equal to the interference limit time, retain the first computing power resource; Step five: calculate the energy consumption of the allocated final computing power resources, and obtain the job task completion progress; Step six: compare the computing power resource energy consumption and the job completion progress to get the energy consumption efficiency, compare the energy consumption efficiency with the set standard to generate the energy consumption evaluation result.

2. A computing power cluster resource consumption evaluation system for computing jobs, characterized in that, A computing power cluster resource consumption evaluation method for computing jobs as claimed in claim 1, comprising a computing power cluster supervision unit, a computing job task interface, a computing power resource allocation unit, an interference resource intervention unit, a computing power resource modification unit, and an energy consumption efficiency evaluation unit; The computing power cluster supervision unit is configured to connect the computing power cluster and acquire total computing power resources of the computing power cluster; The computing job task interface is configured to receive a job task and a computing job completion time requirement, analyze a scale of the computing job task, and generate a job timeliness according to the computing job completion time requirement; The computing power resource allocation unit is configured to acquire the total computing power resources through the computing power cluster supervision unit, acquire the scale of the computing job task through the computing job task interface, and comprehensively analyze the total computing power resources, the scale of the computing job task, and the job timeliness to generate an estimated computing time consumption and a first computing power resource allocation result; The interference resource intervention unit is configured to acquire interference resources including network resources and storage resources, analyze the interference resources and the scale of the job task, and obtain an interference limitation time consumption; The computing power resource correction unit is configured to compare the interference limitation time consumption and the estimated computing time consumption, retain or correct the first computing power resource allocation according to a comparison result, and obtain a final computing power resource; The energy consumption efficiency evaluation unit is configured to analyze a job completion progress and a computing power resource energy consumption to obtain an energy consumption efficiency, and generate an energy consumption evaluation result according to the energy consumption efficiency. 3.The system for evaluating resource consumption of a computing job according to claim 2, wherein, The computing power resource allocation unit is configured to evenly distribute available computing power resources to a total scale of the job task, integrate the evenly distributed available computing power resources according to different computing task scales, and obtain pre-allocation of computing power resources allocated to each computing job task. 4.The system for evaluating resource consumption of a computing job according to claim 2, wherein, The computing power resource correction unit is configured to, when the estimated computing time consumption is less than the interference limitation time consumption, reduce the computing task allocated computing power, so that the estimated computing time consumption is equal to the interference limitation time consumption, and concentrate the reduced computing power to a standby computing power resource.

5. The system of claim 2, wherein: The energy consumption efficiency evaluation unit is configured to acquire a job task completion progress and a duration of a computing job task, calculate a computing power resource energy consumption according to the final computing power resource allocated to the computing job task and the duration of the computing job task, calculate a total energy consumption when all computing job tasks are completed according to the computing power resource energy consumption and the job task completion progress, and calculate a unit energy consumption of the computing job task according to the total energy consumption and the scale of the computing job task. The energy consumption efficiency evaluation unit is configured to standardize the unit energy consumption of the computing job task to obtain a high energy consumption task and a qualified energy consumption task.

6. The system for evaluating resource consumption of a computing job according to claim 5, wherein, The energy consumption efficiency evaluation unit is configured to sum the unit energy consumption of each computing job task to obtain a total unit energy consumption, judge the total unit energy consumption and a set threshold, and generate an energy consumption exceeding signal or an energy consumption qualified signal according to a judgment result.

Images