Resource oversale method, device and equipment for calculation task, medium and vehicle

Abstract

The invention discloses a resource overselling method, device and equipment for a calculation task, a medium and a vehicle, and the method comprises the steps: monitoring a current remaining resource value, scheduling the calculation task to an idle calculation node according to a current resource overselling parameter and the current remaining resource value when the calculation task meets a task scheduling condition, and carrying out the overselling of each type of calculation resources, and when the current resource oversale parameter of the type of computing resources and the actual resource usage amount index are compared and analyzed to determine that the current resource oversale parameter of the type of computing resources meets an adjustment condition, adjusting the current resource oversale parameter of the type of computing resources until the adjustment condition is not met to obtain a new resource oversale parameter. And taking the new resource oversale parameter as the current resource oversale parameter of the computing resource, and returning to execute the step of monitoring the current residual resource value. According to the method, the resource utilization rate and the processing speed can be improved without influencing the stability of the system under the condition that the calculation resource quantity of the calculation node is not changed.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and more particularly, to a method, apparatus, device, medium, and vehicle for oversubscribing resources of computing tasks. Background Art

[0002] Currently, in many scenarios, computing tasks need to be uploaded to computing nodes in the cloud for processing. When processing computing tasks, computing nodes involve the mixed use of exclusive computing resources and computing resources shared with other computing tasks. For example, in the autonomous driving scenario, the recorded autonomous driving data is uploaded to the cloud for preprocessing, involving the mixed use of various computing resources such as the central processing unit (CPU), graphics processing unit (GPU), memory, disk capacity, disk input / output (IO), and input / output operations per second (IOPS).

[0003] Since a computing node may process multiple computing tasks simultaneously, and the computing resources of the computing node are limited, this results in longer processing times for each computing task and may cause computing failures, thereby affecting the stability of the entire system. Therefore, how to improve resource utilization and processing speed as much as possible without affecting the stability of the system while keeping the amount of computing resources of the computing node unchanged has become an urgent problem to be solved. Summary of the Invention

[0004] The present application provides a method, apparatus, device, medium, and vehicle for oversubscribing resources of computing tasks, which can improve resource utilization and processing speed without affecting the stability of the system while keeping the amount of computing resources of the computing node unchanged. The specific technical solutions are as follows.

[0005] In a first aspect, the present application provides a method for oversubscribing resources of computing tasks, which is applied to an oversubscribed resource system. The method includes:

[0006] Determine the current resource oversubscription parameters of various computing resources used when creating a computing task;

[0007] Monitor the current remaining resource values of various computing resources in each computing node of the oversubscribed resource system;

[0008] When the computing task meets the task scheduling conditions, schedule the computing task to an idle computing node according to the current resource oversubscription parameters and the current remaining resource values;

[0009] When the computing task is executed on the idle computing node, collect the total amount of computing tasks executed by the idle computing node at the current moment and the actual resource usage of various computing resources used by each computing task;

[0010] Calculating the actual resource usage metrics for various types of computing resources based on the total amount and the actual resource usage of various types of computing resources used by each computing task;

[0011] For each type of computing resource, when it is determined through comparative analysis of the current resource oversubscription parameter of this type of computing resource and the actual resource usage metric that the current resource oversubscription parameter of this type of computing resource meets the adjustment condition, adjust the current resource oversubscription parameter of this type of computing resource until it no longer meets the adjustment condition to obtain a new resource oversubscription parameter. Take the new resource oversubscription parameter as the current resource oversubscription parameter of this computing resource, and return to execute the step of monitoring the current remaining resource values of various types of computing resources in each computing node of the resource oversubscription system.

[0012] Thus, by comparing and analyzing the current resource oversubscription parameter and the actual resource usage metric, and adjusting the current resource oversubscription parameter until it no longer meets the adjustment condition to obtain a new resource oversubscription parameter, a closed-loop monitoring of the resource oversubscription parameter is achieved, and the purpose of adjusting the resource oversubscription parameter according to the actual resource usage is realized. The new resource oversubscription parameter is in line with the actual situation and will not be too large or too small. Therefore, the processing speed can be improved, and more computing tasks can be successfully processed in a shorter time, thereby improving the resource utilization rate and ensuring the stability of the system.

[0013] Moreover, in the embodiments of the present application, resource oversubscription is set for computing tasks. Compared with the existing method of setting resource oversubscription for computing nodes, the situation where some computing tasks in a computing node cannot perform resource oversubscription is avoided.

[0014] Optionally, the step of determining the current resource oversubscription parameters of various types of computing resources used when creating a computing task includes:

[0015] When creating a computing task, obtain the resource oversubscription parameter of the processor used to create the computing task from the database of the resource oversubscription method;

[0016] Calculate the current resource oversubscription parameters of various types of computing resources used to create the computing task according to the resource oversubscription parameter of the processor.

[0017] Optionally, the step of calculating the actual resource usage metrics for various types of computing resources based on the total amount and the actual resource usage of various types of computing resources used by each computing task includes:

[0018] For each type of computing resource, calculate the sum of the actual resource usage of all computing tasks using this type of computing resource, calculate the quotient between the sum and the total amount, use the quotient as the average value of the actual resource usage of this type of computing resource, and use the maximum value of the actual resource usage of this type of computing resource among all computing tasks as the maximum value of the actual resource usage of this type of computing resource.

[0019] Optionally, the current resource overcommitment parameters for each type of computing resource include the current resource pre-allocation value and the current resource upper limit value. For each type of computing resource, when it is determined through comparative analysis of the current resource overcommitment parameters and the actual resource usage metrics of this type of computing resource that the current resource overcommitment parameters of this type of computing resource meet the adjustment conditions, the steps of adjusting the current resource overcommitment parameters of this type of computing resource until they no longer meet the adjustment conditions to obtain new resource overcommitment parameters and using the new resource overcommitment parameters as the current resource overcommitment parameters of this computing resource include:

[0020] For each type of computing resource, conduct a comparative analysis of the current resource pre-allocation value and the average value of the actual resource usage of this type of computing resource to determine whether the current resource pre-allocation value of this type of computing resource meets the first adjustment condition. If it meets, adjust the current resource pre-allocation value of this type of computing resource according to the preset resource pre-allocation value adjustment rule to obtain the adjusted resource pre-allocation value, use the adjusted resource pre-allocation value as the current resource pre-allocation value of this type of computing resource, and return to execute the step of conducting a comparative analysis of the current resource pre-allocation value and the average value of the actual resource usage of this type of computing resource to determine whether the current resource pre-allocation value of this type of computing resource meets the first adjustment condition until the current resource pre-allocation value of this type of computing resource no longer meets the first adjustment condition to obtain the new resource pre-allocation value, and use the new resource pre-allocation value as the current resource pre-allocation value of this type of computing resource;

[0021] Conduct a comparative analysis of the current resource upper limit value and the maximum value of the actual resource usage of this type of computing resource to determine whether the current resource upper limit value of this type of resource meets the second adjustment condition. If it meets, adjust the current resource upper limit value of this type of computing resource according to the preset resource upper limit value adjustment rule to obtain the adjusted resource upper limit value, use the adjusted resource upper limit value as the current resource upper limit value of this type of computing resource, and return to execute the step of conducting a comparative analysis of the current resource upper limit value and the maximum value of the actual resource usage of this type of computing resource to determine whether the current resource upper limit value of this type of resource meets the second adjustment condition until the current resource upper limit value of this type of computing resource no longer meets the second adjustment condition to obtain the new resource upper limit value, and use the new resource upper limit value as the current resource upper limit value of this type of computing resource.

[0022] Optionally, the step of comparing and analyzing the current resource pre-allocation value and the average value of the actual resource usage of this type of computing resource to determine whether the current resource pre-allocation value of this type of computing resource meets the first adjustment condition includes:

[0023] Determine whether the ratio of the current resource pre-allocation value of this type of computing resource being greater than the average value of the actual resource usage within a preset time period exceeds a preset ratio threshold. If not, determine that the current resource pre-allocation value of this type of computing resource meets the first adjustment condition.

[0024] Optionally, the step of comparing and analyzing the current resource upper limit value and the maximum value of the actual resource usage of this type of resource to determine whether the current resource upper limit value of this type of resource meets the second adjustment condition includes:

[0025] Determine whether the difference between the current resource upper limit value of this type of computing resource and the maximum value of the actual resource usage is within a preset difference range. If not, determine that the current resource upper limit value of this type of computing resource meets the second adjustment condition.

[0026] Optionally, each type of computing resource includes compressible resources and non-compressible resources. The non-compressible resources at least include memory, and the method further includes:

[0027] Monitor whether there are computing tasks that fail due to memory overflow in each computing node;

[0028] If there are, calculate a new memory upper limit value based on the current memory upper limit value of the failed computing task and a preset memory automatic upscaling ratio;

[0029] Use the new memory upper limit value as the current memory upper limit value of the failed computing task.

[0030] Thus, by calculating a new memory upper limit value based on the current memory upper limit value of the failed computing task and a preset memory automatic upscaling ratio, the purpose of memory upscaling for the failed computing task is achieved, so that the failed computing task can be successfully calculated, increasing the total amount of computing tasks running on the computing node and improving resource utilization.

[0031] Moreover, since each type of computing resource in the embodiments of the present application includes compressible resources and non-compressible resources, not only can resource oversubscription of compressible resources be realized, but also resource oversubscription of non-compressible resources can be realized.

[0032] Optionally, after the step of obtaining the new resource upper limit value, the resource oversubscription method for the above computing task further includes:

[0033] For each type of computing resource, calculate the ratio of the new resource pre-allocation value to the new resource upper limit value as the new oversubscription coefficient, and store it in the database.

[0034] Thus, for each type of computing resource, by calculating the ratio of the new resource pre-allocation value to the new resource upper limit value as the new oversubscription coefficient and storing it in the database, the purpose of updating the oversubscription coefficient is achieved. And since the oversubscription ratio is the reciprocal of the oversubscription coefficient, the purpose of adaptively adjusting the oversubscription ratio is achieved.

[0035] Optionally, each type of computing resource consists of computing logical resources and / or data resources.

[0036] Thus, by splitting the computing resources into resources consisting of computing logical resources and / or data resources, compared with the method without splitting, the computing resources can be controlled more precisely.

[0037] In a second aspect, the present application provides a resource oversubscription device for computing tasks, which is applied to a resource oversubscription system. The device includes:

[0038] A current resource oversubscription parameter determination module, configured to determine the current resource oversubscription parameters of various types of computing resources used when creating a computing task;

[0039] A monitoring module, configured to monitor the current remaining resource values of various types of computing resources in each computing node of the resource oversubscription system;

[0040] A scheduling module, configured to schedule the computing task to an idle computing node according to the current resource oversubscription parameters and the current remaining resource values when the computing task meets the task scheduling conditions;

[0041] An acquisition module, configured to acquire the total amount of computing tasks executed by the idle computing node at the current moment and the actual resource usage of various types of computing resources used by each computing task when the idle computing node executes the computing task;

[0042] A calculation module, configured to calculate the actual resource usage metrics of various types of computing resources according to the total amount and the actual resource usage of various types of computing resources used by each computing task;

[0043] An adjustment module, configured to, for each type of computing resource, when it is determined through comparative analysis of the current resource oversubscription parameter and the actual resource usage metric of this type of computing resource that the current resource oversubscription parameter of this type of computing resource meets the adjustment condition, adjust the current resource oversubscription parameter of this type of computing resource until it does not meet the adjustment condition to obtain a new resource oversubscription parameter, use the new resource oversubscription parameter as the current resource oversubscription parameter of this computing resource, and trigger the monitoring module.

[0044] Optionally, the current resource oversubscription parameter determination module includes:

[0045] An acquisition sub-module, configured to, when creating a computing task, acquire the resource oversubscription parameter of the processor used for creating the computing task from the database of the resource oversubscription device;

[0046] A calculation sub-module, configured to calculate the current resource oversubscription parameter of various types of computing resources used for creating the computing task according to the resource oversubscription parameter of the processor.

[0047] Optionally, the calculation module is configured to:

[0048] For each type of computing resource, calculate the sum of the actual resource usage amounts of all computing tasks using this type of computing resource, calculate the quotient between the sum and the total amount, use the quotient as the average value of the actual resource usage amount of this type of computing resource, and use the maximum value of the actual resource usage amount of this type of computing resource among all computing tasks as the maximum value of the actual resource usage amount of this type of computing resource.

[0049] Optionally, the current resource oversubscription parameter of various types of computing resources includes a current resource pre-allocation value and a current resource upper limit value. The adjustment module includes:

[0050] A first comparison and analysis sub-unit, configured to, for each type of computing resource, compare and analyze the current resource pre-allocation value and the average value of the actual resource usage amount of this type of computing resource to determine whether the current resource pre-allocation value of this type of computing resource meets the first adjustment condition. If it meets, trigger the first adjustment sub-unit;

[0051] The first adjustment sub-unit is configured to adjust the current resource pre-allocation value of this type of computing resource according to a preset resource pre-allocation value adjustment rule to obtain an adjusted resource pre-allocation value, use the adjusted resource pre-allocation value as the current resource pre-allocation value of this type of computing resource, trigger the first comparison and analysis sub-unit, until the current resource pre-allocation value of this type of computing resource does not meet the first adjustment condition to obtain a new resource pre-allocation value, and use the new resource pre-allocation value as the current resource pre-allocation value of this type of computing resource;

[0052] A second comparison and analysis sub-unit, configured to compare and analyze the current resource upper limit value and the maximum value of the actual resource usage amount of this type of computing resource to determine whether the current resource upper limit value of this type of resource meets the second adjustment condition. If it meets, trigger the second adjustment sub-unit;

[0053] The second adjustment subunit is configured to adjust the current resource upper limit value of this type of computing resource according to a preset resource upper limit value adjustment rule to obtain an adjusted resource upper limit value, use the adjusted resource upper limit value as the current resource upper limit value of this type of computing resource, trigger the second comparison and analysis subunit, and stop until the current resource upper limit value of this type of computing resource does not meet the second adjustment condition to obtain a new resource upper limit value, and use the new resource upper limit value as the current resource upper limit value of this type of computing resource.

[0054] Optionally, the first comparison and analysis subunit is configured to:

[0055] Determine whether the ratio of the current resource pre-allocation value of this type of computing resource to the average actual resource usage amount within a preset time period exceeds a preset ratio threshold. If not, determine that the current resource pre-allocation value of this type of computing resource meets the first adjustment condition.

[0056] Optionally, the second comparison and analysis subunit is configured to:

[0057] Determine whether the difference between the current resource upper limit value of this type of computing resource and the maximum value of the actual resource usage amount is within a preset difference range. If not, determine that the current resource upper limit value of this type of computing resource meets the second adjustment condition.

[0058] Optionally, each type of computing resource includes compressible resources and incompressible resources. The incompressible resources at least include memory. The resource oversubscription device for the above computing tasks further includes:

[0059] A memory overflow module, configured to monitor whether there are computing tasks that fail due to memory overflow in each computing node. If so, trigger the upgrade module;

[0060] The upgrade module is configured to calculate a new memory upper limit value according to the current memory upper limit value of the failed computing task and a preset memory automatic upgrade ratio;

[0061] The current memory upper limit value determination module is configured to use the new memory upper limit value as the current memory upper limit value of the failed computing task.

[0062] Optionally, the resource oversubscription device for the above computing tasks further includes:

[0063] A storage module, configured to, after obtaining the new resource upper limit value, for each type of computing resource, calculate the ratio of the new resource pre-allocation value to the new resource upper limit value as a new oversubscription coefficient, and store it in the database.

[0064] Optionally, each type of computing resource consists of computing logical resources and / or data resources.

[0065] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method for overbooking resources of a computing task provided in the first aspect of the present application is implemented.

[0066] In a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device includes:

[0067] One or more processors;

[0068] The processor is coupled to a storage device for storing programs;

[0069] When the program is executed by the one or more processors, the electronic device implements the method for overbooking resources of a computing task provided in the first aspect of the present application.

[0070] In a fifth aspect, an embodiment of the present application provides a vehicle, which includes the method for overbooking resources of a computing task provided in the first aspect of the present application, or includes the electronic device provided in the fourth aspect of the present application.

[0071] As can be seen from the above, a method, apparatus, device, medium, and vehicle for resource oversubscription of computing tasks provided by an embodiment of the present application can determine the current resource oversubscription parameters of various types of computing resources used when creating a computing task, monitor the current remaining resource values of various types of computing resources in each computing node of the resource oversubscription system, and when the computing task meets the task scheduling condition, schedule the computing task to an idle computing node according to the current resource oversubscription parameters and the current remaining resource values. When the computing task is executed on the idle computing node, collect the total amount of the computing tasks executed by the idle computing node at the current moment and the actual resource usage of various types of computing resources used by each computing task, calculate the actual resource usage indicators of various types of computing resources based on the total amount and the actual resource usage of various types of computing resources used by each computing task. For each type of computing resource, when it is determined through comparative analysis of the current resource oversubscription parameter and the actual resource usage indicator of this type of computing resource that the current resource oversubscription parameter of this type of computing resource meets the adjustment condition, adjust the current resource oversubscription parameter of this type of computing resource until it does not meet the adjustment condition to obtain a new resource oversubscription parameter, use the new resource oversubscription parameter as the current resource oversubscription parameter of this computing resource, and return to the step of monitoring the current remaining resource values of various types of computing resources in each computing node of the resource oversubscription system. In the present application, by comparing and analyzing the current resource oversubscription parameter and the actual resource usage indicator, and adjusting the current resource oversubscription parameter until it does not meet the adjustment condition to obtain a new resource oversubscription parameter, a closed-loop monitoring of the resource oversubscription parameter is realized, and the purpose of adjusting the resource oversubscription parameter according to the actual resource usage is achieved, so that the new resource oversubscription parameter is in line with the actual situation and will not be too large or too small. Therefore, the processing speed can be improved, and more computing tasks can be successfully processed in a shorter time, thereby improving the resource utilization rate and ensuring the stability of the system.

[0072] Of course, implementing any product or method of the present application does not necessarily require achieving all the above-mentioned advantages simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0074] Figure 1 FIG. 2 is a schematic flowchart of a method for resource oversubscription of computing tasks provided by an embodiment of the present application;

[0075] FIG. 2(a) is a schematic diagram of the actual resource usage indicator of the CPU of the 1877 model processor;

[0076] Figure 2(b) is a schematic diagram of the actual resource usage metrics of the memory of the 1877 model processor;

[0077] Figure 3(a) is a schematic diagram of the CPU usage rate improvement;

[0078] Figure 3(b) is a schematic diagram of the memory usage improvement;

[0079] Figure 4 is a flowchart for resource oversubscription of the resource oversubscription system;

[0080] Figure 5 is a schematic structural diagram of a resource oversubscription device for computing tasks provided by an embodiment of the present application;

[0081] Figure 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0082] Figure 7 is a schematic structural diagram of a vehicle provided by an embodiment of the present application. Detailed implementation manners

[0083] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0084] It should be noted that the terms "include" and "have" in the embodiments of the present application and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, method, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0085] The embodiments of the present application disclose a resource oversubscription method, device, equipment, medium, and vehicle for computing tasks, which can improve resource utilization and processing speed without affecting the stability of the system while the amount of computing resources of the computing node remains unchanged. The embodiments of the present application will be described in detail below.

[0086] Figure 1 is a schematic flowchart of a resource oversubscription method for computing tasks provided by an embodiment of the present application. This method is applied to a resource oversubscription system. The method specifically includes the following steps.

[0087] S110: Determine the current resource overcommitment parameters of various computing resources used when creating a computing task.

[0088] To perform resource overcommitment, it is necessary to determine the current resource overcommitment parameters of various computing resources used when creating a computing task. Exemplarily, the computing task can be a large file processing task or a K8s computing task, where K8s is short for Kubernetes, which is an open-source system used to manage containerized applications on multiple hosts in a cloud platform.

[0089] Among them, step S110 may include:

[0090] When creating a computing task, obtain the resource overcommitment parameters of the processor used to create the computing task from the database of the resource overcommitment system;

[0091] Calculate the current resource overcommitment parameters of various computing resources used to create the computing task based on the resource overcommitment parameters of the processor.

[0092] In order to execute different computing tasks, developers need to create a processor corresponding to the computing task through the resource overcommitment system, set the resource overcommitment parameters of the processor, and save the resource overcommitment parameters of the processor to the database of the resource overcommitment system. Among them, the method for the resource overcommitment system to create a processor is any one of the existing methods for creating a processor, which will not be elaborated here.

[0093] Specifically, the resource overcommitment parameters of the processor include the resource upper limit value and the resource overcommitment coefficient. The resource overcommitment system setting the resource overcommitment parameters of the processor can be as follows:

[0094] When receiving the resource estimated upper limit value provided by the developer, the resource overcommitment system sets the resource upper limit value to the resource estimated upper limit value;

[0095] When the developer does not provide the resource estimated upper limit value, the resource overcommitment system sets the resource upper limit value to the resource upper limit experience value;

[0096] The resource overcommitment system sets the resource overcommitment coefficient to 1.

[0097] Exemplarily, the resource overcommitment parameters of the processor include compressible resource overcommitment parameters and incompressible resource overcommitment parameters. The compressible resource overcommitment parameters at least include the CPU (central processing unit) upper limit value and the CPU overcommitment coefficient, and the incompressible resource overcommitment parameters at least include the memory upper limit value and the memory overcommitment coefficient.

[0098] Among them, setting the resource upper limit value of the compressible resource too large will slow down the computing task, and setting the resource upper limit value of the incompressible resource too large will cause the computing task to fail.

[0099] Specifically, for the CPU upper limit value: If it is set too large, other computing tasks on the same computing node will become slower; if it is set too small, when the CPU used by a certain computing task reaches its upper limit, the current computing task will become slower.

[0100] For the CPU oversubscription coefficient: If it is set too large, the resource utilization rate is not high; if it is set too small, when scheduling too many computing tasks, even if each computing task does not exceed its own CPU upper limit value, there may still be a situation where the cumulative value of the CPUs used by all computing tasks reaches the CPU upper limit value of the computing node, resulting in all computing tasks becoming slower.

[0101] For the memory upper limit value: If it is set too large, when a certain computing task uses an abnormally large amount of memory, all computing tasks on the same computing node are at risk of memory overflow; if it is set too small, when the memory used by a certain computing task reaches the memory upper limit value, the current computing task will experience memory overflow.

[0102] For the memory oversubscription coefficient: If it is set too large, the resource utilization rate is not high; if it is set too small, when scheduling too many computing tasks, even if each computing task does not exceed its own memory upper limit value, there may still be a situation where the cumulative value of the memory used by all computing tasks reaches the memory upper limit value of the computing node, resulting in all computing tasks being at risk of memory overflow.

[0103] Therefore, the database stores the resource oversubscription parameters of different processors. When the resource oversubscription system creates a computing task, it can obtain the resource oversubscription parameters of the processor used to create the computing task from the database.

[0104] After obtaining the resource oversubscription parameters of the processor, the current resource oversubscription parameters of various computing resources used to create the computing task can be calculated according to the resource oversubscription parameters of the processor.

[0105] Among them, the current resource oversubscription parameters of various computing resources include the current resource pre-allocation value and the current resource upper limit value. The above calculation of the current resource oversubscription parameters of various computing resources used to create the computing task according to the resource oversubscription parameters of the processor can include:

[0106] Taking the resource upper limit value of various computing resources as the current resource upper limit value of various computing resources;

[0107] Taking the product of the resource upper limit value of various computing resources and the corresponding resource oversubscription coefficient as the current resource pre-allocation value of various computing resources.

[0108] For example, the upper limit value of the CPU is used as the current resource upper limit value of the CPU, the product of the upper limit value of the CPU and the CPU oversubscription coefficient is used as the current resource pre-allocation value of the CPU, the upper limit value of the memory is used as the current resource upper limit value of the memory, and the product of the upper limit value of the memory and the memory oversubscription coefficient is used as the current resource pre-allocation value of the memory.

[0109] Among them, each type of computing resource consists of computing logical resources and / or data resources. Computing logical resources represent resources related to the processing logic associated with the processor, and data resources represent resources related to the size of the data itself.

[0110] For example: both the CPU and the memory consist of computing logical resources, and the disk consists of computing logical resources and data resources.

[0111] Thus, by splitting the computing resources into resources consisting of computing logical resources and / or data resources, compared with the method of not splitting, it is more accurate to control the computing resources.

[0112] S120: Monitor the current remaining resource values of various types of computing resources in each computing node of the resource oversubscription system.

[0113] The resource oversubscription system may include one or more computing clusters, and each computing cluster includes multiple computing nodes.

[0114] In order to schedule computing tasks to the computing cluster, it is necessary to monitor the current remaining resource values of various types of computing resources in each computing node of the resource oversubscription system. Among them, the monitoring method can be a polling method.

[0115] S130: When the computing task meets the task scheduling conditions, schedule the computing task to an idle computing node according to the current resource oversubscription parameters and the current remaining resource values.

[0116] After monitoring the current remaining resource values, it is necessary to determine whether the computing task meets the task scheduling conditions. When the computing task meets the task scheduling conditions, schedule the computing task to an idle computing node according to the current resource oversubscription parameters and the current remaining resource values.

[0117] Among them, determining whether the computing task meets the task scheduling conditions may include:

[0118] Determine whether the computing task meets the task scheduling conditions according to the task priority of the computing task, the priority of the processor used to create the computing task, and / or the creation time of the computing task.

[0119] Specifically, determining whether the computing task meets the task scheduling conditions according to the task priority of the computing task, the priority of the processor used to create the computing task, and / or the creation time of the computing task can be:

[0120] Determine whether the task priority of the computing task is the same as that of other computing tasks to be scheduled;

[0121] If the task priorities are different, determine whether the task priority of the computing task is the highest;

[0122] If it is the highest, determine that the computing task meets the task scheduling condition;

[0123] If it is not the highest, determine whether the priority of the processor used by the computing task is the same as that of the processors of other computing tasks to be scheduled;

[0124] If the priorities of the processors are different, determine whether the priority of the processor used by the computing task is the highest;

[0125] If it is the highest, determine that the computing task meets the task scheduling condition;

[0126] If it is not the highest, determine whether the creation time of the computing task is the earliest;

[0127] If it is the earliest, determine that the computing task meets the task scheduling condition.

[0128] Since there may be multiple computing tasks to be scheduled at the same time, the task priorities of each computing task to be scheduled may be the same or different. When the computing task with the highest priority cannot be found according to the magnitude of the task priorities, it is determined according to the priority of the processor. When the computing task with the highest priority cannot be found according to the priority of the processor, it is determined according to the creation time.

[0129] When it is determined that the computing task meets the task scheduling condition, the computing task can be scheduled to an idle computing node according to the current resource overcommitment parameter and the current remaining resource value.

[0130] Among them, the current resource overcommitment parameters of various computing resources include the current resource pre-allocation value and the current resource upper limit value. Scheduling the computing task to an idle computing node according to the current resource overcommitment parameter and the current remaining resource value may include:

[0131] Determine the computing nodes in the resource overcommitment system that satisfy that the current remaining resource value of each type of computing resource is greater than the current resource pre-allocation value;

[0132] When there is only one determined computing node, use the determined computing node as the idle computing node and schedule the computing task to the idle computing node;

[0133] When there is more than one determined computing node, select one computing node from the determined computing nodes as the idle computing node according to the preset selection rule and schedule the computing task to the idle computing node.

[0134] For a certain computing node, if the current remaining resource value of each type of computing resource satisfies being greater than the current resource pre-allocation value, it indicates that the computing node can run computing tasks. If there is only one computing node that meets this condition, then this computing node can be used as the idle computing node. If there are multiple ones, then one needs to be selected from them as the idle computing node, and the selection method is a preset selection rule. The preset selection rule can be any selection rule in the prior art, and the embodiments of the present application do not make any limitation thereto.

[0135] S140: When executing a computing task on an idle computing node, collect the total amount of the computing tasks executed by the idle computing node at the current moment and the actual resource usage amounts of each type of computing resource used by each computing task.

[0136] After scheduling the computing task to the idle computing node, the idle computing node can execute the computing task. Since the idle computing node may execute multiple tasks simultaneously, in order to determine whether the previously set current resource overbooking parameter is appropriate, it is necessary to collect the total amount of the computing tasks executed by the idle computing node at the current moment and the actual resource usage amounts of each type of computing resource used by each computing task.

[0137] S150: Calculate the actual resource usage amount indicators of each type of computing resource based on the total amount and the actual resource usage amounts of each type of computing resource used by each computing task.

[0138] After collecting the total amount and the actual resource usage amounts of each type of computing resource used by each computing task, the actual resource usage amount indicators of each type of computing resource can be calculated based on the total amount and the actual resource usage amounts of each type of computing resource used by each computing task.

[0139] Among them, step S150 may include:

[0140] For each type of computing resource, calculate the sum of the actual resource usage amounts of all computing tasks using this type of computing resource, calculate the quotient between the sum and the total amount, use the quotient as the average actual resource usage amount of this type of computing resource, and use the maximum value of the actual resource usage amounts of all computing tasks using this type of computing resource as the maximum actual resource usage amount of this type of computing resource.

[0141] That is to say, the actual resource usage amount indicators include the average actual resource usage amount and the maximum actual resource usage amount.

[0142] For example: FIG. 2(a) is a schematic diagram of the actual resource usage metrics of the CPU of the 1877 model processor. The actual resource usage metrics of the CPU include the average value of the actual CPU usage and the maximum value of the actual CPU usage. In FIG. 2(a), the abscissa is the date and the ordinate is the number of CPU cores; FIG. 2(b) is a schematic diagram of the actual resource usage metrics of the memory of the 1877 model processor. The actual resource usage metrics of the memory include the average value of the actual memory usage and the maximum value of the actual memory usage. In FIG. 2(b), the abscissa is the date and the ordinate is the storage unit GIB of the memory.

[0143] S160: For each type of computing resource, when it is determined through comparative analysis of the current resource overcommitment parameter and the actual resource usage metrics of this type of computing resource that the current resource overcommitment parameter of this type of computing resource meets the adjustment condition, adjust the current resource overcommitment parameter of this type of computing resource until it no longer meets the adjustment condition to obtain a new resource overcommitment parameter, use the new resource overcommitment parameter as the current resource overcommitment parameter of this computing resource, and return to execute step S120.

[0144] After obtaining the actual resource usage metrics, it is possible to conduct comparative analysis of the actual resource usage metrics and the current resource overcommitment parameter, thereby continuously optimizing and adjusting the current resource overcommitment parameter.

[0145] Among them, the current resource overcommitment parameters of various types of computing resources include the current resource pre-allocation value and the current resource upper limit value. Step S160 may include:

[0146] For each type of computing resource, conduct comparative analysis of the current resource pre-allocation value and the average value of the actual resource usage of this type of computing resource to determine whether the current resource pre-allocation value of this type of computing resource meets the first adjustment condition. If it meets, adjust the current resource pre-allocation value of this type of computing resource according to the preset resource pre-allocation value adjustment rule to obtain an adjusted resource pre-allocation value, use the adjusted resource pre-allocation value as the current resource pre-allocation value of this type of computing resource, and return to execute the step of conducting comparative analysis of the current resource pre-allocation value and the average value of the actual resource usage of this type of computing resource to determine whether the current resource pre-allocation value of this type of computing resource meets the first adjustment condition until the current resource pre-allocation value of this type of computing resource no longer meets the first adjustment condition to obtain a new resource pre-allocation value, and use the new resource pre-allocation value as the current resource pre-allocation value of this type of computing resource;

[0147] Compare and analyze the current resource upper limit value and the maximum value of the actual resource usage of this type of computing resource to determine whether the current resource upper limit value of this type of resource meets the second adjustment condition. If it meets, adjust the current resource upper limit value of this type of computing resource according to the preset resource upper limit value adjustment rule to obtain the adjusted resource upper limit value. Take the adjusted resource upper limit value as the current resource upper limit value of this type of computing resource, and return to execute the step of comparing and analyzing the current resource upper limit value and the maximum value of the actual resource usage of this type of computing resource to determine whether the current resource upper limit value of this type of resource meets the second adjustment condition until the current resource upper limit value of this type of computing resource does not meet the second adjustment condition to obtain the new resource upper limit value, and take the new resource upper limit value as the current resource upper limit value of this type of computing resource.

[0148] Continue to refer to FIGS. 2(a) and 2(b). Taking the CPU and memory as examples, the current resource oversubscription parameters of the CPU include the current CPU pre-allocation value and the current CPU upper limit value, and the current resource oversubscription parameters of the memory include the current memory pre-allocation value and the current memory upper limit value.

[0149] For the convenience of viewing, in the embodiments of the present application, avg is used to represent the average value of the actual resource usage, max is used to represent the maximum value of the actual resource usage, request is used to represent the current resource pre-allocation value, and limit is used to represent the current resource upper limit value.

[0150] Among them, the function of request is:

[0151] Whether the idle resources of the computing node can create a computing task is calculated according to request. At the same time, after scheduling the computing task, the computing node will also withhold the resource quantity of request.

[0152] The function of limit is:

[0153] Control the resource usage of a single computing task not to exceed the upper limit, because the computing speed slows down when the compressible resources reach the upper limit, and the computing task fails when the in-compressible resources reach the upper limit.

[0154] The method for judging whether request and limit meet the adjustment condition is as follows:

[0155] 1. Compare and analyze avg and request to determine whether request meets the first adjustment condition:

[0156] Since the total actual used resources = the average value of the actual resource usage × the total number of computing tasks, and the total pre-allocated resources = the sum of the current resource pre-allocation values of all computing tasks, therefore, adjusting request according to avg is to adjust the total pre-allocated resources according to the total actual used resources.

[0157] When avg is much smaller than request, it indicates that the request is set too large, and too much computing resource is pre-allocated, resulting in low resource utilization rate.

[0158] When avg is much larger than request, it indicates that the request is set too small, and too little computing resource is pre-allocated, resulting in mutual influence among the shared computing resources of computing tasks.

[0159] When request exceeds most of the avg, it indicates that the request is set reasonably.

[0160] That is to say, adjustment is needed when the request is set unreasonably. Therefore, when the request is set unreasonably, the first adjustment condition is satisfied.

[0161] Specifically, comparing and analyzing the current resource pre-allocation value and the average value of the actual resource usage of this type of computing resource to determine whether the current resource pre-allocation value of this type of computing resource satisfies the first adjustment condition may include:

[0162] Judging whether the proportion of the current resource pre-allocation value of this type of computing resource in the preset time period that is greater than the average value of the actual resource usage exceeds the preset proportion threshold. If not, it is determined that the current resource pre-allocation value of this type of computing resource satisfies the first adjustment condition.

[0163] Among them, the proportion of the current resource pre-allocation value of this type of computing resource in the preset time period that is greater than the average value of the actual resource usage exceeding the preset proportion threshold indicates that request exceeds most of the avg. Exemplarily, the preset proportion threshold is 90%.

[0164] Correspondingly, when it is determined that the current resource pre-allocation value of this type of computing resource satisfies the first adjustment condition, the current resource pre-allocation value of this type of computing resource is adjusted according to the preset resource pre-allocation value adjustment rule to obtain the adjusted resource pre-allocation value. Among them, the preset resource pre-allocation value adjustment rule is to increase the current resource pre-allocation value according to the preset adjustment ratio.

[0165] Since the adjusted resource pre-allocation value may not necessarily be reasonable, it is necessary to use the adjusted resource pre-allocation value as the current resource pre-allocation value of this type of computing resource, and return to execute the step of comparing and analyzing the current resource pre-allocation value and the average value of the actual resource usage of this type of computing resource to determine whether the current resource pre-allocation value of this type of computing resource meets the first adjustment condition, until the current resource pre-allocation value of this type of computing resource does not meet the first adjustment condition to obtain a new resource pre-allocation value, that is, until the ratio of the current resource pre-allocation value of this type of computing resource to the average value of the actual resource usage within the preset time period exceeds the preset ratio threshold, and then use the new resource pre-allocation value as the current resource pre-allocation value of this type of computing resource. At this time, the current resource pre-allocation value is reasonable.

[0166] 2. Compare and analyze max and limit to determine whether limit meets the second adjustment condition:

[0167] When max is much smaller than limit, it means that the limit is set too large, and computing tasks will respond to each other, resulting in poor resource isolation between computing tasks;

[0168] When max is slightly higher than limit, it means that the limit is set too small, which will slow down or fail the current computing task;

[0169] When limit is slightly higher than max, it means that the limit is set reasonably.

[0170] That is to say, when the limit is set unreasonably, it needs to be adjusted. Therefore, when the limit is set unreasonably, it meets the second adjustment condition.

[0171] Specifically, comparing and analyzing the current resource upper limit value and the maximum value of the actual resource usage of this type of computing resource to determine whether the current resource upper limit value of this type of resource meets the second adjustment condition may include:

[0172] Judge whether the difference between the current resource upper limit value of this type of computing resource and the maximum value of the actual resource usage is within the preset difference range. If not, determine that the current resource upper limit value of this type of computing resource meets the second adjustment condition.

[0173] Correspondingly, when it is determined that the current resource upper limit value of this type of computing resource meets the second adjustment condition, adjust the current resource upper limit value of this type of computing resource according to the preset resource upper limit value adjustment rule to obtain an adjusted resource upper limit value. Among them, the preset resource upper limit value adjustment rule is: adjust the current resource pre-allocation value according to the preset adjustment ratio to reduce the difference between the current resource pre-allocation value and the maximum value of the actual resource usage.

[0174] Since the adjusted resource upper limit value may not necessarily be reasonable, it is necessary to use the adjusted resource upper limit value as the current resource upper limit value of this type of computing resource, and return to execute the step of comparing and analyzing the maximum value of the current resource upper limit value and the actual resource usage of this type of computing resource to determine whether the current resource upper limit value of this type of resource meets the second adjustment condition, until the current resource upper limit value of this type of computing resource does not meet the second adjustment condition to obtain a new resource upper limit value, that is, until the difference between the current resource upper limit value of this type of computing resource and the maximum value of the actual resource usage is within the preset difference range, and then use the new resource upper limit value as the current resource upper limit value of this type of computing resource. At this time, the current resource upper limit value is reasonable.

[0175] Through the method of the embodiments of the present application, after adjusting the current resource oversubscription parameter to be reasonable, the resource utilization rate / resource usage and task throughput have been greatly improved.

[0176] Regarding the resource utilization rate / resource usage:

[0177] For example: Fig. 3(a) is a schematic diagram of the improvement of CPU usage rate. The abscissa of Fig. 3(a) is time, and the ordinate is the CPU usage rate. Referring to Fig. 3(a), on December 31, the current CPU oversubscription parameter is adjusted to be reasonable. It can be seen that the user CPU usage rate has increased from 60% to 90%.

[0178] Fig. 3(b) is a schematic diagram of the improvement of memory usage. The abscissa of Fig. 3(b) is time, and the ordinate is the storage unit GIB of memory. Referring to Fig. 3(b), on December 31, the current memory oversubscription parameter is adjusted to be reasonable. It can be seen that the memory usage has increased from 20GB to 30GB.

[0179] Regarding the task throughput:

[0180] a. For a single 64C128G machine of the DCS (Data Computing Service) main line:

[0181] 1. The single-day task throughput: before adjustment, it was 184, and after adjustment, it was 242, with a 31% increase;

[0182] 2. The single-day bag throughput: before adjustment, it was 2944, and after adjustment, it was 3872, with a 31% increase;

[0183] b. In the case of the total project resources remaining unchanged, the single-day task throughput:

[0184] Before adjustment, it was 5604, and after adjustment, it was 10476, with an 87% increase.

[0185] As can be seen from the above, in this embodiment, the current resource oversubscription parameters of various types of computing resources used when creating a computing task can be determined, the current remaining resource values of various types of computing resources in each computing node of the resource oversubscription system can be monitored, when the computing task meets the task scheduling conditions, the computing task can be scheduled to an idle computing node according to the current resource oversubscription parameters and the current remaining resource values, when the computing task is executed on the idle computing node, the total amount of computing tasks executed by the idle computing node at the current moment and the actual resource usage of various types of computing resources used by each computing task are collected, the actual resource usage metrics of various types of computing resources are calculated based on the total amount and the actual resource usage of various types of computing resources used by each computing task, for each type of computing resource, when it is determined through comparative analysis of the current resource oversubscription parameter and the actual resource usage metric of this type of computing resource that the current resource oversubscription parameter of this type of computing resource meets the adjustment condition, the current resource oversubscription parameter of this type of computing resource is adjusted until it does not meet the adjustment condition to obtain a new resource oversubscription parameter, the new resource oversubscription parameter is used as the current resource oversubscription parameter of this computing resource, and the step of monitoring the current remaining resource values of various types of computing resources in each computing node of the resource oversubscription system is returned. In this application, by comparing and analyzing the current resource oversubscription parameter and the actual resource usage metric, and adjusting the current resource oversubscription parameter until it does not meet the adjustment condition to obtain a new resource oversubscription parameter, the purpose of closed-loop monitoring of the resource oversubscription parameter and adjusting the resource oversubscription parameter according to the actual resource usage is realized, so that the new resource oversubscription parameter is in line with the actual situation and will not be too large or too small. Therefore, the processing speed can be improved, and more computing tasks can be successfully processed in a shorter time, thereby improving the resource utilization rate and ensuring the stability of the system.

[0186] Moreover, in this embodiment of the application, resource oversubscription is set for computing tasks. Compared with the existing method of setting resource oversubscription for computing nodes, the situation where some computing tasks in a computing node cannot perform resource oversubscription is avoided.

[0187] In one implementation, after the above step of obtaining the new resource upper limit value, the resource oversubscription method for the above computing task may further include:

[0188] For each type of computing resource, calculate the ratio of the new resource pre-allocation value to the new resource upper limit value as the new oversubscription coefficient and store it in the database.

[0189] Thus, for each type of computing resource, by calculating the ratio of the new resource pre-allocation value to the new resource upper limit value as the new oversubscription coefficient and storing it in the database, the purpose of updating the oversubscription coefficient is achieved, and since the oversubscription ratio is the reciprocal of the oversubscription coefficient, the purpose of adaptively adjusting the oversubscription ratio is achieved.

[0190] In another implementation, various computing resources include compressible resources and incompressible resources, and the incompressible resources at least include memory. The method for oversubscribing resources of the above computing tasks may further include:

[0191] Monitoring whether there are computing tasks that fail due to memory overflow in each computing node;

[0192] If there are, calculating a new memory upper limit value according to the current memory upper limit value of the failed computing task and a preset automatic memory upgrade ratio;

[0193] Taking the new memory upper limit value as the current memory upper limit value of the failed computing task.

[0194] If there are computing tasks that fail due to memory overflow in the computing node, it means that the current memory upper limit value of the failed computing task is not set large enough. In order to make the computing task succeed, within the maximum number of retries, the failed computing task can be automatically retried after its memory is upgraded. Exemplarily, the maximum number of retries is 5 times.

[0195] Among them, upgrading the memory of the failed computing task means calculating a new memory upper limit value according to the current memory upper limit value of the failed computing task and a preset automatic memory upgrade ratio, and taking the new memory upper limit value as the current memory upper limit value of the failed computing task. Exemplarily, the preset automatic memory upgrade ratio is 1.5 times.

[0196] Thus, by calculating a new memory upper limit value according to the current memory upper limit value of the failed computing task and a preset automatic memory upgrade ratio, the purpose of upgrading the memory of the failed computing task is achieved, so that the failed computing task can be calculated successfully, increasing the total amount of computing tasks running on the computing node and improving resource utilization.

[0197] Moreover, since various computing resources in the embodiments of the present application include compressible resources and incompressible resources, therefore, not only can oversubscribing of compressible resources be realized, but also oversubscribing of incompressible resources can be realized.

[0198] For the convenience of understanding, the oversubscribing of resources of computing tasks will be introduced from the perspective of the system below:

[0199] The resource oversubscribing system includes a scheduler, a resource usage monitoring unit, a data collector, and a computing cluster. The scheduler is communicatively connected to the computing cluster, and the data collector is communicatively connected to the computing cluster and the resource usage monitoring unit respectively. Among them, the computing cluster includes multiple computing nodes.

[0200] Specifically, the scheduler executes steps S110 - S130 and step S160 in the above - mentioned resource over - subscription method for computing tasks, the data collector executes step S140, and the resource usage monitoring unit executes step S150.

[0201] Figure 4 For the flowchart of resource over - subscription for the resource over - subscription system, see Figure 4 , the scheduler determines the current resource over - subscription parameters and monitors the current remaining resource values, then schedules the computing tasks and allocates computing resources. When the computing tasks are being executed, the data collector collects the actual resource consumption, that is, the total amount of the executed computing tasks and the actual resource usage of various types of computing resources used by each computing task, and sends them to the resource usage monitoring unit. The resource usage monitoring unit conducts statistical analysis to obtain the actual resource usage metrics of various types of computing resources and sends them to the scheduler. The scheduler adjusts the pre - resource over - subscription parameters and updates the current resource over - subscription parameters. For details, see the above text and will not be elaborated here.

[0202] Figure 5 For a schematic structural diagram of a resource over - subscription device for computing tasks provided by an embodiment of the present application, see Figure 5 , an embodiment of the present application provides a resource over - subscription device for computing tasks, which is applied to a resource over - subscription system. The device includes:

[0203] A current resource over - subscription parameter determination module 510, configured to determine the current resource over - subscription parameters of various types of computing resources used when creating computing tasks;

[0204] A monitoring module 520, configured to monitor the current remaining resource values of various types of computing resources in each computing node of the resource over - subscription system;

[0205] A scheduling module 530, configured to schedule the computing tasks to idle computing nodes according to the current resource over - subscription parameters and the current remaining resource values when the computing tasks meet the task scheduling conditions;

[0206] An acquisition module 540, configured to collect the total amount of computing tasks executed by the idle computing node at the current moment and the actual resource usage of various types of computing resources used by each computing task when the idle computing node executes the computing tasks;

[0207] A calculation module 550, configured to calculate the actual resource usage metrics of various types of computing resources based on the total amount and the actual resource usage of various types of computing resources used by each computing task;

[0208] An adjustment module 560 is configured to, for each type of computing resource, when it is determined through comparative analysis of the current resource oversubscription parameter and the actual resource usage metric of this type of computing resource that the current resource oversubscription parameter of this type of computing resource meets the adjustment condition, adjust the current resource oversubscription parameter of this type of computing resource until it no longer meets the adjustment condition to obtain a new resource oversubscription parameter, use the new resource oversubscription parameter as the current resource oversubscription parameter of this computing resource, and trigger the monitoring module 520.

[0209] The device provided by the embodiment of the present application can determine the current resource oversubscription parameters of various types of computing resources used when creating a computing task, monitor the current remaining resource values of various types of computing resources in each computing node of the resource oversubscription system, when the computing task meets the task scheduling condition, schedule the computing task to an idle computing node according to the current resource oversubscription parameter and the current remaining resource value, when the computing task is executed on the idle computing node, collect the total amount of computing tasks executed by the idle computing node at the current moment and the actual resource usage of various types of computing resources used by each computing task, calculate the actual resource usage metrics of various types of computing resources based on the total amount and the actual resource usage of various types of computing resources used by each computing task, for each type of computing resource, when it is determined through comparative analysis of the current resource oversubscription parameter and the actual resource usage metric of this type of computing resource that the current resource oversubscription parameter of this type of computing resource meets the adjustment condition, adjust the current resource oversubscription parameter of this type of computing resource until it no longer meets the adjustment condition to obtain a new resource oversubscription parameter, use the new resource oversubscription parameter as the current resource oversubscription parameter of this computing resource, and return to the step of monitoring the current remaining resource values of various types of computing resources in each computing node of the resource oversubscription system. In the present application, by comparing and analyzing the current resource oversubscription parameter and the actual resource usage metric, and adjusting the current resource oversubscription parameter until it no longer meets the adjustment condition to obtain a new resource oversubscription parameter, a closed-loop monitoring of the resource oversubscription parameter is achieved, and the purpose of adjusting the resource oversubscription parameter according to the actual resource usage is realized, so that the new resource oversubscription parameter is in line with the actual situation and will not be too large or too small. Therefore, the processing speed can be improved, and more computing tasks can be successfully processed in a shorter time, thereby improving the resource utilization rate and ensuring the stability of the system.

[0210] Optionally, the current resource oversubscription parameter determination module 510 may include:

[0211] An acquisition sub-module, configured to, when creating a computing task, acquire the resource oversubscription parameter of the processor used to create the computing task from the database of the resource oversubscription device;

[0212] A calculation sub-module, configured to calculate the current resource oversubscription parameters of various types of computing resources used to create the computing task according to the resource oversubscription parameter of the processor.

[0213] Optionally, the computing module 550 may be used to:

[0214] For each type of computing resource, calculate the sum of the actual resource usage of all computing tasks using this type of computing resource, calculate the quotient between the sum and the total amount, use the quotient as the average value of the actual resource usage of this type of computing resource, and use the maximum value of the actual resource usage of this type of computing resource among all computing tasks as the maximum value of the actual resource usage of this type of computing resource.

[0215] Optionally, the current resource oversubscription parameters of each type of computing resource include the current resource pre-allocation value and the current resource upper limit value. The adjustment module 560 may include:

[0216] The first comparison and analysis subunit is used to, for each type of computing resource, compare and analyze the current resource pre-allocation value and the average value of the actual resource usage of this type of computing resource to determine whether the current resource pre-allocation value of this type of computing resource meets the first adjustment condition. If it meets, trigger the first adjustment subunit;

[0217] The first adjustment subunit is used to adjust the current resource pre-allocation value of this type of computing resource according to the preset resource pre-allocation value adjustment rule to obtain the adjusted resource pre-allocation value, use the adjusted resource pre-allocation value as the current resource pre-allocation value of this type of computing resource, trigger the first comparison and analysis subunit, until the current resource pre-allocation value of this type of computing resource does not meet the first adjustment condition to obtain the new resource pre-allocation value, and use the new resource pre-allocation value as the current resource pre-allocation value of this type of computing resource;

[0218] The second comparison and analysis subunit is used to compare and analyze the current resource upper limit value and the maximum value of the actual resource usage of this type of computing resource to determine whether the current resource upper limit value of this type of resource meets the second adjustment condition. If it meets, trigger the second adjustment subunit;

[0219] The second adjustment subunit is used to adjust the current resource upper limit value of this type of computing resource according to the preset resource upper limit value adjustment rule to obtain the adjusted resource upper limit value, use the adjusted resource upper limit value as the current resource upper limit value of this type of computing resource, trigger the second comparison and analysis subunit, until the current resource upper limit value of this type of computing resource does not meet the second adjustment condition to obtain the new resource upper limit value, and use the new resource upper limit value as the current resource upper limit value of this type of computing resource.

[0220] Optionally, the first comparison and analysis subunit may be used to:

[0221] Determine whether the ratio of the current resource pre - allocation value of this type of computing resource within the preset time period to the average actual resource usage exceeds a preset ratio threshold. If not, determine that the current resource pre - allocation value of this type of computing resource meets the first adjustment condition.

[0222] Optionally, the second comparison and analysis subunit can be used to:

[0223] Determine whether the difference between the current resource upper limit value of this type of computing resource and the maximum value of the actual resource usage is within a preset difference range. If not, determine that the current resource upper limit value of this type of computing resource meets the second adjustment condition.

[0224] Optionally, each type of computing resource includes compressible resources and incompressible resources. The incompressible resources at least include memory. The resource over - selling device for the above - mentioned computing tasks may further include:

[0225] A memory overflow module, configured to monitor whether there are computing tasks that fail due to memory overflow in each of the computing nodes. If so, trigger the up - sizing module;

[0226] The up - sizing module is configured to calculate a new memory upper limit value according to the current memory upper limit value of the failed computing task and a preset memory auto - up - sizing ratio;

[0227] The current memory upper limit value determination module is configured to use the new memory upper limit value as the current memory upper limit value of the failed computing task.

[0228] Optionally, the resource over - selling device for the above - mentioned computing tasks may further include:

[0229] A storage module, configured to, after obtaining the new resource upper limit value, for each type of computing resource, calculate the ratio of the new resource pre - allocation value to the new resource upper limit value as a new over - selling coefficient, and store it in the database.

[0230] Optionally, each type of computing resource is composed of computing logical resources and / or data resources.

[0231] The above device embodiment corresponds to the method embodiment and has the same technical effects as the method embodiment. For specific descriptions, refer to the method embodiment. The device embodiment is obtained based on the method embodiment. For specific descriptions, refer to the method embodiment part and will not be elaborated here.

[0232] Figure 6 The following is a schematic structural diagram of an electronic device provided by an embodiment of the present application, as Figure 6 shown, the electronic device includes:

[0233] One or more processors 610;

[0234] The processor 610 is coupled to a storage device 620 for storing programs;

[0235] When the program is executed by the one or more processors 610, the electronic device implements the resource oversubscription method for computing tasks provided in any embodiment of this application.

[0236] Based on the above embodiments, another embodiment of this application provides a vehicle, which includes the resource oversubscription device for computing tasks provided in any embodiment of this application, or includes the electronic device provided in any embodiment of this application.

[0237] Figure 7 Schematic diagram of a vehicle provided in an embodiment of this application. As Figure 7 shown, the vehicle includes a speed sensor 71, an ECU (Electronic Control Unit) 72, a GPS (Global Positioning System) positioning device 73, and a T-Box (Telematics Box) 74. Among them, the speed sensor 71 is used to measure the vehicle speed and use the vehicle speed as the empirical speed for model training; the GPS positioning device 73 is used to obtain the current geographical location of the vehicle; the T-Box 64 can communicate with the server as a gateway; the ECU 72 can execute the above-mentioned data-driven image processing method.

[0238] In addition, the vehicle may further include: a V2X (Vehicle-to-Everything) module 75, a radar 76, and a camera 77. The V2X module 75 is used to communicate with other vehicles, roadside devices, etc.; the radar 76 or the camera 77 is used to sense road environment information in the front and / or other directions to obtain raw point cloud data; the radar 76 and / or the camera 77 can be configured at the front and / or the rear of the vehicle body.

[0239] Based on the above method embodiments, another embodiment of this application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the resource oversubscription method for computing tasks provided in any embodiment of this application.

[0240] Those of ordinary skill in the art can understand that the drawings are only schematic diagrams of one embodiment, and the modules or processes in the drawings are not necessarily essential for implementing this application.

[0241] Those of ordinary skill in the art can understand that the modules in the devices in the embodiments can be distributed in the devices described in the embodiments, or can be correspondingly changed and located in one or more devices different from the present embodiments. The modules in the above embodiments can be combined into one module, or can be further split into multiple sub-modules.

[0242] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A resource oversubscription method for computing tasks, characterized in that, Applied to a resource overbooking system, the method includes: Determine the current resource overbooking parameters of various computing resources used when creating a computing task; Monitor the current remaining resource values of various computing resources in each computing node of the resource overbooking system; When the computing task meets the task scheduling condition, schedule the computing task to an idle computing node according to the current resource overbooking parameters and the current remaining resource values; When the computing task is executed on the idle computing node, collect the total amount of computing tasks executed by the idle computing node at the current moment and the actual resource usage of various computing resources used by each computing task; Calculate the actual resource usage metrics of various computing resources based on the total amount and the actual resource usage of various computing resources used by each computing task; For each type of computing resource, when it is determined through comparative analysis of the current resource overbooking parameter and the actual resource usage metric of this type of computing resource that the current resource overbooking parameter of this type of computing resource meets the adjustment condition, adjust the current resource overbooking parameter of this type of computing resource until it does not meet the adjustment condition to obtain a new resource overbooking parameter, and use the new resource overbooking parameter as the current resource overbooking parameter of this computing resource, and return to execute the step of monitoring the current remaining resource values of various computing resources in each computing node of the resource overbooking system.

2. The method according to claim 1, characterized in that, The step of determining the current resource overbooking parameters of various computing resources used when creating a computing task includes: When creating a computing task, obtain the resource overbooking parameter of the processor used to create the computing task from the database of the resource overbooking method; Calculate the current resource overbooking parameters of various computing resources used to create the computing task based on the resource overbooking parameter of the processor.

3. The method according to claim 1, characterized in that, The step of calculating the actual resource usage metrics of various computing resources based on the total amount and the actual resource usage of various computing resources used by each computing task includes: For each type of computing resource, calculate the sum of the actual resource usage of all computing tasks using this type of computing resource, calculate the quotient between the sum and the total amount, use the quotient as the average actual resource usage of this type of computing resource, and use the maximum value of the actual resource usage of this type of computing resource among all computing tasks as the maximum actual resource usage of this type of computing resource.

4. The method according to claim 3, characterized in that The current resource overbooking parameters of various computing resources include the current resource pre-allocation value and the current resource upper limit value. The step of, for each type of computing resource, when it is determined through comparative analysis of the current resource overbooking parameter and the actual resource usage metric of this type of computing resource that the current resource overbooking parameter of this type of computing resource meets the adjustment condition, adjusting the current resource overbooking parameter of this type of computing resource until it does not meet the adjustment condition to obtain a new resource overbooking parameter and using the new resource overbooking parameter as the current resource overbooking parameter of this computing resource includes: For each type of computing resource, compare and analyze the current resource pre-allocation value and the average actual resource usage of this type of computing resource to determine whether the current resource pre-allocation value of this type of computing resource meets the first adjustment condition. If it meets, adjust the current resource pre-allocation value of this type of computing resource according to the preset resource pre-allocation value adjustment rule to obtain the adjusted resource pre-allocation value. Take the adjusted resource pre-allocation value as the current resource pre-allocation value of this type of computing resource, and return to execute the step of comparing and analyzing the current resource pre-allocation value and the average actual resource usage of this type of computing resource to determine whether the current resource pre-allocation value of this type of computing resource meets the first adjustment condition until the current resource pre-allocation value of this type of computing resource does not meet the first adjustment condition to obtain the new resource pre-allocation value. Take the new resource pre-allocation value as the current resource pre-allocation value of this type of computing resource; Compare and analyze the current resource upper limit value and the maximum actual resource usage of this type of computing resource to determine whether the current resource upper limit value of this type of resource meets the second adjustment condition. If it meets, adjust the current resource upper limit value of this type of computing resource according to the preset resource upper limit value adjustment rule to obtain the adjusted resource upper limit value. Take the adjusted resource upper limit value as the current resource upper limit value of this type of computing resource, and return to execute the step of comparing and analyzing the current resource upper limit value and the maximum actual resource usage of this type of computing resource to determine whether the current resource upper limit value of this type of resource meets the second adjustment condition until the current resource upper limit value of this type of computing resource does not meet the second adjustment condition to obtain the new resource upper limit value. Take the new resource upper limit value as the current resource upper limit value of this type of computing resource.

5. The method according to claim 4, characterized in that, The step of comparing and analyzing the current resource pre-allocation value and the average actual resource usage of this type of computing resource to determine whether the current resource pre-allocation value of this type of computing resource meets the first adjustment condition includes: Judge whether the ratio of the current resource pre-allocation value of this type of computing resource being greater than the average actual resource usage within the preset time period exceeds the preset ratio threshold. If not, determine that the current resource pre-allocation value of this type of computing resource meets the first adjustment condition.

6. The method according to claim 4, wherein The step of comparing and analyzing the current resource upper limit value and the maximum actual resource usage of this type of computing resource to determine whether the current resource upper limit value of this type of resource meets the second adjustment condition includes: Judge whether the difference between the current resource upper limit value of this type of computing resource and the maximum actual resource usage is within the preset difference range. If not, determine that the current resource upper limit value of this type of computing resource meets the second adjustment condition.

7. A resource oversubscription device for computing tasks, characterized in that, Applied to a resource overbooking system, the device includes: A current resource overbooking parameter determination module, configured to determine the current resource overbooking parameters of various types of computing resources used when creating a computing task; A monitoring module, configured to monitor the current remaining resource values of various types of computing resources in each computing node of the resource overbooking system; A scheduling module, configured to schedule the computing task to an idle computing node according to the current resource oversubscription parameter and the current remaining resource value when the computing task meets the task scheduling condition; An acquisition module, configured to acquire the total amount of computing tasks executed by the idle computing node at the current moment and the actual resource usage of various types of computing resources used by each computing task when the computing task is executed on the idle computing node; A calculation module, configured to calculate the actual resource usage metrics of various types of computing resources based on the total amount and the actual resource usage of various types of computing resources used by each computing task; An adjustment module, for each type of computing resource, when it is determined through comparative analysis of the current resource oversubscription parameter and the actual resource usage metric of this type of computing resource that the current resource oversubscription parameter of this type of computing resource meets the adjustment condition, adjust the current resource oversubscription parameter of this type of computing resource until it does not meet the adjustment condition to obtain a new resource oversubscription parameter, use the new resource oversubscription parameter as the current resource oversubscription parameter of this computing resource, and trigger the monitoring module.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method according to any one of claims 1-6.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; The processor is coupled to a storage device for storing a program, When the program is executed by the one or more processors, the electronic device implements the method according to any one of claims 1-6.

10. A vehicle, characterized in that, The vehicle includes the device according to claim 7, or includes the electronic device according to claim 9.

Images