Task resource allocation method and device, electronic equipment and storage medium

Abstract

The invention provides a task resource allocation method and device, electronic equipment and a storage medium, and the method comprises the steps: classifying a to-be-run task according to the use degree of the to-be-run task for each hardware resource category, so as to identify which hardware resource category is intensively required when the to-be-run task is executed; according to the utilization rate of each hardware resource of the platform, calculating to obtain the resource idle degree of each hardware resource of the platform; generating a resource allocation strategy for the to-be-operated task according to the classification of the to-be-operated task and the resource idle degree of each hardware resource of the platform; and according to a resource allocation strategy for the to-be-run task, allocating corresponding hardware resources of the platform to the to-be-run task so as to execute the to-be-run task. Various hardware resources of the platform can be fully utilized, and the execution efficiency of the to-be-run task is improved.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of resource allocation technology, and in particular, to a task resource allocation method, device, electronic device, and storage medium. Background Art

[0002] With the rapid development of information technology, the tasks undertaken by computing platforms are becoming increasingly complex and diverse, and the demand for resources is also increasing. In order to effectively manage these resources and ensure that tasks can be completed in the shortest time, task scheduling has become a key issue. Traditional task scheduling methods are usually based on preset static strategies, such as FCFS (First Come First Served) and SJF (Shortest Job First). Although these methods are simple and easy to implement, they often fail to fully utilize the idle resources of the computing platform.

[0003] In practical applications, the resource usage status of the computing platform changes dynamically. Sometimes some resources may be in a high-load state, while other resources may be relatively idle. Traditional static policy-based task scheduling methods cannot adjust the allocation of tasks according to these real-time changes, resulting in low resource utilization and task execution delays. In addition, different types of tasks have different requirements for resources. For example, computationally intensive tasks require more CPU resources, while I / O intensive tasks have higher requirements for storage and network resources. Therefore, traditional static policy-based task scheduling methods often cannot meet such diverse needs. Summary of the invention

[0004] The present disclosure provides a task resource allocation method, the method comprising:

[0005] Classifying the tasks to be run according to the degree of use of each hardware resource category by the tasks to be run, so as to identify which hardware resource category the tasks to be run mainly require when executing;

[0006] Calculate the resource idleness of each hardware resource of the platform according to the utilization rate of each hardware resource of the platform;

[0007] Generate a resource allocation strategy for the tasks to be run according to the classification of the tasks to be run and the resource idleness of various hardware resources of the platform;

[0008] According to the resource allocation strategy for the task to be run, the corresponding hardware resources of the platform are allocated to the task to be run, so as to execute the task to be run.

[0009] Optionally, the tasks to be run are classified into at least one of the following according to the usage degree of each hardware resource category by the tasks to be run:

[0010] CPU computing resource intensive tasks, memory storage resource intensive tasks, disk storage resource intensive tasks, disk I / O resource intensive tasks and network bandwidth resource intensive tasks.

[0011] Optionally, generating a resource allocation strategy for the task to be run according to the classification of the task to be run and the resource idleness of each hardware resource of the platform includes:

[0012] Determine whether the resource idleness of the hardware resource category of the platform required by the tasks to be run is sufficient according to the classification of the tasks to be run and the resource idleness of each hardware resource of the platform;

[0013] If the platform has sufficient resource free space for the hardware resource category required by the tasks to be run, a resource allocation strategy for the tasks to be run is generated according to the priorities of the tasks to be run and / or a resource allocation algorithm matching the tasks to be run.

[0014] Optionally, the method further includes:

[0015] If the resource idleness of the hardware resource category required by the task to be run is insufficient, the task to be run is placed in a waiting queue until the resource idleness of the hardware resource category required by the task to be run is sufficient;

[0016] Or, the resources occupied by the tasks that have been completed or executed abnormally among the tasks being executed by the platform are recovered, so that the platform has sufficient resource idleness for the hardware resource category required by the tasks to be run.

[0017] Optionally, the task to be executed is an important or urgent task, and the hardware resource categories required by the task to be executed are one or more, and the method further includes:

[0018] Analyze the utilization rate of each hardware resource of the platform to obtain the change trend of each hardware resource of the platform;

[0019] If the platform has insufficient resource free space for at least one hardware resource category among the hardware resource categories that the tasks to be run require, then based on the change trend of the hardware resources of the platform, necessary hardware resources are reserved before the tasks to be run are executed, so that the platform has sufficient resource free space for the hardware resource categories that the tasks to be run require.

[0020] Optionally, the task to be executed is an important or urgent task, and the hardware resource categories required by the task to be executed are one or more, and the method further includes:

[0021] If the platform has insufficient resource free space for at least one hardware resource category among the hardware resource categories required by the tasks to be run, the priority of the tasks to be run is adjusted to be higher than the priority of at least one task in the platform that is using the hardware resource category with insufficient resource free space, so that the platform has sufficient resource free space for the hardware resource categories required by the tasks to be run.

[0022] Optionally, the platform is a distributed system including a plurality of computing nodes; generating a resource allocation strategy for the task to be run according to the classification of the task to be run and the resource idleness of each hardware resource of the platform includes:

[0023] Based on the hardware resource categories required by the tasks to be run, the resource idleness of each hardware resource of each computing node included in the platform, and the load balancing strategy and fault tolerance strategy for each computing node included in the platform, a resource allocation strategy for the tasks to be run is generated to avoid hardware resource overload in any computing node.

[0024] The present disclosure also provides a task resource allocation device, the device comprising:

[0025] A classification unit, used to classify the tasks to be run according to the degree of use of each hardware resource category by the tasks to be run, so as to identify which hardware resource category the tasks to be run centrally require when being executed;

[0026] A calculation unit, configured to calculate the resource idleness of each hardware resource of the platform according to the utilization rate of each hardware resource of the platform;

[0027] A generating unit, configured to generate a resource allocation strategy for the task to be run according to the classification of the task to be run and the resource idleness of each hardware resource of the platform;

[0028] An allocation unit is used to allocate corresponding hardware resources of the platform to the task to be run according to a resource allocation strategy for the task to be run, so as to execute the task to be run.

[0029] The present disclosure also provides an electronic device, comprising a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus;

[0030] The memory stores machine-readable instructions, and the processor executes the above method by calling the machine-readable instructions.

[0031] The present disclosure also provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the above method is implemented.

[0032] Through the embodiments of the present disclosure, the tasks to be run are classified according to the degree of use of each hardware resource category by the tasks to be run, so as to identify which hardware resource category the tasks to be run require when executing; then, the resource idleness of each hardware resource of the platform is calculated according to the utilization rate of each hardware resource of the platform; further, a resource allocation strategy for the tasks to be run is generated according to the classification of the tasks to be run and the resource idleness of each hardware resource of the platform; finally, according to the resource allocation strategy for the tasks to be run, the corresponding hardware resources of the platform are allocated to the tasks to be run to execute the tasks to be run. Accordingly, on the one hand, the present disclosure can accurately evaluate the resource idle state of the platform and provide a reliable basis for subsequent scheduling decisions; on the other hand, the present disclosure takes into account the diversity and resource heterogeneity of the tasks to be run, and combines the resource idle state of the platform to generate a more targeted, more refined and effective resource scheduling strategy, thereby improving the utilization rate of various resources of the platform and the execution efficiency of the tasks to be run. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0034] Figure 1 is a flow chart of a task resource allocation method shown in an exemplary embodiment;

[0035] Figure 2 is a task resource scheduling system architecture diagram shown in an exemplary embodiment;

[0036] Figure 3 is a flow chart of another task resource scheduling method shown in an exemplary embodiment;

[0037] Figure 4 is a hardware structure diagram of an electronic device shown in an exemplary embodiment;

[0038] Figure 5 It is a block diagram of a task resource allocation device shown in an exemplary embodiment. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0040] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0041] With the rapid development of information technology, the tasks undertaken by computing platforms are becoming increasingly complex and diverse, and the demand for resources is also increasing. In order to effectively manage these resources and ensure that tasks can be completed in the shortest time, task scheduling has become a key issue. Traditional task scheduling methods are usually based on preset static strategies, such as FCFS (First Come First Served) and SJF (Shortest Job First). Although these methods are simple and easy to implement, they often fail to fully utilize the idle hardware resources of the computing platform.

[0042] In practical applications, the resource usage status of the computing platform changes dynamically. Sometimes some resources may be in a high-load state, while other resources may be relatively idle. Traditional static policy-based task scheduling methods cannot adjust the allocation of tasks according to these real-time changes, resulting in low hardware resource utilization and task execution delays. In addition, different types of tasks have different requirements for resources. For example, computationally intensive tasks require more CPU resources, while I / O intensive tasks have higher requirements for storage and network resources. Therefore, traditional static policy-based task scheduling methods often cannot meet such diverse needs.

[0043] In order to overcome the limitations of traditional static policy-based task scheduling methods, some dynamic policy-based task scheduling methods have emerged in recent years. These methods monitor the resource usage status of the computing platform in real time and dynamically adjust the allocation of tasks according to the status changes. However, the existing dynamic scheduling methods still have some challenges in implementation, such as how to accurately evaluate the resource idle status of the platform, how to reasonably classify tasks according to their resource requirements, and how to generate effective dynamic scheduling strategies.

[0044] In view of this, the present disclosure aims to propose a technical solution that can generate an adaptive resource scheduling strategy for a task according to the task resource requirements and the platform resource idle state.

[0045] During implementation, first, the tasks to be run are classified according to the degree of use of various hardware resource categories by the tasks to be run, so as to identify which hardware resource categories the tasks to be run require intensively when being executed; then, the resource idleness of various hardware resources of the platform is calculated according to the utilization rate of various hardware resources of the platform; further, a resource allocation strategy for the tasks to be run is generated according to the classification of the tasks to be run and the resource idleness of various hardware resources of the platform; finally, according to the resource allocation strategy for the tasks to be run, the corresponding hardware resources of the platform are allocated to the tasks to be run, so as to execute the tasks to be run.

[0046] For example, there are two tasks to be run that are waiting in line for resource allocation, namely, 3D modeling and rendering tasks, and online multiplayer game tasks. Among them, 3D modeling and rendering tasks need to consume more CPU computing resources when they are executed, that is, the hardware resource category that 3D modeling and rendering tasks focus on when they are executed is CPU computing resources; online multiplayer game tasks need to consume more network bandwidth resources when they are executed, that is, the hardware resource category that online multiplayer game tasks focus on when they are executed is network bandwidth resources. The utilization rate of the platform CPU computing resources is 75%, and the utilization rate of the platform network bandwidth resources is 40%. Then, the resource idleness of the platform CPU computing resources is 25%, and the resource idleness of the platform network bandwidth resources is 60%. Since the resource idleness of the platform network bandwidth resources is relatively high at this time, the resource allocation strategy generated for the two tasks to be run may be to allocate platform network bandwidth resources to the online multiplayer game task first. At this time, the resources are relatively sufficient, which can ensure the smooth operation of the online multiplayer game task, and then allocate CPU computing resources to the 3D modeling and rendering tasks; finally, according to the resource allocation strategy for the two tasks to be run, the corresponding hardware resources are allocated to the two tasks to be run, so as to execute the two tasks to be run.

[0047] It can be seen that in the technical solution of the present disclosure, the tasks to be run are classified according to the degree of use of each hardware resource category by the tasks to be run, so as to identify which hardware resource category the tasks to be run require when executing; then, the resource idleness of each hardware resource of the platform is calculated according to the utilization rate of each hardware resource of the platform; further, a resource allocation strategy for the tasks to be run is generated according to the classification of the tasks to be run and the resource idleness of each hardware resource of the platform; finally, according to the resource allocation strategy for the tasks to be run, the corresponding hardware resources of the platform are allocated to the tasks to be run to execute the tasks to be run. Accordingly, on the one hand, the present disclosure can accurately evaluate the resource idle state of the platform and provide a reliable basis for subsequent scheduling decisions; on the other hand, the present disclosure takes into account the diversity and resource heterogeneity of the tasks to be run, and combines the resource idle state of the platform to generate a more targeted, more refined and effective resource scheduling strategy, thereby improving the utilization rate of various resources of the platform and the execution efficiency of the tasks to be run.

[0048] The present disclosure is described below through specific embodiments in combination with specific application scenarios.

[0049] See also Figure 1 , Figure 1 The following is a flowchart of a task resource allocation method according to an exemplary embodiment. The method may perform the following steps:

[0050] Step 102: Classify the tasks to be run according to the degree of use of each hardware resource category by the tasks to be run, so as to identify which hardware resource category the tasks to be run mainly require when being executed.

[0051] For example, there are two tasks waiting to be run, namely, 3D modeling and rendering tasks, and online multiplayer game tasks. Among them, 3D modeling and rendering tasks need to consume more CPU computing resources when executing, so 3D modeling and rendering tasks are identified as CPU computing resource intensive tasks; online multiplayer game tasks need to consume more network resources when executing, so online multiplayer game tasks are identified as network bandwidth resource intensive tasks.

[0052] Among them, since 3D modeling and rendering tasks usually involve a lot of graphics calculations and complex algorithms, they consume more CPU computing resources and are marked as "CPU computing resource intensive tasks". Since online multiplayer game tasks require real-time data exchange and communication to support interaction between players, they have a high demand for network bandwidth and are marked as "network bandwidth resource intensive tasks".

[0053] In this embodiment, by analyzing the characteristics and requirements of the tasks to be run, the hardware resource categories that they require most during execution are determined. This may involve evaluation of the workload, data flow, processing requirements, etc. of the tasks to be run. Once the tasks to be run are correctly identified, the system can prioritize and allocate resources according to the resource requirements of the tasks to be run, ensuring that critical tasks can obtain sufficient hardware resources, thereby optimizing the overall system performance.

[0054] Step 104: Calculate the resource idleness of each hardware resource of the platform according to the utilization rate of each hardware resource of the platform.

[0055] For example, if the utilization rate of the platform CPU computing resources is 75% and the utilization rate of the platform network bandwidth resources is 40%, then the resource idleness of the platform CPU computing resources is 25% and the resource idleness of the platform network bandwidth resources is 60%.

[0056] Among them, the system monitors the utilization rate of various hardware resources of the platform in real time through the hardware resource monitoring system, including CPU utilization rate, memory utilization rate, storage utilization rate and network bandwidth utilization rate, etc. These resource usage status data can be obtained through the system's built-in monitoring tools or third-party monitoring software, and updated at certain time intervals to ensure the real-time nature of the data. Resource idleness usually refers to the percentage of computing resources (such as CPU, GPU, memory, etc.) that are not used. For example, if the CPU computing resource utilization rate of a system is 75%, then the corresponding CPU computing resource idleness is 25%. There may be other methods for calculating the resource idleness of platform hardware resources, such as weighting the utilization rates of several hardware resources separately, and then calculating the overall resource idleness. This manual does not limit the specific calculation method of the resource idleness of platform hardware resources.

[0057] Step 106: Generate a resource allocation strategy for the task to be run according to the classification of the task to be run and the resource idleness of each hardware resource of the platform.

[0058] For example, the resource idleness of the platform CPU computing resources is 25%, and the resource idleness of the platform network bandwidth resources is 60%. Since the resource idleness of the platform network bandwidth resources is relatively high at this time, the resource allocation strategy generated for the two tasks to be run may be to allocate platform network bandwidth resources to the online multiplayer game task first, at which time the resources are relatively sufficient to ensure the smooth operation of the online multiplayer game task, and then allocate CPU computing resources to the 3D modeling and rendering tasks.

[0059] Among them, the above scheme may include the following technical details: 1. Maintain a task queue to sort the tasks to be run according to the time or priority of task submission. 2. According to the classification of the tasks to be run, evaluate the demand for various hardware resources for each task to be run. 3. Compare the idleness of hardware resources with the resource demand of the tasks to be run to determine which hardware resources are the bottlenecks when the tasks to be run are executed. 4. Assign different priorities to the tasks to be run according to their urgency, importance and user needs. 5. Develop a resource allocation algorithm that can intelligently allocate resources based on the classification of the tasks to be run and the resource idleness of the platform hardware resources. For high-priority or critical tasks to be run, a specific number of resources may need to be reserved to ensure the smooth and priority execution of the tasks. When allocating resources, the dependencies between tasks must also be considered to ensure the smooth execution of the task chain. During the execution of tasks, resource allocation can also be dynamically adjusted based on real-time monitoring data to cope with changes in resource demand.

[0060] Step 108: Allocate corresponding hardware resources of the platform to the task to be run according to the resource allocation strategy for the task to be run, so as to execute the task to be run.

[0061] For example, according to the resource allocation strategy for 3D modeling and rendering tasks and online multiplayer game tasks, platform CPU computing resources are allocated to 3D modeling and rendering tasks, and platform network bandwidth resources are allocated to online multiplayer game tasks to execute 3D modeling and rendering tasks, and online multiplayer game tasks.

[0062] Among them, the resource allocation strategy refers to a strategy formulated based on the classification of tasks to be run and the resource status of the platform, with the aim of ensuring that the tasks to be run can efficiently use the required hardware resources. Tasks to be run are tasks that are ready but have not yet started to execute. They may be waiting for resource allocation or other prerequisites to be met. According to the resource allocation strategy, the platform's hardware resources (such as CPU, memory, storage, network, etc.) are allocated to specific tasks. This process may involve resource reservation, scheduling, and optimization. Once the tasks to be run have obtained the necessary hardware resources, they can start execution. The execution process may include task initialization, running, and monitoring. Platform CPU computing resources refer to the CPU resources on the platform that can be used to perform computing-intensive tasks. These resources can be a single CPU core or multiple cores. According to the resource allocation strategy, 3D modeling and rendering tasks may be allocated to CPU resources with high computing power, while online multiplayer game tasks may be allocated to network resources with high bandwidth and low latency.

[0063] In this embodiment, according to the generated resource allocation strategy, the execution of the task is scheduled to ensure that the task obtains the required resources according to the established strategy. Through these technical details, it can be ensured that the resource allocation strategy not only meets the requirements of the task, but also takes into account the actual situation of the platform resources, thereby improving the operating efficiency and stability of the entire platform.

[0064] In one embodiment shown, the tasks to be run are classified into at least one of the following according to the degree of usage of each hardware resource category by the tasks to be run: CPU computing resource intensive tasks, memory storage resource intensive tasks, disk storage resource intensive tasks, disk I / O resource intensive tasks and network bandwidth resource intensive tasks.

[0065] For example, 3D modeling and rendering tasks mainly use CPU computing resources and are therefore classified as CPU computing resource intensive tasks; online multiplayer game tasks mainly use network bandwidth resources and are therefore classified as network bandwidth resource intensive tasks.

[0066] Among them, in the technical solution of the present disclosure, the tasks to be run are classified and identified as corresponding resource-intensive tasks according to the degree of use of hardware resources by the tasks to be run. If the task mainly relies on the CPU for calculation, such as scientific computing, image processing, 3D modeling and rendering, etc., it is identified as a "CPU computing resource-intensive task". If the task requires a large amount of memory to process data, such as large database operations, data analysis, and compiling large software, it is identified as a "memory storage resource-intensive task". If the task requires a large amount of storage space or frequent read and write operations, such as video editing, big data processing, and database backup, it is identified as a "disk storage resource-intensive task". If the task requires a high-bandwidth and low-latency network connection, such as real-time audio and video transmission, online games, and cloud computing services, it is identified as a "network bandwidth resource-intensive task". If the task involves a large number of disk read and write operations, such as file server and database server operations, it is identified as a "disk I / O resource-intensive task". The purpose of this classification is to be able to allocate the most appropriate platform resources to the tasks to be run according to their resource requirements in the subsequent resource scheduling process.

[0067] In one embodiment shown, the method also includes: generating a resource allocation strategy for the task to be run based on the classification of the task to be run and the resource idleness of each hardware resource of the platform, including: determining whether the resource idleness of the hardware resource category of the platform that is centrally required by the tasks to be run is sufficient based on the classification of the tasks to be run and the resource idleness of each hardware resource of the platform; if the resource idleness of the hardware resource category of the platform that is centrally required by the tasks to be run is sufficient, generating a resource allocation strategy for the task to be run based on the priority of the task to be run and / or a resource allocation algorithm that matches the task to be run.

[0068] For example, the task to be run is an online multiplayer game task, so the task to be run is classified as a "network bandwidth resource intensive task". Since the resource idleness of the platform network bandwidth resources is 60%, according to the judgment of the resource consumption of the online multiplayer game task, the platform network bandwidth resources are sufficient for the online multiplayer game task. Therefore, the corresponding resource allocation strategy can be generated according to the priority assigned by the user to the task to be run and / or the resource allocation algorithm matched to the task to be run.

[0069] Among them, the priority of the task can be determined by a variety of methods. In addition to user specification, the following methods can also be used: 1. Automatically assign priority according to the type of task. For example, real-time tasks usually have higher priority than batch tasks. 2. Judge according to the deadline and urgency of the task. Tasks that are about to reach the deadline will be given a higher priority. 3. For critical tasks or tasks that affect the stability of the system, a higher priority can be given. 4. Considering the task dependencies, if a task is a prerequisite for other tasks, the task can be given a higher priority to ensure that the dependent tasks can be executed smoothly. 5. Determine the priority based on the amount of resources expected to be consumed by the task. Tasks with larger resource consumption may be given a higher priority in order to better manage resource allocation. The resource allocation algorithms for task matching include linear programming, particle swarm optimization, etc. The present disclosure does not limit the method for determining the priority of tasks and the resource allocation algorithm for task matching.

[0070] In one embodiment shown, the method also includes: if the platform has insufficient resource free space for the hardware resource category required by the set of tasks to be run, placing the tasks to be run in a waiting queue until the platform has sufficient resource free space for the hardware resource category required by the set of tasks to be run; or, recycling resources occupied by completed tasks or abnormally executed tasks among the tasks being executed by the platform, so that the platform has sufficient resource free space for the hardware resource category required by the set of tasks to be run.

[0071] For example, the task to be run is a 3D modeling and rendering task, which requires the platform's CPU computing resources during execution. At this time, the utilization rate of the platform's CPU computing resources is 75%, and the resource idleness of the platform's CPU computing resources is 25%. Based on the judgment of the resource consumption of the 3D modeling and rendering tasks, the platform's CPU computing resources are insufficient for the 3D modeling and rendering tasks. Therefore, it is temporarily impossible to allocate platform CPU computing resources to the 3D modeling and rendering tasks. The 3D modeling and rendering tasks are first placed in the waiting queue until the platform has sufficient CPU computing resources to schedule to the 3D modeling and rendering tasks. If the 3D modeling and rendering tasks are important or urgent tasks, they are not suitable for waiting and need to be executed as soon as possible, then the CPU computing resources occupied by the tasks that have been completed or the tasks that have executed abnormally in the tasks being executed by the platform can also be recovered, so that sufficient CPU computing resources can be scheduled to the 3D modeling and rendering tasks.

[0072] Among them, when resources are insufficient, the tasks to be run that cannot be executed immediately will be placed in the waiting queue. These tasks to be run will be scheduled for execution according to certain policies (such as first-in-first-out, execution by priority, etc.) when resources become available. If resources are still tight, the system may terminate the completed or abnormally terminated tasks and reclaim the resources they occupy so that they can be allocated to more urgent or important tasks to be run. In order to realize the dynamic resource allocation strategy that automatically adjusts resource allocation based on real-time resource monitoring data, this may include adjusting the CPU share and memory quota of the virtual machine, or adjusting the resource limit of the container in the container environment. Specifically, a set of task priority and scheduling algorithms can be developed to determine which tasks can wait and which need to be executed immediately, which may involve factors such as the classification type, urgency, resource requirements and expected execution time of the tasks to be run. For important tasks, a part of the resources needs to be reserved for them to ensure that these tasks can obtain the necessary computing power even under high load. This is usually achieved by setting resource reservation policies at the system level. In the case of insufficient resources, the task status can be monitored through mechanisms such as heartbeat monitoring and task status checking, and the completed or abnormally terminated tasks can be automatically identified and the resources occupied by them can be released. The above overall operations can be performed using automated scripts and tools to reduce manual intervention and improve efficiency and accuracy.

[0073] In one embodiment shown, the task to be run is an important or urgent task, and the hardware resource categories concentratedly required by the task to be run are one or more, and the method further includes: analyzing the utilization rate of each hardware resource of the platform to obtain the change trend of each hardware resource of the platform; if the resource idleness of at least one hardware resource category among the hardware resource categories concentratedly required by the tasks to be run on the platform is insufficient, then based on the change trend of the hardware resources of the platform, necessary hardware resources are reserved before the task to be run is executed, so that the resource idleness of the hardware resource categories concentratedly required by the tasks to be run on the platform is sufficient.

[0074] For example, an urgent video transcoding task submitted by a user is received, and the task requires the following four resources at the same time: 1. CPU computing resources for decoding the original video stream and encoding a new video format; 2. Memory resources for storing video frames and encoding buffers; 3. Disk I / O resources for reading original video files and writing transcoded files; 4. Network bandwidth resources for uploading and downloading video files in a cloud environment. In the technical solution disclosed in the present invention, a training model is used to learn the patterns and trends in historical data, and the utilization rate of various hardware resources of the current platform is analyzed by a machine learning algorithm, so as to predict the resource usage in a certain time period in the future. For example, the resource idleness of the CPU computing resources in the time period of running the video transcoding task in the future is insufficient. In this way, adjustments can be made in advance to reserve the CPU computing resources necessary for running the video transcoding task. For example, according to the prediction of the CPU computing resources occupied by running the video transcoding task, 25% of the resource idleness of the CPU computing resources is reserved in advance, so as to optimize resource allocation and ensure that the tasks to be run can be executed efficiently.

[0075] Among them, machine learning algorithms (such as time series analysis, regression analysis, neural networks, etc.) can be used to train models and predict future resource requirements. Cross-validation, grid search and other methods may be needed to tune model parameters and improve prediction accuracy. Through the real-time monitoring system, resource usage is continuously tracked and input into the prediction model for immediate prediction. Based on the prediction results, the resource allocation strategy is automatically adjusted to reserve necessary resources. For resource reservation strategies, a certain proportion of resources can be reserved based on the predicted demand, or a fixed number of resource units can be reserved. At the same time, the cost and benefit of reserved resources need to be considered to avoid waste of resources caused by excessive reservation. At the same time, the reservation strategy is continuously optimized according to the actual resource usage and prediction error. Finally, a feedback loop is established to compare the prediction results with the actual resource usage for model iteration and strategy improvement.

[0076] In one embodiment shown, the task to be run is an important or urgent task, and the hardware resource categories required by the tasks to be run are one or more. The method also includes: if the platform has insufficient resource idleness for at least one hardware resource category among the hardware resource categories required by the tasks to be run, then the priority of the task to be run is adjusted to be higher than the priority of at least one task in the platform that is using the hardware resource category with insufficient resource idleness, so that the platform has sufficient resource idleness for the hardware resource category required by the tasks to be run.

[0077] For example, an urgent video transcoding task submitted by a user is received, and the task requires the following four resources at the same time: 1. CPU computing resources for decoding the original video stream and encoding a new video format; 2. Memory resources for storing video frames and encoding buffers; 3. Disk I / O resources for reading original video files and writing transcoded files; 4. Network bandwidth resources for uploading and downloading video files in a cloud environment. In the technical solution disclosed in the present invention, assuming that the resource idleness of the CPU computing resources necessary for running the video transcoding task is insufficient, the priority of the video transcoding task is adjusted to be higher than the priority of at least one task in the platform that is using the CPU computing resources, such as higher than the priority of image processing tasks and scientific computing tasks that are not important or urgent, so that the resource idleness of the CPU computing resources necessary for running the video transcoding task is sufficient.

[0078] Among them, different priorities are set for the tasks to be run according to their urgency and importance. In the case of limited resources, urgent or important tasks have higher priorities and can preempt the resources of low-priority tasks to ensure that urgent or important tasks to be run can obtain the required resources. By implementing a real-time monitoring system, resource usage is continuously tracked, such as CPU usage, memory usage, disk I / O, and network bandwidth, to timely discover resource bottlenecks. When insufficient resources are detected, the priority of urgent or important tasks to be run is automatically increased, which may require modifying the configuration or rules of the task scheduler. Allowing high-priority tasks to preempt the resources of low-priority tasks usually involves pausing or terminating low-priority tasks to release resources for urgent tasks. For tasks with the same priority, resource usage time can be allocated in a time slice rotation manner to ensure that each task has a fair execution opportunity. The solution can match advanced scheduling algorithms (such as priority scheduling, fair sharing, shortest job first, etc.) to further optimize resource allocation and task execution efficiency.

[0079] In this embodiment, computing resources can be effectively managed to ensure that urgent or important tasks to be run can be given priority when resources are limited, while minimizing the impact on other tasks, which is conducive to providing high-quality services, meeting business needs and improving user experience.

[0080] In one embodiment shown, the platform is a distributed system including multiple computing nodes; the resource allocation strategy for the tasks to be run is generated based on the classification of the tasks to be run and the resource idleness of the various hardware resources of the platform, including: generating a resource allocation strategy for the tasks to be run based on the hardware resource categories required by the tasks to be run, the resource idleness of the various hardware resources of each computing node included in the platform, and the load balancing strategy and fault tolerance strategy for each computing node included in the platform, so as to avoid hardware resource overload in any computing node.

[0081] For example, in a distributed system consisting of multiple computing nodes (such as a cloud service platform), each node has its own CPU, memory, disk space, and network bandwidth. When a user submits a large-scale data analysis task, the task requires a large amount of CPU computing resources, sufficient memory to cache data, and a stable network connection to transmit data packets. In order to process the task, it is necessary to determine how much CPU computing resources, disk storage space, and network bandwidth are required, and then check the CPU usage, available memory, remaining disk space, and network bandwidth utilization on each computing node; then based on resource idleness and load balancing strategy, decide which nodes are most suitable for the task. At the same time, considering the fault tolerance strategy, the system may replicate some parts of the task on multiple nodes, or reserve additional resources as backup to prevent the task from continuing to run when a node fails. Finally, based on the above factors, a resource allocation strategy is generated to indicate how much resources each node should allocate to the task. For example, the computationally intensive part of the task is assigned to node A (because it has high CPU idleness), the data caching task is assigned to node C (because it has sufficient memory and disk space), and the network-intensive operation is assigned to node D (because it has sufficient network bandwidth).

[0082] In this embodiment, through such a strategy, the distributed system can efficiently utilize resources while ensuring that any single node will not be overloaded, thereby maintaining the overall stability and reliability of the system.

[0083] Among them, load balancing is a strategy for evenly distributing tasks among multiple computing nodes to avoid overloading some nodes while idling other nodes. This can be achieved through polling, random selection, minimum number of connections, or weight-based methods. Fault-tolerant design is to ensure that the system can still operate normally when encountering a fault, which usually includes data redundancy, task retry, heartbeat detection, and fault recovery strategies. The present disclosure does not limit the specific methods of load balancing strategies and fault-tolerant designs.

[0084] In order to help those skilled in the art better understand this method, Figure 2 and Figure 3 This method is described.

[0085] See also Figure 2 , Figure 2 FIG. 1 is a diagram showing an architecture of a task resource scheduling system according to an exemplary embodiment. Figure 2 As shown, the task resource scheduling system may include a user interface and an interaction layer. In this layer, the task resource scheduling system may provide a user interface or an API interface so that users or developers can interact with the system. The user interface may be a Web interface, a command line tool, or a mobile application, etc., for displaying the system status, submitting task requests, and viewing task execution results. The resource monitoring module is responsible for monitoring the real-time status data of platform resources, including CPU usage, memory usage, disk I / O, network bandwidth, etc. The resource idleness calculation module is used to calculate the resource idleness of each hardware resource of the platform. The task classification module is used to classify tasks to be run. The scheduling strategy generation module is used to generate a dynamic resource allocation strategy based on the resource idleness of each hardware resource of the platform and the priority of the task and / or the matching resource allocation algorithm. The resource allocation module is used to allocate the corresponding hardware resources of the platform to the task to be run according to the resource allocation strategy for the task to be run, so as to execute the task to be run. The data acquisition and data storage layer is responsible for storing various data generated during the operation of the system, including resource status data, task information, scheduling decision records, etc. At the same time, it is also necessary to record the system log information for troubleshooting and performance optimization. The platform layer refers to the infrastructure of the entire system, which provides resources and services, and supports the data source and monitoring layer, data processing and analysis layer, task management layer, scheduling decision layer, execution layer, data storage and log layer, and user interface and interaction layer. These modules are built on the platform layer, and use the resources and services provided by the platform layer to realize their respective functions, so that each module can operate normally and interact.

[0086] See also Figure 3 , Figure 3 FIG. 1 is a flowchart of another task resource scheduling method shown in an exemplary embodiment. Figure 3As shown, at the beginning of the method disclosed in the present invention, the tasks to be run are first classified according to the degree of use of each hardware resource category by the tasks to be run, and then the resource idleness of each hardware resource of the platform is calculated to determine whether the hardware resources required by the tasks to be run are sufficient. In the case that the hardware resources required by the tasks to be run are sufficient, a targeted resource allocation strategy can be generated according to the classification of the tasks to be run and the resource idleness of each hardware resource of the platform, and the tasks to be run are executed according to the resource allocation strategy. In the case that the hardware resources required by the tasks to be run are insufficient, it is determined whether the tasks to be run are important or urgent. If they are not important or urgent tasks, the tasks to be run are placed in a waiting queue and wait. When the hardware resources required by the tasks to be run are sufficient, a targeted resource allocation strategy is generated again, and the tasks to be run are executed according to the resource allocation strategy; if they are important tasks, the priority of the tasks to be run is adjusted, the corresponding resources of the tasks being run are recovered or resources are reserved for the tasks to be run in advance, and when the hardware resources required by the tasks to be run are sufficient, a targeted resource allocation strategy is generated again, and the tasks to be run are executed according to the resource allocation strategy.

[0087] Corresponding to the above-mentioned embodiment of the task resource allocation method, this specification also provides an embodiment of a task resource allocation device.

[0088] See also Figure 4 , Figure 4 It is a hardware structure diagram of an electronic device shown in an exemplary embodiment. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408 and a non-volatile memory 410, and of course may also include other required hardware. One or more embodiments of this specification can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0089] See also Figure 5 , Figure 5 1 is a block diagram of a task resource allocation device shown in an exemplary embodiment. The task resource allocation device 500 includes:

[0090] The classification unit 502 is used to classify the tasks to be run according to the usage degree of each hardware resource category by the tasks to be run, so as to identify which hardware resource category the tasks to be run mainly require when being executed;

[0091] The calculation unit 504 is used to calculate the resource idleness of each hardware resource of the platform according to the utilization rate of each hardware resource of the platform;

[0092] A generating unit 506, configured to generate a resource allocation strategy for the task to be run according to the classification of the task to be run and the resource idleness of each hardware resource of the platform;

[0093] The allocating unit 508 is configured to allocate corresponding hardware resources of the platform to the task to be run according to the resource allocation strategy for the task to be run, so as to execute the task to be run.

[0094] In this embodiment, the tasks to be run are classified into at least one of the following according to the usage degree of each hardware resource category by the tasks to be run:

[0095] CPU computing resource intensive tasks, memory storage resource intensive tasks, disk storage resource intensive tasks, disk I / O resource intensive tasks and network bandwidth resource intensive tasks.

[0096] In this embodiment, the generating unit includes:

[0097] A determination subunit, configured to determine whether the resource idleness of the hardware resource category of the platform required by the tasks to be run is sufficient according to the classification of the tasks to be run and the resource idleness of each hardware resource of the platform;

[0098] The first generating subunit is used to generate a resource allocation strategy for the task to be run according to the priority of the task to be run and / or a resource allocation algorithm matching the task to be run if the platform has sufficient resource free space for the hardware resource category required by the task to be run.

[0099] In this embodiment, the device further includes:

[0100] A placement unit, configured to place the to-be-run tasks into a waiting queue if the platform has insufficient resource idleness for the hardware resource category required by the to-be-run tasks, until the platform has sufficient resource idleness for the hardware resource category required by the to-be-run tasks;

[0101] The recycling unit is used to recycle the resources occupied by the tasks that have been completed or executed abnormally among the tasks being executed by the platform, so that the platform has sufficient resource idleness for the hardware resource category required by the tasks to be run.

[0102] In this embodiment, the task to be executed is an important or urgent task, the hardware resource categories required by the task to be executed are one or more, and the device further includes:

[0103] An analysis unit, used to analyze the utilization rate of each hardware resource of the platform to obtain the change trend of each hardware resource of the platform;

[0104] A reservation unit is used to reserve necessary hardware resources before the execution of the task to be run, if there is insufficient resource free space for at least one hardware resource category among the hardware resource categories required by the platform for the tasks to be run, based on the change trend of the various hardware resources of the platform, so that the platform has sufficient resource free space for the hardware resource categories required by the tasks to be run.

[0105] In this embodiment, the task to be executed is an important or urgent task, the hardware resource categories required by the task to be executed are one or more, and the device further includes:

[0106] An adjustment unit is used to adjust the priority of the task to be run to be higher than the priority of at least one task in the platform that is using the hardware resource category with insufficient resource free time, if there is insufficient resource free time of at least one hardware resource category among the hardware resource categories required by the platform for the tasks to be run, so that the platform has sufficient resource free time for the hardware resource categories required by the tasks to be run.

[0107] In this embodiment, the platform is a distributed system including multiple computing nodes; the generating unit includes:

[0108] The second generating sub-unit is used to generate a resource allocation strategy for the tasks to be run according to the hardware resource categories required by the tasks to be run, the resource idleness of each hardware resource of each computing node included in the platform, and the load balancing strategy and fault tolerance strategy for each computing node included in the platform, so as to avoid hardware resource overload in any computing node.

[0109] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0110] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Ordinary technicians in this field can understand and implement it without paying creative work.

[0111] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0112] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0113] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0114] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0115] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0116] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0117] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "one", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0119] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0120] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A task resource allocation method, characterized in that: The method comprises: Classifying the tasks to be run according to the degree of use of each hardware resource category by the tasks to be run, so as to identify which hardware resource category the tasks to be run mainly require when executing; Calculate the resource idleness of each hardware resource of the platform according to the utilization rate of each hardware resource of the platform; Generate a resource allocation strategy for the tasks to be run according to the classification of the tasks to be run and the resource idleness of various hardware resources of the platform; According to the resource allocation strategy for the task to be run, the corresponding hardware resources of the platform are allocated to the task to be run, so as to execute the task to be run.

2. The method according to claim 1, characterized in that According to the usage degree of each hardware resource category by the task to be run, the task to be run is classified into at least one of the following: CPU computing resource intensive tasks, memory storage resource intensive tasks, disk storage resource intensive tasks, disk I / O resource intensive tasks and network bandwidth resource intensive tasks.

3. The method according to claim 1, characterized in that According to the classification of the tasks to be run and the resource idleness of various hardware resources of the platform, a resource allocation strategy for the tasks to be run is generated, including: Determine whether the resource idleness of the hardware resource category of the platform required by the tasks to be run is sufficient according to the classification of the tasks to be run and the resource idleness of each hardware resource of the platform; If the platform has sufficient resource free space for the hardware resource category required by the tasks to be run, a resource allocation strategy for the tasks to be run is generated according to the priorities of the tasks to be run and / or a resource allocation algorithm matching the tasks to be run.

4. The method according to claim 3, characterized in that: The method further comprises: If the resource idleness of the hardware resource category required by the task to be run is insufficient, the task to be run is placed in a waiting queue until the resource idleness of the hardware resource category required by the task to be run is sufficient; Or, the resources occupied by the tasks that have been completed or executed abnormally among the tasks being executed by the platform are recovered, so that the platform has sufficient resource idleness for the hardware resource category required by the tasks to be run.

5. The method according to claim 3, characterized in that: The task to be executed is an important or urgent task, the hardware resource categories required by the task to be executed are one or more, and the method further includes: Analyze the utilization rate of each hardware resource of the platform to obtain the change trend of each hardware resource of the platform; If the platform has insufficient resource free space for at least one hardware resource category among the hardware resource categories that the tasks to be run require, then based on the change trend of the hardware resources of the platform, necessary hardware resources are reserved before the tasks to be run are executed, so that the platform has sufficient resource free space for the hardware resource categories that the tasks to be run require.

6. The method according to claim 3, characterized in that: The task to be executed is an important or urgent task, the hardware resource categories required by the task to be executed are one or more, and the method further includes: If the platform has insufficient resource free space for at least one hardware resource category among the hardware resource categories required by the tasks to be run, the priority of the tasks to be run is adjusted to be higher than the priority of at least one task in the platform that is using the hardware resource category with insufficient resource free space, so that the platform has sufficient resource free space for the hardware resource categories required by the tasks to be run.

7. The method according to claim 1, characterized in that The platform is a distributed system including multiple computing nodes; the resource allocation strategy for the tasks to be run is generated according to the classification of the tasks to be run and the resource idleness of various hardware resources of the platform, including: Based on the hardware resource categories required by the tasks to be run, the resource idleness of each hardware resource of each computing node included in the platform, and the load balancing strategy and fault tolerance strategy for each computing node included in the platform, a resource allocation strategy for the tasks to be run is generated to avoid hardware resource overload in any computing node.

8. A task resource allocation device, characterized in that: The device comprises: A classification unit, used to classify the tasks to be run according to the degree of use of each hardware resource category by the tasks to be run, so as to identify which hardware resource category the tasks to be run centrally require when being executed; A calculation unit, configured to calculate the resource idleness of each hardware resource of the platform according to the utilization rate of each hardware resource of the platform; A generating unit, configured to generate a resource allocation strategy for the task to be run according to the classification of the task to be run and the resource idleness of each hardware resource of the platform; An allocation unit is used to allocate corresponding hardware resources of the platform to the task to be run according to a resource allocation strategy for the task to be run, so as to execute the task to be run.

9. An electronic device, characterized in that: It includes a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the method according to any one of claims 1 to 7 by calling the machine-readable instructions.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images