Task scheduling method, computing device and storage medium

Abstract

The invention discloses a task scheduling method, which comprises the following steps of: storing all received to-be-executed tasks into a persistent task queue when the to-be-executed tasks are received; a to-be-executed task is obtained from the persistent task queue at regular time, the to-be-executed task is loaded to the corresponding non-persistent task queue according to the resource pool to which the obtained to-be-executed task belongs, and each resource pool is independently configured with the non-persistent task queue corresponding to the resource pool; determining the cluster load state of each resource pool according to the index monitoring information of the server cluster received in real time; and according to the cluster load state of each resource pool, determining a to-be-executed task issued to the scheduler of the corresponding resource pool in at least one scheduling period from the non-persistent task queue of each resource pool, and issuing the to-be-executed task to the scheduler of the corresponding resource pool. According to the method, the pressure of a cluster scheduler can be effectively relieved, the resource utilization rate is increased, and full-automatic reasonable scheduling of large-scale tasks can be achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cluster management, and in particular to a task scheduling method, a computing device and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence, the amount of business data to be processed is increasingly large, especially in the field of autonomous driving. Therefore, it is necessary to use self-built server clusters or cloud computing server clusters to process and compute business data. Usually, business computing tasks are packaged into images or executable packages, and then a large number of tasks are sorted and resource evaluated by a scheduling module according to a specific strategy. Only when the running conditions are met, the tasks will be allocated to the cluster, bound to the corresponding node and executed. In this way, a large number of tasks can be reasonably run on the server cluster in a certain order.

[0003] With the wide application of Kubernetes (K8S for short), an open source container orchestration platform, clusters are usually managed by K8S, which is responsible for scheduling resource objects such as Pods (the smallest deployable computing unit in Kubernetes). In order to improve resource utilization, a large number of Pods corresponding to tasks are usually deployed to the cluster. However, when the number of Pods to be scheduled reaches a certain threshold, the performance of the K8S scheduler will decrease significantly, resulting in more pending Pods to be scheduled, which brings many challenges to large-scale task scheduling.

[0004] In order to improve the execution efficiency of large-scale tasks in the cluster, the current industry generally adopts the strategy of optimizing Kubernetes cluster, for example, the Koordinator scheduler of Aliyun is an improvement of the original Kubernetes scheduler. However, optimization at the Kubernetes level cannot completely solve the many problems brought by large-scale tasks, such as when a large number of tasks are issued to the cluster, a large number of Pods will still be pending, which will make the cluster scheduling very slow. SUMMARY

[0005] The embodiments of the present application provide a task scheduling method, a computing device and a storage medium to solve the problems of significant decrease in scheduling performance, low resource utilization and poor execution efficiency of the cluster managed based on K8S when executing large-scale tasks in the prior art.

[0006] According to a first aspect of the present application, a task scheduling method is provided, which comprises:

[0007] When a task to be executed is received, the received task to be executed is stored in a persistent task queue;

[0008] Regularly obtain tasks to be executed from the persistent task queue, and load them into the corresponding non-persistent task queue according to the resource pool to which the obtained tasks belong. Each resource pool is independently configured with a non-persistent task queue corresponding to the resource pool.

[0009] Determine the cluster load status of each resource pool based on the real-time server cluster indicator monitoring information received;

[0010] According to the cluster load status of each resource pool, tasks to be executed that are sent to the scheduler of the corresponding resource pool within at least one scheduling cycle are determined from the non-persistent task queue of each resource pool, and sent to the scheduler of the corresponding resource pool.

[0011] According to a second aspect of the present invention, a computing device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in the first aspect above.

[0012] According to a third aspect of the present invention, there is provided a storage medium having a computer program stored thereon, which implements the steps of the method of the first aspect when executed by a processor.

[0013] The task scheduling method of the embodiment of the present invention adopts a multi-level scheduling architecture. First, all tasks to be executed are persistently stored, and the tasks to be executed are loaded into non-persistent high-speed storage media such as memory through a timing mechanism. This effectively integrates the advantages of the full amount of persistent queue information and the fast sorting speed and fast reading and writing of non-persistent queues, which can avoid information loss and improve the task scheduling speed. In addition, the method of the embodiment of the present invention also implements a queue isolation mechanism according to the resource pool to which the task belongs, and realizes efficient scheduling of the hybrid cloud cluster resource pool by allocating non-persistent queues and schedulers to each resource pool, thereby reducing the pressure on the cluster scheduler and improving resource utilization. In addition, the method of the present invention can be implemented by adding a flexible task scheduling system to the upper layer of the cluster, that is, it can be independently deployed and interact with the cluster through the K8S client to complete task level scheduling, thereby truly and effectively reducing the pressure on the cluster scheduler, improving resource utilization, and realizing the reasonable scheduling of large-scale tasks fully automatically, effectively solving the disadvantages of the scheduling scheme of the prior art that it relies entirely on the cluster scheduler and cannot solve the scheduling of large-scale data-intensive tasks. This enables the method of the embodiment of the present invention to achieve good task processing effects in data-intensive task scenarios such as the field of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings described in the following embodiments are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0015] Figure 1 The flow chart of the task scheduling method of an embodiment of the present application is schematically shown;

[0016] Figure 2 The flow chart of the implementation method of step S4 of an embodiment of the present application is schematically shown;

[0017] Figure 3 The effect schematic diagram of the non-persistent task queue allocation of an embodiment of the present application is schematically shown;

[0018] Figure 4 The flow chart of the task resource demand budgeting method based on the topological structure of an embodiment of the present application is schematically shown;

[0019] Figure 5 The topological structure of a certain to-be-executed task and the example effect diagram of the resource demand calculation thereof are schematically shown;

[0020] Figure 6 The flow chart of the task scheduling method of another embodiment of the present application is schematically shown;

[0021] Figure 7 The architecture diagram of the task scheduling system of an embodiment of the present application is schematically shown;

[0022] Figure 8 The principle block diagram of the computing device of an embodiment of the present application is schematically shown;

[0023] Figure 9 The hardware structure schematic diagram of the task scheduling system of another embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0024] In order to make the objects, technical solutions and advantages of the embodiments of the present application more clear, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0025] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other in the case of no conflict.

[0026] In the description of the present application, it should be understood that the orientation or positional relationship indicated by the terms "center", "middle", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "axial", "radial", "circumferential" and the like are based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. The features defined with "first", "second" are used to distinguish the feature names, and not to have special meanings, and in addition, the features defined with "first", "second" can be explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0027] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connection" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integrally connected; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0028] It should also be noted that in this paper, the terms "including", "containing", not only include those elements, but also include other elements not explicitly listed, or include elements inherent to the process, method, article or device. Without more limitations, the elements defined by the statement "including" do not exclude the presence of other identical elements in the process, method, article or device including the elements. The terms used in this paper are generally the terms commonly used by those skilled in the art, and if they are inconsistent with the commonly used terms, the terms in this paper shall prevail.

[0029] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0030] The application will be described in further detail below with reference to the drawings.

[0031] The task scheduling method in the embodiments of the application can be applied in task scheduling of any server cluster and can also be applied in data-intensive large-scale task scheduling scenarios (such as automatic driving scenarios), so that these cluster scheduling systems can optimize cluster scheduler performance, improve cluster resource utilization and scheduling efficiency by using the scheme of the embodiments of the application. Figure 1 The task scheduling method of an embodiment of the application is schematically shown, and the execution subject of the method can be, for example, a processor of a computer, a cloud server, a server in a cluster, and the like, and the embodiments of the application do not limit this. As shown in Figure 1 The method comprises the following steps:

[0032] In step S1, when a to-be-executed task is received, the received to-be-executed task is stored into a persistent task queue.

[0033] In step S2, the to-be-executed task is acquired from the persistent task queue at a time, and the to-be-executed task is loaded into a corresponding non-persistent task queue according to the resource pool to which the to-be-executed task belongs, wherein each resource pool is independently configured with a non-persistent task queue corresponding to the resource pool.

[0034] In step S3, the cluster load state of each resource pool is determined according to real-time received index monitoring information of the server cluster.

[0035] In step S4, the to-be-executed task to be issued to the scheduler of the corresponding resource pool in at least one scheduling period is determined from the non-persistent task queue of each resource pool according to the cluster load state of each resource pool, and the to-be-executed task is issued to the scheduler of the corresponding resource pool.

[0036] The to-be-executed task can be a Workflow (work flow) composed of at least one Pod according to a certain topology structure. The life cycle of the entire Workflow starts from the start of the first batch of Pods and ends until the last batch of Pods is completed. The topology structure enables the Pods to be connected to each other in series and in parallel according to task requirements and expectations, and when the to-be-executed task is issued to the cluster for execution, it will be executed according to the Pod connection relationship defined by the topology structure, that is, only after the previous Pod is executed, the subsequent Pod will start to execute. In step S1, the to-be-executed task can be received from an upper application module such as a third-party platform, an API interface or a WebUI. The persistent queue can be a storage medium such as a database.

[0037] In step S2, the non-persistent queue can be a high-speed storage medium such as memory. Since the persistent queue has the advantage of information fullness, but also has the disadvantage of slow sorting speed and slow reading and writing, while the non-persistent queue has the advantage of fast reading and writing speed, thus by respectively configuring a non-persistent queue for each resource pool, and periodically synchronizing the to-be-executed tasks from the persistent queue, the task and the required resource thereof can be isolated according to the resource pool to which the task belongs, so as to effectively improve the fairness and efficiency of task scheduling execution, and information loss can be avoided. Wherein, the resource pool can be divided according to business requirements, such as according to business direction, which is not limited by the embodiments of the present application, and the specific time period of timing can be set according to requirements and experience values.

[0038] In step S3, the determined cluster load state preferably includes resource utilization and Pod quantity of the server cluster, resource utilization and Pod quantity of the corresponding resource pool, resource utilization and Pod quantity of the current task type. In a specific implementation, the running state information of all Pods in the cluster, the load information of the cluster, the load information of each resource pool, and the load information of each task type can be dynamically obtained by subscribing to the cluster monitoring indicators of the server cluster in real time, and the cluster load state of each resource pool is determined according to the obtained running state information and load information of the Pods. The running state information of the Pods is the object state information provided by the K8S-managed cluster server to the outside, including the state of all Pods in the cluster, from which the number of Pods in the cluster, the number of Pods in each resource pool, and the number of Pods of each task type can be counted. The load information of the cluster includes the total amount and the current usage of various resources of the cluster, such as the total amount and the current usage of CPU resources of the cluster, the total amount and the current usage of GPU resources of the cluster, the total amount and the current usage of memory resources of the cluster, etc. Similarly, the load information of each resource pool includes the total amount and the current usage of various resources allocated to each resource pool, such as the total amount and the current usage of CPU resources of each resource pool, the total amount and the current usage of GPU resources of each resource pool, the total amount and the current usage of memory resources of each resource pool, etc. The load information of each task type includes the total amount and the current usage of various resources allocated to each task type, such as the total amount and the current usage of CPU resources allocated to each task type, the total amount and the current usage of GPU resources allocated to each task, the total amount and the current usage of memory resources allocated to each task, etc. The load information of the cluster, the load information of each resource pool, and the load information of each task type can be obtained from the K8S-managed cluster server by subscribing to the cluster server, and the subscription can be implemented by calling the API of the K8S-managed cluster server. The specific subscription method can refer to related prior art, and will not be described in detail herein. The resource utilization in the embodiment of the present application is calculated based on the total amount and the current usage of each resource, such as the resource utilization of the cluster = used cluster resource amount / total amount of cluster resources * 100%, similarly, the resource utilization of the resource pool = used resource amount of the corresponding resource pool / total amount of resources of the corresponding resource pool * 100%, the resource utilization of a task type = used resource amount of the task type / total amount of resources of the task type * 100%, so the resource utilization of the cluster, the resource utilization of each resource pool, and the resource utilization of each task type can be calculated based on the obtained load information. It should be noted that in the embodiment of the present application, the task type is divided according to the task template used by the to-be-executed task, and the to-be-executed tasks created using the same task template are regarded as the same task type.

[0039] In step S4, the scheduling period refers to a scheduling period defined by a scheduler of the server cluster. In the existing server cluster, the scheduler limits the length of each scheduling period, and in the specific scheduling process, the scheduler of the server cluster schedules the to-be-executed tasks based on the scheduling period. In order to effectively control the number of to-be-scheduled objects on the server cluster, guarantee the scheduling performance and speed of the scheduler of the server cluster, and optimize the resource utilization, the embodiments of the present application determine the to-be-executed tasks to be issued to the scheduler of the corresponding resource pool in each scheduling period according to the cluster load state of each resource pool. As a possible implementation, Figure 2 The implementation method process of step S4 of an embodiment of the present application is schematically shown as follows: Figure 2 As shown in the figure, step S4 can be specifically implemented as comprising:

[0040] Step S41, when it is judged that the cluster load state is in the first gradient range, all the to-be-executed tasks in the non-persistent task queue of the corresponding resource pool are directly issued to the scheduler of the corresponding resource pool, so that the scheduler completes the splitting and scheduling of the to-be-executed tasks in the non-persistent task queue.

[0041] Step S42, when it is judged that the cluster load state is in the second gradient range, the to-be-executed tasks that meet the conditions are determined according to the resource requirements of each to-be-executed task in the non-persistent task queue of the corresponding resource pool, and are issued to the scheduler of the corresponding resource pool in at least one scheduling period.

[0042] Step S43, when it is judged that the cluster load state is in the third gradient range, the to-be-executed tasks are suspended from being issued to the scheduler of the corresponding resource pool.

[0043] The first gradient range, the second gradient range and the third gradient range can be determined according to requirements and experience values. For example, the first gradient range can refer to that the number of Pods in the cluster is less than 70% of the maximum number of Pods that the cluster can bear, the resource utilization of the cluster is less than 70%, the number of Pods in the corresponding resource pool is less than 70% of the maximum number of Pods that the corresponding resource pool can bear, the resource utilization of the corresponding resource pool is less than 70%, the number of Pods of the current task type is less than 70% of the maximum number of Pods that the current task type can bear, and the resource utilization of the current task type is also less than 70%. The second gradient range can refer to that the number of Pods in the cluster is between 70% and 90% of the maximum number of Pods that the cluster can bear, the resource utilization of the cluster is between 70% and 90%, the number of Pods in the corresponding resource pool is between 70% and 90% of the maximum number of Pods that the corresponding resource pool can bear, the resource utilization of the corresponding resource pool is between 70% and 90%, the number of Pods of the current task type is between 70% and 90% of the maximum number of Pods that the current task type can bear, and the resource utilization of the current task type is also between 70% and 90%. The third gradient range can refer to that the number of Pods in the cluster is more than 90% of the maximum number of Pods that the cluster can bear, the resource utilization of the cluster is more than 90%, the number of Pods in the corresponding resource pool is more than 90% of the maximum number of Pods that the corresponding resource pool can bear, the resource utilization of the corresponding resource pool is more than 90%, the number of Pods of the current task type is more than 90% of the maximum number of Pods that the current task type can bear, and the resource utilization of the current task type is also more than 90%.

[0044] As a preferred implementation, in the embodiment of the present application, when scheduling tasks for each resource pool, the Pod quantity and resource utilization of the cluster can be first determined to be in which gradient range, and then the Pod quantity and resource utilization of the resource pool are determined to be in which gradient range, and finally the Pod quantity and resource utilization of the task type are determined to be in which gradient range. As a preferred task flow limiting strategy, only when the Pod quantity and resource utilization of the cluster, the resource pool and the task type are all in the first gradient range, step S41 is executed; otherwise, if one of the indicators, such as the Pod quantity of the cluster or the resource utilization of the cluster or the Pod quantity of the resource pool or the resource utilization of the resource pool or the Pod quantity of the task type or the resource utilization of the task type, is in the third gradient range, step S43 is executed; and in other cases, that is, if at least one indicator is in the second gradient range and no indicator is in the third gradient range, step S42 is executed. Thus, in the embodiment of the present application, the Pod quantity and resource utilization are used as the key load indicators, the correlation between the Pod quantity and the cluster load pressure is considered, and the comprehensive load evaluation is performed based on the three dimensions of the cluster, the resource pool and the task type, so that the multi-dimensional load evaluation mechanism is used to intelligently adjust the task issuing rate, the scheduling pressure of the scheduler at the cluster level is effectively improved, the performance damage of the scheduler at the cluster level in the large-scale task scenario is avoided, and efficient and reasonable task scheduling and calculation can be realized even in the data-intensive task scenario, thereby effectively improving the task processing efficiency and resource utilization of the cluster.

[0045] In step S41, when the cluster load state is in the first gradient range, it can be considered that the cluster load pressure is in a relatively low state, at this time, task flow limiting is not performed, and the tasks are allowed to be issued and scheduled according to the normal state of the scheduler of the cluster server, therefore, in the embodiment of the present application, all the tasks in the non-persistent queue of the corresponding resource pool are issued to the scheduler of the cluster server, so that the scheduler of the cluster server performs specific task decomposition and scheduling processing. When the cluster load pressure reaches the primary warning threshold, that is, when the cluster load state is in the second gradient range, in step S42, the load protection mechanism is automatically triggered, the task flow limiting strategy is started, and the number of tasks to be executed issued to the cluster server is limited, so as to relieve the cluster pressure and ensure the scheduling performance and speed of the scheduler in the cluster. When the cluster load pressure reaches the advanced warning threshold, that is, when the cluster load state is in the third gradient range, in step S43, the task fuse strategy is automatically triggered, and the tasks to be executed are suspended from being issued to the server cluster, thereby effectively preventing the cluster from being overloaded.

[0046] In step S42, when implementing the task flow control strategy, the embodiment of the present application preferably determines the number of to-be-executed tasks or to-be-executed tasks to be issued to the cluster server according to the resource requirement of the to-be-executed tasks, and specifically can compare the resource requirement of the to-be-executed tasks with the current remaining unused resource amount of the corresponding resource pool to determine the number of to-be-executed tasks or to-be-executed tasks to be issued to the resource pool, wherein the determined number of to-be-executed tasks or to-be-executed tasks needs to make the resources of the corresponding resource pool to be able to meet the running conditions of the to-be-executed tasks to be issued. Exemplarily, assuming that the cluster load state corresponding to a resource pool includes that the total amount of CPU resources of the server cluster is 1000 cores, the current usage is 750 cores, the total amount of CPU resources of the resource pool is 100 cores, and the current usage is 75 cores, then the current remaining unused resource amount of the resource pool can be determined to be 25 cores of CPU. Assuming that the to-be-executed tasks in the non-persistent queue include 5, and the resource requirements corresponding to each to-be-executed task are 4 cores of CPU, 32 cores of CPU, 2 cores of CPU, 8 cores of CPU and 8 cores of CPU respectively, then by comparing the resource requirements of the to-be-executed tasks with the current remaining unused resource amount of the resource pool, it can be determined that the four to-be-executed tasks with resource requirements of 4 cores of CPU, 2 cores of CPU, 8 cores of CPU and 8 cores of CPU can be issued to the resource pool at present, i.e. issuing the four to-be-executed tasks can meet the running conditions of the resource pool. Of course, this determination method of the number of to-be-executed tasks is to maximize the current unused resource amount of the resource pool as the running condition, and it can be understood that in other embodiments, other strategies can also be combined to determine the to-be-executed tasks or the number of to-be-executed tasks to be issued, such as in the embodiment that the to-be-executed tasks have scheduling priorities, the priority strategy can be combined to determine the to-be-executed tasks or the number of to-be-executed tasks to be issued that meet the conditions, such as assuming that the priorities of the aforementioned five to-be-executed tasks are that the to-be-executed tasks with resource requirements of 4 cores of CPU and 32 cores of CPU are the first priority, and the remaining three to-be-executed tasks are the second priority, then by combining the priority strategy and comparing the resource requirements of the to-be-executed tasks with the current remaining unused resource amount of the resource pool, it can be determined that the to-be-executed task with resource requirement of 4 cores of CPU can be issued to the resource pool at present to meet the running conditions.Of course, for other possible scenarios, the number of to-be-executed tasks to be issued can also be directly determined according to the resource demand in the to-be-executed task queue and the current amount of remaining unused resources of the corresponding resource pool, and then in the issuing stage, combined with a specific scheduling strategy, it is determined which to-be-issued tasks are issued to the resource pool. For example, in a certain scenario, all to-be-executed tasks in the non-persistent queue corresponding to a certain resource pool have a resource demand of no more than 8-core GPU. At this time, taking the current amount of remaining unused resources of the resource pool as 25-core CPU as an example, the number of to-be-issued tasks can be directly determined by comparing the amount of remaining unused resources with the maximum resource demand of the to-be-executed tasks, i.e., 8-core CPU, to determine that the number of to-be-issued tasks is 3. After determining the number of to-be-executed tasks n, the embodiments of the present application can combine other scheduling strategies such as priority strategies to select to-be-executed tasks to be issued to the corresponding resource pool of the cluster server. The other scheduling strategies here need to be determined in combination with the in-out queue situation and actual demand of the to-be-executed tasks in the non-persistent queue. For example, assuming that the to-be-executed tasks in the non-persistent queue are sorted according to priority, the out-of-queue strategy is first-in last-out, and low-priority tasks are entered first, and high-priority tasks are entered last, then after determining the number of to-be-executed tasks n, the last n to-be-executed tasks in the non-persistent queue can be taken from the non-persistent task queue in the corresponding scheduling period and issued to the scheduler of the corresponding resource pool of the server cluster. Considering that when the number of to-be-executed tasks exceeds n, if the following to-be-executed tasks cannot be satisfied for a long time, it will cause the scheduling period to always only schedule the n to-be-executed tasks with priority in front, and other to-be-executed tasks cannot participate in scheduling, and thus cannot be executed for a long time. In order to improve this situation and ensure the fairness and efficiency of task scheduling, as a possible implementation manner, the embodiments of the present application can also sort the to-be-executed tasks with the same priority in the non-persistent queue according to the task creation timestamp, so that the to-be-executed tasks with earlier creation time under the same priority can be executed preferentially. In other possible implementation manners, the waiting time of the to-be-executed tasks in the non-persistent task queue of each resource pool can also be monitored, and the task priority of the to-be-executed tasks with a waiting time exceeding a specified time threshold can be improved, wherein the way to improve the task priority can be stepwise or gradual, such as increasing the priority of the to-be-executed tasks with a waiting time exceeding 10 minutes by one level each time, or associating the degree of improvement with the waiting time (such as increasing the priority by one level for a waiting time of 10 minutes, and increasing the priority by two levels for a waiting time of 20 minutes).Thus, the embodiment of the present application can realize dynamic and intelligent adjustment of priority in the granularity of resource pool. When it is detected that a low-priority task fails to be successfully scheduled within a specified time threshold, a priority promotion mechanism is automatically triggered, the task priority is gradually increased through a progressive adjustment strategy until the task gets an execution opportunity. This adaptive priority adjustment scheme not only ensures the fairness of low-priority tasks, but also maintains the overall scheduling efficiency of the system, realizes the best balance between resource utilization and task fairness, and can effectively prevent the task "starvation" phenomenon.

[0047] In some preferred embodiments, the non-persistent task queue configured for each resource pool can include more than one, such as two or more, and each non-persistent task queue is respectively assigned a task priority, Figure 3 Such embodiments are schematically shown as follows, Figure 3As shown, in this scenario, the cluster server divides three resource pools, i.e., an A resource pool, a B resource pool and a C resource pool, and each resource pool is respectively assigned a non-persistent queue on the upper layer of the cluster, wherein the non-persistent queue of the A resource pool is three, the non-persistent queue of the B resource pool is four, and the non-persistent queue of the C resource pool is one, these non-persistent queues correspond to a task priority respectively, and the to-be-executed tasks in the non-persistent queue are obtained from the persistent queue in a fixed time, when the to-be-executed task is obtained from the persistent task queue, it will be loaded to the corresponding non-persistent task queue according to the task priority of the to-be-executed task, and when the task scheduling is performed, the to-be-executed tasks of each resource pool of the cluster server delivered to the cluster server will be determined according to the cluster load state, so as to realize intelligent flow limiting delivery and ensure the performance and speed of the scheduler in the cluster server. Among them, as an implementation manner, in the case of including multiple non-persistent queues and respectively assigning a task priority to each non-persistent queue, the present embodiment can further determine the to-be-executed tasks delivered to the scheduler of the corresponding resource pool in at least one scheduling period according to the cluster load state of each resource pool, the task priority of the non-persistent task queue of the corresponding resource pool and the priority scheduling strategy, at this time, after the number of to-be-executed tasks is determined in step S42, the to-be-executed tasks delivered can be determined in combination with the priority scheduling strategy, the priority scheduling strategy can be set according to the demand, for example, can include but is not limited to strict priority (Strict Priority), non-strict priority (Non-strict Priority), static priority (Static Priority), dynamic priority (Dynamic Priority), first-in-first-out (FIFO), short job first (SJF), queue quota (Queue Quota) and round robin scheduling (Round Robin) and the like. Of course, the priority scheduling strategy can also be combined when determining the number of to-be-executed tasks to be delivered, and in other implementation manners, the to-be-executed tasks to be delivered can also be directly determined in combination with the priority scheduling strategy and directly delivered, and the present embodiment does not limit the specific manner. In step S41, the scheduler of the cluster server can also combine the priority scheduling strategy when splitting and scheduling the to-be-executed tasks, so as to ensure the fairness and efficiency of task scheduling. It can be understood that the priority scheduling strategy mentioned in the present embodiment can be combined with the above-mentioned monitoring of the waiting time of the to-be-executed task and the step-by-step or gradual increase of the task priority based on the waiting time, so as to avoid the effect that some tasks cannot be scheduled for a long time.

[0048] It should be noted that the resource requirement of the to-be-executed task in step S42 is fixed when the task is created, and the topology and resource requirement are stored in the persistent queue. When the judgment is made, only the resource requirement of the to-be-executed task needs to be obtained to determine whether the resource is sufficient. There are various resource requirement estimation methods in the prior art, but the resource estimation method in the prior art is often difficult to accurately evaluate the complex topology relationship of the Workflow, which also brings challenges to the resource utilization and task scheduling efficiency. Therefore, the embodiment of the present application also provides a resource requirement method of each to-be-executed task. The method estimates the resource requirement of a complex topology task composed of multiple Pods (container collection Pod is a group of closely related container collections, which is the basic unit of Kubernetes scheduling) by using a hierarchical resource estimation algorithm based on topology sorting, thereby effectively improving the estimation accuracy of the resource requirement of the task with complex topology structure and ensuring that the task can be effectively executed. The resource requirement determination method of the to-be-executed task will be described below. For convenience of description, the to-be-executed task is referred to as a target task. Wherein, Figure 4 The implementation process of the resource requirement determination method of the target task is schematically shown as Figure 4 as shown, which includes:

[0049] Step S31, obtaining the topology structure of the target task, grouping the target task according to the topology structure of the target task according to the topology structure of the target task;

[0050] Step S32, determining the resource requirement of each dependency level of the target task according to the grouping result of the dependency level;

[0051] Step S33, determining the resource requirement of the target task according to the resource requirement of each dependency level.

[0052] The topology of each task to be executed is generated when the task is created, and the topology of the target task can be obtained according to the information when the task is created. After obtaining the topology, the topology of the target task is parsed in step S31 to determine the dependency relationship of each level in the topology, and the pods are intelligently grouped according to the dependency level, that is, the same dependency level is grouped. In step S32, the embodiment of the application adopts a hierarchical aggregation strategy to independently calculate the resource demand of each dependency level. For example, the same type of resources required by each component in the same level can be accumulated, and the total demand of the accumulated same type of resources is taken as the resource demand of the level. In step S33, the embodiment of the application extracts the maximum value of each type of resource demand from the resource demands of all levels as the resource demand of the entire task to be executed. For example, taking the resource demand including CPU demand, GPU demand, memory demand and cache demand as an example, the maximum value of each type of resource demand is extracted from the resource demands of all levels as the resource demand of the entire target task. Specifically, the maximum CPU request, the maximum GPU request, the maximum memory request and the maximum cache request are extracted from all levels, and the maximum values of the key resource indicators of all dependency levels are integrated to form a complete resource demand profile of the target task. Figure 5 An example effect diagram of the topology of a target task and resource demand calculation is schematically shown. The topology of a target task is taken as an example Figure 5 As shown in the figure, the target task can be divided into five dependency levels through the analysis and grouping of step S31. The first dependency level Level1 includes a data reading component and a data collection component, the second dependency level includes a data cleaning A component and a data cleaning B component, the third dependency level includes a data analysis A component, the fourth dependency level includes a data analysis D component, and the fifth dependency level includes a data analysis E component. Assuming that the resource demand of the data reading component is 4C8G0 card (indicating that the required resources are 4 core CPU, 8G memory and 0 cache card), and the resource demand of the data collection component is 2C2G0 card (indicating that the required resources are 2 core CPU, 2G memory and 0 cache card), then the resource demand of the first dependency level can be obtained through step S32, that is, the total demand of the same type of resources of the first dependency level is accumulated as the resource demand of the level. According to the resource demand of each component in the same level, the resource demand of the first dependency level is calculated as 6C10G0 card. Figure 5The required resource conditions of the components marked with the middle marks are sequentially deduced, and the resource requirements of the second dependency level of the target task are determined to be 12C24G2 cards, the resource requirements of the third dependency level are 8C18G4 cards, the resource requirements of the fourth dependency level are 32C16G1 cards, and the resource requirements of the fifth dependency level are 2C4G1 cards. In step S33, the maximum value of each type of resource in the dependency level is taken, and the resource requirements of the to-be-executed task are integrated to be 32C24G4 cards. Since the target task includes multiple sub-tasks associated by a certain topology structure, the sub-tasks are not executed in parallel, but are sequentially dispatched by the scheduler in the server cluster based on the limitation of the topology structure, so that the resource requirement estimation algorithm of the embodiment of the present application, which respectively calculates the resource requirements according to the dependency levels and forms the final task resource requirement according to the maximum value of the divided dependency levels, realizes accurate estimation of resource requirements, avoids resource fragmentation, and helps to establish a scientific task admission mechanism during scheduling, effectively controls the cluster load pressure, and at the same time can improve the success rate of complex topology task delivery and optimize the overall scheduling efficiency.

[0053] In some embodiments, in order to further improve the utilization rate of cluster resources and prevent resource waste, the method of the embodiment of the present application can further include a task type-based flow control scheme. Specifically, the task type-based flow control scheme can be implemented to include monitoring abnormal task types according to the real-time received server cluster index monitoring information, and temporarily suspending the to-be-executed tasks of the task type when an abnormal task type is monitored. Wherein, monitoring abnormal task types can be to monitor the execution time of each task type, when it is found that the execution time of a certain type of task continuously exceeds the normal range, it is determined that there may be user code logic abnormality, that is, it is regarded as an abnormal task type, and flow control is automatically implemented for the type of task, that is, the to-be-executed tasks of the task type are suspended. Therefore, the embodiment of the present application can also avoid the resource depletion and adverse effects on cluster scheduling efficiency caused by task abnormalities.

[0054] Due to the limited local disk capacity of the server, for data-intensive business scenarios such as the autonomous driving industry, it is usually necessary to mount one or more cache systems to speed up the data operation of the Pod. However, the cache system is costly and resource-limited, and if a large number of workflows fail to complete execution for a long time, it will occupy a large amount of cache resources, resulting in waste and exhaustion of resources, which will also bring challenges to resource utilization and task execution efficiency. In order to reduce the risk of abuse of cache resources and further improve the overall resource utilization of the cluster, in some other embodiments, the method of the embodiment of the application can further include a scheme of monitoring the cache resources and automatically triggering the cache resource recycling strategy according to the monitoring situation. Specifically, it can be implemented to include identifying and screening out the task instances occupying cache resources and in idle state according to the real-time received index monitoring information of the server cluster, and restarting the screened task instances and storing the restarted task instances to the persistent task queue. Among them, the index monitoring information subscribed to the server cluster can include the cache resource usage and resource utilization of each task instance, such as the CPU resource utilization of the task instance, so that by judging the cache resource usage and resource utilization of the corresponding task instance, the task instances occupying cache resources and in idle state can be identified and screened out, for example, the task instances with CPU utilization <1% and cache resource usage >0 can be determined as the task instances occupying cache resources but in idle state. Wherein, restarting the task instance includes deleting the task instance from the scheduling queue and releasing resources and recreating the task instance. Preferably, when restarting the task instance, the low-priority task instance can be restarted first or the high-priority task instance can be restarted first according to the demand. By monitoring the cache resources of the task instance and continuously tracking the key indicators such as cache usage, the embodiment of the application realizes the fine-grained supervision of the cluster resources and the effective release of the cache resources. This active cache self-help strategy can significantly improve the overall resource utilization of the cluster and ensure the stability and reliability of the cluster.

[0055] Figure 6 The task scheduling method flow of another embodiment of the application is schematically shown as Figure 6 as shown, which further includes Figure 1 as shown.

[0056] Step S5, listening to the scheduling failure event in the cluster to identify and screen out the target Pod;

[0057] Step S6, when it is judged that the preset condition is met, the screened target Pod is queue-assigned and re-enqueued according to the failure reason.

[0058] In step S5, the scheduling failure event can be monitored through the interface or API provided by K8S to obtain the scheduling failure event in the cluster. For example, the identified target Pod can be a Pod that fails to be scheduled due to insufficient resources. In this case, the Pod that fails to be scheduled due to insufficient resources can be identified by analyzing the failure reason of the scheduling failure event. According to the requirement, the identified target Pod can also be a Pod corresponding to other scheduling failure events. In step S6, the preset condition can be that the number of times of scheduling failure due to insufficient resources reaches a preset value. At this time, the events of scheduling failure due to insufficient resources corresponding to each Pod need to be counted to screen the target Pod. It should be noted that the original scheduling queue architecture of the Koordinator scheduler includes three queues, namely, an active queue, a backoff queue and an unschedulable queue. The active queue is an active queue, from which a Pod is taken out to attempt scheduling at the beginning of each scheduling period. The backoff queue is a queue for temporary retreat, which is sorted according to the length of the retreat time by default, and the Pod with shorter retreat time needs to be in front. The unschedulable queue is used to store the Pods that fail to be scheduled, to wait for resource updates and other related Pod scheduling success events, and to re-schedule the Pods in the queue when receiving these events. In step S6, the queue distribution and re-queue of the screened target Pod according to the failure reason means that the screened target Pod is distributed to the backoff queue or the unschedulable queue according to the specific failure reason (i.e., which resource is insufficient) and is re-queued. Therefore, the embodiment of the present application can not only solve the problems of severe impact on resource utilization, frequent misuse of limited cache resources, long-term scheduling of low-priority tasks and the like caused by the limitation of the design and positioning of the cluster scheduler itself in the large-scale data-intensive task scheduling scenario, improve the resource utilization, relieve the pressure of the cache system, and realize the full-automatic reasonable scheduling of large-scale tasks, but also can further optimize the scheduling strategy of the cluster scheduler to effectively reduce the pressure of the cluster scheduler.

[0059] In the preferred implementation, the embodiments of the present application also respectively deploy an independent scheduler instance for each resource pool, exemplarily a Koordinator scheduler instance. The schedulers of the resource pools are configured to adopt a queue priority scheduling strategy for task scheduling, which includes starting the scheduling task of other queues only when all the Pods in the active queue complete the scheduling attempt. Thus, the embodiments of the present application also optimize the Koordinator scheduler, implement a full-queue coverage scheduling guarantee mechanism, ensure that all tasks in the queue in each scheduling cycle can obtain at least one scheduling opportunity, further improve the scheduling efficiency, and through the resource pool level scheduler isolation, the method of the embodiments of the present application can also significantly improve the overall scheduling performance of the cluster, and achieve a substantial improvement in scheduling throughput.

[0060] Figure 7 The task scheduling system architecture of an embodiment of the present application is schematically shown. As shown in the figure, Figure 7 The server cluster 100 is divided into multiple resource pools according to the business requirements, each resource pool is respectively configured with a scheduler and at least one non-persistent queue, and the task scheduling system 200 is deployed on the upper layer of the server cluster 100, which is configured to interact with the schedulers of each resource pool in the cluster. The task scheduling system 200 includes:

[0061] The queuing module 201 is responsible for loading the to-be-executed tasks from the persistent queue to the non-persistent queue according to a certain strategy, which is specifically implemented as including storing the received to-be-executed tasks to the persistent task queue when receiving the to-be-executed tasks, and periodically obtaining the to-be-executed tasks from the persistent task queue, loading the to-be-executed tasks to the corresponding non-persistent task queue according to the resource pool to which the to-be-executed tasks belong, wherein the priority scheduling strategy can be combined when loading;

[0062] The issuing limiting module 202 is responsible for monitoring the load pressure of the resource pool, and issuing task limitation according to the load pressure, which is specifically implemented as including determining the cluster load state of each resource pool according to the real-time received index monitoring information of the server cluster, and determining the to-be-executed tasks issued to the scheduler of the corresponding resource pool in at least one scheduling cycle from the non-persistent task queue of each resource pool according to the cluster load state of each resource pool, and issuing the to-be-executed tasks to the scheduler of the corresponding resource pool;

[0063] The resource estimation module 203 is configured to be responsible for estimating resource requirements of a task to be executed with a complex topology. Specifically, the resource estimation module 203 can be implemented by including the following steps: grouping the task to be executed according to a topology of the task to be executed; determining resource requirements of each dependency level of the task to be executed according to the grouping result of the dependency levels; and determining resource requirements of the entire task to be executed according to the resource requirements of each dependency level.

[0064] The cache self-help module 204 is configured to monitor cache resources and automatically trigger a cache resource recycling strategy according to a monitoring result. Specifically, the cache self-help module 204 can be implemented by including the following steps: identifying and screening task instances that occupy cache resources and are in an idle state according to real-time received index monitoring information of a server cluster; and restarting the screened task instances and storing the restarted task instances in a persistent task queue.

[0065] The priority dynamic adjustment module 205 is configured to be responsible for adjusting a priority of a long-term unscheduled task. Specifically, the priority dynamic adjustment module 205 can be implemented by including the following steps: monitoring a waiting time of a task to be executed in a non-persistent task queue of each resource pool; and improving a task priority of the task to be executed whose waiting time exceeds a specified time threshold.

[0066] It should be noted that the specific implementation process of each module in the task scheduling system can refer to the description in the foregoing method part, and will not be described here. Since the task scheduling system of the embodiment of the present application is deployed on a server cluster, it interacts with the cluster through a K8S client, completes task level scheduling, and can effectively solve the problems of frequent scheduling stalls in large-scale data-intensive task scheduling scenarios, serious impact on resource utilization, frequent misuse of limited cache resources, and long-term unscheduling of low-priority tasks, effectively reduce the pressure on the cluster scheduler, improve resource utilization, relieve the pressure on the cache system, and realize full-automatic reasonable scheduling of large-scale tasks.

[0067] In other possible embodiments, the task scheduling system can further include a scheduling optimization module configured to interact with a scheduler of a resource pool to listen to a scheduling failure event in the cluster, identify and screen a target Pod from the scheduling failure event, and perform queue allocation and re-queueing on the screened target Pod according to a failure cause by the scheduler of the resource pool when it is determined that a preset condition is met.

[0068] In other embodiments, in the system architecture of the embodiment of the present application, the scheduler of each resource pool can be configured to adopt a queue priority scheduling strategy for task scheduling. The queue priority scheduling strategy includes starting a scheduling task of another queue only when all Pods in an active queue complete a scheduling attempt.

[0069] Figure 8 A computing device of an embodiment of the present application is schematically shown in the figure, which is implemented as comprising:

[0070] a memory 81 for storing executable instructions; and

[0071] a processor 82 communicatively connected with the memory 81, configured to execute the executable instructions stored in the memory, which when executed by the processor implement the steps of the task scheduling method of any of the preceding embodiments.

[0072] In some embodiments, an embodiment of the present application provides a non-volatile computer readable storage medium, in which one or more programs including executable instructions are stored, which can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the task scheduling method of any of the preceding embodiments of the present application.

[0073] In some embodiments, an embodiment of the present application also provides a computer program product, which comprises a computer program stored on a non-volatile computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the task scheduling method of any of the preceding embodiments.

[0074] In some embodiments, an embodiment of the present application also provides an electronic device, which comprises at least one processor, and a memory communicatively connected with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the task scheduling method of any of the preceding embodiments.

[0075] In some embodiments, an embodiment of the present application also provides a storage medium, on which a computer program is stored, which when executed by a processor implements the task scheduling method of any of the preceding embodiments.

[0076] Figure 9 is a schematic diagram of the hardware structure of a task scheduling system provided by another embodiment of the present application, and the task scheduling system can be implemented in the structure shown in the figure, as shown in Figure 9 The task scheduling system comprises:

[0077] one or more processors 610 and a memory 620, Figure 9 of which one processor 610 is taken as an example.

[0078] The task scheduling system can also comprise an input device 630 and an output device 640.

[0079] The processor 610, the memory 620, the input device 630 and the output device 640 can be connected through a bus or other means, Figure 9 The bus connection is taken as an example.

[0080] The memory 620, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the task scheduling method in the embodiment of the application. The processor 610 executes various functions and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, implements the task scheduling method of the above method embodiment.

[0081] The memory 620 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the task scheduling method, etc. In addition, the memory 620 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 can optionally include a memory remotely arranged relative to the processor 610, and these remote memories can be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0082] The input device 630 can receive input digital or character information, and generate signals related to the user settings and function control of the image processing device. The output device 640 can include a display device such as a display screen.

[0083] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, perform the task scheduling method in any of the above method embodiments.

[0084] The above product can execute the method provided by the embodiment of the application, has the function module and the beneficial effect corresponding to the execution method. The technical details not described in detail in the embodiment can be referred to the method provided by the embodiment of the application.

[0085] The electronic device of the embodiment of the application exists in various forms, including but not limited to:

[0086] (1) Mobile communication device: the characteristic of this kind of device is that it has mobile communication function, and the main target is to provide voice and data communication. This kind of terminal includes: smart phone (such as iPhone), multimedia phone, functional phone, and low-end phone, etc.

[0087] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. Such terminals include PDA, MID and UMPC devices, such as iPad.

[0088] (3) Portable entertainment devices: These devices can display and play multimedia content. Such devices include audio and video players (such as iPod), handheld game consoles, electronic books, and smart toys and portable car navigation devices.

[0089] (4) Servers: Devices that provide computing services, the composition of a server includes a processor, a hard disk, a memory, a system bus, etc. The server is similar to a general-purpose computer architecture, but due to the need to provide high-reliability services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, manageability, etc.

[0090] (5) Other electronic devices with data interaction functions.

[0091] The device embodiments described above are only illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs.

[0092] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a general hardware platform, or by hardware. Based on this understanding, the above technical solutions or the part that contributes to the related art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the methods described in each embodiment or some part of the embodiment.

[0093] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A task scheduling method, characterized in that: include: When receiving tasks to be executed, all received tasks to be executed are stored in the persistent task queue; Regularly obtain tasks to be executed from the persistent task queue, and load them into the corresponding non-persistent task queue according to the resource pool to which the obtained tasks belong. Each resource pool is independently configured with a non-persistent task queue corresponding to the resource pool. Determine the cluster load status of each resource pool based on the real-time server cluster indicator monitoring information received; According to the cluster load status of each resource pool, tasks to be executed that are sent to the scheduler of the corresponding resource pool within at least one scheduling cycle are determined from the non-persistent task queue of each resource pool, and sent to the scheduler of the corresponding resource pool.

2. The method according to claim 1, characterized in that Determine, from the non-persistent task queue of each resource pool according to the cluster load status of each resource pool, tasks to be executed that are to be sent to the scheduler of the corresponding resource pool within at least one scheduling cycle, and send the tasks to the scheduler of the corresponding resource pool, including: When it is determined that the cluster load state is within the first gradient range, all pending tasks in the non-persistent task queue of the corresponding resource pool are directly sent to the scheduler of the corresponding resource pool, so that the scheduler completes the splitting and scheduling of the pending tasks in the non-persistent task queue; When it is determined that the cluster load state is in the second gradient range, according to the resource requirements of each to-be-executed task in the non-persistent task queue of the corresponding resource pool, determining the qualified to-be-executed tasks to be sent to the scheduler of the corresponding resource pool within at least one scheduling cycle, and sending them to the scheduler of the corresponding resource pool; When it is determined that the cluster load state is within the third gradient range, the sending of tasks to be executed to the scheduler of the corresponding resource pool is suspended.

3. The method according to claim 2, characterized in that Any task to be executed in the non-persistent task queue is referred to as a target task. The resource requirements of the target task are determined in the following manner: Acquire the topological structure of the target task, and group the target task into dependency hierarchies according to the topological structure of the target task; Determining the resource requirements of each dependency level of the target task based on the result of grouping the dependency levels; The resource requirement of the target task is determined according to the resource requirements of each dependency level.

4. The method according to claim 1, wherein There is at least one non-persistent task queue configured for each resource pool, and each non-persistent task queue corresponds to a task priority; Among them, when the task to be executed is obtained from the persistent task queue, it is loaded into the corresponding non-persistent task queue according to the task priority of the task to be executed; The method further comprises determining, from the non-persistent task queue of each resource pool according to the cluster load status of each resource pool, tasks to be executed that are sent to the scheduler of the corresponding resource pool within at least one scheduling period: According to the cluster load status of each resource pool, the task priority of the non-persistent task queue of the corresponding resource pool and the priority scheduling policy, the tasks to be executed sent to the scheduler of the corresponding resource pool within at least one scheduling cycle are determined.

5. The method according to claim 1, wherein Also includes: Monitor the waiting time of tasks in the non-persistent task queue of each resource pool and increase the priority of tasks whose waiting time exceeds a specified time threshold.

6. The method according to claim 1, characterized in that The cluster load status includes the resource utilization and Pod number of the server cluster, the resource utilization and Pod number of the corresponding resource pool, and the resource utilization and Pod number of the current task type.

7. The method according to claim 1, characterized in that Also includes: Abnormal task types are monitored based on the real-time indicator monitoring information of the server cluster received, and when abnormal task types are detected, pending tasks of that task type are temporarily issued.

8. The method according to claim 1, characterized in that Also includes: According to the indicator monitoring information of the server cluster received in real time, the task instances that occupy cache resources and are in an idle state are identified and screened out, and the screened out task instances are restarted and the restarted task instances are stored back into the persistent task queue.

9. The method according to claim 1, characterized in that Also includes: Monitor scheduling failure events in the cluster and identify and filter target Pods. When it is determined that the preset conditions are met, the filtered target Pods will be assigned to queues and re-queued according to the failure reasons.

10. The method according to claim 9, characterized in that Also includes: Each scheduler is configured to use a queue priority scheduling strategy for task scheduling. The queue priority scheduling strategy includes starting scheduling tasks in other queues only after all Pods in the active queue have completed scheduling attempts.

11. A computing device, characterized in that include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 10.

12. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images