Container scheduling method, apparatus, device, and storage medium

Abstract

Embodiments of the present application disclose a container scheduling method, device, equipment and storage medium. The method comprises: determining a first working group of a first task in a local task state of a to-be-scheduled state, and creating a first container unit corresponding to the first working group, wherein the working group comprises a plurality of workers, each worker corresponds to a container unit, in a case where all first nodes corresponding to the first container unit are successfully determined from a container cluster, the first container unit is attempted to be allocated to the corresponding first node, after determining that all first container units in the first working group have been successfully allocated to the corresponding first node, the local task state of the first task is set to a running state, and a remote task state of the first task in the container cluster is updated. By using the above technical solution, when the container cluster is used to execute a batch processing task, it can be ensured that different workers in the task are started at the same time, and the container scheduling requirement is met.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer technology, and particularly relate to a container scheduling method, device, equipment and storage medium. BACKGROUND

[0002] In computer science, Gang scheduling is a scheduling algorithm for scheduling different threads or processes to different processors (which can be in the same computing device or different computing devices) to run simultaneously in a parallel system. The typical application scenario of Gang scheduling is artificial intelligence computing tasks and big data computing tasks, which may have a group of workers with the same role (or function), referred to as a workgroup, and may also have multiple workgroups. The workers of the same computing task need to be started together and ended together.

[0003] On the other hand, with the increasing prominence of the slow startup, large space occupation and difficulty in migration of virtual machine technology, container technology has emerged as the times require, which can create services by starting containers. Many container management tools have been widely used, such as Kubernetes (K8S), and a cluster based on a container management tool can be referred to as a container cluster, such as a K8S cluster. Containers can be allocated to nodes in a container cluster. With the rapid development of container management tools, more and more users want to migrate artificial intelligence computing and big data computing frameworks to container clusters. However, container management tools are originally designed for resident services, and their schedulers are designed to sequentially schedule container units (such as Pods in K8S, a Pod can contain one or more containers) as the basic scheduling unit, without considering the relationship between containers. This makes it impossible to meet the Gang scheduling needs of these computing tasks, and also makes it impossible to implement batch processing (such as concurrent creation and concurrent scheduling). SUMMARY

[0004] Embodiments of the present application provide a container scheduling method, device, equipment and storage medium, which can optimize the existing container scheduling scheme, so that different workers in a batch processing task can be started simultaneously to meet the container scheduling needs.

[0005] In a first aspect, embodiments of the present application provide a container scheduling method applied to a scheduler corresponding to a container cluster, wherein the container cluster includes a plurality of nodes, and the method comprises:

[0006] determining a first workgroup to be scheduled in a first task, and creating a first container unit corresponding to the first workgroup, wherein the local task state of the first task is a to-be-scheduled state, the number of the first workgroup is at least one, a plurality of first workers are included in a single first workgroup, and each first worker corresponds to a first container unit;

[0007] In a case where all the first container units respectively corresponding to the first nodes are successfully determined from the container cluster, the first container units are attempted to be allocated to the corresponding first nodes;

[0008] After it is determined that all the first container units in all the first work groups have been successfully allocated to the corresponding first nodes, a local task state of the first task is set to a running state, and a remote task state of the first task in the container cluster is updated.

[0009] In a second aspect, an embodiment of the present application provides a container scheduling device, configured in a corresponding scheduler of a container cluster, the container cluster comprising a plurality of nodes, and the device comprising:

[0010] A container unit creation module is configured to determine a first work group to be scheduled in a first task, and create a first container unit corresponding to the first work group, wherein a local task state of the first task is a to-be-scheduled state, a number of the first work groups is at least one, a plurality of first workers are included in a single first work group, and each first worker corresponds to a first container unit;

[0011] A container unit allocation module is configured to, in a case where all the first container units respectively corresponding to the first nodes are successfully determined from the container cluster, attempt to allocate the first container units to the corresponding first nodes;

[0012] A state setting module is configured to, after it is determined that all the first container units in all the first work groups have been successfully allocated to the corresponding first nodes, set a local task state of the first task to a running state, and set a remote task state of the first task in the container cluster to the running state.

[0013] In a third aspect, an embodiment of the present application provides a container scheduling device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the container scheduling method provided by the embodiment of the present application when executing the computer program.

[0014] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, having a computer program stored thereon, and the program is executable by a processor to implement the container scheduling method provided by the embodiment of the present application.

[0015] The container scheduling scheme provided in the embodiments of the present application determines a first worker group of a first task in a local task state of a to-be-scheduled state, and creates a first container unit corresponding to the first worker group, wherein the worker group includes a plurality of workers, and each worker corresponds to a container unit. In a case where all first container units corresponding to first nodes are successfully determined from the container cluster, the first container units are attempted to be allocated to the corresponding first nodes. After it is determined that all first container units in the first worker group have been successfully allocated to the corresponding first nodes, the local task state of the first task is set to a running state, and the remote task state of the first task in the container cluster is updated. By using the above technical scheme, when the batch processing task is executed by using the container cluster, it can be ensured that different workers in the task are started at the same time, and the container scheduling requirement is met. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A flowchart of a container scheduling method provided by the embodiments of the present application is shown.

[0017] Figure 2 A flowchart of another container scheduling method provided by the embodiments of the present application is shown.

[0018] Figure 3 A structure block diagram of a container scheduling device provided by the embodiments of the present application is shown.

[0019] Figure 4 A structure block diagram of a container scheduling device provided by the embodiments of the present application is shown. DETAILED DESCRIPTION

[0020] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures. In addition, the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0021] Figure 1 A flowchart of a container scheduling method provided by the embodiments of the present application is shown. The method can be executed by a container scheduling device, wherein the device can be realized by software and / or hardware, and can be generally integrated in a container scheduling device. The container scheduling device is configured as a scheduler corresponding to a container cluster. The container cluster can be understood as a cluster realized based on a container management tool, and the container management tool can include K8S, etc., which is not limited in particular.

[0022] The tasks involved in this application embodiment are batch processing computing tasks (hereinafter referred to as batch processing tasks or Jobs) that need to meet Gang scheduling requirements. The specific task type is not limited; for example, it could be an artificial intelligence computing task or a big data computing task. A batch processing task contains at least one group of workers with the same role (or function), denoted as a workgroup (Task). When multiple workgroups are included, the roles corresponding to different workgroups are generally different and can be designed according to specific application scenarios. For example, for an information recommendation training task, multiple workgroups can be included. Each workgroup can implement different functional logics such as distributed parameter storage, model loading and saving, anomaly recovery, execution of data and network graph reading, and data allocation and computation. For batch processing tasks, all workers in all workgroups need to start simultaneously, i.e., Gang scheduling is implemented. However, the original scheduler of container clusters schedules containers sequentially using container units as the basic scheduling unit and does not consider the relationships between containers, thus failing to meet Gang scheduling requirements and failing to achieve batch processing computing. Therefore, this disclosure embodiment provides a novel container scheduling scheme to meet the Gang scheduling requirements of batch processing tasks.

[0023] like Figure 1 As shown, the method includes:

[0024] Step 101: Determine the first workgroup to be scheduled in the first task and create the first container unit corresponding to the first workgroup. The local task status of the first task is the scheduling status. The number of first workgroups is at least one. A single first workgroup contains multiple first workers, and each first worker corresponds to a first container unit.

[0025] In the embodiments of the present disclosure, the same task corresponds to two types of task states, which are respectively denoted as a remote task state and a local task state. The remote task state can be understood as a task state stored in a container cluster, and the container cluster can control each node to perform a task-related operation according to the task state; the local task state can be understood as a task state cached locally in a scheduler, and the scheduler can perform a task-related scheduling operation according to the task state. The remote task state and the local task state are generally defined differently. For example, the remote task state can include a starting state (Starting), a running state (Running), a completed state (Completed), and a failed state (Failed), and the local task state can include an unchecked state (Unchecked), a pending state (Pending), a running state (Running), an abnormal state (Broken), a modified state (Modified), a killed state (Killed), and a finish state (Finish). Optionally, a state machine model including each local task state is pre-set in the scheduler, and the state machine model is used to control the task to switch between different local task states according to a state transformation condition. The state transformation condition can be set according to actual needs, such as the execution condition or the scheduling condition of the task. When the local task state changes, the corresponding remote task state can also change, and at this time, the remote task state in the container cluster can be updated by the scheduler.

[0026] Optionally, the scheduler can be used to sequentially process the tasks in each local task state according to a preset task processing order within a preset scheduling period, and update the local task state and / or the remote task state of the corresponding task according to the processing result. When the task in the pending state needs to be processed, the tasks currently in the pending state can be iterated and processed accordingly.

[0027] For example, the first task can be any task in the local task state of the pending state; the first task can include one or more work groups, denoted as a first work group; each first work group includes a plurality of workers, denoted as a first worker; and each first worker corresponds to a container unit, which is the smallest scheduling unit. For example, in K8S, the container unit is a Pod. After determining the pending first work group in the first task, the first container unit corresponding to each first work group is created in batches.

[0028] Step 102: When all the first container units respectively corresponding to the first nodes are successfully determined from the container cluster, the first container unit is attempted to be allocated to the corresponding first node.

[0029] Exemplarily, all the first container units can be tried to find suitable computing nodes according to a preset scheduling algorithm in a workgroup unit, and the found nodes are recorded as first nodes. If all the first container units are successfully scheduled and calculated to find corresponding first nodes, a next action can be performed, that is, the allocation of the first container units is tried.

[0030] Optionally, if any first container unit fails to successfully determine the corresponding first node, the current round of scheduling can be abandoned.

[0031] After it is determined that all the first container units in all the first workgroups have been successfully allocated to the corresponding first nodes, the local task state of the first task is set to a running state, and the remote task state of the first task in the container cluster is updated.

[0032] Exemplarily, if all the first container units in all the first workgroups have been successfully allocated to the corresponding first nodes, it indicates that all the first workers can start normal work, realize Gang scheduling, and update the local task state and the remote task state of the first task. The local task state of the first task is set to a running state, and the remote task state of the first task can be determined according to the definition manner of the task state in the container cluster, such as a starting state or a running state. If any first worker fails to be allocated, the scheduling decision before the first task can be rolled back, for example, all the first container units are tried to find suitable computing nodes according to a preset scheduling algorithm again.

[0033] The container scheduling method provided in the embodiment of the application determines the first workgroup of the first task with a local task state of a to-be-scheduled state, and creates the first container unit corresponding to the first workgroup. The workgroup includes a plurality of workers, and each worker corresponds to a container unit. After successfully determining that all the first container units correspond to the first nodes in the container cluster, the first container unit is tried to be allocated to the corresponding first node. After it is determined that all the first container units in the first workgroup have been successfully allocated to the corresponding first nodes, the local task state of the first task is set to a running state, and the remote task state of the first task in the container cluster is updated. By using the above technical solution, when the container cluster is used to execute a batch processing task, it can be ensured that different workers in the task are started at the same time, and the container scheduling requirement is met.

[0034] In some embodiments, the method can further include: obtaining task information of the created tasks from the container cluster, wherein the task information includes at least one of a remote task state of a task, node information corresponding to the task, container unit information corresponding to the task, and change information of the task; and updating or correcting a local task state of each task according to the task information. This arrangement has the advantage that updating or correcting the local task state according to the task information obtained from the container cluster can ensure the accuracy of the local task state.

[0035] For example, the scheduler can start real-time monitoring of changes (such as addition, deletion, and modification, etc.) of tasks, nodes, and container units in the container cluster after startup or restart, and then obtain the task information. Optionally, the changes can be monitored from the interface service by using a notifier mechanism in the container management tool, change events can be set for different changes, and corresponding callback functions can be registered, the callback functions can be used to update the local cache and correct the local task state. For example, when a deletion event of a container unit in the container cluster is triggered, the local task state of a task to which the container unit belongs can be updated to an abnormal state. In addition to the local task state, the local cache can also include the above-mentioned task information. The node information corresponding to the task can include which tasks are included in the node, the working state of the node, and the resource usage of the node, etc. The container unit information corresponding to the task can include the working state of the container unit, the resource usage of the container unit, and the scheduling history information of the container unit, etc. The change information of the task can include whether the task is modified, the change of the node corresponding to the task, and the change of the container unit corresponding to the task, etc.

[0036] In some embodiments, the determining the first workgroup to be scheduled in the first task comprises: determining a first preset scheduling specification workgroup corresponding to the first task, and performing pre-scheduling processing on the first preset scheduling specification workgroup; if the pre-scheduling fails, determining a second preset scheduling specification workgroup corresponding to the first task, wherein the second preset scheduling specification workgroup occupies less resources than the first preset scheduling specification workgroup; performing pre-scheduling processing on the second preset scheduling specification workgroup, and if the pre-scheduling succeeds, determining the second preset scheduling specification workgroup as the first workgroup to be scheduled in the first task. This setting has the advantage of allowing the configuration of more than two computing specifications for a task, better adapting to the actual situation of computing resources, for example, using more resources to complete the computing task faster when the computing resources are sufficient, and using fewer resources to improve the possibility of task scheduling when the computing resources are scarce. By supporting different resource specification applications for tasks, not only can the computing efficiency be improved, but also the computing resources during the business peak period can be basically met. In addition, by using the pre-scheduling method, it can be found earlier that the Gang scheduling cannot be realized, and unnecessary operations can be reduced.

[0037] For example, pre-scheduling can be understood as performing scheduling calculation without performing actual scheduling operation, that is, no container unit allocation is performed in the pre-scheduling process. When performing scheduling calculation, the current resource state of the container cluster and the current resource application of the task can be calculated. Pre-scheduling success can be understood as determining by pre-scheduling calculation that the current remaining resources of the container cluster can meet the scheduling requirements of the workgroup of the corresponding scheduling specification, and pre-scheduling failure can be understood as determining by pre-scheduling calculation that the current remaining resources of the container cluster cannot meet the scheduling requirements of the workgroup of the corresponding scheduling specification. The preset scheduling specification can be quantified according to the resource occupation, and the first preset scheduling specification and the second preset scheduling specification can be obtained from the task definition corresponding to the first task. Optionally, if the pre-scheduling processing on the second preset scheduling specification workgroup fails, and there is no preset scheduling specification that occupies less resources, the current scheduling is abandoned.

[0038] In some embodiments, the first node corresponding to the current first container unit is determined by: obtaining resource application information of the current first container unit, wherein the resource application information comprises a resource application type and a resource application amount; and determining the first node corresponding to the current first container unit according to a similarity between the resource application information and remaining resource information of each candidate node in the container cluster. This arrangement has the advantage that it enables more rational allocation of resources in the container cluster. The resource application type may, for example, comprise a central processing unit (CPU), a graphics processing unit (GPU) and memory.

[0039] In related technologies, when determining a node to be allocated for a container unit, only whether the resource remaining condition of the node can meet the resource application requirement of the container unit is considered, which can easily lead to a situation where some nodes have a large amount of a certain resource allocated but a large amount of other resources idle, and thus there is a lot of resource fragmentation and the node resource utilization rate is low. In the embodiments of the present application, when determining a node to be allocated for a container unit, the container unit can be allocated to a node with high similarity between the remaining resource information and the resource application information, thereby improving the node resource utilization rate. The similarity can be compared in dimensions such as resource type and resource quantity, and the specific comparison manner is not limited.

[0040] In some embodiments, the determining the first node corresponding to the current first container unit according to the similarity between the resource application information and the residual resource information of each candidate node in the container cluster comprises: obtaining the cluster resource remaining amount corresponding to each resource application type in the container cluster, and obtaining the node resource remaining amount corresponding to each resource application type in each candidate node; determining a first vector according to the resource application amount and the cluster resource remaining amount, wherein each first element in the first vector corresponds to a different resource application type in turn according to a preset order, and the value of the first element is determined according to the ratio of the resource application amount to the cluster resource remaining amount; determining a second vector corresponding to each candidate node according to the node resource remaining amount and the cluster resource remaining amount, wherein each second element in the second vector corresponds to a different resource application type in turn according to the preset order, and the value of the second element is determined according to the ratio of the node resource remaining amount to the cluster resource remaining amount; for each candidate node, the inner product of the first vector and the corresponding second vector is calculated to obtain a node score corresponding to the candidate node; and the candidate node with the highest node score is determined as the first node corresponding to the current first container unit. The advantage of this setting is that the higher the node score, the higher the similarity, and the resource application information and the residual resource information of the node can be quickly and accurately determined according to the node score, thereby improving the container scheduling efficiency and reducing the cost loss caused by resource fragmentation of the computing node. The candidate node can include a node in the container cluster that can meet the resource application information.

[0041] For example, assuming that the current first container unit applies for three resources, the preset order is CPU, GPU, and memory, the overall resource remaining (cluster resource remaining amount) of the current container cluster is A, B, and C, that is, the CPU remaining A, the GPU remaining B, and the memory remaining C, the resource remaining (node resource remaining amount) of a candidate node is a1, b1, and c1, and the resource application amount of the current first container unit is a2, b2, and c2. The first vector can be (a3, b3, c3) = (a1 / A, b1 / B, c1 / C), and the second vector can be (a4, b4, c4) = (a2 / A, b2 / B, c2 / C). The inner product of the first vector and the second vector is calculated to obtain a node score a3a4+b3b4+c3c4.

[0042] Optionally, when determining the first vector and the second vector, normalization processing can also be performed. For example, (a3, b3, c3) can obtain (a5, b5, c5) after normalization processing, which is denoted as the first vector, (a4, b4, c4) can obtain (a6, b6, c6) after normalization processing, which is denoted as the second vector, and the inner product of the first vector and the second vector is calculated to obtain a node score a5a6+b5b6+c5c6.

[0043] In some embodiments, when performing the node score calculation, more vectors can also be determined according to other indicators, and the node score is calculated according to the inner product of the more vectors, so as to make the node score more accurate. For example, a third vector can be determined according to the resource utilization of the candidate node. Specifically, for each candidate node, the inner product of the first vector and the corresponding second vector is calculated to obtain the node score of the corresponding candidate node, including: for each candidate node, the inner product of the first vector, the corresponding second vector and the corresponding third vector is calculated to obtain the node score of the corresponding candidate node; wherein each third element in the third vector corresponds to a different resource application type according to the preset order, and the value of the third element is determined according to the resource utilization of the corresponding candidate node. The advantage of such setting is that the three-phase inner product algorithm is used to calculate the node score, so that the resource utilization of the candidate node can participate in the calculation of the node score, and the node score can be more accurately calculated.

[0044] For example, continuing the above example, assuming that the resource utilization of the candidate node is (a0, b0, c0), the node score can be represented as a0a5a6+b0b5b6+c0c5c6.

[0045] In some embodiments, the number of the first work groups is multiple, the functions corresponding to different first work groups are different, and the life cycle control strategies corresponding to at least two first work groups are different, wherein the life cycle control strategy includes whether to allow failure retry during running. The advantage of such setting is that different life cycle control strategies can be configured for work groups with different functions, so that the work of the work group is more flexible, and the computing efficiency is improved. For example, some work groups may fail to calculate due to container unit being evicted, node exception or network exception, etc. during work, if the function of the work group is auxiliary computing function, it will not affect the success or failure of the whole task, so failure retry can be allowed, if the work group is a work group for implementing a resident type function, such as a parameter service work group, failure retry can not be allowed.

[0046] With the expansion of the computing power of the cluster, the number of times of hardware failure, network failure or downtime of the computing device also increases. When these situations occur, the entire task can be restarted as much as possible to ensure the normal progress of the task, and the container unit on the abnormal node can be scheduled to another node. In the embodiments of the present application, for the task in the running state, the local restart and off-site rescheduling of the container unit are counted, and the task state is updated in time. In some embodiments, the method further comprises: performing state counting on a second container unit in a second working group in a second task, wherein the local task state of the second task is in a running state; adding the second container unit with a state exception and the second container unit with a local restart number greater than a preset restart number threshold to a preset failure queue; in a working group unit, if the sum of the container unit rescheduling number corresponding to any second working group and the length of the preset failure queue is greater than a preset threshold, the remote task state of the second task is set to a failure state. The advantage of such setting is that for the task in the running state, the container unit restart and rescheduling of the whole task in the task are quantified in a working group unit, and when the problem is serious, the remote task state is set to a failure state in time, thereby avoiding wasting too much time and resources.

[0047] In some embodiments, the method further comprises: determining a target third working group to be processed in a third task, wherein the local task state of the third task is in an abnormal state, the target third working group contains an abnormal worker and the corresponding life cycle control strategy includes failure retry during running; performing container unit reconstruction and rescheduling processing on the abnormal worker in the target third working group, and updating the local task state and the remote task state of the third task according to the processing result. The advantage of such setting is that for the task in the abnormal state, the working group containing the abnormal worker and allowing failure retry is found in time, and the container unit reconstruction and rescheduling are performed on the abnormal worker, so that the task has the opportunity to recover normally, off-site disaster recovery is achieved, the entire task is restarted, and the task execution efficiency is improved.

[0048] For example, the abnormal worker can be understood as a worker whose corresponding container unit local restart number is greater than a preset restart number threshold. Further, for the abnormal worker, because the corresponding container unit local restart number on the same node is too large, it indicates that the node may have problems such as mounting exception, network exception or component exception, which causes the container unit to be unable to start normally. Therefore, the node can be added to a preset blacklist to prevent other container units from being allocated to the possibly abnormal node, thereby causing frequent rescheduling.

[0049] In some embodiments, the method can further include: performing a check, admission, or garbage collection, or the like on the task, without limitation to the specific timing and details of the operations.

[0050] Figure 2 A flowchart of another container scheduling method is provided in the embodiments of the present application, which is optimized based on the above-mentioned optional embodiments. For example, the method can include:

[0051] In step 201, when a new task is detected, the new task is checked and admitted.

[0052] For example, the scheduler can obtain relevant information in the K8S cluster from the interface service (APIServer) through the informer mechanism provided by K8S. When there is a first created task, or a task that is created again after the scheduler is restarted, or a task that is edited and modified, it can be considered that a new task is detected. In the embodiments of the present application, the check and admission of the task can be implemented internally in the scheduler, so as to avoid introducing single point of failure risk. For example, the check can include field checking (such as task name, resource application, security context, and life cycle control field, etc.) and information completion, or the like, of the task according to the preset specification, and the admission can include deciding to accept or reject the task according to the check result. For example, when the resource application or the number of task replicas of the new task exceeds the corresponding maximum limit, the task can be rejected.

[0053] In step 202, task information of the created tasks is obtained from the container cluster, and the local task state of each task is updated or corrected according to the task information.

[0054] For example, when the scheduler is restarted, the task information of the created tasks can be obtained from the container cluster, and according to the current working state and other information of each working group in the task, it can be determined that the task should be in which local task state. For example, if it is determined that each working group meets the task definition and is in a normal working state, the local task state can be corrected to a running state.

[0055] In step 203, a task in the checking state is processed, and the local task state and / or the remote task state of the corresponding task are updated according to the processing result.

[0056] For example, the scheduler can be configured to process tasks in each local task state in a preset task processing order in a preset scheduling period, and update the local task state and / or the remote task state of the corresponding task according to the processing result. The local task state includes Unchecked, Pending, Running, Broken, Modified, Killed, and Finish. The preset task processing order can be Unchecked, Pending, Running, Broken, Modified, and Killed. A state machine model including each local task state is preconfigured in the scheduler, and the state transformation condition in the state machine model is used to control the switching of the task between different local task states.

[0057] The task in the Unchecked state can be a newly submitted task, or a task that is re-listened to by the scheduler after the scheduler is restarted, a task received in a pod creation event when the workgroup is not expanded (i.e., not created), etc. There is no scheduled pod in the Pending state, and the workgroup can not be expanded, and the pod can not be created. In the Running state, all pod instances in the local cache exist. In the Broken state, there are missing or unscheduled pods in the local cache. In the Modified state, all pods are to be cleaned up due to task redefinition. The Killed state indicates that the task is actively deleted by the outside. The Finish state indicates that the task is completed, and all pods are to be cleaned up. In this state, the task is successfully observed or ended by the outside, and the scheduler does not process it separately.

[0058] Optionally, garbage collection can be performed before processing tasks in different local task states. For example, orphan container units (container units that are not normally cleaned up) introduced due to node downtime, node Kubelet component suspension, or node network failure, etc. can be subjected to garbage collection. For example, an orphan pod is listened to by using the above listening mechanism, and is put into a delayed deletion queue. The expired pod is cleaned up.

[0059] For example, for a task in the unchecked state, it can be determined whether the local cache task object is empty (e.g., set to empty when the task is deleted). If it is empty, garbage collection is performed on the related resources such as the task, the workgroup, the worker, and the configuration file. If it is not empty, the legality of the task can be checked. If the check fails, the remote task state is updated to the failed state, and garbage collection is performed on the corresponding resources. If the check passes, it can be determined whether the workgroup of the task in the local cache exists. If it does not exist, the task may exist or the scheduler may be restarted. In this case, the actual state of the task in the cluster is adjusted (corrected). If it exists, the current scheduling specification of the workgroup is obtained according to the task definition and the scheduling specification downgrade record, the information of each workgroup in the local cache and the workgroup information in the workgroup are counted and supplemented, and the local task state and / or the remote task state are updated or adjusted according to the counting and supplementing results.

[0060] Step 204, for a first task in the local task state of the to-be-scheduled state, a workgroup corresponding to a first preset scheduling specification is determined and pre-scheduling processing is performed. If the pre-scheduling fails, a workgroup corresponding to a second preset scheduling specification after downgrade is determined and pre-scheduling processing is performed. If the pre-scheduling succeeds, the workgroup of the second preset scheduling specification is determined as the first workgroup to be scheduled in the first task.

[0061] For example, for a task in the local task state of the to-be-scheduled state, it can be determined whether the workgroup information in the local cache is empty. If it is empty, it is initialized, two-specification workgroups are obtained according to the task definition, that is, the first preset scheduling specification and the second preset scheduling specification, and then pre-scheduling is performed using the specification that occupies more resources to ensure work efficiency. When the pre-scheduling fails, downgrade is performed, and pre-scheduling is performed using the specification that occupies less resources. After the pre-scheduling succeeds, the to-be-scheduled workgroup is obtained, and information can be completed, such as creating or updating the Hosts file (system file) of each Pod of the task corresponding to the Configmap (configuration map object) and the Service (service) corresponding to the task. If the to-be-scheduled workgroup is a degraded workgroup, the resource downgrade flag in the remote task state field in the container cluster can be updated to true. If the workgroup information in the local cache is not empty, the information can be completed first, and then step 205 is executed.

[0062] Step 205, a first container unit corresponding to the first workgroup is created, and in a case where all first container units respectively correspond to first nodes are successfully determined from the container cluster, the first container unit is attempted to be allocated to the corresponding first node.

[0063] In the determination of the first node corresponding to each first container unit, the three-phase inner product algorithm described above can be used to calculate the node score, and the appropriate node is selected for the container unit according to the node score. Details are not repeated here.

[0064] Step 206, after determining that all first container units in all first working groups have been successfully assigned to the corresponding first node, the local task state of the first task is set to the running state, and the remote task state of the first task in the container cluster is updated.

[0065] Step 207, the state of the second container unit in the second working group in the second task with the local task state in the running state is counted, and the local task state and / or remote task state of the corresponding task is updated according to the processing result.

[0066] For example, for the task with the local task state in the Running state, all working groups under the task can be traversed to count the number of normally running and state abnormal (problem exists, needs to be deleted) container units. If the total number of currently existing container units is inconsistent with the number of replicas in the task definition, the remote task state can be set to the failure state. For the second container unit with a state abnormal, it is added to the preset failure queue; if there is a second container unit with a local restart number greater than a preset restart number threshold, the second container unit is also added to the preset failure queue. In units of working groups, if the sum of the container unit rescheduling number corresponding to any second working group and the length of the preset failure queue is greater than a preset threshold, the remote task state of the second task is set to the failure state. For example, when the node in the container cluster appears abnormal, the container cluster can add a deletion flag to the container unit on the node, and the scheduler can add the container unit in the preset failure queue carrying the deletion flag to the to-be-deleted queue. For example, if all second working groups voluntarily exit, and the total number of successfully calculated container units is equal to the number of replicas, the remote task state of the second task can be set to the completion state, and the local task state can be updated to the completion state. If the sum of the number of normally running container units and the number of successfully calculated container units is equal to the number of replicas, the local task state remains in the running state. If none of the above conditions are met, the remote task state can be set to the starting state.

[0067] For example, the to-be-deleted queue is traversed, and the container units in it are deleted, which can keep the local task state unchanged, that is, it does not actively switch to the abnormal state, and waits for the deletion event of the container unit in the container cluster to be triggered, and then updates the local task state of the task to which the container unit belongs to the abnormal state. For tasks other than tasks with the remote task state in the starting state and the running state, it is a successfully calculated task or a failed calculation task, after updating the task state, the working groups and workers of the task are cleaned up.

[0068] Step 208, for the third task with the local task state being the abnormal state, determining a target third workgroup to be processed, performing container unit reconstruction and rescheduling processing for abnormal workers in the target third workgroup, and updating the local task state and / or the remote task state of the third task according to the processing result.

[0069] For example, for a task with a local task state being a Broken state, the container units in each workgroup are checked to determine whether they are absent or unscheduled. If so, the rescheduling frequency and the local restart frequency are counted according to the scheduling history record. If the rescheduling frequency is greater than a preset frequency threshold, the remote task state is updated to a failure state, the local task state is updated to an end state, and the workgroup and workers of the task are cleaned up. Otherwise, the container units are placed in a to-be-processed workgroup queue. The to-be-processed workgroup queue (a queue formed by the target third workgroup) is traversed, and container unit reconstruction is performed for abnormal workers therein. If the reconstruction fails, the task or the container cluster may have a problem, and the current scheduling can be skipped, and a retry can be performed after a preset waiting duration. If the reconstruction succeeds, the reconstructed container units are rescheduled, and the local task state and the remote task state of the third task are updated according to the rescheduling result.

[0070] Step 209, sequentially processing tasks with a local task state being a modified state and a killed state, and updating the local task state and / or the remote task state of the corresponding task according to the processing result.

[0071] For example, for a task with a local task state being a Modified state, the task can be verified. If the verification fails, the remote task state is updated to a failure state. If the verification succeeds, the local task state can be updated to a to-be-scheduled state. The workgroup and workers corresponding to the task before modification are cleaned up.

[0072] For example, for a task with a local task state being a Killed state, the workgroup and workers corresponding to the task can be cleaned up, the service and configuration graph corresponding to the task are cleaned up, and finally the task object in the local cache is removed.

[0073] For example, when all tasks have been processed according to the preset task processing order, in the next scheduling period, tasks in each local task state can be sequentially processed according to the preset task processing order, and the local task state and / or the remote task state of the corresponding task is updated according to the processing result.

[0074] The container scheduling method provided by the embodiments of the present application can realize an integrated Gang scheduling scheme of task verification and admission control, task control and task scheduling functions based on a state machine inside a scheduler, reduce the instability introduced by single point failure of the system, improve the overall maintainability of the system, improve the availability of the system when a node fails and the utilization rate of system resources through disaster recovery scheduling, and the tasks do not have to wait or fail as a whole. By supporting different resource specification applications of tasks, the resources in the container cluster are flexibly utilized, the resource utilization during a business peak period is realized, the computing efficiency is improved, and the business computing resources during the business peak period are basically met. When performing container scheduling, a three-phase inner product scoring algorithm is used to match appropriate nodes for each container unit, which greatly reduces the cost loss caused by resource fragmentation of the computing nodes.

[0075] Figure 3 A structural block diagram of a container scheduling device provided by the embodiments of the present application is shown in FIG. 3. The device is configured in a corresponding scheduler of a container cluster and can be realized by software and / or hardware and generally integrated in a container scheduling device. The container scheduling device can perform container scheduling by executing the container scheduling method. Figure 3

[0076] The container unit creation module 301 is configured to determine a first work group to be scheduled in a first task and create a first container unit corresponding to the first work group. The local task state of the first task is a to-be-scheduled state, the number of the first work groups is at least one, each first work group contains a plurality of first workers, and each first worker corresponds to a first container unit.

[0077] The container unit allocation module 302 is configured to, in a case where all first container units corresponding to the first container units are successfully determined from the container cluster, attempt to allocate the first container units to corresponding first nodes.

[0078] The state setting module 303 is configured to, after determining that all first container units in all first work groups have been successfully allocated to corresponding first nodes, set the local task state of the first task to a running state and update the remote task state of the first task in the container cluster.

[0079] ​The container scheduling apparatus provided in the embodiments of the present application determines a first worker group of a first task in a local task state of a to-be-scheduled state, and creates a first container unit corresponding to the first worker group, wherein the worker group comprises a plurality of workers, each worker corresponds to a container unit, in a case where all first nodes corresponding to the first container unit are successfully determined from a container cluster, the first container unit is attempted to be allocated to the corresponding first node, after it is determined that all first container units in the first worker group have been successfully allocated to the corresponding first node, the local task state of the first task is set to a running state, and a remote task state of the first task in the container cluster is updated. By using the above technical solution, when a batch processing task is executed by using a container cluster, it can be ensured that different workers in the task are started at the same time, and the container scheduling requirement is met.

[0080] The container scheduling apparatus provided in the embodiments of the present application can be integrated into the container scheduling device. Figure 4 A structure block diagram of a container scheduling device provided in the embodiments of the present application is provided. The container scheduling device 400 comprises a memory 401, a processor 402, and a computer program stored in the memory 401 and executable on the processor 402, and the processor 402 implements the container scheduling method provided in the embodiments of the present application when executing the computer program.

[0081] The embodiments of the present application also provide a storage medium comprising computer executable instructions for executing the container scheduling method provided in the embodiments of the present application when executed by a computer processor.

[0082] The container scheduling apparatus, device and storage medium provided in the above embodiments can execute the container scheduling method provided in any of the embodiments of the present application, and have the corresponding function modules and beneficial effects of executing the method. Technical details not described in the above embodiments can be referred to the container scheduling method provided in any of the embodiments of the present application.

Claims

1. A container scheduling method, characterized in that, The method, applied to a scheduler corresponding to a container cluster containing multiple nodes, includes: The first workgroup to be scheduled in the first task is determined, and the first container unit corresponding to the first workgroup is created. The local task state of the first task is the state to be scheduled. The local task state is the task state cached locally in the scheduler. The number of the first workgroup is at least one. A single first workgroup contains multiple first workers, and each first worker corresponds to a first container unit. If the first node corresponding to each of the first container units is successfully determined from the container cluster, an attempt is made to assign the first container unit to the corresponding first node; After confirming that all first container units in all first workgroups have been successfully assigned to their corresponding first nodes, the local task status of the first task is set to running status, and the remote task status of the first task in the container cluster is updated to set the remote task status of the first task to start status or running status. The remote task status is the task status stored in the container cluster, and the remote task status includes at least one of start status, running status, completed status, and failed status.

2. The method according to claim 1, characterized in that, The process of determining the first working group to be scheduled in the first task includes: Determine the working group of the first preset scheduling specification corresponding to the first task, and perform pre-scheduling processing for the working group of the first preset scheduling specification; If pre-scheduling fails, a second preset scheduling specification working group corresponding to the first task is determined, wherein the resources occupied by the second preset scheduling specification working group are less than the resources occupied by the first preset scheduling specification working group. Pre-schedule processing is performed on the workgroup with the second preset scheduling specification. If the pre-schedule is successful, the workgroup with the second preset scheduling specification is determined as the first workgroup to be scheduled in the first task.

3. The method according to claim 1, characterized in that, The first node corresponding to the current first container unit is determined in the following way: Obtain the resource request information of the current first container unit, wherein the resource request information includes the resource request type and the resource request quantity; The first node corresponding to the current first container unit is determined based on the similarity between the resource request information and the remaining resource information of each candidate node in the container cluster.

4. The method according to claim 3, characterized in that, The step of determining the first node corresponding to the current first container unit based on the similarity between the resource request information and the remaining resource information of each candidate node in the container cluster includes: Obtain the remaining amount of cluster resources in the container cluster corresponding to each resource request type, and obtain the remaining amount of node resources in each candidate node corresponding to each resource request type; A first vector is determined based on the resource request amount and the remaining cluster resources, wherein each first element in the first vector corresponds to a different resource request type in a preset order, and the value of the first element is determined based on the ratio of the resource request amount to the remaining cluster resources. The second vector corresponding to each candidate node is determined based on the remaining node resources and the remaining cluster resources, wherein each second element in the second vector corresponds to a different resource request type in a predetermined order, and the value of the second element is determined based on the ratio of the remaining node resources to the remaining cluster resources. For each candidate node, the inner product of the first vector and the corresponding second vector is calculated to obtain the node score of the corresponding candidate node. The candidate node with the highest node score is determined as the first node corresponding to the current first container unit.

5. The method according to claim 4, characterized in that, The step of calculating the inner product of the first vector and the corresponding second vector for each candidate node to obtain the node score for that candidate node includes: For each candidate node, the inner product of the first vector, the corresponding second vector, and the corresponding third vector is calculated to obtain the node score of the corresponding candidate node. In this context, each third element in the third vector corresponds to a different resource application type according to the preset order, and the value of the third element is determined based on the resource utilization rate of the corresponding candidate node.

6. The method according to claim 1, characterized in that, Also includes: Perform status statistics on the second container unit in the second work group of the second task, wherein the local task status of the second task is running. Add the second container unit with an abnormal status, and the second container unit with a local restart count greater than the preset restart count threshold, to the preset failure queue; If, on a workgroup basis, the sum of the number of reschedulings of the container unit corresponding to any second workgroup and the length of the preset failure queue is greater than a preset threshold, then the remote task status of the second task is set to a failure status.

7. The method according to claim 1, characterized in that, Also includes: Identify the target third workgroup to be processed in the third task, wherein the local task status of the third task is an abnormal state, the target third workgroup contains abnormal workers and the corresponding lifecycle control policy includes allowing retry during operation. For the abnormal workers in the target third working group, container unit reconstruction and rescheduling are performed, and the local task status and / or remote task status of the third task are updated according to the processing results.

8. A container scheduling device, characterized in that, A scheduler configured for a container cluster containing multiple nodes, the device comprising: The container unit creation module is used to determine the first workgroup to be scheduled in the first task and create the first container unit corresponding to the first workgroup. The local task status of the first task is the scheduling status, which is the task status cached locally in the scheduler. The number of the first workgroup is at least one, and a single first workgroup contains multiple first workers, with each first worker corresponding to a first container unit. The container unit allocation module is used to attempt to allocate the first container unit to the corresponding first node when the first node corresponding to each first container unit is successfully determined from the container cluster. The status setting module is used to set the local task status of the first task to running status after determining that all first container units in all first workgroups have been successfully allocated to the corresponding first nodes, and to update the remote task status of the first task in the container cluster to set the remote task status of the first task to start status or running status. The remote task status is the task status stored in the container cluster, and the remote task status includes at least one of start status, running status, completed status, and failed status.

9. A container scheduling device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images