Task processing method, and apparatus

Abstract

The present application provides a task processing method and an apparatus. The method comprises: identifying at least one idle processing unit in a processing unit cluster; processing historical information of a first task on the basis of the processing unit cluster, and predicting a future idle duration of the at least one idle processing unit, wherein the future idle duration is a duration during which the at least one idle processing unit is not required to process the first task; and when the future idle duration is greater than or equal to a required duration of a second task, instructing the at least one idle processing unit to process the second task, wherein the priority of the second task is lower than the priority of the first task, and the required duration of the second task is a duration required for the at least one idle processing unit to process the second task. The method can improve the resource utilization rate of a cluster.

Description

Task processing method and device

[0001] The present application claims priority to a Chinese patent application No. 202410634507.6, filed on May 21, 2024, entitled “A resource utilization method and device”, and a Chinese patent application No. 202410799780.4, filed on June 19, 2024, entitled “A task processing method and device”, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of computer, and in particular to a task processing method and device. BACKGROUND

[0003] With the development of artificial intelligence (AI), the intelligence level of AI models is getting higher and higher, which can be used to save manpower and improve business processing efficiency. Therefore, more and more industries begin to apply AI models to related businesses.

[0004] The application of AI models involves model inference and model training. Among them, model inference is the process of using AI models to process businesses, which directly faces users and requires a high service level agreement (SLA). In order to guarantee the SLA of model inference, more resources are configured for the cluster used to execute model inference. Generally, a business has a peak period and a trough period. In the peak period of the business, the task quantity of model inference is large, and the resources in the cluster can be fully utilized. In the trough period of the business, the task quantity of model inference is small, and the resource utilization rate of the cluster is low, resulting in resource waste. SUMMARY

[0005] The present application provides a task processing method and device, which can improve the resource utilization rate of the cluster.

[0006] In a first aspect, a task processing method is provided. The method is applied to a control device in a computing system, the computing system further comprising a processing unit cluster, wherein processing units in the processing unit cluster are capable of processing different tasks at different time instants under the indication of the control device; the method comprises: identifying at least one idle processing unit in the processing unit cluster; predicting a future idle duration of the at least one idle processing unit based on historical information of the processing unit cluster processing a first task, the future idle duration being a duration during which the at least one idle processing unit is not required to process the first task; in a case where the future idle duration is greater than or equal to a required duration of a second task, instructing the at least one idle processing unit to process the second task; wherein the second task has a lower priority than the first task, and the required duration of the second task is a duration required for the at least one idle processing unit to process the second task.

[0007] The first task is a high-priority task, and the second task is a low-priority task. For example, the first task can be a model inference task, and the second task can be a model training task.

[0008] The idle processing unit is a current idle processing unit in the processing unit cluster, and is an idle processing unit of the processing unit cluster in addition to meeting the processing requirement of the high-priority task. In other words, the idle processing unit is an idle processing unit of the processing unit cluster in a case where there is no high-priority task to be processed or the processing unit cluster is in a waiting state.

[0009] The method can identify an idle processing unit of the cluster in addition to meeting the processing requirement of the high-priority task, and predict a duration during which the idle processing unit is not required to process the high-priority task in the future. Then, in a case where the duration is greater than or equal to a required duration of processing the low-priority task, the idle processing unit is instructed to process the low-priority task, so that the processing of the low-priority task can be completed before the idle processing unit is required to process the high-priority task, and thus the high-priority task can be processed in time when the high-priority task arrives. In this way, the resource utilization is improved, and the future arrival of the high-priority task can be processed in time.

[0010] In a possible implementation, the method further comprises: identifying an idle processing capability of the at least one idle processing unit; and in a case where the future idle duration is greater than or equal to the required duration of the second task, instructing the at least one idle processing unit to process the second task comprises: in a case where the future idle duration is greater than or equal to the required duration of the second task, and the idle processing capability is greater than or equal to a required processing capability of the second task, instructing the at least one idle processing unit to process the second task, wherein the required processing capability of the second task is a capability required for processing the second task.

[0011] The idle processing unit is a processing unit in an idle state, and is also the maximum available processing capacity of the processing unit. When the required processing capacity of the second task is less than or equal to the idle processing capacity of the idle processing unit, the idle processing unit is instructed to process the second task, so that the processing of the second task is completed before the end of the future idle duration.

[0012] In a possible implementation, the second task is a task with the maximum required processing capacity among the multiple tasks, wherein the priority of each task in the multiple tasks is less than the priority of the first task, and the future idle duration is greater than or equal to the required duration of each task in the multiple tasks.

[0013] If there are multiple low-priority tasks waiting to be processed by the processing unit cluster, and the required duration of each task is less than or equal to the future idle duration of the idle processing unit, and the required processing capacity of each task is less than or equal to the processing capacity of the idle processing unit, the idle processing unit is instructed to process the task with the maximum required processing capacity among the multiple tasks. In this way, the computing resources of the idle processing unit can be fully utilized, and the resource utilization rate is improved.

[0014] In a possible implementation, the second task is a task with the longest required processing duration among the multiple tasks; wherein the priority of each task in the multiple tasks is less than the priority of the first task, and the future idle duration is greater than or equal to the required duration of each task in the multiple tasks.

[0015] If there are multiple low-priority tasks waiting to be processed by the processing unit cluster, and the required duration of each task in the multiple tasks is less than or equal to the future idle duration of the idle processing unit, the idle processing unit is instructed to process the task with the longest required processing duration among the multiple tasks. In this way, the computing resources of the idle processing unit can be fully utilized, and the resource utilization rate is improved.

[0016] In a possible implementation, the second task is a task with the highest priority among the multiple tasks; wherein the priority of each task in the multiple tasks is less than the priority of the first task, and the future idle duration is greater than or equal to the required duration of each task in the multiple tasks.

[0017] If there are multiple low-priority tasks waiting to be processed by the processing unit cluster, and the required duration of each task in the multiple tasks is less than or equal to the future idle duration of the idle processing unit, and the multiple tasks include tasks with different priorities, the idle processing unit is instructed to process the task with the highest priority among the multiple tasks. In this way, the task with the highest priority among the multiple tasks can be processed preferentially.

[0018] In a possible implementation, the method further includes: in a case where there is a third task to be processed and the processing unit cluster does not have enough idle processing units to process the third task, expelling a fourth task currently processed by the processing unit cluster from the processing unit cluster to make the processing unit cluster have enough idle processing units to process the third task, wherein the priority of the fourth task is lower than the priority of the third task.

[0019] In this implementation, when a high-priority task arrives, if the processing unit cluster does not have enough idle processing units to process the high-priority task, a low-priority task currently being processed can be expelled to vacate a processing unit in time, so that the high-priority task is processed in time.

[0020] In a possible implementation, the fourth task is a task with the lowest priority among tasks currently processed by the processing unit cluster.

[0021] In this implementation, the task with the lowest priority is preferentially expelled, so that the resources in the processing unit cluster are reasonably used.

[0022] In a possible implementation, the fourth task is a task with the longest remaining processing duration among tasks currently processed by the processing unit cluster.

[0023] In this implementation, the task with the longest remaining processing duration is preferentially expelled, so that a task that is about to be completed is not expelled, and the cost of task expulsion is reduced.

[0024] In a possible implementation, the fourth task is a task with the shortest processed duration among tasks currently processed by the processing unit cluster.

[0025] In this implementation, the task with the shortest processed duration is preferentially expelled, so that a task that is about to be completed is not expelled, and the cost of task expulsion is reduced.

[0026] In a possible implementation, the fourth task is a task with the least number of interruptions among tasks currently processed by the processing unit cluster.

[0027] In this implementation, the task with the least number of interruptions is preferentially expelled, so that the task is not interrupted for multiple times, and user experience is thus avoided from being affected.

[0028] In a possible implementation, the fourth task is a task that occupies a processing unit with the best performance among tasks currently processed by the processing unit cluster.

[0029] In this implementation, the processing unit with the best performance can be vacated, so that the processing unit can be used to process a high-priority task, and the processing effect and efficiency of the high-priority task are thus guaranteed.

[0030] In a possible implementation, the processing units in the processing unit cluster access different networks when processing different tasks.

[0031] In this way, network isolation between different tasks is achieved, and the security of task processing is ensured.

[0032] In a second aspect, a control device is provided. The computing system in which the device is located also includes a processing unit cluster. Under the indication of the device, the processing units in the processing unit cluster can process different tasks at different times. The device includes: an identification module configured to identify at least one idle processing unit in the processing unit cluster; a prediction module configured to predict a future idle duration of the at least one idle processing unit based on historical information of the processing unit cluster processing a first task, the future idle duration being a duration during which the at least one idle processing unit does not need to process the first task; and an indication module configured to instruct the at least one idle processing unit to process a second task if the future idle duration is greater than or equal to a required duration of the second task. The priority of the second task is lower than the priority of the first task, and the required duration of the second task is a duration required for the at least one idle processing unit to process the second task.

[0033] In a possible implementation, the identification module is further configured to identify an idle processing capability of the at least one idle processing unit, and the indication module is configured to instruct the at least one idle processing unit to process the second task if the future idle duration is greater than or equal to the required duration of the second task, and the idle processing capability is greater than or equal to a required processing capability of the second task, wherein the required processing capability of the second task is a capability required for processing the second task.

[0034] In a possible implementation, the second task is a task with the largest required processing capability among a plurality of tasks, wherein the priority of each task in the plurality of tasks is lower than the priority of the first task, and the future idle duration is greater than or equal to a required duration of each task in the plurality of tasks.

[0035] In a possible implementation, the second task is a task with the longest required processing duration or the highest priority among a plurality of tasks, wherein the priority of each task in the plurality of tasks is lower than the priority of the first task, and the future idle duration is greater than or equal to a required duration of each task in the plurality of tasks.

[0036] In a possible implementation, the indication module is configured to, in a case where there is a third task to be processed and the processing unit cluster does not have enough idle processing units to process the third task, evict a fourth task currently processed by the processing unit cluster from the processing unit cluster to make the processing unit cluster have enough idle processing units to process the third task, wherein the priority of the fourth task is lower than the priority of the third task.

[0037] In a possible implementation, the fourth task is a task with the lowest priority among the tasks currently processed by the cluster of processing units; or,

[0038] The fourth task is a task with the longest remaining processing duration among the tasks currently processed by the cluster of processing units; or,

[0039] The fourth task is a task with the least number of interruptions among the tasks currently processed by the cluster of processing units; or,

[0040] The fourth task is a task occupying the most optimal performance processing unit among the tasks currently processed by the cluster of processing units.

[0041] In a possible implementation, the processing units in the cluster of processing units access different networks when processing different tasks.

[0042] In a possible implementation, the first task is a model inference task, and the second task is a model training task.

[0043] In a third aspect, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method provided in the first aspect.

[0044] In a fourth aspect, a computer-readable storage medium is provided, including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method provided in the first aspect.

[0045] In a fifth aspect, a computer program product including instructions is provided, when the instructions are run by a computing device cluster, the computing device cluster executes the method provided in the first aspect.

[0046] The beneficial effects of the second aspect to the fifth aspect can refer to the beneficial effects of the first aspect described above, and will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0047] FIG. 1 is a schematic diagram of a computing system provided by an embodiment of the present application;

[0048] FIG. 2 is a schematic diagram of a computing system provided by an embodiment of the present application;

[0049] FIG. 3 is a schematic diagram of a control device provided by an embodiment of the present application;

[0050] FIG. 4 is a schematic diagram of a control device provided by an embodiment of the present application;

[0051] FIG. 5 is a schematic diagram of a task processing method provided by an embodiment of the present application;

[0052] FIG. 6 is a schematic diagram of a task processing method provided by an embodiment of the present application;

[0053] FIG. 7 is a flowchart of a task processing method provided by an embodiment of the present application;

[0054] FIG. 8 is a schematic diagram of a control device provided by an embodiment of the present application;

[0055] FIG. 9 is a schematic diagram of a computing device provided by an embodiment of the present application;

[0056] FIG. 10 is a schematic diagram of a computing device cluster provided by an embodiment of the present application;

[0057] FIG. 11 is a schematic diagram of a computing device cluster provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] The schemes provided by the embodiments of the present application will be described below with reference to the drawings. In the embodiments of the present application, “multiple” refers to two or more, and “multiple types” refers to two or more types. “First”, “second”, and the like are only used to distinguish similar objects, and do not necessarily describe a specific order or number of objects.

[0059] To facilitate understanding of the schemes provided by the embodiments of the present application, the technical terms that may be involved in the embodiments of the present application are first introduced.

[0060] Artificial intelligence (AI): is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing, and expert systems.

[0061] Machine learning (ML): is a technology that specifically studies how a computer simulates or implements human learning behavior to acquire new knowledge or skills, reorganizes existing knowledge structure to continuously improve the performance of the computer itself. Machine learning is the core of artificial intelligence and is the fundamental approach to making computers intelligent. Machine learning involves inputting data into a machine learning algorithm and allowing it to learn from the data, so as to make accurate predictions or classifications on new data.

[0062] Checkpoint: refers to periodically saving the weights or other information of a model during model training.

[0063] Service-level agreement (SLA): refers to the agreement or contract between the service provider and the customer on the quality, level, performance, etc. of the service. SLA can include multiple indicators, such as timeliness, reliability, response time, repair time, etc.

[0064] Model training: refers to the task of optimizing the model through machine learning algorithms and a large amount of data, so that it can better predict and make decisions. Model training is an offline task, and the SLA requirement is relatively low.

[0065] Model inference: refers to the task of using the intelligence obtained through model training to infer and judge unknown information under given conditions for the model that has completed model training. Model inference is an online task, and the model directly faces the customer and interacts with the user, so the SLA requirement of model inference is higher.

[0066] Cluster: refers to a cluster of processing units used to perform tasks. Common clusters include Kubernetes clusters. Different clusters can be used to perform different tasks. Among them, the cluster performing the model inference task can be called inference cluster, and the cluster performing the model training task can be called training cluster. A cluster can be composed of multiple processing units. For example, a cluster can be composed of one or more computing nodes. Among them, a computing node can serve as a processing unit, or a computing node includes multiple processing units.

[0067] Processing unit: the smallest computing unit to perform a task. A processing unit processes one task at a time, not two tasks at the same time. A processing unit can process different tasks at different times. Processing units can be hardware such as graphics processing units (GPUs) or neural processing units (NPUs), or virtual computing instances such as containers.

[0068] Computing node (node): a worker node in a cluster, used to provide a running environment for tasks, responsible for running and managing containerized applications. Computing nodes can be hardware such as servers, or virtual computing devices such as virtual machines (VMs).

[0069] Kubernetes cluster: refers to a cluster managed by Kubernetes. Kubernetes is a container orchestration tool. Using Kubernetes can facilitate the scheduling and orchestration of containers. Kubernetes can regard a large number of servers as a huge server, and run an application program on the huge server. No matter how many servers are in the Kubernetes cluster, the method of deploying an application program on Kubernetes is the same.

[0070] Compared with model training, model inference has higher SLA requirements, and model inference has crest and trough phenomena. In order to guarantee the SLA during the crest, the inference cluster is configured with more processing units, etc. During the crest, the processing units in the inference cluster can be fully utilized. However, during the trough, there are a large number of idle processing units in the inference cluster, resulting in resource waste.

[0071] In one scheme, during the trough of model inference, part of the processing units in the inference cluster are migrated to the training cluster, so that the part of the processing units participate in model training, to improve resource utilization. During the crest of model inference, the part of the processing units are migrated back to the inference cluster from the training cluster, to guarantee the SLA of model inference. The migration of processing units between different clusters involves processing unit removal, addition, etc., and takes a long time. Since the processing units do not process tasks during the migration of the processing units, the improvement of the resource utilization is limited by this scheme. Moreover, when there is a sudden model inference task, the processing units cannot be migrated back to the inference cluster in time, so that the inference cluster is difficult to cope with the sudden model inference task.

[0072] In one scheme, the priority of a model inference task is set to be higher than that of a model training task, then the tasks are added to a scheduling queue based on the priority of the tasks, and the tasks in the scheduling queue are processed in turn. This method can make the model inference task be processed before the model training task. However, if the model inference task arrives during the processing of the model training task, the model inference task needs to wait until the model training task is processed, so as to start processing the model inference task, thereby increasing the time delay of the model inference task and affecting the SLA of the model inference task.

[0073] Embodiments of the present application provide a task processing method. In the method, idle processing units of a cluster can be identified when the processing demand of a high-priority task is met, and a length of time when the idle processing units are predicted to be unnecessary to process the high-priority task in the future. Then, in the case that the length of time is greater than or equal to a length of time required to process a low-priority task, the idle processing units are instructed to process the low-priority task. In this way, while improving resource utilization, the high-priority task arriving in the future can be processed in time.

[0074] Next, the task processing method provided by the embodiments of the present application is described in detail.

[0075] FIG. 1 shows a computing system 100 that can be used to implement the method. The computing system 100 includes a control device 110 and a processing unit cluster 120. The processing unit cluster 120 can be referred to as cluster 120 for short. The cluster 120 can include a plurality of processing units, such as processing unit 121, processing unit 122, etc. The control device 110 is used to instruct the processing units in the cluster 120 to perform tasks. Under the instruction of the control device 110, the processing units can perform different tasks. Among them, one processing unit performs the same task at the same time, and can perform different tasks at different times.

[0076] In some embodiments, as shown in FIG. 2, the control device 110 can receive a plurality of tasks initiated by a user through a console, such as A-type tasks, B-type tasks, etc. Among them, the priority of the A-type task is higher than the priority of the B-type task. For example, the A-type task can be an online task, such as a model inference task. The B-type task can be an offline task, such as a model training task.

[0077] The control device 110 can instruct the processing units in the cluster 120 to process different tasks according to the task processing method provided by the embodiments of the present application.

[0078] In some embodiments, as shown in FIG. 2, in order to achieve the foregoing object, the control device 110 can include a resource manager 111 and an evictor 112.

[0079] The resource manager 111 can calculate the processing units that can be used to process the B-type task and the duration of using the processing units to process the B-type task based on the resource yielding policy. Among them, the processing units that can be used to process the B-type task are also called idle processing units, specifically, the idle processing units of the cluster 120 in addition to processing the A-type task, that is, the idle processing units under the premise of meeting the processing demand of the A-type task. The duration of using the processing units to process the B-type task is also called the future idle duration of the idle processing units, specifically, the duration of not needing the idle processing units to process the A-type task in the future.

[0080] In some embodiments, the resource allocation strategy can identify a free processing unit, and then calculate a future free duration of the free processing unit. The free processing unit can be identified based on a current running status of each processing unit in the cluster 120. The future free duration of the free processing unit can be calculated based on historical information of the cluster 120 processing the class A task. The historical information of the cluster 120 processing the class A task can include a time when the class A task historically arrived at the cluster 120 (i.e., a time when the cluster 120 historically started processing the class A task), load information of the cluster 120 processing the class A task, etc. The load information of the cluster 120 processing the class A task can be used to determine whether the free processing unit needs to process the class A task. In the case that the free processing unit does not need to process the class A task, a time when the free processing unit will start processing the class A task can be determined based on the time when the class A task historically arrived at the cluster 120. Thus, the future free duration of the free processing unit can be calculated.

[0081] The resource allocation strategy can also identify a priority of the task and determine whether the task is a class A task. In one example, each task corresponds to relevant information of the task defined by a user. In one example, the relevant information of the task can be a priority. Thus, the priority of the task can be directly used to determine whether the task is a class A task. In another example, the relevant information of the task can include a real-time requirement of the task, a response delay requirement of the task, a task label classification, an SLA, etc. The priority of the task can be calculated based on the custom information of the task. Then, the priority of the task can be used to determine whether the task is a class A task. For example, the relevant information of the task is custom information, e.g., information defined by a user for the task.

[0082] The resource manager 111 can instruct the free processing unit to process a class B task in the future free duration based on a task filling strategy. The task filling strategy instructs the free processing unit to process a task whose required duration is less than or equal to the future free duration of the free processing unit. Thus, the free processing unit can be guaranteed to finish processing the class B task before or when the free processing unit is needed to process the class A task, so that the class A task does not need to wait for execution and the class B task does not need to be frequently interrupted.

[0083] In some embodiments, there are multiple class B tasks to be processed, and the required duration of each of the multiple tasks is less than or equal to the future free duration of the free processing unit. In this case, the task filling strategy instructs the free processing unit to process the task with the longest required duration among the multiple tasks. Thus, the computing resources of the free processing unit can be fully utilized, thereby improving resource utilization.

[0084] In some embodiments, the task filling strategy instructs the idle processing unit to process a task whose required processing capability is less than or equal to the processing capability of the idle processing unit. Here, the processing capability of the idle processing unit refers to the processing capability of the processing unit when it is in an idle state, which can be referred to as idle processing capability. The idle processing capability of a processing unit is also the maximum available processing capability of the processing unit. In one example of this embodiment, there are multiple B-type tasks to be processed, and the required processing capability of each of the multiple tasks is less than or equal to the processing capability of the idle processing unit. In this case, the task filling strategy instructs the idle processing unit to process the task with the largest required processing capability among the multiple tasks. In this way, the computing resources of the idle processing unit can be fully utilized, thereby improving resource utilization.

[0085] The resource manager 111 can guarantee the quality of service of the tasks based on a service guarantee strategy, such as an SLA. Specifically, the A-type tasks and the B-type tasks are different types of tasks, for example, the A-type tasks are model inference tasks and the B-type tasks are model training tasks. The B-type tasks can be further prioritized, i.e., the priorities of different B-type tasks can also be different. There are multiple B-type tasks to be processed, and the priority of each of the multiple tasks is lower than the A-type task described above, and the required duration of each of the multiple tasks is less than the idle duration described above. In this case, the service guarantee strategy instructs the idle processing unit to process the task with the highest priority among the multiple tasks, so that the task with the highest priority among the multiple tasks is processed first, thereby guaranteeing the quality of service of the task. Illustratively, the resource manager 111 can calculate the priority of a task based on the relevant information of the task. The relevant information of the task can include real-time requirements, response delay requirements, task label classification, SLA, etc. The priorities of the multiple tasks can be calculated based on the relevant information of the tasks, and the priorities of the multiple tasks can be sorted. The relevant information of the task can include real-time requirements, response delay requirements, task label classification, SLA, etc.

[0086] The resource manager 111 can provide network face isolation capability for the tasks based on a network isolation policy to guarantee the isolation between different tasks. When instructing the processing unit to process different tasks, the processing unit accesses different networks. That is, the network accessed by the processing unit when processing different tasks is different. For example, the processing unit cluster 120 can include a network C1 and a network C2. The network C1 corresponds to an A-class task, and the network C2 corresponds to a B-class task. When instructing the processing unit to process an A-class task, the processing unit accesses the network C1. When instructing the processing unit to process a B-class task, the processing unit accesses the network C2. For example, each processing unit has multiple network interfaces, and different network interfaces access different networks. When instructing the processing unit to process a task, the processing unit enables the network interface corresponding to the task and disables other network interfaces, so that the processing unit accesses the network corresponding to the task.

[0087] In some embodiments, the processing unit is connected to the network C1 by default, so as to switch the network when the A-class task arrives without the need to switch the network again, thereby improving the timeliness of processing the A-class task. When instructing the processing unit to process a B-class task, the network connected to the processing unit is switched from the network C1 to the network C2. When the B-class task is processed, the network connected to the processing unit is switched from the network C2 to the network C1.

[0088] In some embodiments, as shown in FIG. 3, the control device 110 further includes a control plane 113. The resource manager 111 implements related functions through the control plane 113. For example, the resource manager 111 can make a specific task processing decision based on a resource release policy, a task filling policy, a service guarantee policy, or a network isolation policy, and then send the task processing decision to the control plane 113. The control plane 113 controls the processing unit in the cluster 120 to process the related task based on the task processing decision.

[0089] The required duration of the B-class task is calculated based on experience or experiment and can not be completely accurate. In addition, the processing unit can fail when processing the B-class task, and the like, so that the B-class task can not be processed completely when the future idle duration ends. In this case, the evacuator 112 evacuates the B-class task from the processing unit based on a resource recycling policy when the future idle duration ends. Here, the processing unit is the processing unit that processes the B-class task based on the resource release policy.

[0090] The resource recycling policy can instruct the processing units to evict the tasks according to a specified eviction manner. The eviction manner can be specified by a user. For example, the eviction manner can be one of graceful eviction and direct eviction. The graceful eviction means that the processing unit performs at least one checkpoint on the task before evicting the task, so as to keep an intermediate state result of the task. When the task is processed again, the processing can be started from the intermediate state result, so as to improve the processing efficiency of the task. The direct eviction means that the task is directly deleted, so as to quickly recycle the resource.

[0091] As described above, the future idle duration of the idle processing unit is predicted based on the historical information of the A-class tasks. However, an unexpected situation can occur, which causes the number of A-class tasks to suddenly increase, and there can be not enough idle processing units in the cluster 120. In this case, the B-class task being processed needs to be evicted from the processing unit, so that the cluster 120 has enough idle processing units to process the suddenly increased A-class tasks.

[0092] The evacuator 112 can select the processing unit based on the conflict resolution policy, and instruct the selected processing unit to evict the task being processed.

[0093] In some embodiments, as shown in FIG. 4, the conflict resolution policy can include a service guarantee eviction policy. As described above, the priorities of different B-class tasks can also be different. There are multiple B-class tasks being processed in the cluster 120. The service guarantee policy is to evict the task with the lowest priority among the multiple tasks. That is, based on the service guarantee policy, the evacuator 112 confirms the task with the lowest priority among the multiple tasks, and then instructs the processing unit processing the task with the lowest priority to evict the task.

[0094] In one example of this embodiment, if the priorities of the multiple tasks are the same, the task with the least number of interruptions among the multiple tasks is evicted.

[0095] In some embodiments, as shown in FIG. 4, the conflict resolution policy can include a cost eviction policy. There are multiple B-class tasks being processed in the cluster 120. The cost eviction policy is to evict the task with the least number of interruptions among the multiple tasks, or to evict the task with the longest remaining processing time among the multiple tasks, or to evict the task with the shortest start processing time. The remaining processing time of the task can be estimated according to the information of the same type of tasks processed in the past.

[0096] In some embodiments, as shown in FIG. 4, the conflict resolution policy can include a performance guarantee eviction policy. There are multiple B-type tasks currently being processed in the cluster 120. Different tasks in the multiple tasks can occupy processing units with different performance. The cost eviction policy is to evict a task in the multiple tasks that occupies a processing unit with optimal performance. For example, the performance can be network performance. Specifically, the network performance can be the connection performance of the processing unit. The connection performance of each processing unit with other processing units can be identified based on the network topology information between the processing units in the cluster 120. There are two connection modes between the processing units in the cluster 120, which are full mesh and partition mesh. The connection performance of the full mesh is greater than that of the partition mesh. The full mesh means that any two processing units in the cluster are directly connected. The partition mesh means that some processing units in the cluster are not directly connected, and the information exchanged between them needs to be forwarded by other processing units in the cluster.

[0097] Through the performance guarantee eviction policy, the B-type task occupying the high-performance processing unit can be evicted from the high-performance processing unit, so that the high-performance processing unit can be used to process the A-type task, thereby guaranteeing the SLA of the A-type task.

[0098] In some embodiments, the evictor 112 can obtain a specific eviction decision based on the conflict resolution policy described above, and then send the eviction decision to the control plane 113. The control plane 113 instructs the relevant processing unit to evict the task based on the eviction decision.

[0099] Through the above scheme, the user can uniformly issue A-type tasks and B-type tasks to the cluster 120.

[0100] When the resources of the cluster 120 are sufficient to simultaneously process all the tasks issued by the user, the control device 110 can allocate processing units to all the tasks to process all the tasks. In some embodiments, the control device 110 can allocate processing units to the tasks in order of priority from high to low, so as to guarantee that the high-priority tasks are allocated processing units first and guarantee the timely processing of high-priority tasks.

[0101] When the cluster 120 cannot simultaneously process all the tasks, and a task waiting queue appears. The control device 110 instructs the cluster 120 to process the A-type tasks first. Specifically, it can be divided into the following two cases.

[0102] Case 1, the A-class task waiting queue is not empty (i.e., there is an A-class task waiting for processing), and the cluster 120 has a processing unit (e.g., the processing unit 123) currently processing a task which is a B-class task. For case 1, as shown in FIG. 5, at step 51, the evictor 112 in the control device 110 instructs the processing unit 123 to evict the B-class task, i.e., to expel the B-class task currently being processed by the processing unit 123 from the processing unit 123. And at step 52, the control plane 123 switches the network connected by the processing unit 123 from the network C2 to the network C1. Then, at step 53, the resource manager 111 instructs the processing unit 123 to process an A-class task. Wherein, under the instruction of the resource manager 111, the processing unit 123 acquires an A-class task from the A-class task waiting queue and processes the acquired A-class task.

[0103] Case 2, the A-class task waiting queue is empty (i.e., there is no A-class task waiting for processing), the B-class task waiting queue is not empty (i.e., there is a B-class task waiting for processing), and the cluster 120 has an idle processing unit (e.g., the processing unit 123). For case 2, as shown in FIG. 6, at step 61, the resource manager 111 can calculate the future idle duration of the processing unit 123. And at step 62, the control plane 113 switches the network connected by the processing unit 123 from the network C1 to the network C2. Then, at step 63, the resource manager 111 instructs the processing unit 123 to process a B-class task. Wherein, under the instruction of the resource manager 111, the processing unit 123 acquires a B-class task from the B-class task waiting queue and processes the acquired B-class task. Finally, at step 64, when the future idle duration of the processing unit 123 ends, the resource of the processing unit 123 is recycled.

[0104] In this way, by the above scheme, the idle resources of the cluster can be used to process low-priority tasks without affecting the processing of high-priority tasks, which not only guarantees the SLA of high-priority tasks, but also improves the resource utilization of the cluster.

[0105] Based on the above description, the embodiment of the present application provides a task processing method. The method can be executed by the control device 110 in the computing system 100. As shown in FIG. 7, the method comprises the following steps.

[0106] At step 701, at least one idle processing unit in the processing unit cluster 120 is identified. Wherein, the idle processing unit is an idle processing unit of the processing unit cluster 120 in addition to processing A-class tasks. Specifically, the idle processing unit is an idle processing unit in the processing unit cluster 120 in the case that there is no A-class task in the waiting state. In step 701, the control device 110 identifies a processing unit that is currently not processing a task in the case that there is no A-class task in the waiting state, to obtain the at least one idle processing unit.

[0107] At step 702, the control device 110 predicts a future idle duration of the at least one idle processing unit based on historical information of the processing unit cluster 120 processing the first task, where the future idle duration is a duration that the at least one idle processing unit does not need to process the first task. In some embodiments, the first task is an A-class task. The future idle duration is a duration that the at least one idle processing unit processes a single A-class task (i.e., one A-class task instance). The control device 110 can calculate the future idle duration based on the resource yielding policy described above. For details, please refer to the description above, which will not be repeated here.

[0108] At step 703, the control device 110 instructs the at least one idle processing unit to process a second task if the future idle duration is greater than or equal to a required duration of the second task, where the second task has a priority lower than that of the first task, and the required duration of the second task is a duration that the at least one idle processing unit needs to process the second task. In some embodiments, the second task is a B-class task. For example, the second task is one or more B-class task instances.

[0109] In some embodiments, at step 703, the control device 110 can calculate the required duration of the second task based on historical information of the processing unit cluster 120 processing B-class tasks.

[0110] In some embodiments, the method further comprises identifying an idle processing capability of the at least one idle processing unit. As described above, the idle processing capability of the idle processing unit is the processing capability of the processing unit in the idle state, which is also the maximum available processing capability of the processing unit. The idle processing capability of the processing unit can be obtained by querying the specifications of the processing unit.

[0111] In this embodiment, step 703 is specifically: instructing the at least one idle processing unit to process the second task if the future idle duration is greater than or equal to the required duration of the second task, and the idle processing capability is greater than or equal to a required processing capability of the second task, where the required processing capability of the second task is a capability required to process the second task. For example, the control device 110 can calculate the required processing capability of the second task based on historical information of the processing unit cluster 120 processing B-class tasks.

[0112] In one example of this embodiment, the second task is a task with the maximum required processing capability among a plurality of tasks, where each task in the plurality of tasks has a priority lower than that of the first task, and the future idle duration is greater than or equal to a required duration of each task in the plurality of tasks.

[0113] In this example, there are multiple class B tasks currently waiting for the processing unit cluster 120 to process, and the required duration of each task is less than or equal to the future idle duration of the at least one processing unit, and the required processing capacity of each task is less than or equal to the processing capacity of the at least one processing unit. The second task is the task with the largest required processing capacity among the multiple tasks. In this way, the computing resources of the idle processing units can be fully utilized, and the resource utilization rate is improved. For details, please refer to the above description of the task filling strategy.

[0114] In some embodiments, the second task is the task with the longest required processing duration or the highest priority among the multiple tasks; wherein the priority of each task in the multiple tasks is less than the priority of the first task, and the future idle duration is greater than or equal to the required duration of each task in the multiple tasks. For details, please refer to the above description of the service guarantee strategy and the task filling strategy.

[0115] In some embodiments, the method further comprises: in the case that a third task to be processed is obtained, and there are not enough idle processing units in the processing unit cluster to process the third task, expelling a fourth task currently processed by the processing unit cluster from the processing unit cluster, so that there are enough idle processing units in the processing unit cluster to process the third task; wherein the priority of the fourth task is less than the priority of the third task. The third task belongs to class A tasks, and can be one or more class A task instances. The fourth task belongs to class B tasks, and can be one or more class B task instances.

[0116] In one example of this embodiment, the fourth task is the task with the lowest priority among the tasks currently processed by the processing unit cluster 120. For details, please refer to the above description of the service guarantee expulsion strategy.

[0117] In another example of this embodiment, the fourth task is the task with the longest remaining processing duration among the tasks currently processed by the processing unit cluster 120. For details, please refer to the above description of the cost expulsion strategy.

[0118] In yet another example of this embodiment, the fourth task is the task with the least number of interruptions among the tasks currently processed by the processing unit cluster 120. For details, please refer to the above description of the cost expulsion strategy.

[0119] In yet another example of this embodiment, the fourth task is the task that occupies the most optimal performance processing unit among the tasks currently processed by the processing unit cluster 120. For details, please refer to the above description of the performance guarantee expulsion strategy.

[0120] In some embodiments, the processing units in the processing unit cluster 120 access different networks when processing different tasks. This can be specifically referred to the network isolation strategy described above.

[0121] In some embodiments, the first task is a model inference task, and the second task is a model training task.

[0122] In summary, the task processing method provided by the embodiments of the present application uses the idle processing units after processing high-priority tasks to process low-priority tasks, and the time length required by the processing units to process the low-priority tasks is less than or equal to the time length during which the processing units do not need to process the high-priority tasks, thereby improving the resource utilization rate while avoiding the influence of processing the low-priority tasks on the SLA of the high-priority tasks.

[0123] Based on the above description, the embodiments of the present application provide a control device 800. The computing system where the device 800 is located also includes a processing unit cluster, wherein the processing units in the processing unit cluster can process different tasks at different times under the indication of the device 800. As shown in FIG. 8, the device 800 includes:

[0124] An identification module 810 is configured to identify at least one idle processing unit in the processing unit cluster;

[0125] A prediction module 820 is configured to predict a future idle time length of the at least one idle processing unit based on historical information of the processing unit cluster processing a first task, the future idle time length being a time length during which the at least one idle processing unit does not need to process the first task;

[0126] An indication module 830 is configured to instruct the at least one idle processing unit to process a second task if the future idle time length is greater than or equal to a required time length of the second task, wherein the priority of the second task is lower than the priority of the first task, and the required time length of the second task is a time length required by the at least one idle processing unit to process the second task.

[0127] In some embodiments, the identification module 810 is further configured to identify an idle processing capability of the at least one idle processing unit, and the indication module 830 is configured to instruct the at least one idle processing unit to process the second task if the future idle time length is greater than or equal to the required time length of the second task, and the idle processing capability is greater than or equal to a required processing capability of the second task, wherein the required processing capability of the second task is a capability required to process the second task.

[0128] In one example of the embodiment, the second task is a task with the largest required processing capability among the plurality of tasks, wherein a priority of each task in the plurality of tasks is less than the priority of the first task, and the future idle duration is greater than or equal to a required duration of each task in the plurality of tasks.

[0129] In some embodiments, the second task is a task with the longest required processing duration or the highest priority among the plurality of tasks; wherein a priority of each task in the plurality of tasks is less than the priority of the first task, and the future idle duration is greater than or equal to a required duration of each task in the plurality of tasks.

[0130] In some embodiments, the indication module 830 is configured to: in a case where a third task needs processing and the processing unit cluster does not have enough idle processing units to process the third task, evict a fourth task currently processed by the processing unit cluster from the processing unit cluster, so that the processing unit cluster has enough idle processing units to process the third task; wherein a priority of the fourth task is less than a priority of the third task.

[0131] In one example of the embodiment, the fourth task is a task with the lowest priority among the tasks currently processed by the processing unit cluster; or, the fourth task is a task with the longest remaining processing duration among the tasks currently processed by the processing unit cluster; or, the fourth task is a task with the least number of interruptions among the tasks currently processed by the processing unit cluster; or, the fourth task is a task that occupies the most performance processing units among the tasks currently processed by the processing unit cluster.

[0132] In some embodiments, the processing units in the processing unit cluster access different networks when processing different tasks.

[0133] In some embodiments, the first task is a model inference task, and the second task is a model training task.

[0134] The identification module 810, the prediction module 820, and the indication module 830 can be implemented by software or by hardware. For example, the implementation of the identification module 810 is described below. The implementation of the prediction module 820 and the indication module 830 can be similar to the implementation of the identification module 810.

[0135] As an example of a software functional unit, the identification module 810 can include code running on a compute instance. The compute instance can include at least one of a physical host (computing device), a virtual machine, a container. Further, the compute instance can be one or more. For example, the identification module 810 can include code running on multiple hosts / virtual machines / containers. It is noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ) or in different AZs, each of which includes one data center or multiple data centers in close geographical proximity. Typically, a region can include multiple AZs.

[0136] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same VPC or in multiple VPCs. Typically, a VPC is set up within a region, and a communication gateway needs to be set up in each VPC for cross-zone communication between two VPCs in the same region or between VPCs in different regions, and the interconnection between the VPCs is achieved through the communication gateway.

[0137] As an example of a hardware functional unit, the identification module 810 can include at least one computing device, such as a server, etc. Alternatively, the identification module 810 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. The PLD can be implemented by a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0138] The multiple computing devices included in the identification module 810 can be distributed in the same region or in different regions. The multiple computing devices included in the identification module 810 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the identification module 810 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, etc.

[0139] It should be noted that in other embodiments, the identifying module 810 can be configured to perform any of the steps of the method shown in FIG. 7, the predicting module 820 can be configured to perform any of the steps of the method shown in FIG. 7, and the indicating module 830 can be configured to perform any of the steps of the method shown in FIG. 7. The steps that the identifying module 810, the predicting module 820, and the indicating module 830 are responsible for implementing can be specified as desired, and the entire functionality of the apparatus 800 can be implemented by the identifying module 810, the predicting module 820, and the indicating module 830 each implementing different steps of the method shown in FIG. 7.

[0140] The present application also provides a computing device 900. As shown in FIG. 9, the computing device 900 includes a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other through the bus 902. The computing device 900 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 900.

[0141] The bus 902 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 9, but it does not mean that there is only one bus or only one type of bus. The bus 902 can include a path for transmitting information between various components (e.g., the memory 906, the processor 904, the communication interface 908) of the computing device 900.

[0142] The processor 904 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0143] The memory 906 can include volatile memory, such as random access memory (RAM), and non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid-state drive (SSD).

[0144] The memory 906 stores executable program code that is executed by the processor 904 to implement the functions of the aforementioned identification module 810, prediction module 820, and indication module 830, respectively, and thus implement the method shown in FIG. 7. That is, the memory 906 stores instructions for implementing the method shown in FIG. 7.

[0145] The communication interface 908 uses a transceiving module such as, but not limited to, a network interface card or a transceiver to enable communication between the computing device 900 and other devices or communication networks.

[0146] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smartphone.

[0147] As shown in FIG. 10, the computing device cluster includes at least one computing device 900. The memory 906 in one or more computing devices 900 in the computing device cluster can store the same instructions for implementing the method shown in FIG. 7.

[0148] In some possible implementations, the memory 906 in one or more computing devices 900 in the computing device cluster can also respectively store partial instructions for implementing the method shown in FIG. 7. In other words, the combination of one or more computing devices 900 can collectively execute the instructions for implementing the method shown in FIG. 7.

[0149] It should be noted that the memory 906 in different computing devices 900 in the computing device cluster can store different instructions for respectively implementing part of the functions of the apparatus 800. That is, the instructions stored in the memory 906 in different computing devices 900 can implement the functions of one or more of the identification module 810, the prediction module 820, and the indication module 830.

[0150] In some possible implementations, one or more of the computing devices in the cluster of computing devices can be connected through a network. The network can be a wide area network, a local area network, or the like. FIG. 11 illustrates one possible implementation. As shown in FIG. 11, two computing devices 900A and 900B are connected through a network. Specifically, the computing devices are connected to the network through a communication interface in each of the computing devices. In this type of possible implementation, the memory 906 in the computing device 900A stores instructions for performing the functions of the identification module 810. Meanwhile, the memory 906 in the computing device 900B stores instructions for performing the functions of the prediction module 820 and the indication module 830.

[0151] It should be understood that the functions of the computing device 900A shown in FIG. 11 can also be performed by multiple computing devices 900. Similarly, the functions of the computing device 900B can also be performed by multiple computing devices 900.

[0152] Embodiments of the present application also provide another cluster of computing devices. The computing devices in the cluster of computing devices can be connected in a similar manner as described above with reference to the cluster of computing devices shown in FIG. 10 and FIG. 11. The difference is that the memory 906 in one or more of the computing devices 900 in the cluster of computing devices can store the same instructions for performing the method shown in FIG. 7.

[0153] In some possible implementations, the memory 906 in one or more of the computing devices 900 in the cluster of computing devices can also store partial instructions for performing the method shown in FIG. 7, respectively. In other words, a combination of one or more of the computing devices 900 can collectively perform the instructions for performing the method shown in FIG. 7.

[0154] Embodiments of the present application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform the method shown in FIG. 7.

[0155] Embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium or a host migration device such as a data center containing one or more available media that a computing device can store. The available media can be a magnetic medium (e.g., a floppy diskette, a hard disk drive, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive), or the like. The computer readable storage medium contains instructions that instruct a computing device to perform the method shown in FIG. 7.

[0156] Finally, it should be noted that: the above examples are used to illustrate the technical solutions of the present application, but not limited to them; although the present application is described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A task processing method characterized by, The method is applied to a control device in a computing system, the computing system further comprising a processing unit cluster, wherein, under the indication of the control device, processing units in the processing unit cluster can process different tasks at different time instants; the method comprises: identifying at least one idle processing unit in the processing unit cluster; predicting a future idle duration of the at least one idle processing unit based on historical information of the processing unit cluster processing a first task, the future idle duration being a duration during which the at least one idle processing unit is not required to process the first task; in a case where the future idle duration is greater than or equal to a required duration of a second task, instructing the at least one idle processing unit to process the second task; wherein a priority of the second task is lower than a priority of the first task, and the required duration of the second task is a duration required for the at least one idle processing unit to process the second task.

2. The method of claim 1, wherein: the method further comprises identifying an idle processing capability of the at least one idle processing unit; the case where the future idle duration is greater than or equal to the required duration of the second task comprises a case where the future idle duration is greater than or equal to the required duration of the second task, and the idle processing capability is greater than or equal to a required processing capability of the second task, and the required processing capability of the second task is a capability required for processing the second task.

3. The method of claim 2, wherein, the second task is a task with a greatest required processing capability among a plurality of tasks, wherein a priority of each task in the plurality of tasks is lower than the priority of the first task, and the future idle duration is greater than or equal to a required duration of each task in the plurality of tasks.

4. The method according to any one of claims 1-3, characterized in that, the second task is a task with a greatest required processing duration or a highest priority among a plurality of tasks; wherein a priority of each task in the plurality of tasks is lower than the priority of the first task, and the future idle duration is greater than or equal to a required duration of each task in the plurality of tasks.

5. The method according to any one of claims 1-4, characterized in that, the method further comprises: in a case where there is a third task to be processed, and the processing unit cluster does not have enough idle processing units to process the third task, evicting a fourth task currently processed by the processing unit cluster from the processing unit cluster, so that the processing unit cluster has enough idle processing units to process the third task; wherein a priority of the fourth task is lower than a priority of the third task.

6. The method of claim 5, wherein: the fourth task is a task with a lowest priority among tasks currently processed by the processing unit cluster; or the fourth task is a task with a greatest remaining processing duration among tasks currently processed by the processing unit cluster; or the fourth task is a task with a least number of interruptions among tasks currently processed by the processing unit cluster; or the fourth task is a task with a least number of interruptions among tasks currently processed by the processing unit cluster; or The fourth task is a task that occupies the optimal performance processing unit among tasks currently processed by the processing unit cluster.

7. The method according to any one of claims 1 to 6, characterized in that, Processing units in the processing unit cluster access different networks when processing different tasks.

8. The method according to any one of claims 1-7, characterized in that, The first task is a model inference task, and the second task is a model training task.

9. A control device characterized by comprising: The computing system where the device is located further includes a processing unit cluster, wherein, under indication of the device, processing units in the processing unit cluster can process different tasks at different time instants; the device includes: An identification module configured to identify at least one idle processing unit in the processing unit cluster; A prediction module configured to predict, based on historical information of the processing unit cluster processing a first task, a future idle duration of the at least one idle processing unit, the future idle duration being a duration during which the at least one idle processing unit does not need to process the first task; An indication module configured to instruct the at least one idle processing unit to process a second task if the future idle duration is greater than or equal to a required duration of the second task; wherein a priority of the second task is lower than a priority of the first task, and the required duration of the second task is a duration required for the at least one idle processing unit to process the second task.

10. The device of claim 9, wherein: The identification module is further configured to identify an idle processing capability of the at least one idle processing unit; The indication module is configured to instruct the at least one idle processing unit to process the second task if the future idle duration is greater than or equal to the required duration of the second task, and the idle processing capability is greater than or equal to a required processing capability of the second task, wherein the required processing capability of the second task is a capability required for processing the second task.

11. The apparatus of claim 10, wherein, The second task is a task with the greatest required processing capability among a plurality of tasks, wherein a priority of each task in the plurality of tasks is lower than the priority of the first task, and the future idle duration is greater than or equal to a required duration of each task in the plurality of tasks.

12. The apparatus of any one of claims 9-11, wherein, The second task is a task with the longest required processing duration or the highest priority among a plurality of tasks; wherein a priority of each task in the plurality of tasks is lower than the priority of the first task, and the future idle duration is greater than or equal to a required duration of each task in the plurality of tasks.

13. The apparatus of any one of claims 9-12, wherein, The indication module is configured to evict, from the processing unit cluster, a fourth task currently processed by the processing unit cluster to make the processing unit cluster have sufficient idle processing units to process a third task if the third task needs to be processed and the processing unit cluster does not have sufficient idle processing units to process the third task; wherein a priority of the fourth task is lower than a priority of the third task.

14. The device of claim 13, wherein: The fourth task is a task with the lowest priority among tasks currently processed by the processing unit cluster; or The fourth task is a task with the longest remaining processing duration among tasks currently processed by the processing unit cluster; or The fourth task is a task with the least number of interruptions among the tasks currently processed by the cluster of processing units; or The fourth task is a task that occupies the best performance processing unit among the tasks currently processed by the cluster of processing units.

15. The apparatus of any one of claims 9-14, wherein, The processing units in the cluster of processing units access different networks when processing different tasks.

16. The apparatus of any one of claims 9-15, wherein, The first task is a model inference task, and the second task is a model training task.

17. A cluster of computing devices, characterized in that, comprise at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of claims 1-8.

18. A computer-readable storage medium, characterized in that, comprise computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any one of claims 1-8.

19. A computer program product comprising instructions, characterized in that, The instructions, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any one of claims 1-8.

Images