Task scheduling method applicable to large language model (LLM), and computing device, storage medium and computer program product

Abstract

Provided in the present disclosure are a task scheduling method applicable to a large language model (LLM), and a computing device, a storage medium and a computer program product. The method comprises: when determining that a target task processing node currently executing a target task meets a resource reporting condition, a target second task scheduler sending a target running resource of the target task processing node to a first task scheduler; when determining that the target running resource meets a task migration condition, the first task scheduler determining a migration task processing node from among a plurality of task processing nodes, and sending node information of the migration task processing node to the target second task scheduler; and the target second task scheduler determining the migration task processing node on the basis of the node information, determining the target task currently executed by the target task processing node, as well as task data of the target task, and migrating the target task and the task data in batches to the migration task processing node for execution, so as to obtain a task processing result of the target task.

Description

[0001] This disclosure claims priority to Chinese Patent Application No. 202410494141.7, filed with the Chinese Patent Office on April 23, 2024, entitled "Task Scheduling Method, Computing Device, Storage Medium, and Computer Program Product Applicable to Large Language Models (LLMs)," the entire contents of which are incorporated herein by reference. Technical Field This disclosure relates to the field of computer technology, and particularly to a task scheduling method, computing device, storage medium, and computer program product applicable to Large Language Models (LLMs). Background Art With the continuous development of computer technology, task processing requires the use of nodes responsible for task processing. However, during task processing, uneven load distribution among nodes leads to low resource utilization, resulting in a significant waste of computer resources. Therefore, how to rationally schedule tasks to avoid low resource utilization has become an urgent problem to be solved. In view of the above, this disclosure provides a task scheduling method, computing device, storage medium, and computer program product suitable for Large Language Models (LLM). One or more embodiments of this specification also relate to another task scheduling method, a task processing system, a task scheduling apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies of low resource utilization in the prior art.According to a first aspect of this disclosure, a task scheduling method is provided, applied to a task processing system. The task processing system includes a task scheduling node and multiple task processing nodes. A first task scheduler is deployed in the task scheduling node, and a second task scheduler is deployed in each task processing node. A target second task scheduler, upon determining that the target task processing node currently executing the target task meets resource reporting conditions, determines the target running resources of the target task processing node and sends the target running resources to the first task scheduler. The target second task scheduler is deployed on the target task processing node, which is any one of the multiple task processing nodes. The first task scheduler, upon determining that the target running resources meet task migration conditions, determines a migration task processing node from the multiple task processing nodes and sends the node information of the migration task processing node to the target second task scheduler. The target second task scheduler determines the migration task processing node based on the node information, determines the target task currently being executed by the target task processing node, and determines the task data of the target task. The target task and the task data are migrated to the migration task processing node in batches for execution, so as to obtain the task processing result of the target task. According to a second aspect of this disclosure, a task processing system is provided, comprising a task scheduling node and multiple task processing nodes. A first task scheduler is deployed in the task scheduling node, and a second task scheduler is deployed in each task processing node. A target second task scheduler is configured to, upon determining that the target task processing node currently executing the target task meets resource reporting conditions, determine the target running resources of the target task processing node and send the target running resources to the first task scheduler. The target second task scheduler is deployed on the target task processing node, which is any one of the multiple task processing nodes. The first task scheduler is configured to, upon determining that the target running resources meet task migration conditions, determine a migration task processing node from the multiple task processing nodes and send the node information of the migration task processing node to the target second task scheduler. The target second task scheduler is configured to determine the migration task processing node based on the node information, determine the target task currently being executed by the target task processing node, and determine the task data of the target task. The target task and the task data are migrated to the migration task processing node in batches for execution, so as to obtain the task processing result of the target task.According to a third aspect of this disclosure, another task scheduling method is provided, applied to a second task scheduler deployed in multiple task processing nodes, comprising: determining the target running resources of the target task processing node when it is determined that the target task processing node currently executing the target task meets resource reporting conditions, wherein the target second task scheduler is deployed on the target task processing node, and the target task processing node is any one of the multiple task processing nodes; sending the target running resources to a first task scheduler, wherein the first task scheduler is deployed on a task scheduling node; receiving node information of a migrated task processing node sent by the first task scheduler, wherein the migrated task processing node is determined by the first task scheduler from the multiple task processing nodes when it is determined that the target running resources meet task migration conditions; determining the migrated task processing node based on the node information; determining the target task currently being executed by the target task processing node, and the task data of the target task; and migrating the target task and the task data to the migrated task processing node in batches for execution to obtain the task processing result of the target task. According to a fourth aspect of this disclosure, a task scheduling apparatus is provided, applied to a second task scheduler deployed in multiple task processing nodes, comprising: a resource determination module configured to determine a target running resource of the target task processing node when it is determined that the target task processing node currently executing the target task meets resource reporting conditions, wherein the target second task scheduler is deployed on the target task processing node, and the target task processing node is any one of the multiple task processing nodes; a resource sending module configured to send the target running resource to a first task scheduler, wherein the first task scheduler is deployed on a task scheduling node; an information receiving module configured to receive node information of a migrating task processing node sent by the first task scheduler, wherein the migrating task processing node is determined by the first task scheduler from the multiple task processing nodes when it is determined that the target running resource meets task migration conditions; a task determination module configured to determine the migrating task processing node based on the node information, determine the target task currently being executed by the target task processing node, and the task data of the target task; and a task migration module. The target task and its data are configured to be migrated to the migration task processing node in batches for execution, so as to obtain the task processing result of the target task.According to a fifth aspect of this disclosure, a computing device is provided, comprising: a memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, wherein the computer programs / instructions, when executed by the processor, implement the steps of the aforementioned task scheduling method or another task scheduling method. According to a sixth aspect of this disclosure, a computer-readable storage medium is provided storing computer programs / instructions, wherein the computer programs / instructions, when executed by a processor, implement the steps of the aforementioned task scheduling method or another task scheduling method. According to a seventh aspect of this disclosure, a computer program product is provided, comprising computer programs / instructions, wherein the computer programs / instructions, when executed by a processor, implement the steps of the aforementioned task scheduling method or another task scheduling method. The task scheduling method provided in one or more embodiments of this specification is applied to a task processing system. The task processing system includes a task scheduling node and multiple task processing nodes. A first task scheduler is deployed in the task scheduling node, and a second task scheduler is deployed in each task processing node. The target second task scheduler, upon determining that the target task processing node currently executing the target task meets resource reporting conditions, determines the target running resources of the target task processing node and sends the target running resources to the first task scheduler. The target second task scheduler is deployed on the target task processing node, which is any one of the multiple task processing nodes. The first task scheduler, upon determining that the target running resources meet task migration conditions, determines a migration task processing node from the multiple task processing nodes and sends the node information of the migration task processing node to the target second task scheduler. The target second task scheduler determines the migration task processing node based on the node information, determines the target task currently being executed by the target task processing node, and the task data of the target task. The target task and the task data are migrated to the migration task processing node in batches for execution, so as to obtain the task processing result of the target task.Specifically, this task scheduling method, through the cooperation of a first task scheduler deployed in each task scheduling node and a second task scheduler deployed in each task processing node, accurately and quickly identifies task processing nodes that meet the task migration conditions, achieving accurate detection of the current running status of task processing nodes. Furthermore, the second task scheduler migrates the target task currently being executed by the target task processing node and its task data to the migration task processing node in batches, thereby rationally scheduling tasks, achieving load balancing among task processing nodes, and avoiding the problem of low resource utilization of task processing nodes due to load imbalance, thus saving a significant amount of computer resources. Figure 1 is a schematic diagram illustrating the application of a task scheduling method according to an embodiment of this specification; Figure 2 is a flowchart illustrating a task scheduling method according to an embodiment of this specification; Figure 3 is a schematic diagram illustrating a scheduling strategy in a task scheduling method according to an embodiment of this specification; Figure 4 is a schematic diagram illustrating multiple scheduling strategies in a task scheduling method according to an embodiment of this specification; Figure 5 is a schematic diagram illustrating a request migration in a task scheduling method according to an embodiment of this specification; Figure 6 is a schematic diagram illustrating a handshake process in a task scheduling method according to an embodiment of this specification; Figure 7 is a flowchart illustrating the processing procedure of a task scheduling method according to an embodiment of this specification; Figure 8 is a flowchart illustrating another task scheduling method according to an embodiment of this specification; Figure 9 is a structural block diagram of a computing device according to an embodiment of this specification. Detailed Description: Many specific details are set forth in the following description to provide a thorough understanding of this specification. However, this specification can be implemented in many ways other than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific embodiments disclosed below. The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another.For example, without departing from the scope of one or more embodiments of this specification, "first" may also be referred to as "second," and similarly, "second" may also be referred to as "first." Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination." Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In one or more embodiments of this specification, "large model" refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. Large models, also known as foundational models, are pre-trained on large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. These models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and Multimodal Pre-trained Models. In practical applications, large models require only a small number of samples to fine-tune the pre-trained model before being applied to different tasks. Large models can be widely used in Natural Language Processing (NLP), Computer Vision, and other fields. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Major application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. Therefore, in one or more of the following embodiments, the language inference task can be understood as a language inference task performed using a large model. In one or more of the following embodiments, the language inference model and the large language model can be understood as large models.First, the terminology used in one or more embodiments of this specification will be explained. Large Language Model (LLM): refers to a large-scale pre-trained language model, such as the GPT (Generated Pretrained Transformer) series of models. These models typically have a large number of parameters and are trained on massive amounts of text data.

[0002] Pref (prompt) is used to set the timer for the task.

[0003] Decode: Refers to the text that a generative language model generates step by step during the reasoning process. At each step, the model predicts the next most likely token based on the generated text fragments.

[0004] KV Cache: To avoid recalculating the key-value vector every time a token is generated through sampling, pre-calculated k and v values ​​are cached, saving significant computation time during LLM inference. Dynamic Batching: A technique used in large model inference that allows data processing with varying batch sizes. This improves hardware resource utilization, especially for parallel computing devices like GPUs or TPUs, thereby increasing processing efficiency. Load Balancing: Distributing workloads (such as network traffic or requests) across multiple servers or resources to optimize resource usage, maximize throughput, minimize response time, and ensure high system availability. Service Level Objective (SLO): A target used to define and quantify the level of service a service provider promises to its customers. SLOs are typically a set of specific, measurable targets that specify the performance standards the service should achieve, such as response time and throughput. Tail Latency: The phenomenon where a small number of responses in a system have latency higher than the average latency. These responses with above-average latency are called tail latencies. In large-scale distributed systems, tail latency has a particularly severe impact, as the system may need to wait for a tail-latency response before returning to the user. In recent years, large language models (LLMs) have brought generative artificial intelligence to new heights. Their near-human generative capabilities have led to their rapid application across various fields, inspiring numerous imaginings for future applications and profoundly impacting people's lives and work. Unlike general deep learning models, LLMs (i.e., the aforementioned large models) can handle a variety of tasks with specific input text (i.e., prompts) by increasing model parameters, dataset size, and advanced pre-training methods. Therefore, LLMs can serve as a foundation model for various tasks; for example, a large language model can support chatbots, summarization, encoding, AI assistants, and other functions. The diversity of service applications leads to different service requirements from user requests. Request sequence length is an important service requirement. Currently, LLMs support continuously increasing sequence lengths, with the maximum sequence length increasing from 32k to 128k. o This trend will continue because longer sequence lengths allow LLMs to support more tasks. For example, summary tasks require sufficiently long input sequence lengths, while writing tasks require sufficiently long output lengths. Another service requirement is the expected latency of requests. The expected latency of requests varies depending on the task. For example, interactive tasks (such as personal assistants) have lower expected latency than summary tasks.

[0005] LLMs possess characteristics that distinguish them from general deep learning models, necessitating optimization of existing inference service systems to address these characteristics. The first characteristic is workload heterogeneity. By learning as much knowledge as possible from any domain, LLMs are general-purpose models. People can use LLMs to accomplish different tasks and even build applications for various scenarios based on them; this characteristic leads to heterogeneity in user requests, meaning that the input length, output length, and expected latency of requests vary. For example, a task of summarizing long texts might introduce significant input length, and the latency of the first token is more important to the user experience. The second characteristic is execution unpredictability. Responding to LLM requests requires running the model multiple times, with the model iteratively generating tokens; however, it is impossible to know in advance how many tokens will ultimately be generated. o Furthermore, iterative generation incurs significant GPU or CPU memory consumption, which dynamically increases with the number of tokens. Therefore, request execution times and resource requirements are unpredictable. These characteristics inherently make LLM a multi-tenant, dynamic environment, providing heterogeneous and unpredictable workloads across multiple instances. This behavior is similar to modern operating systems hosting processes with dynamic working sets and varying priorities across multiple kernels. Managing such a system presents complex requirements that existing inference service systems cannot meet. Moreover, on the one hand, current LLM inference engines focus on maximizing throughput for a single instance. On the other hand, current service schedulers primarily target non-LLM models, whose requests are homogeneous and execute deterministically step-by-step. Therefore, LLM presents the following challenges to inference service systems:

[0006] 1. As memory consumption becomes unpredictable, the system can hardly provide performance isolation for requests. Memory contention can cause performance interference or even preemption of some requests in a batch, resulting in unstable latency and service level target (SLO) violations, thus greatly impairing the user experience.

[0007] 2. Different request lengths and memory requirements lead to memory fragmentation between instances, introducing conflicting scheduling goals: running requests tend to be load-balanced to reduce preemption, thus increasing fragmentation, while new requests require contiguous space to reduce queuing latency. This conflict is difficult for the scheduler to resolve because the arrival time and length of requests are unpredictable.

[0008] 3. Requests in different applications and scenarios naturally have different latency targets. For example, online chatbots are interactive applications and therefore have strict SLO constraints. Conversely, offline applications (such as evaluation, scoring, or data manipulation) are less sensitive to latency. However, current LLM inference systems typically treat all model requests equally and cannot distinguish their priorities, which limits their ability to satisfy different request SLOs. To address these issues, this specification provides two solutions. The first solution is a language model inference service system that uses dynamic programming to implement a scheduler that is aware of the request input length. The second solution is a large model inference service system that incorporates a hybrid scheduling strategy in pipelined parallelism to improve the throughput of the inference system. However, both solutions have significant drawbacks. The language model inference service system uses dynamic programming to implement a scheduler that is aware of the request input length. The large model inference service system incorporates a hybrid scheduling strategy in pipelined parallelism to improve the throughput of the inference system. However, neither of them considers the heterogeneity of generative large language model request execution and the unpredictability of decoding. Based on this, this specification provides a task scheduling method, and also relates to another task scheduling method, a task processing system, a task scheduling device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments. Referring to Figure 1, Figure 1 shows an application diagram of a task scheduling method provided according to one embodiment of this specification. As shown in Figure 1, in the task scheduling method provided by one or more embodiments of this specification, a user can send an LLM request to a server 104 through a terminal 102. The server 104 is a task processing system, and the server 104 processes the LLM request as follows: First, the global scheduler sends the LLM request to the instance for execution; and the local scheduler configured in the instance can receive the LLM request sent by the global scheduler, thereby executing the LLM request. Secondly, during the execution of LLM requests by an instance, the local scheduler reports the instance's load information to the global scheduler based on the scheduling policy. The global scheduler, upon determining that the current instance is under load based on the instance load information, identifies a target instance for the load instance during the request migration process and sends a request migration instruction to the load instance. Finally, upon receiving the request migration instruction, the instance determines the requests to be migrated from the local scheduler and migrates the requests from the load instance to the target instance for execution, obtaining the request execution result.This task scheduling method reschedules requests to improve load balancing and isolation, alleviate resource fragmentation, and differentiate between request priorities and SLOs. It achieves rescheduling of requests and their memory states using an efficient and scalable migration mechanism. Referring to Figure 2, which shows a flowchart of a task scheduling method according to an embodiment of this specification, this method is applied to a task processing system. The task processing system includes task scheduling nodes and multiple task processing nodes. A first task scheduler is deployed in each task scheduling node, and a second task scheduler is deployed in each task processing node. The task scheduling method specifically includes the following steps: Step 202: Targeting the second task scheduler, if the target task processing node currently executing the target task meets the resource reporting conditions, the target running resources of the target task processing node are determined, and the target running resources are sent to the first task scheduler. The target second task scheduler is deployed on the target task processing node, which can be any one of the multiple task processing nodes. The task processing system can be understood as a system that processes target tasks; the task processing system differs depending on the target task. When the task scheduling method provided in this specification is applied to a language reasoning scenario, the target task is a language reasoning task, and the task processing system can be understood as a language reasoning task processing system. This system effectively performs language reasoning on the user-provided language reasoning question (language reasoning task) to obtain a language reasoning answer that meets the user's needs. When the task scheduling method provided in this specification is applied to an image processing scenario or a text processing scenario, the target task is an image processing task (e.g., image-to-text task, 2D image to 3D image task, etc.) or a text processing task (e.g., text-to-image task, text translation task, etc.). This task processing system can be understood as an image processing task processing system or a text processing task processing system. This system processes the user-provided image (image processing task) or text (text processing task) to meet the user's needs. A task scheduling node can be understood as a node used to schedule the target task. The task scheduling node can be a physical server, a virtual server, or a terminal, without specific limitations. In one or more embodiments provided in this specification, the task processing system includes one task scheduling node and multiple task processing nodes. This task processing cluster consists of task processing nodes and task scheduling nodes. This task processing cluster can be a task processing system. A task processing node can be understood as a node used to process the target task.The task processing node can be a physical server, a virtual server, or a terminal, without specific limitations. In one or more embodiments provided in this specification, the task processing node is configured with a task processing model. The task processing model can be understood as a model for processing the target task, and the task processing model can be set according to the actual application scenario. The task processing model is different depending on the target task. For example, if the target task is a language reasoning task, the model is a language reasoning model, or the aforementioned Large Language Models (LLMs). If the target task is an image processing task or a text processing task, the model is an image processing model used to perform the image processing task; or a text processing model used to perform the text processing task. In one or more embodiments provided in this specification, the task processing node can be an instance. For the explanation of whether the task processing node is a physical server, a virtual server, or a terminal, please refer to the corresponding or relevant content in the explanation of the task processing node as an instance in one or more embodiments provided in this specification, and this specification does not impose specific limitations here. The first task scheduler can be understood as a scheduler deployed in the task scheduling node, which can schedule the target tasks received by the task processing system. Specifically, the first task scheduler can schedule target tasks received by the task processing system to one of multiple task processing nodes. For example, the first task scheduler can be a global scheduler in the task processing system. The second task scheduler can be understood as a scheduler deployed in the task processing nodes. Each task processing node can be configured with a second task scheduler. For example, the second task scheduler can be a local scheduler. In one or more embodiments provided in this specification, the first task scheduler and the second task scheduler can be task scheduling software modules or task scheduling hardware modules. For example, the first task scheduler and the second task scheduler can be task scheduling software modules for scripts, programs, processes, etc., or task scheduling hardware modules for entity plugins, devices, equipment, etc. The application of the task scheduling method provided in this specification in a large language model inference service scenario is used as an example to illustrate the task scheduling method. In this scenario, the task processing system is a language reasoning task processing system, the task processing node can be an instance (which can be called a model instance or model reasoning instance), the first task scheduler can be the global scheduler in the language reasoning task processing system, and the second task scheduler can be the local scheduler in the instance.Based on this, the task scheduling task provided in one or more embodiments of this specification designs a scalable architecture that combines a cluster-level global scheduler and a distributed instance-level local scheduler (i.e., ll um let) to effectively achieve runtime rescheduling. In one or more embodiments provided in this specification, the first task scheduler and the second task scheduler are connected through a narrow interface, and the first task scheduler executes scheduling tasks only based on the runtime resources reported by the second task scheduler. It should be noted that in the task scheduling task provided in one or more embodiments of this specification, a separation point between the two levels is defined, and they are connected through a narrow interface. A "narrow interface" refers to APIs (Application Programming Interfaces) that are simple in design, highly abstract in function, and refined. Such interfaces only provide the most basic function calls and interaction methods, allowing users or developers to access and manage cloud resources in the simplest way, without needing to understand the underlying implementation details. The narrow interface design principle helps to improve the abstraction level and service quality of cloud computing services, and promotes efficient and flexible cloud resource management and application development. Furthermore, the global scheduler in one or more embodiments of this specification makes all scheduling decisions only based on the instance memory (GPU memory or CPU memory) load. In this way, the global scheduler does not need to monitor the status of every running request, which changes frequently during iterative inference execution, thus enabling runtime rescheduling decisions. The load is calculated by Ilumlet based on the request status and the scheduling policy used. The global scheduler uses load information to schedule new requests, trigger cross-instance migrations, and control instance auto-scaling. This global scheduler and the local scheduler together constitute a dynamic scheduler in a large language model inference service. This dynamic scheduler reschedules requests (i.e., target tasks) between model instances. The scheduling of requests by this dynamic scheduler can be compared to context switching between CPU cores in operating system process management. The scheduler can dynamically optimize unpredictable workloads in various ways. The resource reporting conditions can be set according to the actual application scenario; the manual does not specify these conditions. For example, the resource reporting conditions can determine whether the target task processing node is under load or idle. Alternatively, if the second task scheduler receives a resource reporting instruction from the first task scheduler, it determines that the target task processing node meets the resource reporting conditions; otherwise, it determines that the target task processing node does not meet the conditions. The target running resource can be understood as data characterizing the resource usage of the target task processing node; for example, the target running resource could be load information.The target runtime resources include, but are not limited to, GPU memory usage and CPU memory usage. In one or more embodiments provided in this specification, before determining the target runtime resources of the target task processing node when it is determined that the target task processing node currently executing the target task meets the resource reporting conditions, the method further includes: the first task scheduler, upon receiving the target task, determining the target runtime resources of each task processing node; determining an idle task processing node from among the task processing nodes based on the target runtime resources and the target task; and sending the target task to the idle task processing node for execution. Wherein, determining an idle task processing node from among the task processing nodes based on the target runtime resources and the target task can be understood as sorting each task processing node based on the target runtime resources (e.g., ascending or descending order), obtaining a sorting result, and determining an idle task processing node from the sorting result whose runtime resources can handle the target task. Alternatively, based on the target runtime resources, idle task processing nodes can be determined from the task processing nodes. This can be understood as determining task processing nodes whose target runtime resources are greater than or equal to a preset idle resource threshold and using them as idle task processing nodes. Furthermore, if there are multiple idle task processing nodes, any one can be selected. This idle task processing node can be understood as a node with remaining runtime resources capable of executing the target task. Specifically, a user can send a target task to the task processing system via a terminal. After receiving the target task, the task processing system can obtain the target runtime resources of each task processing node through a first task scheduler, and determine idle task processing nodes from the task processing nodes based on the target runtime resources. Then, the target task is sent to the idle task processing node for execution, thereby achieving load balancing by rationally scheduling the target task to the idle task processing node according to the target runtime resources of each task processing node. Following the above example, in one or more embodiments provided in this specification, the scheduling strategy of the task scheduling method aims to achieve the following objectives: First, improve pref2 and decode latency by reducing queuing delays, preemption, and interference. The second objective is to improve cost efficiency by adapting to changes in cluster load. Given that the benefits of rescheduling are also related to cluster load, which can be limited under overload or underload conditions, this solution integrates automatic instance scaling to maintain appropriate cluster load, thereby saving costs and maximizing the benefits of rescheduling.Specifically, the language inference task processing system, upon receiving a language inference task (i.e., an LLM request) sent by a user through a terminal, can determine the load information of each instance through a global scheduler and select an idle instance from multiple instances based on this load information to execute the LLM request (or simply the request). Specifically, one or more task scheduling methods provided in this specification employ a load balancing scheduling strategy for the received LLM request (i.e., the target task). This scheduling strategy distributes the request to the instance with the highest availability. For this availability, a metric is introduced to measure the availability of an instance, defined as F = (MZV) / B, where M is the total memory (i.e., the target runtime resource), V is the virtual usage per request, and B is the batch size. When (MZV) has measured the free space, it is divided by the batch size, as the latter determines the consumption rate, i.e., the number of new decode tokens in an iteration. Therefore, this metric indicates how many more iterations the batch processing can run. Then, each incoming request is distributed to the instance with the highest availability. Because the virtual usage of requests may exceed the physical usage, negative availability can occur, for example, in the presence of queued or high-priority requests. This negative availability helps to automatically treat these instances as overloaded and avoid sending requests to them. Availability metrics are also used to guide migration and automatic scaling. It should be noted that, regarding the automatic scaling strategy: the task scheduling method in one or more embodiments provided in this specification adjusts the number of instances based on the cluster load (i.e., the average memory usage of all instances at normal priority). This strategy keeps the memory usage of instances within the range [x, y], and adds or terminates an instance after a period of time less than x or greater than y, respectively. The scheduler selects the instance with the fewest running requests for scaling down. In one or more embodiments provided in this specification, determining the target running resources of the target task processing node when it is determined that the target task processing node currently executing the target task meets the resource reporting conditions includes: the target second task scheduler determining the current running resources of the target task processing node currently executing the target task; and determining the current running resources as the target running resources when it is determined that the current running resources are greater than or equal to a resource load threshold or less than or equal to a resource idle threshold. Here, the current running resources can be understood as data characterizing the current actual resource usage of the target task processing node; for example, the current running resources can be actual load information. The current running resources include, but are not limited to, GPU memory usage, CPU memory usage, etc.Following the previous example, in one or more embodiments provided in this specification, the unpredictability of LLM request execution is taken into account. In response to an LLM request, the instance needs to run the model multiple times, and the model iteratively generates tokens; however, it is impossible to know in advance how many tokens will ultimately be generated. oFurthermore, iterative generation incurs significant memory consumption, which dynamically increases with the number of tokens. Therefore, request execution time and resource requirements are unpredictable. To address these issues, the local scheduler can employ a load balancing strategy. While the global scheduler can consider container memory usage for load balancing during initial scheduling, the uncertainty of the decode length in LLM requests leads to unpredictable GPU memory consumption during LLM request execution, resulting in ineffective load balancing. A runtime rescheduling of LLM requests using a load balancing strategy is necessary to compensate for poor load balancing by real-time monitoring of actual request memory usage. Based on this, the local scheduler detects the current load status of instances (GPU memory usage or CPU memory usage) by acquiring resource load data for instances currently executing LLM requests and determining whether this resource load data is greater than or equal to a resource load threshold (e.g., 90%) or less than or equal to a resource idle threshold (e.g., 10%). If the resource load data is determined to be greater than or equal to the resource load threshold, then the instance is considered to be under load. In this case, LLM request migration needs to be performed promptly to ensure the smooth execution of LLM requests and avoid inefficient LLM request execution due to instance load. If the resource load data is determined to be less than or equal to the resource idle threshold, then the instance is considered to be in an idle state. The instance's resources are not being utilized to their full potential, leading to resource waste. Therefore, LLM request migration needs to be performed promptly, and the instance should be deleted after the migration is complete to release its resources and improve resource utilization. In one or more embodiments provided in this specification, determining the target running resources of the target task processing node when it is determined that the target task processing node currently executing the target task meets the resource reporting conditions includes: the target second task scheduler determining a priority task from the target task processing nodes currently executing the target task; determining the task running resources of the priority task and the task running resource threshold corresponding to the priority task; and, if the task running resources are greater than or equal to the task running resource threshold, adjusting the current running resources of the target task processing node based on the node resource threshold of the target task processing node to obtain the target running resources. Here, priority task execution can be understood as executing tasks with higher priority. In one or more embodiments provided in this specification, requests under different applications and scenarios naturally have different latency targets.For example, online chatbots are interactive applications and therefore have strict SLO constraints. Conversely, offline applications (such as evaluation, scoring, or data manipulation) are less sensitive to latency. However, given that LLM inference systems typically treat all requests from the model equally and cannot distinguish their priorities, there are limitations in satisfying SLOs for different requests. Based on this, in one or more embodiments provided in this specification, the task scheduling method introduces a new priority objective derived from the new requirements of LLM. Priority provides a systematic approach for serving certain requests with higher urgency (e.g., from more interactive applications) within the same LLM. This approach provides an interface for applications to specify the priorities of different SLOs, including scheduling priority and execution priority. Requests with higher scheduling priority will be scheduled first to reduce queuing latency. Requests with higher execution priority will receive lower instance load to speed up their execution. Currently, this approach supports two priority classes: high priority and normal priority, to demonstrate its ability to favor high-priority requests, but the task scheduling method provided in one or more embodiments of this specification is also designed to accommodate more priorities. Regarding priority: For a request with high execution priority, this solution will reserve a certain amount of space in the instance memory. Refer to Figure 3, which is a schematic diagram of the scheduling strategy in a task scheduling method provided in one embodiment of this specification. Figure 3 shows how the physical and virtual resource usage of an instance is allocated under different scheduling strategies. Based on the content of part (c) in Figure 3, priority is used to prevent the instance running the request from exceeding a given actual load level. This can be achieved by adding a margin (i.e., a task execution resource threshold) to the physical usage of high-priority requests. When there are multiple high-priority requests on an instance, the margin will be allocated among them. The margin for high-priority requests is defined as the margin required to maintain ideal decoding speed (i.e., no visible interference with latency). The margin for normal requests is zero. This solution can also support more execution priorities, depending on the specified margin size. When the margin for high-priority requests is exhausted (i.e., task execution resources are greater than or equal to the task execution resource threshold), the load balancing strategy will migrate other normal requests away because the total virtual usage (i.e., virtual resource usage) of the instance is too large. Therefore, it is necessary to adjust the current actual load information of the instance to the resource threshold to obtain the adjusted load information. Subsequently, the adjusted load information is sent to the global scheduler, thereby triggering the LLM request migration.In Figure 3, part (a) can be understood as request scheduling through a load balancing scheduling strategy; part (b) can be understood as request scheduling through a de-fragmentation scheduling strategy, thereby freeing up more resources for processing requests in the queue; and part (d) can be understood as request scheduling when an instance is scaled down (i.e., the instance is about to be shut down) through a scaling-down scheduling strategy. In one or more embodiments provided in this specification, determining the target running resources of the target task processing node when it is determined that the target task processing node currently executing the target task meets the resource reporting conditions includes: the target second task scheduler determining the resource requirements of the tasks to be executed in the task execution queue, and the current running resources of the target task processing node; and, if the resource requirements are greater than the current running resources, adjusting the current running resources of the target task processing node based on the node resource threshold of the target task processing node to obtain the target running resources. The task execution queue can be understood as a queue where the target task processing node stores tasks to be executed during task execution. After the current task is completed, the target task processing node can retrieve tasks to be executed from the task execution queue. Tasks to be executed can be understood as tasks assigned to the target task processing node for execution but not yet started. Resource requirements can be understood as information representing the runtime resources needed to execute the task to be executed. Continuing the example above, to achieve these goals in a distributed scheduling architecture, this solution introduces a virtual usage abstraction to translate various goals into simple load metrics for instances. This solution notes that the above rescheduling scenarios can be divided into two categories: load balancing and creating contiguous free space (defractionation, prioritization, instance removal). Despite the different goals, it can be assumed that the virtual load on instances unifies them as load balancing. To create free space on an instance, it is only necessary to set the virtual usage of certain requests, making the instance virtually overloaded, which then triggers a load balancing strategy to migrate requests to other instances. This observation allows for the establishment of a simple load balancing strategy as a foundation, combined with a set of rules to set the virtual usage of requests under different conditions. See Figure 3 for details; this solution exemplifies some scenarios in Figure 3. Under normal circumstances, the virtual usage of a request is the same as its physical memory usage, to achieve regular load balancing. During the execution of a request by an instance, requests need to be queued. In this situation, if the instance's remaining runtime resources are insufficient to meet the resource demands of the request at the head of the queue, the instance's current runtime resources can be adjusted to meet the load, thereby triggering request scheduling.After scheduling is complete, the instance can use more runtime resources to process the head-of-line queue request. Specifically, for the head-of-line queue request on the instance, a positive virtual usage (i.e., a node resource threshold) is allocated to reflect its resource requirements, specifically the physical memory needed by pref i II, even though the actual physical usage of the request is zero. Therefore, queuing requests increase the instance's virtual usage (i.e., the target runtime resources obtained after adjusting the current runtime resources of the target task processing node using the node resource threshold). Subsequently, the global scheduler's load balancing strategy triggers migration to achieve load balancing (actually, defragmenting the queue requests), as shown in Figure 3(b). There are many heuristics to explore for setting virtual usage, which controls the trade-off between reducing queuing latency and load balancing. For example, gradually increasing the virtual usage of a queue request until it reaches its actual memory requirement. This scheme currently uses a simple rule: directly using its actual requirement, prioritizing the reduction of queuing latency. This rule is based on the observation in this scheme that queuing latency is the determining factor for end-to-end latency. The evaluation of this scheme also shows that, due to the high flexibility of migration, this rule retains the advantages of load balancing. Based on the content of one or more of the above embodiments, this scheme uses dynamic batching and dynamic memory allocation techniques commonly used in high-throughput large-model inference service systems. In addition, request rescheduling is used to address unpredictable workload dynamics under various circumstances and adjustments are made for different scheduling objectives. Figure 4 is a schematic diagram of multiple scheduling strategies in a task scheduling method provided by one embodiment of this specification. Based on Figure 4, the first strategy is load balancing (part a in Figure 4) to reduce the number of requests on instances with excessive memory consumption. Although load balancing can be considered based on memory usage during initial scheduling, it cannot achieve good results due to the uncertainty of decode length. Rescheduling can compensate for this by real-time sensing of the actual memory usage of requests. At the same time, as mentioned earlier, load balancing also leads to higher memory fragmentation and longer input queue latency. Therefore, the task scheduling method provided by one or more embodiments of this specification also reschedules requests as needed for fragmentation (fragmentation defragmentation in part b of Figure 4), that is, by moving requests to other locations to create contiguous space on instances. Another objective is to reschedule certain requests by priority (part c in Figure 4) to reduce load and avoid disruption. This rescheduling can provide additional dedicated resources to high-priority requests without pre-allocating machines.Finally, this scheme will also reschedule requests during automatic scaling, for example, to terminate instances more quickly (part d in Figure 4) or to saturate new instances more quickly. Step 204: The first task scheduler, when determining that the target running resource meets the task migration conditions, determines a migration task processing node from the plurality of task processing nodes, and sends the node information of the migration task processing node to the target second task scheduler. In one or more embodiments provided in this specification, determining that the target running resource meets the task migration conditions includes: the first task scheduler, when determining that the target running resource is greater than or equal to a resource load threshold, determines that the target running resource meets the task migration conditions; determining the migration task processing node from the plurality of task processing nodes when determining that the target running resource meets the task migration conditions includes: generating a new task processing node based on the target running resource, and determining the new task processing node as the migration task processing node. Here, a new task processing node can be understood as a newly generated task processing node, for example, an instance generated based on an instance scaling strategy. Following the previous example, the task scheduling method provided in this manual requires the global scheduler to scale up or down instances using an automatic scaling scheduling strategy to terminate instances more quickly or saturate new instances more quickly. Based on this, the execution method of the automatic scaling scheduling strategy includes the following for automatic scaling: First, after the global scheduler generates a new instance (adding a task processing node) through automatic scaling, it determines the high-load instance (i.e., the target task processing node) from multiple instances based on the instance load information of each instance (excluding the new instance); it then sends a migration instruction to the high-load instance, instructing it to migrate its executed LLM requests to the new instance. Thus, when the new instance starts, it is automatically saturated using a load balancing strategy. In one or more embodiments provided in this specification, determining that the target running resource meets the task migration conditions includes: the first task scheduler, when determining that the target running resource is less than or equal to a resource idle threshold, determining that the target running resource meets the task migration conditions; after migrating the target task and the task data to the migration task processing node in batches for execution to obtain the task processing result of the target task, the method further includes: the first task scheduler deleting the target task processing node. Following the above example, in order to terminate instances more quickly or saturate new instances more quickly, the task scheduling method provided in this specification requires the global scheduler to scale the instances using an automatic scaling scheduling strategy.Based on this, the execution method of the automatic scaling-up scheduling strategy includes the following for automatic scaling-down: First, the global scheduler, based on the instance load information of each instance, identifies low-load instances (i.e., instances with very low load, such as instances with GPU memory utilization of 10%) from multiple instances; and then, from multiple instances, determines a target instance for this low-load instance during the request migration process; Second, the global scheduler adds a dummy request with unlimited virtual usage to the low-load instance, thereby triggering the low-load instance to perform a request migration operation; Third, the global scheduler sends a migration instruction to the low-load instance, instructing the low-load instance to migrate all LLM requests it is executing to the target instance; Finally, the global scheduler, after determining that the request migration is complete, deletes the low-load instance, thereby releasing the underutilized GPU memory. In one or more embodiments provided in this specification, determining the task processing node to migrate from the plurality of task processing nodes includes: the first task scheduler determining the node running resources of each task processing node; and migrating the task processing node from the plurality of task processing nodes based on the target running resources and the node running resources of the target task processing node. Continuing with the above example, in a language inference task processing system, when it is determined that an LLM request migration is needed, the system can determine the load information of each instance through a global scheduler, and select an idle instance from multiple instances based on the load information as the target instance (i.e., the migration task processing node) to which the LLM request is migrated, for the purpose of migrating the LLM request. After determining the idle instance, the information of the idle instance is sent to the local scheduler to execute the request migration. It should be noted that, regarding this migration strategy: this scheme periodically triggers a migration strategy cycle. In each round, this scheme selects the source and target instances by selecting a candidate set of source and target instances whose degrees of freedom are less than or greater than a given threshold. This scheme pairs instances from two groups by repeatedly selecting the two instances with the lowest and highest degrees of freedom values ​​and setting them to the corresponding migration states. Each source instance's `ll_uml_et` begins continuously requesting migration to the target instance until it is no longer in a migration state. When selecting a request to migrate, `ll_uml_et` prefers requests with lower priority and shorter sequence lengths. In the next round, if the number of instances during the migration period no longer exceeds a threshold, the migration state is canceled and migration stops.Step 206: The target second task scheduler determines the migration task processing node based on the node information, determines the target task currently being executed by the target task processing node and the task data of the target task, and migrates the target task and the task data to the migration task processing node in batches for execution, so as to obtain the task processing result of the target task. In one or more embodiments provided in this specification, the step of determining the target task currently being executed by the target task processing node and the task data of the target task, and migrating the target task and the task data to the migration task processing node in batches for execution to obtain the task processing result of the target task, includes: the target second task scheduler determining the target task currently being executed by the target task processing node and confirming multiple task data of the target task; wherein, the multiple task data are data generated in each stage during the execution of the target task through multiple stages; determining task data to be migrated from the multiple task data, generating a data migration instruction based on the task data to be migrated, and sending the data migration instruction to the migration task processing node; upon receiving a migration accuracy completion instruction sent by the migration task processing node, migrating the task data to be migrated to the migration task processing node; upon receiving a data migration completion instruction sent by the migration task processing node, continuing to execute the step of determining the task data to be migrated from the multiple task data until the multiple task data are migrated to the migration task processing node. Continuing the previous example, migration decisions are not based on specific requests, but on instances. The global scheduler simply pairs source and target instances based on load and sets them to specific states to trigger migration. Ilumlets decide which requests to migrate and executes the migration automatically. Each instance's control module consists of a local scheduler and a migration coordinator. In addition to functions similar to those in existing systems, such as queuing, batch processing, and memory block management, a significant new task for the local scheduler is calculating the instance's memory load. The load is not just the physical memory being used, but the sum of requested virtual memory usage. When triggered, the local scheduler is also responsible for deciding which requests to migrate. Based on the selected requests, the migration coordinator coordinates with the local scheduler and other instances. Rescheduling requests can incur significant time costs and performance penalties. This approach leverages a key feature of LLM inference to address this challenge: the KV cache is append-only. LLM inference iteratively appends the output of the current iteration's token to existing tokens, which are then set as input for the next round.In this way, the inference engine continuously adds the token KV tensor computed in the current round to the KV cache, while the KV tensors generated in previous rounds remain unchanged. This migration mechanism leverages the inherent appending property of the KV cache to parallelize inference computation and KV cache transfer. Since previously generated KV caches will not be modified in subsequent iterations, this scheme can safely transfer previously decoded KV caches while computing for new decodes. oThus, this scheme achieves near-zero and fixed pause times for rescheduling requests. As shown in Figure 5, which is a schematic diagram of multiple request migrations in a task scheduling method provided in one embodiment of this specification, at the start of the migration, the source instance begins transferring the KV cache memory blocks that have completed iterations (this memory block can be understood as the "generated key-value cache" in Figure 5, which can be stored in video memory), while computation (i.e., inference computation in Figure 5) continues (stage 0). When the previous KV cache block transfer is complete, more tokens (i.e., the "generated key-value cache" in Figure 5) are calculated in stage 0. Then it switches to stage 1 to copy the KV cache generated in stage 0 and continues computation thereafter. Transfer is usually much faster than computation, so the number of new blocks is usually small, allowing them to be transferred in a very short time. Finally (stage N), the KV cache transfer time is roughly the same as the time for one iteration computation. Computation of the request is paused by removing it from the current batch and transferring the remaining blocks, completing the transfer of all KV caches. Upon completion, the migration is complete and the request is resumed on the target instance. Although the copying time depends on the sequence length, the pause time for this request is only the KV cache flushing time generated by its Nth iteration, which is negligible regardless of the sequence length. This migration method borrows from the migration concept introduced in virtual machines (VMs), which gradually reduces the working set to minimize downtime. Since the working set is appended but not modified, dirty page tracking is not required during VM migration. However, LLM services present additional challenges. First, both the source and target instances are continuously processing requests, which may exhaust instance memory during migration, leading to request preemption. Second, requests may complete midway through the migration due to unpredictable execution (e.g., generating EOS tokens) and dynamic batching. To handle such exceptions and ensure the correctness of asynchronous computation and memory copying processes, this scheme introduces fine-grained coordination between participating instances, i.e., a handshake process, as shown in Figure 6, which is a schematic diagram of the handshake process in a task scheduling method provided by an embodiment of this specification. Before each stage, the source instance issues a pre-allocation request, specifying the number of target memory blocks to migrate to, to ensure that the target has sufficient space. The target will attempt to allocate and reserve the memory blocks; if successful or unsuccessful, the target will notify the source instance to continue or terminate the migration (i.e., "Accept / Reject" in Figure 6) and clean up its state accordingly.Similarly, after each stage, the source instance checks whether the request being migrated has been completed or preempted. If so, the source instance notifies the target to terminate and releases the reserved memory block; otherwise, the source continues to the next stage. After the last stage is completed, the source releases its local memory block and notifies the target to commit. In one or more embodiments provided in this specification, determining the target task currently being executed by the target task processing node and the task data of the target task includes: determining the other tasks (excluding the priority task) among the multiple tasks currently being executed by the target task processing node as the target task, and determining the task data of the target task; or determining the priority migration task among the multiple tasks currently being executed by the target task processing node as the target task, and determining the task data of the target task. Continuing with the above example, based on the above scheduling strategy, it can be seen that when executing the priority scheduling strategy, the local scheduler can determine normal priority requests (meaning requests that are not high priority requests) in the instance as requests that need to be migrated. In cases where there are multiple normal priority requests, the shorter normal priority request can be prioritized as the request to be migrated. This solution provides an interface for the application to specify the priorities of different SLOs, including scheduling priority and execution priority. Requests with higher scheduling priority will be scheduled first to reduce queuing latency. When scaling down an instance based on a scaling-down scheduling strategy, all requests in the instance can be identified as requests that need to be migrated. When implementing a load balancing strategy, if there is a request with higher scheduling priority among the requests in the instance, then that request with higher scheduling priority is identified as the request that needs to be migrated. If there is no request with higher scheduling priority among the requests in the instance, that is, all requests in the instance have normal scheduling priority, then the shorter normal priority request can be prioritized as the request that needs to be migrated. In one or more embodiments provided in this specification, when the first task scheduler generates a new task processing node, it sends the received new target task to the new task processing node for execution. The new target task can be understood as the target task newly received by the first task scheduler.Based on this, the first task scheduler, upon generating a new task processing node, sends the received new target task to the new task processing node for execution. This can be understood as the first task scheduler sending the newly received target task to the new task processing node for execution, instead of the target task processing node, thereby sharing the task execution burden of the target task processing node and avoiding excessive load on the target task processing node. For example, when a new instance starts, the global scheduler in one or more embodiments provided in this specification will automatically saturate it through a load balancing strategy, that is, sending the newly received target task to the new instance for execution. In one or more embodiments provided in this specification, the first task scheduler, upon determining to delete the target task processing node, sends a load task generated based on preset load running resources to the target task processing node, and upon receiving the target running resources generated based on the load task from the target second task scheduler, determines that the target running resources meet the task migration conditions. It should be noted that, upon determining that the target runtime resources meet the task migration conditions, the first task scheduler and the target second task scheduler can perform a task migration operation on the target task processing node. After the task migration operation is completed, the target task processing node is deleted. The preset load runtime resources can be understood as resources that can cause severe load on the target task processing node and require a target task migration operation. For example, the preset load runtime resources could be 100% memory usage or unlimited virtual memory usage. The load task can be understood as a task that uses the preset load runtime resources as its execution resources. Preset load runtime resources can be added to the load task. When the target task processing node receives this load task, it will mistakenly believe that the runtime resources required to execute the load task are the preset load runtime resources, thus causing the target task processing node's target runtime resources to reach a high load level. Subsequently, when the target runtime resources generated by the load task are sent to the first task scheduler, the first task scheduler will determine that the target runtime resources meet the task migration conditions, thereby triggering the task migration step for the target task processing node. After completing the migration step, the target task processing node can be deleted.For example, when instance termination is required, a fake request (i.e., a load task) with unlimited virtual usage (i.e., preset load running resources) can be added to the instance. This fake request can cause the instance's running resources to reach a high load level. When the local scheduler in the instance sends the high-load running resources to the global scheduler, the global scheduler will trigger a migration operation for that instance, thereby migrating the remaining requests in the instance to other instances. After the request migration is completed, the instance can be deleted; thus, during the deletion of the target task processing node, the target task executed by the target task processing node will not be deleted, avoiding the problem of the target task being unable to execute. In one or more embodiments provided in this specification, the target task is a language inference task sent by the user; after the step of migrating the target task and the task data to the migration task processing node in batches for execution to obtain the task processing result of the target task, it further includes: the migration task processing node sending the language inference result of the language inference task to the user. In one or more embodiments provided in this specification, the task scheduling method, through the cooperation of a first task scheduler deployed in the task scheduling node and a second task scheduler deployed in each task processing node, accurately and quickly identifies task processing nodes that meet the task migration conditions, and accurately detects the current running status of the task processing nodes. Furthermore, the second task scheduler migrates the target task currently being executed by the target task processing node and its task data to the migration task processing node in batches, thereby rationally scheduling tasks, achieving load balancing among the task processing nodes, and avoiding the problem of low resource utilization of each task processing node due to load imbalance, thus saving a significant amount of computer resources. The following description, in conjunction with Figure 7, uses the application of the task scheduling method provided in this specification in a large language model inference service scenario as an example to further illustrate the task scheduling method. Figure 7 shows a flowchart of the processing procedure of a task scheduling method provided in one embodiment of this specification, specifically including the following steps: Step 702: The global scheduler receives a request sent by the user and stores the received new request locally. This request can be an LLM request. Step 704: The global scheduler sends the LLM request to the instance for execution. Specifically, the global scheduler first determines the instance load information of each instance stored locally; this instance load information is actively uploaded by the local scheduler in each embodiment according to a preset time period.It should be noted that, in addition to proactively uploading instance load information according to a preset time period, the local scheduler will also immediately upload instance load information when it determines that an instance meets the scheduling policy of step 708. Next, idle instances (i.e., instances with higher idle resources) are identified from multiple instances. Then, the received LLM requests are sent to the idle instances. Specifically, this scheme uses a load balancing strategy that distributes requests to the instance with the highest availability. Based on this, a metric is introduced to measure the availability of an instance, defined as F = (MZV) / B, where M is the total memory, V is the virtual usage per request, and B is the batch size. When (MZV) has measured the free space, it is divided by the batch size, because the latter determines the consumption rate, i.e., the number of new decodetokens in the iteration. Therefore, this metric indicates how many more iterations the batch processing can run. Then, each incoming request is distributed to the instance with the highest availability. Step 706: The local scheduler in the instance's configured Logic Module (ILM) can receive LLM requests sent by the global scheduler and send the LLM requests to the executor to execute the LLM requests. Specifically, the task scheduling method provided in this specification designs a scalable architecture that combines a cluster-level global scheduler and a distributed instance-level local scheduler (i.e., Logic Module) to effectively implement runtime rescheduling. The global scheduler makes all scheduling decisions based solely on the instance's memory load. This eliminates the need for the global scheduler to monitor the status of every running request, which changes frequently during iterative inference execution, thus enabling runtime rescheduling decisions. The instance's load is calculated by the Logic Module based on the request status and the scheduling policy used, and reported to the global scheduler. Based on this, the local scheduler can receive LLM requests sent by the global scheduler and send the LLM requests to the executor within the instance; the executor, utilizing the GPU resources allocated to the instance, executes the LLM request through the Large Language Model (LLM) within the instance. Step 708: During the execution of an LLM request by an instance, the local scheduler reports the instance's load information to the global scheduler based on the scheduling policy. Since the instance load information is calculated by Ilumet based on the request status and the scheduling policy used, the scheduling policy in the task scheduling method provided in this specification can be of various types, as detailed below. First scheduling policy: Load balancing scheduling policy.During the initial scheduling by the global scheduler, although container memory usage can be considered for load balancing, the uncertainty of the decode length in LLM requests makes GPU memory consumption unpredictable during request execution, resulting in ineffective load balancing. A load balancing scheduling strategy is needed to reschedule LLM requests at runtime, thereby mitigating the problem of poor load balancing by real-time awareness of actual memory usage. Based on this, the load balancing scheduling strategy is executed as follows: First, the local scheduler monitors the instance's GPU physical memory usage (i.e., load information) in real time. Second, if the local scheduler determines that the GPU physical memory usage is greater than or equal to a preset threshold (e.g., 90%), it immediately and proactively sends the instance's GPU physical memory usage to the global scheduler, causing the global scheduler to trigger request migration and relieve the load pressure on the instance. The second scheduling strategy is a priority scheduling strategy. The task scheduling method provided in this specification addresses the issue that LLM inference systems typically treat all model requests equally, leading to a lack of targeted prioritization for LLM requests with high Scheduler Response (SLO) requirements. Therefore, a new priority objective is introduced, stemming from new requirements of LLM. Priority refers to a system approach that provides services for certain requests with higher urgency (e.g., from more interactive applications) within the same LLM. Specifically, an interface is provided to the application to specify the priorities of different SLOs, including scheduling priority and execution priority. Based on this, the priority scheduling strategy is executed as follows: First, the local scheduler, upon receiving a high-priority request from the global scheduler, allocates a certain amount of GPU memory space from the GPU memory resources corresponding to the instance (for executing the high-priority request), thereby preventing the instance running the high-priority request from exceeding the given actual load level. This allocation of execution space can be achieved by adding a margin (i.e., execution space size) to the physical usage of high-priority requests. It should be noted that when there are multiple high-priority requests on an instance, the margin will be distributed among them. The remaining capacity for normal priority requests is zero. Secondly, when the remaining capacity for high-priority requests is exhausted, and the total virtual memory usage of the instance is too high (e.g., 120% GPU memory usage), the local scheduler immediately and proactively sends the instance's total virtual memory usage to the global scheduler. This causes the global scheduler to trigger request migration, moving other normal requests out to ensure the normal execution of high-priority requests.The total virtual usage refers to: the remaining amount allocated to high-priority requests (GPU memory space allocated to high-priority requests) + the GPU memory resources actually used by normal-priority requests. The third scheduling strategy is the defractional scheduling strategy. As mentioned earlier, the above scheduling strategies may lead to higher GPU memory fragmentation and longer input queue latency. Therefore, requests will be rescheduled as needed to address fragmentation issues. This involves moving requests to other instances for execution, thereby creating contiguous GPU memory space on the current instance and defragmenting the queue of requests. Based on this, the defractional scheduling strategy is executed as follows: First, the local scheduler determines the request execution queue in the instance. For the head request in this queue (i.e., the request about to be executed in the queue), the local scheduler allocates a positive virtual usage (e.g., 100% GPU memory utilization) to the head request to reflect its resource requirements (i.e., the physical memory required by the pref i II), even though the actual physical usage of the head request is zero. Finally, because queuing requests increases the virtual usage of instances, leading to excessive virtual usage (e.g., 100% GPU memory usage), the local scheduler immediately and proactively sends the instance's virtual usage to the global scheduler. This causes the global scheduler to trigger request migration, thereby defragmenting the queued requests. Furthermore, the task scheduling method provided in this specification also includes a fourth scheduling strategy: the automatic scaling scheduling strategy. The task scheduling method provided in this specification will reschedule requests during the automatic scaling of instances, thereby terminating instances more quickly or saturating new instances more quickly. Based on this, the execution method of the automatic scaling scheduling strategy is as follows: The scheduling strategy for automatic scaling includes: First, after the global scheduler generates new instances through automatic scaling, it will determine high-load instances (i.e., instances with high load) from multiple instances based on the instance load information of each instance (excluding the new instance); Second, it sends migration instructions to the high-load instances, instructing them to migrate their executed LLM requests to the new instances. Therefore, when a new instance starts, it is automatically saturated through a load balancing strategy.The scheduling strategy for automatic scaling down includes the following steps: First, the global scheduler, based on the instance load information of each instance, identifies low-load instances (i.e., instances with very low load, such as instances with 10% GPU memory utilization) from multiple instances; and then, from multiple instances, determines a target instance for this low-load instance during the request migration process. Second, the global scheduler adds a dummy request with unlimited virtual usage to the low-load instance, thereby triggering the low-load instance to perform a request migration operation. Third, the global scheduler sends a migration instruction to the low-load instance, instructing the low-load instance to migrate all LLM requests it is executing to the target instance. Finally, the global scheduler, upon determining that the request migration is complete, deletes the low-load instance, thereby releasing the underutilized GPU memory. Step 710: The global scheduler, upon determining that the current instance is in a loaded state based on the instance load information, determines a target instance for the loaded instance (an instance in a loaded state) during the request migration process and sends a request migration instruction to the loaded instance. Specifically, in each round, this scheme selects source and target instances by choosing a candidate set of source and target instances whose degrees of freedom (i.e., instance load information) are less than or greater than a given threshold. This scheme repeatedly pairs instances from the two groups by selecting the two instances with the lowest and highest degrees of freedom values ​​and sets them to the corresponding migration states. Step 712: The global scheduler simultaneously sends a request to receive instruction to the target instance, instructing the target instance to receive the data migrated from the load instance. Step 714: The migration coordinator in the instance, upon receiving the request migration instruction, determines the requests that need to be migrated from the local scheduler. It should be noted that the global scheduler only pairs source and target instances based on load and sets them to specific states to trigger migration. Ilumets decides which requests to migrate and automatically executes the migration. The control module (Ilumets) of each instance consists of a local scheduler and a migration coordinator. In addition to calculating the instance's memory load, the local scheduler is also responsible for determining which requests to migrate according to the above scheduling strategy when a request migration is triggered. Based on the selected request, the migration coordinator will coordinate with the local scheduler and other instances, and perform the requested migration according to the requested migration instructions. Based on this, the migration coordinator first determines the LLM requests that need to be migrated from the local scheduler. It should be noted that, based on the above scheduling strategy, when executing a priority scheduling strategy, the local scheduler can determine normal priority requests (meaning requests that are not high priority requests) in the instance as requests that need to be migrated.In cases where there are multiple normal priority requests, the shorter normal priority request can be selected as the request to be migrated. When scaling down an instance based on a scaling-down scheduling strategy, all requests in the instance can be identified as requests to be migrated. When executing a load balancing strategy, if there is a request with a higher scheduling priority among the instance's requests, that request with the higher scheduling priority is identified as the request to be migrated. If there is no request with a higher scheduling priority among the instance's requests, that is, all requests in the instance have normal scheduling priorities, then the shorter normal priority request can be selected as the request to be migrated. Next, the request data for the LLM requests to be migrated is determined from the executor. Step 716: Migrate the requests in the load instance to the target instance for execution and obtain the request execution results. First, at the start of the migration, the source instance (i.e., the load instance) will begin transferring the KVcache memory blocks that have completed iterations, and computation can continue (Phase 0). Second, when the previous KVcache block transfer is complete, more tokens will be calculated in Phase 0. Then it switches to Phase 1 and replicates the KVcache generated in Phase 0, and continues computation in Phase 1 thereafter. This continues until all tokens from all phases (Phase-N) have been migrated to the target instance, thus completing the request migration. Finally, the migration request is executed in the target instance to obtain the execution result corresponding to the LLM request. It should be noted that, to handle such exceptions and ensure the correctness of asynchronous computation and memory copying, this scheme introduces fine-grained coordination, i.e., a handshake process, between participating instances. Before each phase, the source instance issues a pre-allocation request, specifying the number of target memory blocks to migrate to, ensuring the target has sufficient space. The target will attempt to allocate and reserve memory blocks; if successful or unsuccessful, the target will notify the source instance to continue or terminate the migration and clean up its state accordingly. Similarly, after each phase, the source instance checks whether the request being migrated has been completed or preempted; if so, the source instance will notify the target to terminate and release the reserved memory blocks; otherwise, the source will continue to the next phase. After the last phase is completed, the source releases its local memory blocks and notifies the target to commit. Based on the above, the task scheduling method provided in one or more embodiments of this specification provides a dynamic scheduler for large language model inference services. This task scheduling method solves the heterogeneity and unpredictability of LLM requests by rescheduling requests among model instances. By rescheduling requests, the inference service system can dynamically optimize unpredictable workloads in various ways.This method improves tail latency by several orders of magnitude and saves significant GPU resources under similar tail latency by accelerating the processing efficiency of high-priority requests. Furthermore, this task scheduling method reschedules requests to improve load balancing and isolation, alleviate resource fragmentation, and differentiate between request priorities and SLO. It achieves rescheduling of requests and their memory states using an efficient and scalable migration mechanism, and applies it to a dynamic scheduling strategy, unifying various rescheduling scenarios. Referring to Figure 8, Figure 8 shows a flowchart of another task scheduling method according to an embodiment of this specification. This task scheduling method is applied to a second task scheduler deployed in multiple task processing nodes, and includes: Step 802: If the target task processing node currently executing the target task meets the resource reporting conditions, determine the target running resources of the target task processing node, wherein the target second task scheduler is deployed on the target task processing node, and the target task processing node is any one of the multiple task processing nodes; Step 804: Send the target running resources to a first task scheduler, wherein the first task scheduler is deployed on a task scheduling node; Step 806: Receive node information of a migrating task processing node sent by the first task scheduler, wherein the migrating task processing node is determined by the first task scheduler from the multiple task processing nodes after determining that the target running resources meet the task migration conditions; Step 808: Determine the migrating task processing node based on the node information, determine the target task currently being executed by the target task processing node, and the task data of the target task; Step 810: The target task and its data are migrated in batches to the migration task processing nodes for execution to obtain the task processing results of the target task. Another task scheduling method in one or more embodiments of this specification, through the cooperation of a first task scheduler deployed in a task scheduling node and a second task scheduler deployed in each task processing node, accurately and quickly identifies task processing nodes that meet the task migration conditions, achieving accurate detection of the current running status of the task processing nodes; furthermore, the second task scheduler migrates the target task currently being executed by the target task processing node and its task data in batches to the migration task processing nodes for execution, thereby rationally scheduling tasks, achieving load balancing among the task processing nodes, and avoiding the problem of low resource utilization of each task processing node due to load imbalance, thus saving a significant amount of computer resources.Corresponding to the above method embodiments, this specification also provides a task processing system embodiment. The system includes a task scheduling node and multiple task processing nodes. A first task scheduler is deployed in the task scheduling node, and a second task scheduler is deployed in each task processing node. The target second task scheduler is configured to, upon determining that the target task processing node currently executing the target task meets resource reporting conditions, determine the target running resources of the target task processing node and send the target running resources to the first task scheduler. The target second task scheduler is deployed on the target task processing node, which is any one of the multiple task processing nodes. The first task scheduler is configured to, upon determining that the target running resources meet task migration conditions, determine a migration task processing node from the multiple task processing nodes and send the node information of the migration task processing node to the target second task scheduler. The target second task scheduler is configured to determine the migration task processing node based on the node information, determine the target task currently being executed by the target task processing node, and determine the task data of the target task. The target task and its data are migrated to the migration task processing nodes in batches for execution to obtain the task processing results of the target task. Optionally, the first task scheduler and the second task scheduler are connected via a narrow interface, and the first task scheduler executes the scheduling task only based on the running resources reported by the second task scheduler. Optionally, the first task scheduler is further configured to, upon receiving the target task, determine the target running resources of each task processing node; based on the target running resources and the target task, determine an idle task processing node from among the task processing nodes; and send the target task to the idle task processing node for execution. Optionally, the first task scheduler is further configured to, if the target running resources are greater than or equal to a resource load threshold, determine that the target running resources meet the task migration conditions; the first task scheduler is further configured to generate a new task processing node based on the target running resources, and determine the new task processing node as the migration task processing node. Optionally, the first task scheduler is further configured to determine that the target running resource meets the task migration conditions if the target running resource is less than or equal to a resource idle threshold; the first task scheduler is further configured to delete the target task processing node.Optionally, the target second task scheduler is further configured to determine the current running resources of the target task processing node currently executing the target task; and, if the current running resources are determined to be greater than or equal to a resource load threshold, or less than or equal to a resource idle threshold, determine the current running resources as the target running resources. Optionally, the target second task scheduler is further configured to determine priority execution tasks from the target task processing nodes currently executing the target task; determine the task running resources of the priority execution tasks, and the task running resource threshold corresponding to the priority execution tasks; and, if the task running resources are greater than or equal to the task running resource threshold, adjust the current running resources of the target task processing node based on the node resource threshold of the target task processing node to obtain the target running resources. Optionally, the target second task scheduler is further configured to determine the resource requirements of tasks to be executed in the task execution queue, and the current running resources of the target task processing node; if the resource requirements are greater than the current running resources, adjust the current running resources of the target task processing node based on the node resource threshold of the target task processing node to obtain the target running resources. Optionally, the first task scheduler is further configured to determine the node running resources of each task processing node; based on the target running resources and the node running resources of the target task processing node, migrate task processing nodes from the task processing nodes. Optionally, the target second task scheduler is further configured to: determine the target task currently being executed by the target task processing node; and confirm multiple task data of the target task; wherein the multiple task data are data generated in each stage during the execution of the target task through multiple stages; determine task data to be migrated from the multiple task data; generate a data migration instruction based on the task data to be migrated; and send the data migration instruction to the migration task processing node; upon receiving a migration accuracy completion instruction from the migration task processing node, migrate the task data to be migrated to the migration task processing node; upon receiving a data migration completion instruction from the migration task processing node, continue executing the step of determining the task data to be migrated from the multiple task data until the multiple task data are migrated to the migration task processing node.Optionally, the target second task scheduler is further configured to identify, from among the multiple tasks currently being executed by the target task processing node, the tasks other than the priority task to be executed as the target task, and to determine the task data of the target task; or to identify, from among the multiple tasks currently being executed by the target task processing node, the priority migration task as the target task, and to determine the task data of the target task. Optionally, the migration task processing node is further configured to send the language inference result of the language inference task to the user. Optionally, the first task scheduler, when generating a new task processing node, sends the received new target task to the new task processing node for execution; and / or the first task scheduler, when determining to delete the target task processing node, sends a load task generated based on preset load running resources to the target task processing node, and upon receiving the target running resources generated based on the load task sent by the target second task scheduler, determines that the target running resources meet the task migration conditions. The task scheduling system provided in one or more embodiments of this specification, through the cooperation between a first task scheduler deployed in a task scheduling node and a second task scheduler deployed in each task processing node, accurately and quickly identifies task processing nodes that meet the task migration conditions, achieving accurate detection of the current running status of task processing nodes. Furthermore, the second task scheduler migrates the target task currently being executed by the target task processing node and its task data to the migration task processing node in batches, thereby rationally scheduling tasks, achieving load balancing among task processing nodes, and avoiding the problem of low resource utilization of task processing nodes due to load imbalance, thus saving a significant amount of computer resources. The above is an illustrative scheme of a task scheduling system according to this embodiment. It should be noted that the technical solution of this task scheduling system and the technical solution of the above-described task scheduling method belong to the same concept. Details not described in detail in the technical solution of the task scheduling system can be found in the description of the technical solution of the above-described task scheduling method.Corresponding to the above method embodiments, this specification also provides a task scheduling device embodiment. This task scheduling device is applied to a second task scheduler deployed in multiple task processing nodes, comprising: a resource determination module, configured to determine the target running resources of the target task processing node when it is determined that the target task processing node currently executing the target task meets resource reporting conditions, wherein the target second task scheduler is deployed on the target task processing node, and the target task processing node is any one of the multiple task processing nodes; a resource sending module, configured to send the target running resources to a first task scheduler, wherein the first task scheduler is deployed on the task scheduling node; an information receiving module, configured to receive node information of a migrating task processing node sent by the first task scheduler, wherein the migrating task processing node is determined by the first task scheduler from the multiple task processing nodes when it is determined that the target running resources meet task migration conditions; a task determination module, configured to determine the migrating task processing node based on the node information, determine the target task currently being executed by the target task processing node, and the task data of the target task; and a task migration module. The task scheduling device is configured to migrate the target task and its data to the migration task processing node in batches for execution, thereby obtaining the task processing result of the target task. One or more embodiments of this specification provide a task scheduling device that, through the cooperation of a first task scheduler deployed in the task scheduling node and a second task scheduler deployed in each task processing node, accurately and quickly identifies task processing nodes that meet the task migration conditions, achieving accurate detection of the current running status of the task processing nodes. Furthermore, the second task scheduler migrates the target task currently being executed by the target task processing node and its task data to the migration task processing node in batches, thereby rationally scheduling tasks, achieving load balancing among the task processing nodes, and avoiding the problem of low resource utilization of each task processing node due to load imbalance, thus saving a significant amount of computer resources. The above is an illustrative scheme of a task scheduling device according to this embodiment. It should be noted that the technical solution of this task scheduling device belongs to the same concept as the technical solution of the other task scheduling method described above. Details not described in detail in the technical solution of the task scheduling device can be found in the description of the technical solution of the other task scheduling method described above. Figure 9 shows a structural block diagram of a computing device 900 according to one embodiment of this specification. Components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. oThe processor 920 is connected to the memory 910 via a bus 930, and the database 950 is used to store data. The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 940 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface. In one embodiment of this specification, the aforementioned components of computing device 900, as well as other components not shown in FIG. 9, may also be interconnected, for example, via a bus. It should be understood that the block diagram of the computing device shown in FIG. 9 is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed. The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 900 can also be a mobile or stationary server.The processor 920 is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the aforementioned task scheduling method or another task scheduling method. The various embodiments in this specification are described in a progressive manner; similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the computing device embodiments, since they are substantially similar to one or more task scheduling method embodiments, the description is relatively simple; relevant details can be referred to in the description of one or more task scheduling method embodiments. An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions, which, when executed by a processor, implement the steps of the aforementioned task scheduling method or another task scheduling method. The various embodiments in this specification are described in a progressive manner; similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the computer-readable storage medium embodiments, since they are substantially similar to one or more task scheduling method embodiments, the description is relatively simple; relevant details can be referred to in the description of one or more task scheduling method embodiments. This specification also provides a computer program product in one embodiment, including a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned task scheduling method or another task scheduling method. The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the aforementioned task scheduling method or another task scheduling method. Details not described in detail in the technical solution of the computer program product can be found in the description of the aforementioned task scheduling method or another task scheduling method. Specific embodiments of this specification have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. The computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms, etc.The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals. It should also be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps may be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. In the above embodiments, the descriptions of each embodiment have different focuses; for parts not described in detail in a certain embodiment, refer to the relevant descriptions of other embodiments. The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this disclosure. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of this disclosure, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

23 claims 1. A task scheduling method applied to a task processing system, the task processing system including a task scheduling node and multiple task processing nodes, wherein a first task scheduler is deployed in the task scheduling node, and a second task scheduler is deployed in each task processing node, wherein... The target second task scheduler, upon determining that the target task processing node currently executing the target task meets the resource reporting conditions, determines the target running resources of the target task processing node and sends the target running resources to the first task scheduler. The target second task scheduler is deployed on the target task processing node, which is any one of the plurality of task processing nodes. The first task scheduler, upon determining that the target running resources meet the task migration conditions, determines a migration task processing node from the plurality of task processing nodes and sends the node information of the migration task processing node to the target second task scheduler. The target second task scheduler, based on the node information, determines the migration task processing node, the target task currently being executed by the target task processing node, and the task data of the target task, and migrates the target task and the task data to the migration task processing node in batches for execution to obtain the task processing result of the target task.

2. The task scheduling method according to claim 1, wherein the first task scheduler and the second task scheduler are connected through a narrow interface, and the first task scheduler executes scheduling tasks only according to the running resources reported by the second task scheduler.

3. The task scheduling method according to claim 1 or 2, further comprising, before determining the target running resources of the target task processing node when it is determined that the target task processing node currently executing the target task meets the resource reporting conditions: Upon receiving the target task, the first task scheduler determines the target running resources for each task processing node. Based on the target operating resources and the target task, an idle task processing node is determined from the task processing nodes; the target task is then sent to the idle task processing node for execution.

4. The task scheduling method according to any one of claims 1-3, wherein determining that the target running resource meets the task migration conditions includes: The first task scheduler determines that the target running resource meets the task migration conditions if it determines that the target running resource is greater than or equal to the resource load threshold. The step of determining the migration task processing node from the plurality of task processing nodes when it is determined that the target running resources meet the task migration conditions includes: the first task scheduler generating a new task processing node based on the target running resources, and determining the new task processing node as the migration task processing node.

5. The task scheduling method according to any one of claims 1-4, wherein determining that the target running resource meets the task migration conditions includes: The first task scheduler determines that the target running resource meets the task migration conditions if it determines that the target running resource is less than or equal to the resource idle threshold. After migrating the target task and the task data to the migration task processing node in batches for execution to obtain the task processing result of the target task, the method further includes: the first task scheduler deleting the target task processing node.

6. The task scheduling method according to any one of claims 1-5, wherein, when it is determined that the target task processing node currently executing the target task meets the resource reporting conditions, the target task processing node is determined to be the target task processing node. Operating resources, including: The target second task scheduler determines the current running resources of the target task processing node currently executing the target task; If it is determined that the current running resource is greater than or equal to the resource load threshold, or the current running resource is less than or equal to the resource idle threshold, the current running resource is determined as the target running resource.

7. The task scheduling method according to any one of claims 1-5, wherein determining the target running resources of the target task processing node when it is determined that the target task processing node currently executing the target task meets the resource reporting conditions includes: The target second task scheduler determines the priority task to be executed from the target task processing nodes currently executing the target task; Determine the task execution resources for the priority task and the task execution resource threshold corresponding to the priority task; if the task execution resources are greater than or equal to the task execution resource threshold, adjust the current execution resources of the target task processing node based on the node resource threshold of the target task processing node to obtain the target execution resources.

8. The task scheduling method according to any one of claims 1-5, wherein determining the target running resources of the target task processing node when it is determined that the target task processing node currently executing the target task meets the resource reporting conditions includes: The target second task scheduler determines the resource requirements of the tasks to be executed in the task execution queue, as well as the current running resources of the target task processing node; If the resource requirement exceeds the current operating resources, the current operating resources of the target task processing node are adjusted based on the node resource threshold of the target task processing node to obtain the target operating resources.

9. The task scheduling method according to any one of claims 1-8, wherein determining the migration task processing node from the plurality of task processing nodes includes: The first task scheduler determines the node running resources of each task processing node; Based on the target running resources and the node running resources of the target task processing node, the task processing node is migrated from each task processing node.

10. The task scheduling method according to any one of claims 1-9, wherein determining the target task currently being executed by the target task processing node and the task data of the target task, and migrating the target task and the task data to the migrating task processing node in batches for execution, to obtain the task processing result of the target task, comprises: The target second task scheduler determines the target task currently being executed by the target task processing node and confirms multiple task data of the target task; wherein, the multiple task data are data generated in each stage during the execution of the target task through multiple stages; it determines the task data to be migrated from the multiple task data, generates a data migration instruction based on the task data to be migrated, and sends the data migration instruction to the migration task processing node; upon receiving a migration accuracy completion instruction from the migration task processing node, it migrates the task data to be migrated to the migration task processing node; upon receiving a data migration completion instruction from the migration task processing node, it continues to execute the step of determining the task data to be migrated from the multiple task data until the multiple task data are migrated to the migration task processing node.

11. The task scheduling method according to any one of claims 1, 2, and 3, wherein determining the target task currently being executed by the target task processing node and the task data of the target task includes: The target second task scheduler identifies the tasks currently being executed by the target task processing node, excluding the priority task, as the target task, and determines the task data of the target task; or the target second task scheduler identifies the priority migration task among the multiple tasks currently being executed by the target task processing node as the target task, and determines the task data of the target task.

12. The task scheduling method according to any one of claims 1-11, wherein the target task is a language inference task sent by a user; after migrating the target task and the task data to the migration task processing node in batches for execution to obtain the task processing result of the target task, the method further includes: The migration task processing node sends the language reasoning result of the language reasoning task to the user.

13. The task scheduling method according to any one of claims 1-12, further comprising: The first task scheduler, upon generating a new task processing node, sends the received new target task to the new task processing node for execution; And / or the first task scheduler, upon determining to delete the target task processing node, sends a load task generated based on a preset load running resource to the target task processing node, and upon receiving the target running resource generated based on the load task from the target second task scheduler, determines that the target running resource meets the task migration conditions.

14. A task scheduling method, applied to a second task scheduler deployed in multiple task processing nodes, comprising: determining the target running resources of the target task processing node when it is determined that the target task processing node currently executing the target task meets the resource reporting conditions, wherein, The target task processing node is any one of the plurality of task processing nodes; the target runtime resource is sent to the first task scheduler, wherein the first task scheduler is deployed on the task scheduling node; node information of the migration task processing node sent by the first task scheduler is received, wherein the migration task processing node is determined by the first task scheduler from the plurality of task processing nodes after determining that the target runtime resource meets the task migration conditions; the migration task processing node is determined according to the node information, the target task currently being executed by the target task processing node is determined, and the task data of the target task is determined; the target task and the task data are migrated to the migration task processing node in batches for execution to obtain the task processing result of the target task.

15. A task processing system, comprising a task scheduling node and multiple task processing nodes, wherein a first task scheduler is deployed in the task scheduling node, and a second task scheduler is deployed in each task processing node, wherein... A second target task scheduler is configured to, upon determining that the target task processing node currently executing the target task meets the resource reporting conditions, determine the target running resources of the target task processing node and send the target running resources to the first task scheduler. The second target task scheduler is deployed on the target task processing node, which is any one of the plurality of task processing nodes. The first task scheduler is configured to, upon determining that the target running resources meet the task migration conditions, determine the migration task processing node from the plurality of task processing nodes and send the node information of the migration task processing node to the second target task scheduler. The second target task scheduler is configured to, based on the node information, determine the migration task processing node, the target task currently being executed by the target task processing node, and the task data of the target task, and migrate the target task and the task data to the migration task processing node in batches. Line 26 is used to obtain the task processing result of the target task.

16. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the task scheduling method according to any one of claims 1 to 13 or the task scheduling method according to any one of claims 14.

17. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the task scheduling method according to any one of claims 1 to 13 or the task scheduling method according to any one of claims 14.

18. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the task scheduling method of any one of claims 1 to 13 or the task scheduling method of any one of claims 14.

Images