Task scheduling method and system and related equipment

Abstract

The invention provides a task scheduling method and system and related equipment, and can be applied to the technical field of computers. The method comprises the steps that after a target task comprising a plurality of sub-tasks is obtained, computing resources needed for executing each sub-task are obtained, and target computing resources needed for executing the target task are determined according to the computing resources corresponding to each sub-task. Then, available computing resources of each computing cluster in the multiple computing clusters are obtained, a target computing cluster meeting the target computing resources is allocated to the target task according to the available computing resources of each computing cluster, and the target task is sent to the target computing cluster to be executed. Therefore, according to the task scheduling method, the target task can be allocated to the computing cluster with sufficient resources or relatively low resource utilization rate for execution, so that reasonable allocation and utilization of the cluster resources are realized, and the execution efficiency of the target task is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a task scheduling method, system and related equipment. Background Technology

[0002] With the development of the big data era, existing data warehouse systems are becoming increasingly large. Since data warehouses are deployed on computing device clusters, the number of computing device clusters and the number of devices within those clusters will only increase as the data warehouse system expands.

[0003] Existing data warehouses need to process and analyze massive amounts of job packages every day. Specifically, the data warehouse management system allocates each job package to the corresponding computing devices for execution, which mainly consists of two stages:

[0004] (1) First stage: The management system allocates the received Job packages.

[0005] The management system requires that all query requests (such as Structured Query Language (SQL) statements) in the received Job package access the same computing device cluster. In this way, after receiving the Job package, the management system will directly put the Job package into the task queue of the computing device cluster to be accessed, and the computing device cluster will select a computing device from all the computing devices under its management to execute the Job package according to its own task management mechanism.

[0006] (2) Second stage: The management system redistributes the Job packages according to security requirements.

[0007] The management system monitors the resources of computing devices. If a central processing unit (CPU) or disk overload occurs, the management system issues a task cleanup command to the computing device cluster containing that device. Upon receiving the cleanup command, the computing device cluster cleans up the job packages on all the computing devices it manages and then reassigns job packages to all computing devices for execution.

[0008] Although the management system's allocation mode can complete the allocation of all job packages, it will result in some computing device clusters having very long task queues and others having very short task queues, making it difficult to effectively utilize computing device resources. As a result, job packages cannot make full use of available computing device resources for efficient execution, leading to low job package execution efficiency. Summary of the Invention

[0009] To address the aforementioned technical problems, this application provides a task scheduling method, system, and related equipment, which can allocate tasks to computing clusters with sufficient resources or low resource utilization, thereby achieving reasonable allocation and utilization of cluster resources and improving task execution efficiency.

[0010] Firstly, a task scheduling method is provided. This method includes: obtaining a target task comprising multiple subtasks; obtaining the computing resources required to execute each subtask; determining the target computing resources required to execute the target task based on the computing resources corresponding to each subtask; then obtaining the available computing resources of each of the multiple computing clusters; allocating a target computing cluster that meets the target computing resources for the target task based on the available computing resources of each computing cluster; and sending the target task to the target computing cluster for execution.

[0011] In the above scheme, the target computing resources required to execute the target task are first determined based on the subtasks in the target task. Then, based on the global cluster resource status, a target computing cluster that meets the target computing resources is allocated to the target task. This allows the target task to be assigned to a computing cluster with sufficient resources or low resource utilization for execution, thereby achieving reasonable allocation and utilization of cluster resources, effectively avoiding resource contention and resource bottlenecks, and improving the execution efficiency of the target task.

[0012] In some possible implementations, obtaining the computational resources required to execute each subtask includes: inputting each subtask into an artificial intelligence (AI) model to obtain the computational resources required to execute each subtask. This AI model is obtained by training using the resource consumption data of historical subtasks.

[0013] In the above scheme, using the resource consumption of historical subtasks to calculate the computing resources required to execute each subtask enables fine-grained calculation of the computing resources needed for each subtask, ensuring accurate and reliable results. This allows for fine-grained and reliable calculation of the target computing resources needed to execute the target task based on the computing resources corresponding to each subtask. When the target computing resources for each target task can be calculated in a fine-grained manner, each target task can be allocated more rationally. This allows multiple computing clusters to accommodate and execute more tasks when global cluster resources are limited, further improving the utilization rate of cluster resources and the execution efficiency of tasks.

[0014] In some possible implementations, the target computing cluster is the first computing cluster whose available computing resources satisfy the target computing resources, or the target computing cluster satisfies the condition that, after subtracting the target computing resources from the available computing resources of the target computing cluster, the difference between the available computing resources of the multiple computing clusters is minimized.

[0015] The above scheme provides two methods for determining the target computing cluster. The first method quickly identifies the target computing cluster corresponding to the target task, improving scheduling efficiency and making it suitable for scenarios involving a large number of tasks. The second method considers the impact of target task allocation on the differences in available computing resources among multiple computing clusters, minimizing these differences to facilitate load balancing.

[0016] In some possible implementations, sending the target task to the target computing cluster for execution includes: allocating a target cluster node that meets the target computing resources based on the available computing resources of each cluster node in the target computing cluster, and sending the target task to the target cluster node for execution.

[0017] In the above scheme, not only can the target task be assigned to the target computing cluster that meets the target computing resources, but the target task can also be further assigned to the target cluster nodes that meet the target computing resources in the target computing cluster, thus achieving more refined task scheduling.

[0018] In some possible implementations, the target computing resources mentioned above include one or more of processor resources, memory resources, disk resources, and network resources. The available computing resources mentioned above include one or more of available processor resources, memory resources, disk resources, and network resources.

[0019] In the above scheme, the specific content of computing resources can be flexibly determined according to user needs, making it suitable for various scenarios.

[0020] Secondly, a task scheduling system is provided, comprising a first acquisition unit, a computing unit, a second acquisition unit, and a sending unit. The first acquisition unit acquires a target task, which includes multiple subtasks. The computing unit acquires the computing resources required to execute each subtask and determines the target computing resources required to execute the target task based on the computing resources corresponding to each subtask. The second acquisition unit acquires the available computing resources of each of the multiple computing clusters. The sending unit allocates a target computing cluster that meets the target computing resources to the target task based on the available computing resources of each computing cluster and sends the target task to the target computing cluster for execution.

[0021] In some possible implementations, the aforementioned computing unit is specifically used to input each subtask into an artificial intelligence (AI) model to obtain the computing resources required to execute each subtask. This AI model is obtained by training using the resource consumption data of historical subtasks.

[0022] In some possible implementations, the target computing cluster is the first computing cluster whose available computing resources satisfy the target computing resources, or the target computing cluster satisfies the condition that, after subtracting the target computing resources from the available computing resources of the target computing cluster, the difference between the available computing resources of the multiple computing clusters is minimized.

[0023] In some possible implementations, the aforementioned sending unit is specifically used to allocate a target cluster node that meets the target computing resources for the target task based on the available computing resources of each cluster node in the target computing cluster, and send the target task to the target cluster node for execution.

[0024] In some possible implementations, the target computing resources mentioned above include one or more of processor resources, memory resources, disk resources, and network resources. The available computing resources mentioned above include one or more of available processor resources, memory resources, disk resources, and network resources.

[0025] Thirdly, a computing system is provided, the computing system including a processor and a memory, the processor being configured to execute instructions stored in the memory to cause the computing system to perform the method as described in any of the first aspects.

[0026] Fourthly, a computer program product containing instructions is provided, which, when executed by a computing device, cause the computing device to perform the method as described in any of the first aspects.

[0027] Fifthly, a computer-readable storage medium is provided, characterized in that it includes computer program instructions, which, when executed by a computing device, perform the method as described in any of the first aspects.

[0028] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0029] Figure 1 This is an architecture diagram of a task scheduling system provided in an embodiment of this application;

[0030] Figure 2 This is a flowchart illustrating a task scheduling method provided in an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of the structure of a computing system provided in this application;

[0032] Figure 4 This is a schematic diagram of another computing system provided in this application;

[0033] Figure 5 This is a schematic diagram of another computing system provided in this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0035] To address the problem of low task execution efficiency caused by low cluster resource utilization, this application provides a task scheduling system. This system calculates the target computing resources required to execute a target task after acquiring it. Next, it obtains the available computing resources of each of multiple computing clusters, allocates a target computing cluster that meets the required computing resources for the target task based on the available resources of each cluster, and then sends the target task to the target computing cluster for execution. This system can allocate target tasks to clusters with sufficient resources or low resource utilization based on the target computing resources and the overall cluster resource status, achieving reasonable allocation and utilization of cluster resources, effectively avoiding resource contention and bottlenecks, and improving the execution efficiency of target tasks.

[0036] See Figure 1 , Figure 1 This is an architecture diagram of a task scheduling system provided in an embodiment of this application. Figure 1 As shown, the architecture includes a client 100, a task scheduling system 200, and a computing cluster 300. Communication between the client 100 and the task scheduling system 200, and between the task scheduling system 200 and the computing cluster 300, can be achieved via wired or wireless means.

[0037] Client 100 generates the target task and sends it to task scheduling system 200. The target task is typically created by the user on client 100 based on business requirements. The target task includes multiple subtasks. Subtasks can be data requests, specifically business-related functional requests (such as annual report generation). To implement business-related functional requests, data stored in computing cluster 300 is usually required. Therefore, data requests can be Structured Query Language (SQL) statements, which retrieve data from computing cluster 300 and complete business-related functional requests based on the retrieved data. Further, data requests include creation requests (such as CREATE statements), query requests (such as SELECT statements), update requests (such as INSERT, UPDATE, and DELETE statements), etc. For ease of understanding, the following description uses SQL statements as an example of data requests. When the data request is an SQL statement, the target task can be a job package used to execute a series of related operations in computing cluster 300. Furthermore, multiple data requests within the target task can be used to access the same computing cluster 300 or different computing clusters 300. The computing cluster 300 accessed by the data request can be identified by the specific syntax, keywords, or the name of the specified data table or computing cluster 300 used in the data request.

[0038] The task scheduling system 200 is used to determine the computing cluster 300 suitable for executing the target task based on the target task, and push the target task to the computing cluster 300.

[0039] Computing cluster 300 is used to store data and execute subtasks within the target task. Computing cluster 300 is a cluster of computing devices, such as an Oracle cluster, Gauss cluster, PostgreSQL database, etc. Computing cluster 300 consists of one or more cluster nodes. Based on their roles and functions within computing cluster 300, cluster nodes can be categorized into compute nodes, storage nodes, and network nodes. Computation nodes are responsible for executing data requests and typically include computing devices, primarily utilizing processors (such as central processing units (CPUs), graphics processing units (GPUs), and data processing units (DPUs) and memory. Storage nodes are responsible for storing data and typically include storage devices. Network nodes are responsible for network communication and data transmission and typically include network devices (such as routers, switches, firewalls, and gateways), handling packet routing, forwarding, filtering, and communication between different networks. These cluster nodes work together to achieve high availability, performance, and reliability for computing cluster 300.

[0040] In some possible implementations, client 100 can be deployed on computing devices or terminal devices. Computing devices are electronic devices used for computing, processing, and storing data, including servers, supercomputers, personal computers, workstations, mobile devices, etc. Terminal devices are electronic devices used to access computing devices, including personal computers, smartphones, handheld devices, tablets, mobile laptops, all-in-one handheld devices, smart conferencing devices, smart advertising devices, etc.

[0041] In some possible implementations, the task scheduling system 200 can be deployed on a computing device.

[0042] In some possible implementations, the computing cluster 300 can be deployed on computing devices and storage devices. Storage devices are used to store data, including hard drives, memory, storage arrays, and so on.

[0043] In some possible implementations, the client 100, task scheduling system 200, and computing cluster 300 can be deployed on different computing devices, or on the same computing device. When the client 100, task scheduling system 200, and computing cluster 300 are deployed on different computing devices, at least two of them can be deployed on different computing devices within the same computing device cluster, or on different computing devices within different computing clusters. The computing device cluster can include multiple of the aforementioned computing devices. The specific deployment can be determined based on the actual application scenario, and this application does not impose specific limitations.

[0044] In some possible implementations, the task scheduling system 200 includes multiple units. For example, in... Figure 1 In the task scheduling system 200, there are a first acquisition unit 210, a calculation unit 220, a second acquisition unit 230, a sending unit 240, and a training unit 250.

[0045] The first acquisition unit 210, the calculation unit 220, the second acquisition unit 230, the sending unit 240, and the training unit 250 can be implemented in software or in hardware. For example, the implementation of the calculation unit 220 will be described below. Similarly, the implementation of the first acquisition unit 210, the second acquisition unit 230, the sending unit 240, and the training unit 250 can refer to the implementation of the calculation unit 220.

[0046] As an example of a software functional unit, computing unit 220 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, computing unit 220 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0047] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0048] As an example of a hardware functional unit, computing unit 220 may include at least one computing device, such as a server. Alternatively, computing unit 220 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0049] The computing unit 220 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the computing devices in the computing unit 220 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the computing devices in the computing unit 220 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0050] The functions of each unit in the task scheduling system 200 are explained below.

[0051] The first acquisition unit 210 is used to acquire the target task and input the target task into the calculation unit 220 and the sending unit 240 respectively.

[0052] The computing unit 220 is used to obtain the computing resources required to execute each subtask in the target task. Specifically, the computing unit 220 is used to input each subtask into the artificial intelligence (AI) model and receive the computing resources required to execute each subtask calculated by the AI ​​model for each subtask. This process can be seen in the following... Figure 2 In the task scheduling method, step S202 involves the computing unit inputting each subtask into the AI ​​model to obtain the computing resources required to execute each subtask.

[0053] The computing unit 220 is also used to determine the target computing resources required to execute the target task based on the computing resources corresponding to each subtask, and input the target computing resources into the sending unit 240. This process can be seen in the following... Figure 2 In the task scheduling method, step S202 is the process by which the computing unit determines the target computing resources required to execute the target task based on the computing resources corresponding to each subtask.

[0054] The second acquisition unit 230 is used to acquire the available computing resources of each of the multiple computing clusters 300, and input the available computing resources of each computing cluster 300 into the sending unit 240. The available computing resources of the computing cluster 300 include the available computing resources of one or more cluster nodes within the computing cluster 300. This process can be described in the following... Figure 2 The execution process of step S203 in the task scheduling method.

[0055] The sending unit 240 is used to allocate a target computing cluster that meets the target computing resources according to the available computing resources of each computing cluster 300, and send the target task to the target computing cluster for execution. The target computing cluster is computing cluster 300. This process can be seen in the following... Figure 2 The execution process of step S204 in the task scheduling method.

[0056] The sending unit 240 is specifically used to allocate target computing clusters that meet the target computing resources for the target task based on the available computing resources of each computing cluster 300, and then allocate target cluster nodes that meet the target computing resources for the target task based on the available computing resources of each cluster node in the target computing cluster, and send the target task to the target cluster node for execution. This process can be seen in the following... Figure 2 In the task scheduling method, step S204 involves the sending unit, after determining the target computing cluster, allocating a target cluster node that meets the target computing resources for the target task based on the available computing resources of each cluster node in the target computing cluster, and then sending the target task to the target cluster node for execution.

[0057] Training unit 250 is used to train the AI ​​model used by computing unit 220. The AI ​​model is obtained after training using the resource consumption of historical subtasks. This process can be seen below. Figure 2 The training process of the AI ​​model in step S202 of the task scheduling method.

[0058] Training unit 250 is also used to update the AI ​​model. This process is similar to that of training the AI ​​model, and can also be seen below. Figure 2 The task scheduling method includes step S202, which describes the training process of the AI ​​model. The AI ​​model is updated based on the resource consumption of each subtask in the actual execution of the target task. Since the number of target tasks can dynamically increase, the number of subtasks also dynamically increases. The training unit 250 continuously updates the AI ​​model using these dynamically added subtasks, further improving the accuracy of the AI ​​model.

[0059] It should be noted that the resource consumption of the historical subtasks used by training unit 250 and the trained AI model can be stored in a storage device 260 (e.g., Figure 1 (As shown), or, stored in multiple storage devices 260. Storage devices 260 can be file systems, storage systems, databases, etc., and this application does not specifically limit their use. Furthermore, as can be seen from the foregoing, the task scheduling system 200 can be deployed in a single computing device or a cluster of computing devices. When the task scheduling system 200 is deployed in a single computing device, the storage device 260 can be deployed in the memory of that computing device. When the task scheduling system 200 is deployed in a cluster of computing devices, the storage device 260 can be deployed in the same or different computing devices as other units, and this application does not specifically limit their use.

[0060] It should be understood that the units in the aforementioned task scheduling system 200 are based on... Figure 1 The first acquisition unit 210 to the training unit 250 of the task scheduling system 200 are used as examples for illustration. In practical applications, the task scheduling system 200 may include more or fewer units, and this application does not make specific limitations.

[0061] It should be understood that the above Figure 1 The architecture of the task scheduling system is illustrated using one client 100 communicating with the task scheduling system 200 and three computing clusters 300 as an example. In practical applications, the number of clients 100 can be multiple, and the number of computing clusters 300 can be more or less; this application does not impose specific limitations. When there are multiple clients 100, one client 100 can generate a target task and send it to the task scheduling system 200, while another client 100 generates another target task and sends it to the task scheduling system 200.

[0062] In summary, the task scheduling system 200 provided in this application first calculates the target computing resources required to execute the target task, then allocates target computing clusters that meet the target computing resources according to the available computing resources of each computing cluster, then sends the target task to the target computing cluster, and finally sends it to a suitable target cluster node within the target computing cluster for execution. Therefore, the task scheduling system 200 can allocate target tasks to computing clusters with sufficient resources or low resource utilization based on the global cluster resource status, realizing the rational allocation and utilization of cluster resources, effectively avoiding resource contention and resource bottlenecks, and improving the execution efficiency of the target task.

[0063] See Figure 2 , Figure 2 This is a flowchart illustrating a task scheduling method provided in an embodiment of this application. Figure 2 As shown in the embodiments of this application, the task scheduling method includes:

[0064] S201: The client sends the target task to the task scheduling system. The target task includes multiple subtasks.

[0065] Accordingly, the first acquisition unit in the task scheduling system receives the target task from the client. The client can be... Figure 1 In the client 100, the task scheduling system can be Figure 1 In the task scheduling system 200, the first acquisition unit can be Figure 1 The first acquisition unit 210 in the task scheduling system 200.

[0066] A target task (such as a Job package) is a unit of work containing multiple subtasks (such as SQL statements) used to execute a series of related operations within a cluster. Furthermore, the multiple subtasks within the target task may access the same computing cluster or different computing clusters. The computing cluster can be... Figure 1 The computing cluster is 300. For example, suppose the target task S includes multiple subtasks such as subtask S1, subtask S2, and subtask S3. It is possible that all subtasks in the target task S (including subtask S1, subtask S2, and subtask S3) access the Oracle cluster, or that subtask S1 accesses the Oracle cluster, subtask S2 accesses the Gauss cluster, and subtask S3 and the remaining subtasks access the PostgreSQL database.

[0067] S202: The computing unit in the task scheduling system obtains the computing resources required for each subtask in the target task, and determines the target computing resources required for the target task based on the computing resources corresponding to each subtask.

[0068] The computing unit can be Figure 1 The computing unit 220 in the task scheduling system 200.

[0069] The computing resources required to execute a subtask describe the specifications and quantity of resources required to execute the subtask while ensuring its successful completion and performance requirements. These resources include one or more of the following: processor resources (such as CPU, GPU, DPU, etc.), memory resources, disk resources, network resources, etc.

[0070] The target computing resources required to execute the target task describe the specifications and quantity of resources required to execute the target task while ensuring its successful completion and performance requirements. These resources include one or more of the following: processor resources (such as CPU, GPU, DPU, etc.), memory resources, disk resources, network resources, etc.

[0071] In some possible implementations, the computing unit obtains the computing resources required to execute each subtask in the target task, including: the computing unit inputs each subtask into the AI ​​model to obtain the computing resources required to execute each subtask.

[0072] The AI ​​model described above is obtained by training using the resource consumption of historical subtasks. Historical subtasks can be, for example, SQL statements, and their resource consumption can be, for example, the resource consumption of a computing node executing an SQL statement, including one or more of the following: processor resources, memory resources, disk resources, network resources, etc. The historical subtasks and their resource consumption are stored in the task scheduling system's storage device. The storage device can be... Figure 1 The storage device 260 in the task scheduling system 200. The computing node can be... Figure 1 The computing cluster has 300 cluster nodes.

[0073] In some possible implementations, the AI ​​model includes a vector representation model and a prediction model; therefore, the training process of the AI ​​model includes training both the vector representation model and the prediction model. Furthermore, the training process of the AI ​​model is executed by a training unit within a task scheduling system. The training unit can be... Figure 1 The training unit 250 in the task scheduling system 200.

[0074] Vector representation models are used to take historical subtasks as training data and map the training data to a vector space, resulting in vectors of the training data. Vector representation models can be feature representation models such as the bag-of-words model or word vector model. Therefore, the process of inputting training data into a vector representation model to obtain the trained vector representation model and the vectors of the training data can be similar to the process of representing the original data as feature vectors or matrices using feature representation models such as the bag-of-words model or word vector model. For the sake of brevity, this will not be elaborated further here.

[0075] The predictive model uses the resource consumption of historical subtasks as labels for the training data. Based on the vector representation of the training data output by the model and the labels, it learns the relationship between the training data and the labels to accurately predict the computational resources required to execute new subtasks. The predictive model can be a decision tree, random forest, or neural network (such as a feedforward neural network or a recurrent neural network). Therefore, the process of inputting the vector of training data and the vector of labels into the predictive model to obtain the trained predictive model can be similar to the process of training a decision tree, random forest, or neural network using a training set. For the sake of brevity, this will not be elaborated further here.

[0076] In one specific implementation, to eliminate the influence of units and improve interpretability, the resource consumption of historical subtasks can first be normalized. Then, the normalized resource consumption of historical subtasks can be used as labels for training the prediction model. In a more specific implementation, a performance baseline can be used to normalize the resource consumption of historical subtasks. Here, the performance baseline refers to the processor resources, memory resources, disk resources, network resources, etc., used by the database to query 1 gigabyte (GB) of data during normal operation.

[0077] Therefore, the trained AI model includes a trained vector representation model and a trained prediction model. After the training unit completes the training of the AI ​​model, it can store the trained AI model in the storage device of the task scheduling system.

[0078] Therefore, the aforementioned computing unit can input each subtask into the trained AI model and receive the computational resources required to execute each subtask, calculated separately by the trained AI model. Specifically, the computing unit inputs a subtask into the trained vector representation model to obtain the vector of that subtask; then, the trained vector representation model inputs the vector of that subtask into the trained prediction model to obtain the label of that subtask; finally, the trained prediction model sends the label of that subtask to the computing unit. The label of the subtask indicates the computational resources required to execute that subtask.

[0079] After acquiring the computing resources required to execute each subtask, the computing unit determines the target computing resources needed to execute the objective task based on the computing resources corresponding to each subtask. Specifically, the computing unit adds up the computing resources corresponding to all subtasks, obtains the sum, and then uses this sum as the target computing resources needed to execute the objective task. To better suit practical applications, the sum can be rounded down to represent the target computing resources as integers, simplifying data processing and calculations and facilitating mathematical operations and comparisons later. Rounding operations can include, for example, flooring and flooring. Flooring indicates rounding a real number to the nearest larger integer. Flooring indicates rounding a real number to the nearest smaller integer.

[0080] The following example continues, using subtasks S1, S2, S3... as subtasks in the target task S, and CPU resources, memory resources, disk resources, and network resources as the computing resources required to execute the subtasks. This example illustrates the process of obtaining the computing resources required to execute subtasks S1, S2, S3... and determining the target computing resources R required to execute the target task S based on the computing resources corresponding to subtasks S1, S2, S3...

[0081] First, input subtasks S1, S2, S3, etc. into the trained AI model, and receive the CPU resources, memory resources, disk resources, and network resources required to execute subtasks S1, S2, S3, etc., calculated by the trained AI model.

[0082] For subtask S1, the trained AI model inputs subtask S1 into the trained vector representation model to obtain the vector X1 of subtask S1. Then, the vector X1 is input into the trained prediction model to obtain the label T1 of subtask S1, specifically T1 = [R11, R12, R13, R14], where R11 represents the CPU resources required to execute subtask S1; R12 represents the memory resources required to execute subtask S1; R13 represents the disk resources required to execute subtask S1; and R14 represents the network resources required to execute subtask S1.

[0083] For subtask S2, the trained AI model inputs subtask S2 into the trained vector representation model to obtain the vector X2 of subtask S2. Then, the vector X2 is input into the trained prediction model to obtain the label T2 of subtask S2, specifically T2 = [R21, R22, R23, R24], where R21 represents the CPU resources required to execute subtask S2; R22 represents the memory resources required to execute subtask S2; R23 represents the disk resources required to execute subtask S2; and R24 represents the network resources required to execute subtask S2.

[0084] For subtask S3, the trained AI model inputs subtask S3 into the trained vector representation model to obtain the vector X3 of subtask S3. Then, the vector X3 is input into the trained prediction model to obtain the label T3 of subtask S3, specifically T3 = [R31, R32, R33, R34], where R31 represents the CPU resources required to execute subtask S3; R32 represents the memory resources required to execute subtask S3; R33 represents the disk resources required to execute subtask S3; and R34 represents the network resources required to execute subtask S3.

[0085] ...

[0086] It should be understood that the process of inputting other subtasks in the target task S into the trained AI model and receiving the CPU resources, memory resources, disk resources, and network resources required to execute other subtasks calculated by the trained AI model is similar to the process described above of inputting subtasks S1, S2, and S3 into the trained AI model respectively and receiving the CPU resources, memory resources, disk resources, and network resources required to execute subtasks S1, S2, and S3 respectively calculated by the trained AI model. For the sake of brevity, it will not be elaborated further here.

[0087] It should be understood that when the prediction model is trained using the resource consumption of the normalized historical subtasks as labels for the training data, labels T1, T2, T3, etc., also represent the normalized computational resources.

[0088] Then, the corresponding elements in labels T1, T2, T3, ... are added together to obtain the sum R0. Specifically, R0 = [R1, R2, R3, R4], where R1 represents the CPU resources required to execute the target task S, R1 = R11 + R21 + R31 + ...; R2 represents the memory resources required to execute the target task S, R2 = R12 + R22 + R32 + ...; R3 represents the disk resources required to execute the target task S, R3 = R13 + R23 + R33 + ...; and R4 represents the network resources required to execute the target task S, R4 = R14 + R24 + R34 + ...

[0089] Finally, the summation result R0 is used as the target computing resource R required to execute the target task S. Alternatively, the summation result R0 is rounded up so that R1, R2, R3, and R4 in R0 are all integers, and then the rounded R0 is used as the target computing resource R required to execute the target task S.

[0090] S203: The second acquisition unit in the task scheduling system acquires the available computing resources of each computing cluster in multiple computing clusters.

[0091] The second acquisition unit can be Figure 1 The second acquisition unit 230 in the task scheduling system 200. The computing cluster can be... Figure 1 The computing cluster is 300.

[0092] A computing cluster is a cluster of computing devices, such as an Oracle cluster, a Gauss cluster, or a PostgreSQL database. A computing cluster consists of multiple cluster nodes. These nodes can be compute nodes, storage nodes, network nodes, etc.

[0093] The available computing resources of a computing cluster include one or more of the following: available processor resources, memory resources, disk resources, network resources, etc. For the sake of brevity, resource pools are used below to describe the available computing resources of a computing cluster. For example, resource pool Z1 describes the available computing resources of computing cluster J1, resource pool Z2 describes the available computing resources of computing cluster J2, and resource pool Z3 describes the available computing resources of computing cluster J3. Furthermore, there is a correspondence between the available computing resources described by the resource pools and the target computing resources. For example, if the target computing resources include CPU resources and memory resources, then the resource pool describes the available CPU resources and memory resources of the computing cluster. If the target computing resources include CPU resources, memory resources, disk resources, and network resources, then the resource pool describes the available CPU resources, memory resources, disk resources, and network resources of the computing cluster.

[0094] In some possible implementations, the second acquisition unit acquires the available computing resources of each computing cluster in multiple computing clusters, including: the second acquisition unit actively collects the available computing resources of each cluster node, and aggregates the available computing resources of cluster nodes belonging to the same computing cluster to obtain the aggregate result as the available computing resources (i.e., resource pool) of the computing cluster.

[0095] The available computing resources of a cluster node can be either unused computing resources or unused computing resources that meet security requirements. When the available computing resources of a cluster node are unused, the available computing resources of the entire computing cluster are considered unused. Similarly, when the available computing resources of a cluster node are unused while meeting security requirements, the available computing resources of the entire computing cluster are considered unused while meeting security requirements. The process by which the second acquisition unit actively collects the available computing resources of a cluster node can be similar to the process of using monitoring software such as Zabbix, Ganglia, Nagios, and Prometheus to monitor the resource utilization of computing devices in real time. For the sake of brevity, this will not be elaborated upon here.

[0096] In a specific implementation, to better suit practical applications, after the second acquisition unit actively collects the available computing resources of each cluster node, it can first round down the available computing resources of the cluster nodes, and then collect the available computing resources of the cluster nodes that have been rounded down and belong to the same computing cluster to obtain the available computing resources of the computing cluster (i.e., the resource pool). This allows the resource pool to be represented by integers, which makes it easier to simplify data processing and calculation when using the resource pool later, and makes it easier to perform mathematical operations and comparisons.

[0097] The following section continues to use computing clusters J1, J2, and J3 as multiple computing clusters, and takes the available computing resources described by the resource pool as available CPU resources, memory resources, disk resources, and network resources as examples to introduce the process of obtaining the resource pool for each computing cluster in multiple computing clusters.

[0098] Assume that computing cluster J1 includes cluster nodes J11, J12 and J13; computing cluster J2 includes cluster nodes J21, J22 and J23; and computing cluster J3 includes cluster nodes J31, J32 and J33.

[0099] First, actively collect the available computing resources of each cluster node.

[0100] For cluster node J11, obtain the available computing resources Z11 of cluster node J11, specifically Z11 = [Z111, Z112, Z113, Z114], where Z111 represents the available CPU resources of cluster node J11; Z112 represents the available memory resources of cluster node J11; Z113 represents the available disk resources of cluster node J11; and Z114 represents the available network resources of cluster node J11.

[0101] For cluster node J12, obtain the available computing resources Z12 of cluster node J12, specifically Z12 = [Z121, Z122, Z123, Z124], where Z121 represents the available CPU resources of cluster node J12; Z122 represents the available memory resources of cluster node J12; Z123 represents the available disk resources of cluster node J12; and Z124 represents the available network resources of cluster node J12.

[0102] For cluster node J13, obtain the available computing resources Z13 of cluster node J13, specifically Z13 = [Z131, Z132, Z133, Z134], where Z131 represents the available CPU resources of cluster node J131; Z132 represents the available memory resources of cluster node J31; Z133 represents the available disk resources of cluster node J31; and Z134 represents the available network resources of cluster node J13.

[0103] For cluster nodes J21, J22, and J23, obtain the corresponding available computing resources Z21, Z22, and Z23 respectively, specifically Z21 = [Z211, Z212, Z213, Z214], Z22 = [Z221, Z222, Z223, Z224], and Z23 = [Z231, Z232, Z233, Z234]. The meanings of each element in Z21, Z22, and Z23 are similar to those of each element in Z11 above. For the sake of brevity, they will not be elaborated further here.

[0104] For cluster nodes J31, J32, and J33, obtain the corresponding available computing resources Z31, Z32, and Z33 respectively, specifically Z31 = [Z311, Z312, Z313, Z314], Z32 = [Z321, Z322, Z323, Z324], and Z33 = [Z331, Z332, Z333, Z334]. The meanings of each element in Z31, Z32, and Z33 are similar to those of each element in Z11 mentioned above. For the sake of brevity, they will not be elaborated further here.

[0105] Subsequently, the available computing resources of cluster nodes belonging to the same computing cluster are aggregated, and the aggregated result is used as the resource pool of the computing cluster.

[0106] Since cluster nodes J11, J12, and J13 belong to computing cluster J1, the corresponding elements in the available computing resources Z11, Z12, and Z13 are added together to obtain the sum Z10, which serves as the resource pool Z1 of computing cluster J1. Specifically, Z10 = [Z101, Z102, Z103, Z104], where Z101 represents the available CPU resources of computing cluster J1, Z101 = Z111 + Z121 + Z131; Z102 represents the available memory resources of computing cluster J1, Z102 = Z112 + Z122 + Z132; Z103 represents the available disk resources of computing cluster J1, Z103 = Z113 + Z123 + Z133; and Z104 represents the available network resources of computing cluster J1, Z104 = Z114 + Z124 + Z134.

[0107] Since cluster nodes J21, J22, and J23 belong to computing cluster J2, the corresponding elements in the available computing resources Z21, Z22, and Z23 are added together to obtain the result Z20, which serves as the resource pool Z2 of computing cluster J2. Specifically, Z20 = [Z201, Z202, Z203, Z204], where Z201 represents the available CPU resources of computing cluster J2, Z201 = Z211 + Z221 + Z231; Z202 represents the available memory resources of computing cluster J2, Z202 = Z212 + Z222 + Z232; Z203 represents the available disk resources of computing cluster J2, Z203 = Z213 + Z223 + Z233; and Z204 represents the available network resources of computing cluster J2, Z204 = Z214 + Z224 + Z234.

[0108] Since cluster nodes J31, J32, and J33 belong to computing cluster J3, the corresponding elements in the available computing resources Z31, Z32, and Z33 are added together to obtain the sum Z30, which serves as the resource pool Z3 for computing cluster J3. Specifically, Z30 = [Z301, Z302, Z303, Z304], where Z301 represents the available CPU resources of computing cluster J3, Z301 = Z311 + Z321 + Z331; Z302 represents the available memory resources of computing cluster J3, Z302 = Z312 + Z322 + Z332; Z303 represents the available disk resources of computing cluster J3, Z303 = Z313 + Z323 + Z333; and Z304 represents the available network resources of computing cluster J3, Z304 = Z314 + Z324 + Z334.

[0109] It should be understood that the number of computing clusters mentioned above is three, each computing cluster includes three cluster nodes, and the resource pool describes the available CPU resources, memory resources, disk resources and network resources. This is just an example. In actual applications, the number of computing clusters can be more or less, the number of cluster nodes in a computing cluster can be more or less, and the available computing resources described in the resource pool can be more or less. No specific limitation is made here.

[0110] S204: The sending unit in the task scheduling system allocates a target computing cluster that meets the target computing resources according to the available computing resources of each computing cluster, and sends the target task to the target computing cluster for execution.

[0111] The sending unit can be Figure 1 The sending unit 240 in the task scheduling system 200.

[0112] In some possible implementations, the sending unit allocates target computing clusters that meet the target computing resources for the target task based on the available computing resources of each computing cluster. Implementation methods 1 and 2 are as follows:

[0113] Implementation Method 1: Select the first computing cluster whose available computing resources meet the target computing resources as the target computing cluster.

[0114] Implementation Method 2: Select the computing cluster with the smallest difference in available computing resources among multiple computing clusters as the target computing cluster, where the remaining available computing resources are the difference between the available computing resources and the target computing resources.

[0115] In implementation method 1 of step S204, each resource pool is matched with the target computing resource. If each available resource in a resource pool satisfies the resource requirements of each resource in the target computing resource, then the resource pool satisfies the target computing resource. In this case, the computing cluster corresponding to the resource pool is designated as the target computing cluster, and the matching of the remaining resource pools with the target computing resource is stopped. The matching order of the resource pools is determined by the user.

[0116] The following section continues, using resource pools Z1, Z2, and Z3 as multiple resource pools, and the target computing resources R required to execute the target task S as an example, to introduce the process of determining the target computing cluster using implementation method 1. The matching order of the resource pools is resource pool Z1, resource pool Z2, and resource pool Z3, respectively.

[0117] For the target computing resource R = [R1, R2, R3, R4] and the resource pool Z1 = [Z101, Z102, Z103, Z104], if Z101≥R1, Z102≥R2, Z103≥R3, and Z104≥R4 are all true, then the computing cluster J1 corresponding to the resource pool Z1 is taken as the target computing cluster; if at least one of Z101≥R1, Z102≥R2, Z103≥R3, and Z104≥R4 is not true, then the resource pool Z2 is matched with the target computing resource R. For resource pool Z2 = [Z201, Z202, Z203, Z204], if Z201≥R1, Z202≥R2, Z203≥R3, and Z204≥R4 are all true, then the computing cluster J2 corresponding to resource pool Z2 is taken as the target computing cluster; if at least one of Z201≥R1, Z202≥R2, Z203≥R3, and Z204≥R4 is not true, then resource pool Z3 is matched with the target computing resource R. For resource pool Z3 = [Z301, Z302, Z303, Z304], if Z301≥R1, Z302≥R2, Z303≥R3, and Z304≥R4 are all true, then the computing cluster J3 corresponding to resource pool Z3 is taken as the target computing cluster; if at least one of Z301≥R1, Z302≥R2, Z303≥R3, and Z304≥R4 is not true, then it means that there is no computing cluster that meets the target computing resources.

[0118] In implementation method 2 of step S204, the minimum difference between multiple resource pools is taken as the first objective function, and the computing cluster whose remaining available computing resources satisfy the first objective function is taken as the target computing cluster. The calculation formula for the first objective function C1 is:

[0119]

[0120] in, This indicates that A is selected to minimize the first objective function C1. i M represents the number of resource pools, M∈N + A i Let A represent the i-th resource pool that satisfies the target computing resources. i =(a p ), a p Indicates resource pool A i The available resources in the p-th item; R represents the target computing resources, R = (r v ), r v A represents the resource requirement of the v-th item in the target computing resource R; i -R indicates resource pool A i The remaining available computing resources after satisfying the target computing resource R, A i -R=(a p -rp ); (A i -R)′ is (A i The result of performing min-max normalization on R) (a p -r p )′ is (a p -r p The result of performing min-max normalization is min((a) p -r p )) represents the minimum value of all available or remaining available resources for the p-th item, max((a p -r p B' represents the maximum value of all available or remaining available resources for the p-th item; j It is B j The result of performing min-max normalization, B j B represents the j-th resource pool in the resource pool group. j =(b q ), b q Indicates resource pool B j Available resources in item q, b′ q It is b q The result of performing min-max normalization, min(b) q ) represents the minimum value of all available or remaining available resources of the q-th item, max(b q ) represents the maximum value of all available or remaining available resources for the q-th item; |(A i -R)′-B′ j | indicates calculation (A) i -R)′-B′ j The model,

[0121] The following section continues using resource pools Z1, Z2, and Z3 as multiple resource pools, and the target computing resource R required to execute the target task S as an example, to introduce the process of determining the target computing cluster using implementation method 2. Here, the target computing resource R = [R1, R2, R3, R4], resource pool Z1 = [Z101, Z102, Z103, Z104], resource pool Z2 = [Z201, Z202, Z203, Z204], and resource pool Z3 = [Z301, Z302, Z303, Z304].

[0122] Assuming that Z101≥R1, Z102≥R2, Z103≥R3, and Z104≥R4 are all true, and Z201≥R1, Z202≥R2, Z203≥R3, and Z204≥R4 are all true, and Z301≥R1 and Z302≥R2 in Z301≥R1, Z302≥R2, Z303≥R3, and Z304≥R4 are not true, then resource pools Z1 and Z2 are both resource pools that satisfy the target computing resources, and resource pool Z3 is a resource pool that does not satisfy the target computing resources.

[0123] Therefore, in the first objective function C1, resource pool A i For resource pool Z1 or resource pool Z2, resource pool B j It can be resource pool Z1, resource pool Z2, or resource pool Z3.

[0124] When resource pool A i When the resource pool is Z1, the specific calculation formula for the first objective function C1 is as follows:

[0125]

[0126] in,

[0127] Z001 min =min((Z101-R1),Z201,Z301),Z001 max =max((Z101-R1),Z201,Z301);

[0128]

[0129] Z002 min =min((Z102-R2),Z202,Z302),Z002 max =max((Z102-R2),Z202,Z302);

[0130]

[0131] Z003 min =min((Z103-R3),Z203,Z303),Z003 max =max((Z103-R3),Z203,Z303);

[0132]

[0133] Z004 min =min((Z104-R4),Z204,Z304),Z004 max =max((Z104-R4),Z204,Z304).

[0134] When resource pool A i When the resource pool is Z2, the specific calculation formula for the first objective function C1 is as follows:

[0135]

[0136] in,

[0137] Z005 min =min(Z101,(Z201-R1),Z301),Z005 max =max(Z101,(Z201-R1),Z301);

[0138]

[0139] Z006 min =min(Z102,(Z202-R2),Z302),Z006 max =max(Z102,(Z202-R2),Z302);

[0140]

[0141] Z007 min =min(Z103,(Z203-R3),Z303),Z007 max =max(Z103,(Z203-R3),Z303);

[0142]

[0143] Z008 min =min(Z104,(Z204-R4),Z304),Z008 max =max(Z104,(Z204-R4),Z304).

[0144] After calculating C1(Z1) and C1(Z2) respectively, if C1(Z1)≤C1(Z2), then the computing cluster J1 corresponding to resource pool Z1 is taken as the target computing cluster; if C1(Z1)>C1(Z2), then the computing cluster J2 corresponding to resource pool Z2 is taken as the target computing cluster.

[0145] It should be understood that the above-described implementation methods 1 and 2 for determining the target computing cluster are merely examples. In practical applications, any implementation method that can allocate target computing clusters that meet the target computing resources for the target task based on the available computing resources of each computing cluster is within the scope of protection of this application.

[0146] After identifying the target computing cluster, the sending unit sends the target task to the target computing cluster for execution. For example, if the target computing cluster is computing cluster J1, then the target task is sent to computing cluster J1.

[0147] Furthermore, the sending unit sends the target task to the target computing cluster for execution, including: after determining the target computing cluster, the sending unit allocates a target cluster node that meets the target computing resources according to the available computing resources of each cluster node in the target computing cluster, and sends the target task to the target cluster node for execution.

[0148] In some possible implementations, the sending unit allocates target cluster nodes that meet the target computing resources to the target task based on the available computing resources of each cluster node in the target computing cluster. This can be done using methods 1 and 2 as described below:

[0149] Method 1: Select the first cluster node in the target computing cluster whose available computing resources meet the target computing resources as the target cluster node.

[0150] Method 2: Select the cluster node in the target computing cluster whose remaining available computing resources satisfy the smallest difference between the available computing resources of multiple cluster nodes as the target cluster node, where the remaining available computing resources are the difference between the available computing resources and the target computing resources.

[0151] In step S204, method 1, the available computing resources of each cluster node in the target computing cluster are matched one by one with the target computing resource. If each available resource of a cluster node satisfies the resource requirements of each resource in the target computing resource, then the available computing resources of that cluster node satisfy the target computing resource, and that cluster node is designated as the target cluster node. The matching of the available computing resources of the remaining cluster nodes with the target computing resource is then stopped. The matching order of the cluster nodes is determined by the user.

[0152] The following section continues using computing cluster J1 as the target computing cluster, comprising cluster nodes J11, J12, and J13, and the target computing resources R required to execute the target task S as an example, to illustrate the process of determining the target cluster nodes using determination method 1. The matching order of the cluster nodes is as follows: cluster node J11, cluster node J12, and cluster node J13.

[0153] For the target computing resources R = [R1, R2, R3, R4], and the available computing resources Z11 = [Z111, Z112, Z113, Z114] of cluster node J11, if Z111≥R1, Z112≥R2, Z113≥R3, and Z114≥R4 are all true, then cluster node J11 is taken as the target cluster node; if at least one of Z111≥R1, Z112≥R2, Z113≥R3, and Z114≥R4 is not true, then the available computing resources Z12 of cluster node J12 are matched with the target computing resources R. For the available computing resources Z12 = [Z121, Z122, Z123, Z124] of cluster node J12, if Z121≥R1, Z122≥R2, Z123≥R3, and Z124≥R4 are all true, then cluster node J12 is taken as the target cluster node; if at least one of Z121≥R1, Z122≥R2, Z123≥R3, and Z124≥R4 is not true, then the available computing resources Z13 of cluster node J13 are matched with the target computing resources R. For the available computing resources Z13 = [Z131, Z132, Z133, Z134] of cluster node J13, if Z131≥R1, Z132≥R2, Z133≥R3, and Z134≥R4 are all true, then cluster node J13 is taken as the target cluster node; if at least one of Z131≥R1, Z132≥R2, Z133≥R3, and Z134≥R4 is not true, it means that there is no cluster node in computing cluster J1 that satisfies the target computing resources.

[0154] In step S204, method 2, the second objective function is to minimize the difference in available computing resources among multiple cluster nodes in the target computing cluster. Cluster nodes in the target computing cluster whose remaining available computing resources satisfy the second objective function are designated as target cluster nodes. The formula for calculating the second objective function C2 is as follows:

[0155]

[0156] in, This indicates that D is selected to minimize the second objective function C2. i K represents the number of cluster nodes in the target computing cluster, K∈N + ;D i D represents the available computing resources of the i-th cluster node in the target computing cluster that satisfy the target computing resources. i =(d p ), d p D represents available computing resources i The available resources in the p-th item; R represents the target computing resources, R = (r v ), r v D represents the resource requirement of the v-th item in the target computing resource R;i -R indicates available computing resources D i The remaining available computing resources, D, after satisfying the target computing resource R. i -R=(d p -r p );(D i -R)′ is (D i -R) is the result of minimum-maximum normalization. (d p -r p )′ is (d p -r p The result of performing min-max normalization is min((d) p -r p )) represents the minimum value of the available or remaining available resources of item p across all cluster nodes in the target computing cluster, max((d p -r p E' represents the maximum value of the available or remaining available resources of item p across all nodes in the target computing cluster; j It is E j The result of performing min-max normalization, E j This represents the available computing resources of the j-th cluster node in the target computing cluster. e q The available computing resources E are represented j Available resources in item q, e′ q It is e q The result of performing min-max normalization, min(e q ) represents the minimum value of the available or remaining available resources of the q-th item across all cluster nodes in the target computing cluster, max(e q ) represents the maximum value of the available or remaining available resources of item q across all cluster nodes in the target computing cluster; |(D i -R)′-E′ j | indicates calculation (D) i -R)′-E′ j The model,

[0157] The following section continues using computing cluster J1 as the target computing cluster, which includes cluster nodes J11, J12, and J13. Using the target computing resources R required to execute the target task S as an example, the process of determining the target cluster nodes using determination method 2 is introduced. Here, the target computing resources R = [R1, R2, R3, R4], the available computing resources Z11 of cluster node J11 = [Z111, Z112, Z113, Z114], the available computing resources Z12 of cluster node J12 = [Z121, Z122, Z123, Z124], and the available computing resources Z13 of cluster node J13 = [Z131, Z132, Z133, Z134].

[0158] Assuming that Z111≥R1, Z112≥R2, Z113≥R3, and Z114≥R4 are all true, and Z121≥R1, Z122≥R2, Z123≥R3, and Z124≥R4 are all true, and Z131≥R1 and Z132≥R2 in Z131≥R1, Z132≥R2, Z133≥R3, and Z134≥R4 are not true, then cluster nodes J11 and J12 are cluster nodes in computing cluster J1 that meet the target computing resources, and cluster node J13 is a cluster node in computing cluster J1 that does not meet the target computing resources.

[0159] Therefore, in the second objective function C2, the available computational resources D i For cluster node J11's available computing resources Z11 or cluster node J12's available computing resources Z12, cluster node E j The available computing resources Z11 for cluster node J11, Z12 for cluster node J12, or Z13 for cluster node J13.

[0160] In available computing resources D i When the available computing resources Z11 for cluster node J11 are given, the specific calculation formula for the second objective function C2 is as follows:

[0161]

[0162] in,

[0163] Z010 min =min((Z111-R1),Z121,Z131),Z010 max =max((Z111-R1),Z121,Z131);

[0164]

[0165] Z020 min=min((Z112-R2),Z122,Z132),Z020 max =max((Z112-R2),Z122,Z132);

[0166]

[0167] Z030 min =min((Z113-R3),Z123,Z133),Z030 max =max((Z113-R3),Z123,Z133);

[0168]

[0169] Z040 min =min((Z114-R4),Z124,Z134),Z040 max =max((Z114-R4),Z124,Z134).

[0170] In available computing resources D i When the available computing resources Z12 for cluster node J12 are given, the specific calculation formula for the second objective function C2 is as follows:

[0171]

[0172] in,

[0173] Z050 min =min(Z111,(Z121-R1),Z131),Z050 max =max(Z111,(Z121-R1),Z131);

[0174]

[0175] Z060 min =min(Z112,(Z122-R2),Z132),Z060 max =max(Z112,(Z122-R2),Z132);

[0176]

[0177] Z070 min =min(Z113,(Z123-R3),Z133),Z070 max =max(Z113,(Z123-R3),Z133);

[0178]

[0179] Z080 min =min(Z114,(Z124-R4),Z134),Z080 max =max(Z114,(Z124-R4),Z134).

[0180] After calculating C2(J11) and C2(J12) respectively, if C2(J11)≤C2(J12), then cluster node J11 is taken as the target cluster node; if C2(J11)>C2(J12), then cluster node J12 is taken as the target cluster node.

[0181] It should be understood that the above-mentioned methods 1 and 2 for determining the target cluster node are merely examples. In practical applications, any implementation method that can allocate target cluster nodes that meet the target computing resources based on the available computing resources of each cluster node in the target computing cluster is within the protection scope of this application.

[0182] After determining the target cluster node, the sending unit sends the target task to the target cluster node for execution. For example, if the target cluster node is cluster node J11, then the target task is sent to cluster node J11.

[0183] In summary, by implementing the embodiments of this application, after receiving the target task, the required target computing resources for executing the target task are first calculated based on the subtasks within the target task. Then, target computing clusters that meet the required computing resources are allocated to the target task according to the available computing resources of each computing cluster. The target task is then pushed to the target computing cluster and finally to a suitable target cluster node within the target computing cluster for execution. Therefore, this technical solution can allocate the target task to computing clusters with sufficient resources or low resource utilization based on the global cluster resource status, achieving reasonable allocation and utilization of cluster resources, effectively avoiding resource contention and resource bottlenecks, and improving the execution efficiency of the target task.

[0184] Furthermore, this technical solution provides two methods for determining the target computing cluster. In the first method, the first computing cluster whose available computing resources satisfy the target computing resources is selected as the target computing cluster. This allows for rapid identification of the target computing cluster corresponding to the target task, improving the scheduling efficiency of the target task and making it suitable for scenarios involving scheduling a large number of tasks. In the second method, the computing cluster with the smallest difference in available computing resources among multiple computing clusters is selected as the target computing cluster. This considers the impact of target task allocation on the differences in available computing resources among multiple computing clusters, minimizing these differences and thus facilitating load balancing among the multiple computing clusters. Additionally, this technical solution also provides two methods for determining the target cluster nodes. The advantages of these two methods are similar to those of the two methods for determining the target computing cluster, and will not be elaborated upon here.

[0185] Furthermore, this technical solution uses the resource consumption of historical subtasks to calculate the computing resources required to execute each subtask. This allows for fine-grained calculation of the computing resources needed for each subtask, ensuring accurate and reliable results. This enables precise and reliable calculation of the target computing resources needed to execute the target task based on the computing resources corresponding to each subtask. When the target computing resources for each target task can be calculated in a fine-grained manner, each target task can be allocated more rationally. This allows multiple computing clusters to accommodate and execute more tasks even with limited global cluster resources, further improving cluster resource utilization and task execution efficiency. Moreover, the specific content of the target computing resources and available computing resources of the computing cluster obtained in this technical solution can be flexibly determined according to user needs, making it applicable to various scenarios.

[0186] See Figure 3 , Figure 3 This is a schematic diagram of the structure of a computing system provided in this application. Figure 3 As shown, the computing system provided in this application includes a computing device 600. The computing device 600 can be a server or a terminal device. The computing device 600 includes a bus 601, a processor 602, a memory 603, and a communication interface 604. The processor 602, the memory 603, and the communication interface 604 communicate with each other via the bus 601. It should be understood that this application does not limit the number of processors and memories in the computing device 600.

[0187] Bus 601 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The Unified Bus is also known as the Lingqu Bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus 601 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 601 may include a path for transmitting information between various components of the computing device 600 (e.g., memory 603, processor 602, communication interface 604).

[0188] The processor 602 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0189] Memory 603 may include volatile memory, such as random access memory (RAM). Processor 602 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 603 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0190] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 603, or two or more types of storage media can be configured to realize the function of memory 603. This application does not limit this.

[0191] The memory 603 stores executable program code, and the processor 602 executes the executable program code to implement the aforementioned functions. Figure 1 The task scheduling system 200 utilizes the functions of the first acquisition unit 210, the calculation unit 220, the second acquisition unit 230, and the sending unit 240 to respectively execute the aforementioned functions. Figure 2 The steps in the task scheduling method are executed by the first acquisition unit, the calculation unit, the second acquisition unit, and the sending unit in the task scheduling system.

[0192] Alternatively, the memory 603 stores executable program code, and the processor 602 executes the executable program code to implement the aforementioned functionality. Figure 1 The task scheduling system 200 performs the functions of the training unit 250, thereby executing the aforementioned... Figure 2 The steps in the task scheduling method are executed by the training unit in the task scheduling system.

[0193] In summary, the memory 603 stores information for performing the aforementioned tasks. Figure 2 The instructions for the task scheduling method. For the sake of brevity, they will not be elaborated here.

[0194] The communication interface 604 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 600 and other computing devices or communication networks.

[0195] See Figure 4 , Figure 4 This is a schematic diagram of another computing system provided in this application. The computing system provided in this application includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0196] like Figure 4 As shown, the computing system includes at least one computing device 600. The memory 603 in one or more computing devices 600 within the computing system may store the same memory for performing the aforementioned tasks. Figure 2 The instructions for the steps executed by the task scheduling system in the task scheduling method.

[0197] In some possible implementations, the memory 603 of one or more computing devices 600 in the computing system may also store memory for executing the aforementioned commands. Figure 2 The task scheduling method involves a subset of instructions executed by the task scheduling system. In other words, a combination of one or more computing devices 600 can jointly execute instructions for performing the aforementioned steps. Figure 2 The instructions for the steps executed by the task scheduling system in the task scheduling method.

[0198] It should be noted that the memory 603 in different computing devices 600 in the computing system can store different instructions, which are used to execute the aforementioned instructions respectively. Figure 1 The task scheduling system 200 has some of the functions. That is, the instructions stored in the memory 603 of different computing devices 600 can realize the functions of one or more units among the first acquisition unit 210, the computing unit 220, the second acquisition unit 230, and the sending unit 240.

[0199] See Figure 5 , Figure 5 This is a schematic diagram of another computing system provided in this application. In some possible implementations, one or more computing devices in the computing system can be connected via a network. The network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 5 As shown, the two computing devices 600A and 600B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this type of possible implementation, the memory 603 in computing device 600A stores instructions for executing the functions of the first acquisition unit 210 and the computing unit 220. The memory 603 in computing device 600B stores instructions for executing the functions of the second acquisition unit 230 and the sending unit 240.

[0200] Figure 5 The connection method of the computing system shown can be that, considering that the task scheduling method provided in this application needs to receive a large number of target tasks, and the target tasks contain a large number of subtasks, the functions implemented by the first acquisition unit 210 and the computing unit 220 are handed over to the computing device 600A for execution.

[0201] It should be understood that Figure 5 The functions of computing device 600A shown can also be performed by multiple computing devices 600. Similarly, the functions of computing device 600B can also be performed by multiple computing devices 600.

[0202] This application also provides another computing system. The interconnections between the computing devices in this system can be similarly referenced. Figure 4 and Figure 5 The connection method of the computing system differs in that the memory 603 in one or more computing devices 600 within this computing system can store the same information used to execute the aforementioned... Figure 2The instructions for the steps executed by the task scheduling system in the task scheduling method.

[0203] In some possible implementations, the memory 603 of one or more computing devices 600 in the computing system may also store memory for executing the aforementioned commands. Figure 2 The task scheduling method involves a set of instructions executed by the task scheduling system. In other words, a combination of one or more computing devices 600 can jointly execute instructions for performing tasks. Figure 2 The instructions for the steps executed by the task scheduling system in the task scheduling method.

[0204] It should be noted that the memory 603 in different computing devices 600 in the computing system can store different instructions, which are used to execute the aforementioned instructions respectively. Figure 1 The task scheduling system 200 has some of the functions. That is, the instructions stored in the memory 603 of different computing devices 600 can realize the functions of one or more units among the first acquisition unit 210, the computing unit 220, the second acquisition unit 230, and the sending unit 240.

[0205] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the aforementioned... Figure 2 The steps executed by the task scheduling system in the task scheduling method.

[0206] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any usable medium that a computing device can store, or a data storage device such as a data center containing one or more usable media. The usable medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that direct a computing device to perform the aforementioned actions. Figure 2 The steps executed by the task scheduling system in the task scheduling method.

[0207] It should be understood that in the embodiments of this application, "when," "...when," and "if" all refer to the device making corresponding processing under certain objective circumstances, and are not time-limited, nor do they require the device to make a judgment action, nor do they imply any other limitations.

[0208] It should be understood that in the embodiments of this application, "simultaneous" does not necessarily require that they occur at the same time, minute, second, or even at the same moment. When the times of occurrence are slightly different, they can also be understood as occurring "simultaneously".

[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A task scheduling method, characterized in that, The method includes: Obtain the target task, which includes multiple subtasks; Obtain the computing resources required to execute each subtask; The target computing resources required to execute the target task are determined based on the computing resources corresponding to each subtask. Obtain the available computing resources for each of the multiple computing clusters; Assign a target computing cluster that meets the target computing resources to the target task based on the available computing resources of each computing cluster; The target task is sent to the target computing cluster for execution.

2. The method according to claim 1, characterized in that, The step of obtaining the computing resources required to execute each subtask includes: Each subtask is input into the artificial intelligence (AI) model to obtain the computing resources required to execute each subtask; The AI ​​model was obtained by training it using the resource consumption of historical subtasks.

3. The method according to claim 1 or 2, characterized in that, The target computing cluster is the first computing cluster whose available computing resources satisfy the target computing resource requirement, or the target computing cluster satisfies the requirement that, after subtracting the target computing resource from the available computing resources of the target computing cluster, the difference between the available computing resources of the plurality of computing clusters is minimized.

4. The method according to any one of claims 1 to 3, characterized in that, Sending the target task to the target computing cluster for execution includes: Assign target cluster nodes that meet the target computing resources to the target task based on the available computing resources of each cluster node in the target computing cluster; The target task is sent to the target cluster node for execution.

5. The method according to any one of claims 1 to 4, characterized in that, The target computing resources include one or more of processor resources, memory resources, disk resources, and network resources, and the available computing resources include one or more of available processor resources, memory resources, disk resources, and network resources.

6. A task scheduling system, characterized in that, include: The system comprises a first acquisition unit, a calculation unit, a second acquisition unit, and a sending unit. The first acquisition unit is used to acquire a target task, which includes multiple subtasks; The computing unit is used to obtain the computing resources required to execute each subtask; The computing unit is also configured to determine the target computing resources required to execute the target task based on the computing resources corresponding to each subtask; The second acquisition unit is used to acquire the available computing resources of each of the multiple computing clusters; The sending unit is configured to allocate a target computing cluster that meets the target computing resources for the target task based on the available computing resources of each computing cluster; The sending unit is further configured to send the target task to the target computing cluster for execution.

7. The system according to claim 6, characterized in that, The computing unit is specifically used to input each subtask into the artificial intelligence (AI) model to obtain the computing resources required to execute each subtask, wherein the AI ​​model is obtained by training using the resource consumption of historical subtasks.

8. The system according to claim 6 or 7, characterized in that, The target computing cluster is the first computing cluster whose available computing resources satisfy the target computing resource requirement, or the target computing cluster satisfies the requirement that, after subtracting the target computing resource from the available computing resources of the target computing cluster, the difference between the available computing resources of the plurality of computing clusters is minimized.

9. The system according to any one of claims 6 to 8, characterized in that, The sending unit is specifically used to allocate a target cluster node that meets the target computing resources according to the available computing resources of each cluster node in the target computing cluster, and send the target task to the target cluster node for execution.

10. The system according to any one of claims 6 to 9, characterized in that, The target computing resources include one or more of processor resources, memory resources, disk resources, and network resources, and the available computing resources include one or more of available processor resources, memory resources, disk resources, and network resources.

11. A computing system, characterized in that, The computing system includes a processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing system to perform the method as described in any one of claims 1 to 5.

12. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1 to 5.

13. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1 to 5.

Images