Task scheduling method and device, and computer device

Abstract

The application relates to a task scheduling method, device and computer equipment. The method comprises the following steps: obtaining task information of each to-be-scheduled task and resource states of each computing node in a computing cluster; wherein the task information of each to-be-scheduled task at least comprises data volume and data distribution information of processing data required by the to-be-scheduled task; determining execution time lengths of each computing node for executing each to-be-scheduled task according to the task information of each to-be-scheduled task and the resource states of each computing node; determining load imbalance degrees corresponding to each candidate scheduling strategy according to the execution time lengths of each computing node for executing each to-be-scheduled task; the candidate scheduling strategy comprises node allocation of each to-be-scheduled task; selecting a target scheduling strategy from each candidate scheduling strategy according to the load imbalance degrees corresponding to each candidate scheduling strategy; and scheduling each to-be-scheduled task based on the target scheduling strategy. The method can improve resource utilization and task processing efficiency.

Description

Technical Field

[0001] This application relates to the field of task processing technology, and in particular to a task scheduling method, apparatus and computer equipment. Background Technology

[0002] With the rapid development of big data, cloud computing, and artificial intelligence technologies, dynamic modeling is increasingly being applied in scientific computing, financial analysis, industrial simulation, and other scenarios. Dynamic modeling typically involves processing large amounts of real-time data and calculating complex mathematical models, and its computational load increases exponentially with model accuracy and data scale. Therefore, computationally intensive tasks such as dynamic modeling have extremely high demands for load balancing.

[0003] The scheduling methods used in the relevant technologies are only suitable for processing static tasks or scenarios with relatively balanced task loads. However, they cannot adapt to the load fluctuations during the execution of computationally intensive tasks, resulting in technical problems such as low resource utilization and low task processing efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a task scheduling method, apparatus, and computer equipment that can improve resource utilization and task processing efficiency in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a task scheduling method, including:

[0006] Obtain the task information for each task to be scheduled and the resource status of each computing node in the computing cluster; wherein, the task information for each task to be scheduled includes at least the amount of data to be processed and the data distribution information.

[0007] Based on the task information of each task to be scheduled and the resource status of each computing node, determine the execution time of each task to be scheduled on each computing node.

[0008] Based on the execution time of each scheduled task on each computing node, the degree of load imbalance corresponding to each candidate scheduling strategy is determined; wherein, the candidate scheduling strategy includes the node allocation of each scheduled task.

[0009] Based on the degree of load imbalance corresponding to each candidate scheduling strategy, select the target scheduling strategy from the candidate scheduling strategies;

[0010] Based on the target scheduling strategy, each task to be scheduled is scheduled.

[0011] In one embodiment, the execution time of each computing node for each scheduled task is determined based on the task information of each scheduled task and the resource status of each computing node. This includes: for each scheduled task, a weighted summation of the data volume and data distribution information of the data to be processed by the scheduled task is performed to obtain the target task load of the scheduled task; for each computing node, the execution time of each scheduled task is determined based on the target task load and corresponding data volume of each scheduled task, as well as the resource status of the computing node.

[0012] In one embodiment, the target task load of the task to be scheduled is obtained by weighted summation of the data volume and data distribution information of the data to be processed by the task to be scheduled, including: weighted summation of the data volume and data distribution information of the task to be scheduled according to the target weight coefficient; wherein, the target weight coefficient is obtained by processing the task information of each historical task of the computing cluster and the task information of each task to be scheduled.

[0013] In one embodiment, the task information of each historical task and the task information of each task to be scheduled in the computing cluster are processed, including: for each historical task, a weighted sum of the data volume and data distribution information of the data to be processed by the historical task is performed according to the initial weight coefficient to obtain the initial task load of the historical task; for each task to be scheduled, a weighted sum of the data volume and data distribution information of the data to be processed by the task to be scheduled is performed according to the initial weight coefficient to obtain the initial task load of the task to be scheduled; a first computational complexity distribution information is determined according to the initial task load of each historical task; a second computational complexity distribution information is determined according to the initial task load of each task to be scheduled; and a target weight coefficient is determined according to the first computational complexity distribution information, the second computational complexity distribution information, and the initial weight coefficient.

[0014] In one embodiment, determining the target weight coefficient based on the first computational complexity distribution information, the second computational complexity distribution information, and the initial weight coefficient includes: determining the relative entropy between the first computational complexity distribution information and the second computational complexity distribution information; if the relative entropy is greater than a preset threshold, adjusting the initial weight coefficient to obtain a new initial weight coefficient, and based on the new initial weight coefficient, returning to execute the operation of weighted summation of the data volume and data distribution information of the historical task to be processed for each historical task according to the initial weight coefficient to obtain the initial task load of the historical task; if the relative entropy is less than or equal to the preset threshold, using the initial weight coefficient as the target weight coefficient.

[0015] In one embodiment, the load imbalance degree corresponding to each candidate scheduling strategy is determined based on the execution time of each scheduled task executed by each computing node. This includes: for each candidate scheduling strategy, determining the task allocation of each computing node corresponding to the candidate scheduling strategy based on the node allocation of each scheduled task in the candidate scheduling strategy; for each computing node, determining the cumulative execution time of the assigned scheduled tasks executed by the computing node based on the execution time of each scheduled task executed by the computing node and the task allocation of the computing node; determining the average execution time based on the cumulative execution time corresponding to each computing node and the total number of computing nodes in the computing cluster; and determining the load imbalance degree corresponding to the candidate scheduling strategy based on the difference between the cumulative execution time of each computing node and the average execution time.

[0016] In one embodiment, selecting a target scheduling strategy from the candidate scheduling strategies based on the degree of load imbalance corresponding to each candidate scheduling strategy includes: selecting the minimum degree of load imbalance from each degree of load imbalance; and determining the candidate scheduling strategy corresponding to the selected minimum degree of load imbalance as the target scheduling strategy.

[0017] In one embodiment, scheduling each task to be scheduled is performed based on a target scheduling strategy, including: scheduling each task to be scheduled based on the target scheduling strategy, and determining the new resource status of each computing node based on the execution resource consumption data of the first task to be scheduled that was scheduled before the target time, for a target time in the scheduling process; adjusting the node allocation of the second task to be scheduled in the target scheduling strategy based on the new resource status of each computing node to obtain a new target scheduling strategy; wherein, the second task to be scheduled is the task to be scheduled other than the first task to be scheduled; and scheduling the second task to be scheduled based on the new target scheduling strategy after the target time.

[0018] Secondly, this application also provides a task scheduling device, comprising:

[0019] The acquisition module is used to acquire the task information of each task to be scheduled and the resource status of each computing node in the computing cluster; wherein, the task information of each task to be scheduled includes at least the amount of data and data distribution information of the data to be processed by the task to be scheduled.

[0020] The first determining module is used to determine the execution time of each computing node for each scheduled task based on the task information of each scheduled task and the resource status of each computing node.

[0021] The second determining module is used to determine the degree of load imbalance corresponding to each candidate scheduling strategy based on the execution time of each task to be scheduled on each computing node; wherein, the candidate scheduling strategy includes the node allocation of each task to be scheduled.

[0022] The selection module is used to select the target scheduling strategy from the candidate scheduling strategies based on the degree of load imbalance corresponding to each candidate scheduling strategy.

[0023] The scheduling module is used to schedule each task to be scheduled based on the target scheduling strategy.

[0024] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0025] Obtain the task information for each task to be scheduled and the resource status of each computing node in the computing cluster; wherein, the task information for each task to be scheduled includes at least the amount of data to be processed and the data distribution information.

[0026] Based on the task information of each task to be scheduled and the resource status of each computing node, determine the execution time of each task to be scheduled on each computing node.

[0027] Based on the execution time of each scheduled task on each computing node, the degree of load imbalance corresponding to each candidate scheduling strategy is determined; wherein, the candidate scheduling strategy includes the node allocation of each scheduled task.

[0028] Based on the degree of load imbalance corresponding to each candidate scheduling strategy, select the target scheduling strategy from the candidate scheduling strategies;

[0029] Based on the target scheduling strategy, each task to be scheduled is scheduled.

[0030] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0031] Obtain the task information for each task to be scheduled and the resource status of each computing node in the computing cluster; wherein, the task information for each task to be scheduled includes at least the amount of data to be processed and the data distribution information.

[0032] Based on the task information of each task to be scheduled and the resource status of each computing node, determine the execution time of each task to be scheduled on each computing node.

[0033] Based on the execution time of each scheduled task on each computing node, the degree of load imbalance corresponding to each candidate scheduling strategy is determined; wherein, the candidate scheduling strategy includes the node allocation of each scheduled task.

[0034] Based on the degree of load imbalance corresponding to each candidate scheduling strategy, select the target scheduling strategy from the candidate scheduling strategies;

[0035] Based on the target scheduling strategy, each task to be scheduled is scheduled.

[0036] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0037] Obtain the task information for each task to be scheduled and the resource status of each computing node in the computing cluster; wherein, the task information for each task to be scheduled includes at least the amount of data to be processed and the data distribution information.

[0038] Based on the task information of each task to be scheduled and the resource status of each computing node, determine the execution time of each task to be scheduled on each computing node.

[0039] Based on the execution time of each scheduled task on each computing node, the degree of load imbalance corresponding to each candidate scheduling strategy is determined; wherein, the candidate scheduling strategy includes the node allocation of each scheduled task.

[0040] Based on the degree of load imbalance corresponding to each candidate scheduling strategy, select the target scheduling strategy from the candidate scheduling strategies;

[0041] Based on the target scheduling strategy, each task to be scheduled is scheduled.

[0042] The aforementioned task scheduling method, apparatus, and computer equipment acquire task information for each task to be scheduled and resource status of each computing node in the computing cluster; determine the execution time for each computing node to execute each task to be scheduled based on the task information for each task to be scheduled and the resource status of each computing node; determine the degree of load imbalance corresponding to each candidate scheduling strategy based on the execution time for each computing node to execute each task to be scheduled; and select a target scheduling strategy from the candidate scheduling strategies based on the degree of load imbalance corresponding to each candidate scheduling strategy. The above scheme, by incorporating the data volume and distribution information of each task to be scheduled, as well as the resource status of each computing node, into the process of determining the target scheduling strategy, schedules each task based on the target scheduling strategy. This overcomes the technical problem that related technologies struggle to respond in real time to load fluctuations of the tasks to be scheduled and significant resource differences between computing nodes. Furthermore, it balances the load of the computing cluster, thereby avoiding idle or overloaded computing nodes. This not only improves the throughput and processing efficiency of computing nodes for the tasks to be scheduled, but also enhances the resource utilization of computing nodes, thus ensuring the stability and accuracy of the execution of the tasks to be scheduled under high load and high concurrency application scenarios. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a diagram illustrating the application environment of a task scheduling method in one embodiment;

[0045] Figure 2 This is a flowchart illustrating a task scheduling method in one embodiment;

[0046] Figure 3 This is a flowchart illustrating the process of determining the target weight coefficient in one embodiment;

[0047] Figure 4 This is a flowchart illustrating the process of determining the target weight coefficient in another embodiment;

[0048] Figure 5 This is a flowchart illustrating the process of determining the degree of load imbalance in one embodiment;

[0049] Figure 6 This is a schematic diagram illustrating the scheduling process for each task to be scheduled in one embodiment;

[0050] Figure 7 This is a flowchart illustrating the task scheduling method in another embodiment;

[0051] Figure 8 This is a structural block diagram of a task scheduling device in one embodiment;

[0052] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0055] In related technologies, load balancing relies on static scheduling strategies or scheduling strategies based on simple scheduling algorithms, such as round-robin allocation based on task queues, fixed resource allocation rules, or scheduling based on task priority. However, for computationally intensive tasks such as dynamic modeling, these methods are difficult to detect frequent fluctuations in task load and differences in computing resources. This can easily lead to some nodes in the computing cluster being overloaded while others are idle, resulting in low resource utilization and affecting the overall system performance.

[0056] Furthermore, the strategies employed by these technologies typically rely on experience or randomly set parameters, lacking real-time sensing of the computational complexity of the task and the resource usage of the computing cluster. Consequently, it is difficult to achieve fine-grained load balancing, leading to reduced task processing efficiency and wasted time and computing power.

[0057] Based on this, embodiments of this application provide a task scheduling method, which can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on the cloud or other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. In this embodiment, server 104 can be a server in a computing cluster used to manage task-related matters (e.g., task scheduling). Optionally, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0058] In one exemplary embodiment, such as Figure 2 As shown, a task scheduling method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S201 to S205. Among them:

[0059] S201, obtain the task information of each task to be scheduled and the resource status of each computing node in the computing cluster.

[0060] Among them, the tasks to be scheduled are computational tasks waiting for the server to allocate computing nodes. For example, a task to be scheduled can be an iteration in model training, a matrix operation, or a data classification process.

[0061] In some embodiments, in response to a scheduling request or scheduling instruction, task information for each task to be scheduled and the resource status of each computing node in the computing cluster are obtained, wherein the scheduling request or scheduling instruction may come from the terminal.

[0062] For example, task information includes multi-dimensional information related to the computational complexity of the task to be scheduled. For instance, the task information for each task to be scheduled includes at least the amount of data to be processed and data distribution information. The amount of data to be processed quantifies the scale of data that the task needs to process; for different tasks, the amount of data can be represented by file size, matrix dimensions, number of training samples, etc. Data distribution information reflects the internal data structure of the task to be scheduled, such as the density of the data distribution, numerical distribution characteristics, and the distribution of data volume in each dimension of the matrix.

[0063] Optionally, the computing cluster can be a cluster of multiple computing nodes connected by a network, which can be used as a whole computing resource pool to execute computing tasks. Each computing node has a processor, storage unit and network interface for executing computing tasks. The processor can include various types of CPUs (Central Processing Units) and various types of GPUs (Graphics Processing Units).

[0064] It should be noted that the computing cluster can be a local computing cluster or a virtual cluster in the cloud. Optionally, when the computing cluster is a local computing cluster, the computing unit can be various types of servers or terminals that integrate processors, storage units, and network interfaces; when the computing cluster is a virtual cluster, the computing node can be a virtual machine, and this application embodiment does not limit this.

[0065] Resource status reflects the resource availability of a computing node. For example, resource status can include network bandwidth status, CPU status, memory status, and GPU status. Specifically, network bandwidth status can include the percentage of available network bandwidth; CPU status can include the percentage of available CPU resources; memory status can include the percentage of available memory; and GPU status can include the percentage of available GPU resources.

[0066] In some embodiments, task information for each scheduled task in the computing task list can be obtained by parsing the task description file from the terminal; and for each computing node, the available network bandwidth ratio, CPU available resource ratio, GPU available resource ratio, and memory available ratio of the computing node can be obtained from the computing cluster's operation monitoring device, and the obtained available network bandwidth ratio, CPU available resource ratio, GPU available resource ratio, and memory available ratio can be integrated to obtain the resource status of the computing node.

[0067] In some embodiments, the resource state of computing node j can be represented in vector form, i.e., the resource state R of computing node j. j It can be expressed as the following formula (1).

[0068] (1)

[0069] in, Indicates the percentage of available CPU resources. Indicates the percentage of available GPU resources, Indicates the percentage of available memory. This indicates the percentage of available network bandwidth.

[0070] S202, based on the task information of each task to be scheduled and the resource status of each computing node, determine the execution time of each task to be scheduled on each computing node.

[0071] In one embodiment, regarding the execution time of the scheduled task i on computing node j, the computation time required for computing node j to process the data of the scheduled task i can be determined first based on the task information of the scheduled task i and the CPU status, memory status, and GPU status of computing node j's resources. Then, based on the amount of data to be processed by the scheduled task i and the network bandwidth of computing node j's resources, the transmission time required to transmit the data to be processed in the scheduled task i to computing node j can be determined. Finally, the execution time of computing node j for the scheduled task i can be obtained by summing the computation time and the transmission time.

[0072] S203, determine the degree of load imbalance corresponding to each candidate scheduling strategy based on the execution time of each scheduled task on each computing node.

[0073] Among them, the degree of load imbalance is used to quantify the uniformity of the distribution of execution time of all computing nodes executing the scheduled tasks after scheduling each scheduled task according to the corresponding candidate scheduling strategy.

[0074] In some embodiments, a larger load imbalance value indicates a greater difference in the total execution time of different computing nodes, which may indicate that some computing nodes in the computing cluster are too idle while others are overloaded; conversely, a smaller load imbalance value indicates a smaller difference in the total execution time of different computing nodes, which may indicate that the idle time of each computing node in the computing cluster is relatively balanced.

[0075] Candidate scheduling strategies can include the node allocation for each task to be scheduled.

[0076] Optionally, gradient descent or genetic algorithms can be used to iteratively generate multiple candidate scheduling strategies, traversing all node allocations for different tasks to be scheduled as much as possible. Furthermore, to save computing power and time, pre-defined scheduling rules can be used to generate multiple candidate scheduling strategies based on the task information of each task to be scheduled and the resource status of each computing node.

[0077] For example, for each candidate scheduling strategy, the load of each computing node can be determined based on the execution time of each scheduled task on each computing node, and the load imbalance corresponding to the candidate scheduling strategy can be determined based on the load of each computing node.

[0078] S204. Select the target scheduling strategy from the candidate scheduling strategies based on the degree of load imbalance corresponding to each candidate scheduling strategy.

[0079] In some embodiments, the load imbalance levels can be sorted in descending order, and the load imbalance levels can be compared with preset thresholds in turn. The candidate scheduling strategy corresponding to the first load imbalance level that is less than the preset threshold is determined as the target scheduling strategy.

[0080] S205 schedules each task to be scheduled based on the target scheduling strategy.

[0081] Optionally, after determining the target scheduling strategy, each task to be scheduled can be assigned to the corresponding computing node according to the node allocation of each task to be scheduled corresponding to the target scheduling strategy, and executed in the computing node.

[0082] The task scheduling method in the above embodiments obtains the task information of each task to be scheduled and the resource status of each computing node in the computing cluster; determines the execution time of each computing node for each task to be scheduled based on the task information of each task to be scheduled and the resource status of each computing node; determines the load imbalance degree corresponding to each candidate scheduling strategy based on the execution time of each computing node for each task to be scheduled; and selects a target scheduling strategy from the candidate scheduling strategies based on the load imbalance degree corresponding to each candidate scheduling strategy. This scheme, by incorporating the data volume and distribution information of the data to be processed by each task to be scheduled and the resource status of each computing node in the process of determining the target scheduling strategy, schedules each task to be scheduled based on the target scheduling strategy. On the one hand, it overcomes the technical problem that related technologies struggle to respond in real-time to load fluctuations of tasks to be scheduled and significant resource differences between computing nodes; on the other hand, it can balance the load of the computing cluster, thereby avoiding idle or overloaded computing nodes. This not only improves the throughput and processing efficiency of computing nodes for tasks to be scheduled but also enhances the resource utilization of computing nodes, thus ensuring the stability and accuracy of task execution under high load and high concurrency application scenarios.

[0083] Based on the above embodiments, in some embodiments, S204 may specifically include: selecting the minimum load imbalance degree from each load imbalance degree; and determining the candidate scheduling strategy corresponding to the selected minimum load imbalance degree as the target scheduling strategy.

[0084] For example, the essence of load imbalance is the cumulative execution time of the computing node with the longest cumulative execution time in the quantified computing cluster and the total execution time of the computing cluster. Therefore, the smaller the value of load imbalance, the smaller the difference in the total execution time of different computing nodes and the faster the total execution time of the computing cluster. This can be interpreted as the idle time of each computing node in the computing cluster being relatively balanced and the total execution time of each task to be scheduled being relatively fast under the current candidate scheduling policy. Therefore, the minimum load imbalance among the various load imbalance degrees can be used as the target scheduling policy.

[0085] In one embodiment, S202 specifically includes: for each task to be scheduled, performing a weighted summation of the amount of data and data distribution information of the data to be processed by the task to be scheduled to obtain the target task load of the task to be scheduled; for each computing node, determining the execution time of each task to be scheduled by the computing node based on the target task load and the corresponding amount of data of each task to be scheduled, as well as the resource status of the computing node.

[0086] Among them, the target task load is used to quantify the computational complexity of the task to be scheduled. The larger the value of the target task load, the higher the computational complexity of the task to be scheduled, and correspondingly, the higher the resource consumption required to execute the task to be scheduled.

[0087] Optionally, for each task to be scheduled, a preset weighting coefficient can be used to perform a weighted summation of the amount of data and data distribution information that the task needs to process, so as to obtain the target task load of the task to be scheduled.

[0088] Furthermore, taking computing node j as an example, based on the target task load of the task i to be scheduled... and the corresponding amount of data and the resource status R of the computing node j Determine the execution time of the scheduled task i on computing node j. The process can be expressed as the following formula (2).

[0089] (2)

[0090] in, This refers to the computation time required for computing node j to process the data of the scheduled task i. To calculate the performance coefficient of node j, To compute the speedup ratio of the GPU in node j, This refers to the transmission time required to transfer the processing data of task i to computing node j, as mentioned above. This represents a coefficient related to network transmission.

[0091] This embodiment calculates the target task load for each scheduled task by weighted summing of the data volume and distribution information required for processing. For each computing node, the execution time for each scheduled task is determined based on its target task load, corresponding data volume, and resource status. Since the target task load is obtained by weighted summing of the data volume and distribution information required for processing, this overcomes the limitation of related technologies that rely solely on data volume for estimation. Furthermore, combining the computing node's resource status with execution time prediction improves the matching accuracy between scheduled tasks and computing nodes, providing a more reliable basis for subsequent target scheduling decisions. This facilitates applications in cloud computing, edge computing, and multi-computing cluster scenarios.

[0092] In one embodiment, the target task load of the task to be scheduled is obtained by weighted summation of the data volume and data distribution information of the data to be processed by the task to be scheduled, including: weighted summation of the data volume and data distribution information of the data to be processed by the task to be scheduled according to the target weight coefficient.

[0093] Optionally, the target task load of task i to be scheduled. The calculation process can be expressed as the following formula (3).

[0094] (3)

[0095] in, Let be the amount of data that task i needs to process. This provides the data distribution information for the data that task i needs to process. This is a predefined mapping function used to quantify the complexity of the data distribution. and They are respectively and The corresponding target weight coefficient. Optionally, the target weight coefficient can be a pre-set weight coefficient, or it can be obtained by processing the task information of each historical task and each task to be scheduled in the computing cluster. Historical tasks can be tasks that the computing cluster has already completed before the target scheduling strategy for each task to be scheduled is determined.

[0096] It should be noted that the number of historical tasks and the number of tasks to be scheduled can be the same or different, and this application embodiment does not limit this.

[0097] For example, based on the above embodiments, such as Figure 3As shown, the process of processing the task information of each historical task and the task information of each task to be scheduled in the computing cluster specifically includes the following steps S301-S305. Wherein:

[0098] S301, for each historical task, the initial task load of the historical task is obtained by weighted summation of the amount of data and data distribution information of the data to be processed by the historical task according to the initial weight coefficient.

[0099] In one embodiment, the initial weighting coefficients corresponding to the amount of data to be processed in historical tasks and the data distribution information can be preset based on experience. For example, the initial weighting coefficient corresponding to the amount of data to be processed in historical tasks can be 1, while the initial weighting coefficient corresponding to the data distribution information can be 0.

[0100] After determining the initial weight coefficients, the above formula (3) can be used to sum the data volume and data distribution information of the historical task according to the initial weight coefficients to obtain the initial task load of the historical task.

[0101] S302, for each task to be scheduled, the amount of data and data distribution information of the data to be processed by the task to be scheduled are weighted and summed according to the initial weight coefficient to obtain the initial task load of the task to be scheduled.

[0102] For example, based on the initial weight coefficients, the amount of data and data distribution information of the data to be processed by the task to be scheduled are weighted and summed to obtain the initial task load of the task to be scheduled. See S301 above, which will not be repeated here.

[0103] S303, determine the first computational complexity distribution information based on the initial task load of each historical task.

[0104] Among them, the first computational complexity distribution information can represent the distribution of the computational complexity of historical tasks in different computational complexity intervals.

[0105] For example, the complexity distribution of each historical task can be determined based on the computational complexity range to which the initial task load of each historical task belongs, which is also known as the first computational complexity distribution information.

[0106] S304. Determine the second computational complexity distribution information based on the initial task load of each task to be scheduled.

[0107] Similar to S303 above, the complexity distribution of each task to be scheduled can be analyzed based on the initial task load of each task to be scheduled, which is the second computational complexity distribution information.

[0108] Among them, the second computational complexity distribution information can represent the distribution of the computational complexity of the task to be scheduled in different computational complexity intervals.

[0109] S305. Determine the target weight coefficient based on the first computational complexity distribution information, the second computational complexity distribution information, and the initial weight coefficient.

[0110] Optionally, the convergence of the initial weighting coefficients can be determined based on the difference between the first and second computational complexity distribution information. Specifically, if the difference is greater than a preset difference, it indicates a significant difference between the computational complexity of the historical task and the computational complexity of the task to be scheduled. In other words, the initial weighting coefficients are insufficient to accurately calculate the target task load of the task to be scheduled. Therefore, the initial weighting coefficients can be adjusted, and the process can return to step S301 above until the difference is less than or equal to the preset difference. At this point, the current initial weighting coefficients can be determined as the target weighting coefficients.

[0111] In this embodiment, for each historical task, the initial task load is obtained by weighting and summing the data volume and data distribution information of the data to be processed for the historical task based on the initial weight coefficient. For each task to be scheduled, the initial task load is obtained by weighting and summing the data volume and data distribution information of the data to be processed for the task to be scheduled based on the initial weight coefficient. Based on the initial task load of each historical task, a first computational complexity distribution is determined. Based on the initial task load of each task to be scheduled, a second computational complexity distribution is determined. Based on the first computational complexity distribution information, the second computational complexity distribution information, and the initial weight coefficient, a target weight coefficient is determined. Thus, by combining the computational complexity distribution information of the historical tasks and the tasks to be scheduled in the computing cluster, the target weight coefficient is determined. Based on the target weight coefficient, the data volume and data distribution information of the data to be processed for the task to be scheduled are weighted and summed, which can obtain a target task load of the task to be scheduled with high reliability and accuracy. This provides a scientific and reliable data source for determining the execution time and is conducive to enhancing the computing cluster's computing capability for tasks to be scheduled with frequent load fluctuations.

[0112] In one embodiment, such as Figure 4 As shown, S305 specifically includes the following S401-S404. Wherein:

[0113] S401, determine the relative entropy between the first computational complexity distribution information and the second computational complexity distribution information.

[0114] Relative entropy is used to quantify the degree of difference between two probability distributions. The relative entropy between the first computational complexity distribution information and the second computational complexity distribution information can reflect the degree of deviation between the first computational complexity distribution information and the second computational complexity distribution information. The larger the value of the relative entropy, the more serious the deviation between the first computational complexity distribution information and the second computational complexity distribution information.

[0115] For example, the relative entropy between the first computational complexity distribution information and the second computational complexity distribution information. The calculation process can be expressed as the following formula (4).

[0116] (4)

[0117] Where P represents the first computational complexity distribution information, Q represents the second computational complexity distribution information, x represents the index of the computational complexity interval, and X represents the set of computational complexity intervals. This represents the probability that the computational complexity of a historical task falls within the computational complexity interval x. This represents the probability that the computational complexity of the task to be scheduled falls within the computational complexity interval x.

[0118] S402, determine the relationship between the relative entropy and the preset threshold. If the relative entropy is greater than the preset threshold, execute S403 as follows; otherwise, if the relative entropy is less than or equal to the preset threshold, execute S404 as follows.

[0119] It should be noted that the preset threshold can be flexibly set according to actual needs such as resource utilization requirements of the current application scenario. Therefore, this application embodiment does not limit it in this regard.

[0120] S403, adjust the initial weight coefficient to obtain a new initial weight coefficient, and based on the new initial weight coefficient, return to execute the operation for each historical task, which is to perform a weighted summation of the amount of data and data distribution information that the historical task needs to process according to the initial weight coefficient, to obtain the initial task load of the historical task.

[0121] For example, if the relative entropy is determined to be greater than a preset threshold, the initial weight coefficient can be adjusted based on the difference between the relative entropy and the preset threshold to obtain a new initial weight coefficient, and then the process returns to execute S301 above to obtain a new relative entropy. The larger the difference between the relative entropy and the preset threshold, the wider the adjustment range of the initial weight coefficient.

[0122] Furthermore, if the new relative entropy is less than or equal to the preset threshold, it means that the target task load can be accurately obtained based on the new initial weight coefficient. In this case, the new initial weight coefficient can be used as the target weight coefficient. Conversely, if the new relative entropy is greater than the preset threshold, the new initial weight coefficient is adjusted according to the difference between the new relative entropy and the preset threshold, and the above S301 is executed again until the obtained relative entropy is less than or equal to the preset threshold.

[0123] In one embodiment, based on adjusting the initial weight coefficients, the number of historical tasks can be further increased, and appropriate adjustments can be made to obtain the parameters of the new preset mapping function, the performance coefficients of the computing nodes, the speedup ratio of the GPU, and the coefficients related to network transmission, so as to obtain a new relative entropy.

[0124] S404, use the initial weight coefficients as the target weight coefficients.

[0125] If the relative entropy is less than or equal to a preset threshold, it means that the target task load can be accurately obtained based on the current initial weight coefficient. Therefore, the current initial weight coefficient can be used as the target weight coefficient to continue to execute subsequent operations.

[0126] This embodiment determines the relative entropy between the first and second computational complexity distribution information. Based on the relationship between the relative entropy and a preset threshold, the initial weight coefficients are iteratively adjusted until the relative entropy is less than or equal to the preset threshold. Therefore, the target weight coefficients can respond in real-time to changes in the target task load of the task to be scheduled, thereby ensuring the accuracy and reliability of the target scheduling strategy.

[0127] Optional, such as Figure 5 As shown, S203 above determines the load imbalance degree corresponding to each candidate scheduling strategy based on the execution time of each scheduled task on each computing node. Specifically, it may include the following S501-S504. Wherein:

[0128] S501, for each candidate scheduling strategy, determine the task allocation of each computing node corresponding to the candidate scheduling strategy based on the node allocation of each task to be scheduled in the candidate scheduling strategy.

[0129] In some embodiments, since the candidate scheduling strategy includes the node allocation of each task to be scheduled, the task allocation of each computing node corresponding to the candidate scheduling strategy can be determined based on the node allocation of each task to be scheduled in the candidate scheduling strategy.

[0130] For example, the task allocation for each computing node corresponding to the candidate scheduling strategy includes computing node 1 {task 2 to be scheduled, task 3 to be scheduled}, computing node 2 {task 5 to be scheduled}, computing node 3 {task 4 to be scheduled}, computing node 4 {task 1 to be scheduled}, computing node 5 {none}, etc.

[0131] S502, for each computing node, determine the cumulative execution time of the assigned scheduled tasks by the computing node based on the execution time of each scheduled task executed by the computing node and the task allocation of the computing node.

[0132] For example, for each computing node, the execution time of each assigned task can be determined based on the execution time of each scheduled task and the task allocation of the computing node; then, the cumulative execution time of each assigned task can be obtained by summing the execution times of each assigned task.

[0133] Taking the task allocation of compute node 1 as compute node 1 {task 2 to be scheduled, task 3 to be scheduled} as an example, the cumulative execution time of compute node in executing the assigned tasks to be scheduled can be expressed as E. 21 +E 31 .

[0134] S503 determines the average execution time based on the cumulative execution time of each computing node and the total number of computing nodes in the computing cluster.

[0135] Optionally, the average execution time can be determined based on the ratio between the sum of the cumulative execution times of each computing node and the total number of computing nodes in the computing cluster.

[0136] S504. Based on the difference between the cumulative execution time and the average execution time of each computing node, determine the degree of load imbalance corresponding to the candidate scheduling strategy.

[0137] In some embodiments, for each computing node, a first value can be obtained based on the difference between the cumulative execution time of the computing node and the average execution time calculated above. The first values ​​of all computing nodes in the computing cluster are traversed and the squares of each first value are summed to obtain a second value. Then, the load imbalance degree corresponding to the candidate scheduling strategy can be obtained based on the square root of the ratio between the second value and the total number of computing nodes.

[0138] Specifically, the calculation process of the load imbalance degree U can be expressed as the following formula (5).

[0139] (5)

[0140] Where J is the total number of computing nodes in the computing cluster, and I is the total number of tasks to be scheduled assigned to computing node j. This represents the average execution time.

[0141] This embodiment determines the load imbalance degree corresponding to the candidate scheduling strategy based on the difference between the cumulative execution time and the average execution time of each computing node. This enables the determination of the target scheduling strategy based on the load imbalance degree to overcome the technical bottleneck of static scheduling ignoring the complex and variable target task load. Furthermore, determining the target scheduling strategy based on the load imbalance degree can avoid the technical problem of reduced resource utilization caused by some computing nodes being idle while others are overloaded, effectively improving the node parallelism rate of the computing cluster and saving task execution time.

[0142] Based on the above embodiments, such as Figure 6 As shown, S204 above schedules each task to be scheduled based on the target scheduling strategy, which may specifically include the following S601-S603.

[0143] S601 schedules each task to be scheduled based on the target scheduling strategy, and determines the new resource status of each computing node based on the execution resource consumption data of the first task to be scheduled that was scheduled before the target time, for the target time in the scheduling process.

[0144] Among them, the execution resource consumption data is used to quantify the resources consumed by the computing node corresponding to the first scheduled task after executing the first scheduled task. For example, the execution resource consumption data may include CPU usage time, memory usage, etc.

[0145] Optionally, the target time can be determined based on a preset fine-tuning frequency during the scheduling process. For example, if the fine-tuning frequency is once every 10 seconds, then the target time can be 10 seconds, 20 seconds, etc., after the scheduling starts.

[0146] For example, for each first task to be scheduled, the execution resource consumption data generated by the computing node corresponding to the first task to be scheduled during the execution of the first task to be scheduled can be determined based on the node allocation of the first task to be scheduled; furthermore, the new resource status of the corresponding computing node at the target time can be determined based on the execution consumption data.

[0147] S602, based on the new resource status of each computing node, adjust the node allocation of the second task to be scheduled in the target scheduling strategy to obtain a new target scheduling strategy.

[0148] Among them, the second task to be scheduled is the task to be scheduled other than the first task to be scheduled.

[0149] In some embodiments, for each first task to be scheduled, the node allocation of each second task to be scheduled can be fine-tuned according to the difference between the new resource state of the computing node corresponding to the first task to be scheduled and the resource state of the computing node before scheduling, based on the difference between each first task to be scheduled and according to the pre-set fine-tuning rules, so as to obtain a new target scheduling strategy.

[0150] S603, after the target time, schedules the second task to be scheduled based on the new target scheduling strategy.

[0151] Based on the above embodiments, after the target time, each second task to be scheduled can continue to be scheduled based on the new target scheduling strategy obtained above.

[0152] In this embodiment, during the scheduling process, the resource status of each computing node is re-determined based on the execution resource consumption data of the first scheduled task that has been scheduled. Then, the node allocation of the unscheduled second scheduled task is adjusted according to the new resource status of each computing node. After the adjustment, the second scheduled task is scheduled according to the new target scheduling strategy. Thus, a dynamic task scheduling is realized, which can adaptively adjust the target scheduling strategy according to the changes in resource status during task execution, thereby improving the stability and robustness of long-term scheduling.

[0153] Based on the above embodiments, in an exemplary embodiment, an optional task scheduling method is provided, such as... Figure 7 As shown, it may include:

[0154] S701 obtains task information for each task to be scheduled and the resource status of each computing node in the computing cluster.

[0155] S702, based on the target weight coefficient, the data volume and data distribution information of the data to be processed by the task to be scheduled are weighted and summed to obtain the target task load of the task to be scheduled.

[0156] The target weight coefficient is obtained by processing the task information of each historical task and the task information of each task to be scheduled in the computing cluster. Specifically, the task information of each historical task and each task to be scheduled in the computing cluster is processed, including: for each historical task, a weighted sum of the data volume and data distribution information of the data to be processed by the historical task is performed based on the initial weight coefficient to obtain the initial task load of the historical task; for each task to be scheduled, a weighted sum of the data volume and data distribution information of the data to be processed by the task to be scheduled is performed based on the initial weight coefficient to obtain the initial task load of the task to be scheduled; a first computational complexity distribution is determined based on the initial task load of each historical task; and a second computational complexity distribution is determined based on the initial task load of each task to be scheduled; the relative entropy between the first and second computational complexity distribution information is determined; if the relative entropy is greater than a preset threshold, the initial weight coefficient is adjusted to obtain a new initial weight coefficient, and based on the new initial weight coefficient, the operation of performing a weighted sum of the data volume and data distribution information of the data to be processed by the historical task based on the initial weight coefficient to obtain the initial task load of the historical task is returned; if the relative entropy is less than or equal to the preset threshold, the initial weight coefficient is used as the target weight coefficient.

[0157] S703 determines the execution time of each scheduled task for each computing node based on the target task load and corresponding data volume of each scheduled task, as well as the resource status of the computing node.

[0158] S704, for each candidate scheduling strategy, determine the task allocation of each computing node corresponding to the candidate scheduling strategy based on the node allocation of each scheduled task in the candidate scheduling strategy; and for each computing node, determine the cumulative execution time of the computing node executing the assigned scheduled tasks based on the execution time of the computing node executing each scheduled task and the task allocation of the computing node; determine the average execution time based on the cumulative execution time of each computing node and the total number of computing nodes in the computing cluster, and determine the load imbalance degree corresponding to the candidate scheduling strategy based on the difference between the cumulative execution time of each computing node and the average execution time.

[0159] S705 selects the minimum load imbalance level from various load imbalance levels.

[0160] S706, determine the candidate scheduling strategy corresponding to the minimum load imbalance as the target scheduling strategy.

[0161] S707 schedules each task to be scheduled based on the target scheduling strategy, and determines the new resource status of each computing node based on the execution resource consumption data of the first task to be scheduled that was scheduled before the target time, for the target time in the scheduling process.

[0162] S708, based on the new resource status of each computing node, adjusts the node allocation for the second scheduled task in the target scheduling strategy to obtain a new target scheduling strategy.

[0163] S709, after the target time, schedules the second task to be scheduled based on the new target scheduling strategy.

[0164] The specific processes of S701-S709 described above can be found in the description of the above method embodiments. Their implementation principles and technical effects are similar, and will not be repeated here.

[0165] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0166] Based on the same inventive concept, this application also provides a task scheduling apparatus for implementing the task scheduling method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more task scheduling apparatus embodiments provided below can be found in the limitations of the task scheduling method described above, and will not be repeated here.

[0167] In one exemplary embodiment, such as Figure 8 As shown, a task scheduling device is provided, comprising: an acquisition module 810, a first determination module 820, a second determination module 830, a selection module 840, and a scheduling module 850, wherein:

[0168] The acquisition module 810 is used to acquire the task information of each task to be scheduled and the resource status of each computing node in the computing cluster; wherein, the task information of each task to be scheduled includes at least the amount of data and data distribution information of the data to be processed by the task to be scheduled.

[0169] The first determining module 820 is used to determine the execution time of each computing node for each scheduled task based on the task information of each scheduled task and the resource status of each computing node.

[0170] The second determining module 830 is used to determine the degree of load imbalance corresponding to each candidate scheduling strategy based on the execution time of each scheduled task executed by each computing node; wherein, the candidate scheduling strategy includes the node allocation of each scheduled task.

[0171] The selection module 840 is used to select the target scheduling strategy from the candidate scheduling strategies based on the degree of load imbalance corresponding to each candidate scheduling strategy.

[0172] The scheduling module 850 is used to schedule each task to be scheduled based on the target scheduling strategy.

[0173] In some embodiments, the first determining module 820 may include:

[0174] The summation unit is used to perform a weighted summation of the amount of data and data distribution information required to process the data of each task to be scheduled, so as to obtain the target task load of the task to be scheduled.

[0175] The determination unit is used to determine the execution time of each scheduled task for each computing node, based on the target task load and corresponding data volume of each scheduled task, as well as the resource status of the computing node.

[0176] In some embodiments, the summation unit is specifically used to perform a weighted summation of the amount of data and data distribution information of the data to be processed by the task to be scheduled, based on the target weight coefficient, to obtain the target task load of the task to be scheduled; wherein, the target weight coefficient is obtained by processing the task information of each historical task of the computing cluster and the task information of each task to be scheduled.

[0177] In some embodiments, the second determining module 830 is specifically configured to, for each candidate scheduling strategy, determine the task allocation of each computing node corresponding to the candidate scheduling strategy based on the node allocation of each scheduled task in the candidate scheduling strategy; for each computing node, determine the cumulative execution time of the computing node executing the allocated scheduled tasks based on the execution time of the computing node executing each scheduled task and the task allocation of the computing node; determine the average execution time based on the cumulative execution time corresponding to each computing node and the total number of computing nodes in the computing cluster; and determine the load imbalance degree corresponding to the candidate scheduling strategy based on the difference between the cumulative execution time of each computing node and the average execution time.

[0178] In some embodiments, the selection module 840 is specifically used to select the minimum load imbalance degree from each load imbalance degree; and to determine the candidate scheduling strategy corresponding to the selected minimum load imbalance degree as the target scheduling strategy.

[0179] In some embodiments, the scheduling module 850 is specifically used to schedule each task to be scheduled based on a target scheduling strategy, and, for a target time in the scheduling process, determine the new resource status of each computing node based on the execution resource consumption data of the first task to be scheduled that was scheduled before the target time; adjust the node allocation of the second task to be scheduled in the target scheduling strategy based on the new resource status of each computing node to obtain a new target scheduling strategy; wherein, the second task to be scheduled is the task to be scheduled other than the first task to be scheduled; after the target time, schedule the second task to be scheduled based on the new target scheduling strategy.

[0180] Each module in the aforementioned task scheduling device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0181] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores candidate scheduling policies. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a task scheduling method.

[0182] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0183] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0184] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0185] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0188] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A task scheduling method, characterized in that, The method includes: Obtain the task information for each task to be scheduled and the resource status of each computing node in the computing cluster; wherein, the task information for each task to be scheduled includes at least the amount of data and data distribution information of the data to be processed by the task to be scheduled; Based on the task information of each task to be scheduled and the resource status of each computing node, determine the execution time of each task to be scheduled on each computing node. Based on the execution time of each scheduled task on each computing node, the degree of load imbalance corresponding to each candidate scheduling strategy is determined; wherein, the candidate scheduling strategy includes the node allocation of each scheduled task. Based on the degree of load imbalance corresponding to each of the candidate scheduling strategies, a target scheduling strategy is selected from each of the candidate scheduling strategies. Based on the target scheduling strategy, each task to be scheduled is scheduled.

2. The method according to claim 1, characterized in that, The step of determining the execution time of each computing node for each scheduled task based on the task information of each task to be scheduled and the resource status of each computing node includes: For each task to be scheduled, the data volume and data distribution information of the data to be processed by the task to be scheduled are weighted and summed to obtain the target task load of the task to be scheduled. For each computing node, the execution time of each scheduled task is determined based on the target task load and corresponding data volume of each scheduled task, as well as the resource status of the computing node.

3. The method according to claim 2, characterized in that, The step of weighted summing of the data volume and data distribution information of the data to be processed by the task to be scheduled to obtain the target task load of the task to be scheduled includes: Based on the target weight coefficient, the data volume and data distribution information of the data to be processed by the task to be scheduled are weighted and summed to obtain the target task load of the task to be scheduled; wherein, the target weight coefficient is obtained by processing the task information of each historical task of the computing cluster and the task information of each task to be scheduled.

4. The method according to claim 3, characterized in that, The processing of task information for each historical task and task information for each task to be scheduled in the computing cluster includes: For each historical task, the initial task load is obtained by weighting and summing the amount of data and data distribution information of the data to be processed for the historical task according to the initial weight coefficient. For each task to be scheduled, the amount of data and data distribution information of the data to be processed by the task to be scheduled are weighted and summed according to the initial weight coefficient to obtain the initial task load of the task to be scheduled. Based on the initial task load of each historical task, determine the first computational complexity distribution information; and, The second computational complexity distribution information is determined based on the initial task load of each task to be scheduled. The target weight coefficient is determined based on the first computational complexity distribution information, the second computational complexity distribution information, and the initial weight coefficient.

5. The method according to claim 4, characterized in that, The step of determining the target weight coefficient based on the first computational complexity distribution information, the second computational complexity distribution information, and the initial weight coefficient includes: Determine the relative entropy between the first computational complexity distribution information and the second computational complexity distribution information; If the relative entropy is greater than a preset threshold, the initial weight coefficient is adjusted to obtain a new initial weight coefficient. Based on the new initial weight coefficient, the operation of weighting and summing the amount of data and data distribution information of the data to be processed for each historical task according to the initial weight coefficient is returned to obtain the initial task load of the historical task. If the relative entropy is less than or equal to a preset threshold, the initial weight coefficient is used as the target weight coefficient.

6. The method according to any one of claims 1-5, characterized in that, The step of determining the load imbalance degree corresponding to each candidate scheduling strategy based on the execution time of each scheduled task on each computing node includes: For each candidate scheduling strategy, the task allocation of each computing node corresponding to the candidate scheduling strategy is determined based on the node allocation of each task to be scheduled in the candidate scheduling strategy. For each computing node, the cumulative execution time of the assigned scheduled tasks is determined based on the execution time of each scheduled task executed by the computing node and the task allocation of the computing node. The average execution time is determined based on the cumulative execution time of each computing node and the total number of computing nodes in the computing cluster. The degree of load imbalance corresponding to the candidate scheduling strategy is determined based on the difference between the cumulative execution time of each computing node and the average execution time.

7. The method according to any one of claims 1-5, characterized in that, The step of selecting a target scheduling strategy from the candidate scheduling strategies based on the load imbalance degree corresponding to each candidate scheduling strategy includes: Select the minimum load imbalance level from among the various load imbalance levels; The candidate scheduling strategy corresponding to the minimum load imbalance selected is determined as the target scheduling strategy.

8. The method according to any one of claims 1-5, characterized in that, The scheduling of each task to be scheduled based on the target scheduling strategy includes: Based on the target scheduling strategy, each task to be scheduled is scheduled, and for the target time in the scheduling process, the new resource status of each computing node is determined according to the execution resource consumption data of the first task to be scheduled that was scheduled before the target time. Based on the new resource status of each computing node, the node allocation of the second task to be scheduled in the target scheduling strategy is adjusted to obtain a new target scheduling strategy; wherein, the second task to be scheduled is the task to be scheduled other than the first task to be scheduled among the tasks to be scheduled. After the target time, the second task to be scheduled is scheduled based on the new target scheduling strategy.

9. A task scheduling device, characterized in that, The device includes: The acquisition module is used to acquire the task information of each task to be scheduled and the resource status of each computing node in the computing cluster; wherein, the task information of each task to be scheduled includes at least the amount of data and data distribution information of the data to be processed by the task to be scheduled; The first determining module is used to determine the execution time of each computing node for each scheduled task based on the task information of each scheduled task and the resource status of each computing node. The second determining module is used to determine the degree of load imbalance corresponding to each candidate scheduling strategy based on the execution time of each task to be scheduled on each computing node; wherein, the candidate scheduling strategy includes the node allocation of each task to be scheduled. The selection module is used to select a target scheduling strategy from the candidate scheduling strategies based on the degree of load imbalance corresponding to each candidate scheduling strategy. The scheduling module is used to schedule each task to be scheduled based on the target scheduling strategy.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images