Dynamic task scheduling apparatus and method based on computing resources
By using a dynamic task scheduling system based on computing resources, the shortcomings of platforms such as Kubernetes in scheduling resource-sensitive tasks are addressed, achieving flexible and stable task scheduling and high availability. This reduces the risk of task failure due to insufficient resources and simplifies the operation and maintenance process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- COWA TECHNOLOGY CO LTD
- Filing Date
- 2023-03-16
- Publication Date
- 2026-07-21
AI Technical Summary
Existing cloud computing management platforms such as Kubernetes suffer from insufficient flexibility in task scheduling, difficulties in operation and maintenance, inability to respond proactively to tasks that are sensitive to specific resources, and a tendency for tasks to fail or be re-executed due to excessive resource consumption.
A dynamic task scheduling system based on computing resources is adopted, including a control module, a computing module, an Etcd and a MySQL database server. Through the scheduler's high availability architecture, task resource requirement assessment, load balancing, node circuit breaker checks and overload protection mechanisms, customized scheduling and resource-sensitive response for different tasks are achieved.
It improves the flexibility and stability of the scheduling algorithm, avoids task failures due to insufficient resources, reduces network I/O and scheduling latency, supports high availability and containerized deployment, and simplifies the operation and maintenance process.
Smart Images
Figure CN116225656B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cloud computing technology, and more specifically, to a dynamic task scheduling device and method based on computing resources. Background Technology
[0002] With the continuous expansion of the internet, the computing power of traditional single-node servers is no longer sufficient to handle the ever-increasing volume of data. Therefore, developers need to manage and schedule large amounts of computing resources in a unified manner, forming a resource pool to provide services to end users. Kubernetes (K8s) and OpenStack, as the most widely used cloud computing management platforms in the industry, offer good resource management and scheduling solutions. As business volume grows, enterprises have increasingly demanding task scheduling needs for scenarios such as artificial intelligence model training and simulation. However, as a general-purpose task and resource scheduling platform, it does not perform well for tasks that are sensitive to resources such as CPU, memory, and I / O.
[0003] For example, patent CN112241368A discloses an automated model training method and apparatus based on Kubernetes, which achieves the scheduling of model training tasks through Kubernetes. Its working principle is as follows: Figure 1 As shown, a high-level model training management platform was built based on Kubernetes. After creating a training task, a pod was created by calling the Kubernetes API. Upon receiving the pod creation request, the Kubernetes scheduler immediately finds an available node and schedules the task to that node. The Kubernetes node daemon is responsible for starting the container and executing the task. However, this solution has the following drawbacks: it is heavily dependent on Kubernetes, making deployment difficult and inflexible; it uses Kubernetes' built-in scheduling algorithm, making customization difficult and unable to respond proactively to tasks sensitive to specific resources; after Kubernetes schedules a task to a designated node, a sudden increase in the consumption of non-compressible resources such as memory may lead to OutOfMemoryError (OOM) or pod eviction, causing the task to fail or be re-executed.
[0004] As mentioned above, building related task systems based on distributed scheduling platforms such as Kubernetes has become the mainstream solution in the industry. Although the system construction is relatively simple, the scheduling algorithm has poor flexibility and the operation and maintenance are also very difficult. Summary of the Invention
[0005] To address the above issues, this proposal presents a dynamic task scheduling system and related algorithms based on computing resources. This scheme possesses a complete design architecture, can be customized for different tasks, actively responds to tasks sensitive to specific resources, overcomes the shortcomings of unified scheduling algorithms, and improves flexibility. To achieve the above objectives, this invention adopts the following technical solution:
[0006] A dynamic task scheduling device based on computing resources includes: a control module, a computing module, an Etcd, and a MySQL database server; the control module includes multiple schedulers, the computing module includes multiple nodes, and each node includes a node monitoring module and a task processing module.
[0007] The scheduler is used to execute tasks according to the task scheduling calculation module submitted by the user. The scheduler completes the election function of the primary scheduler and the backup scheduler based on the high availability and strong data consistency characteristics provided by Etcd, thereby realizing the high availability of the scheduler. Under normal service conditions, only the primary scheduler is responsible for scheduling services. When the primary scheduler fails, one of the backup schedulers will immediately detect the failure and be elected as the primary scheduler to continue processing services.
[0008] This invention provides a high availability architecture. The scheduler is based on the Raft distributed consensus protocol provided by Etcd and forms a master-slave cluster. When the master scheduler fails, the standby scheduler will be immediately elected as the master scheduler to continue processing.
[0009] Furthermore, the monitoring module is responsible for communicating with the scheduler and transmitting the current node resource status to the current master scheduler. After the master scheduler sends the task to the monitoring module, the monitoring module is responsible for preparing the data required for the task and starting the task. After the task is completed, the task execution result is sent back to the master scheduler.
[0010] Furthermore, the scheduler also includes a task resource requirement assessment module. For different tasks, in order to achieve better scheduling results, the module assesses the sensitive resources of the tasks submitted by the user and calculates the required amount of resources. This amount of resources will serve as an important criterion for subsequent node selection.
[0011] Furthermore, the scheduler also includes a node pre-selection module, which is used to find nodes among the currently online nodes whose remaining resources meet the requirements based on the evaluation results of the task resource requirement assessment module. When the node pool is large, the search starting point and step size are determined by random numbers. When the number of nodes that meet the requirements reaches a certain number, the pre-selection can be stopped and the pre-selection results can be passed to the next step.
[0012] Furthermore, the scheduler also includes a load balancing module for scoring pre-selected nodes. The scoring depends on the various resources required by the task, with the product of the resources and their weights serving as a positive parameter. The node penalty value serves as a negative parameter, calculated from the number of failures using an exponentially weighted moving average algorithm. The higher the number of failures in a short period, the higher the penalty value; otherwise, the penalty value will gradually decrease to 1 as the task is successfully executed.
[0013] Meanwhile, the number of tasks currently being executed by a node is also included as a negative parameter in the calculation, aiming to balance the load. In addition, to avoid scheduling a large number of tasks to the same node under heavy load, the algorithm first randomly selects two nodes before evaluating the node score, evaluates the scores of the selected nodes, and selects the better node.
[0014] Furthermore, the load balancing module uses the following functions to obtain the weight, penalty value, and node score:
[0015] The weights are moving average weights, calculated using the following formula (Formula 1):
[0016] w = e (last-now) / lifetime ;
[0017] Where e is a mathematical constant, last is the time when the node was last selected, now is the current time, and lifetime is a custom time period. The time lookup value within this period has a greater impact, and it reaches its half-life after lifetime×ln(2).
[0018] The penalty value is calculated using the following formula (Formula 2):
[0019] penty = old × w + new × (1 - w);
[0020] Where old is the penalty value calculated in the previous calculation, new is the penalty value in the current calculation (1000 for failure, 1 for success), and w is the weight calculated by Formula 1.
[0021] The node scoring formula is as follows:
[0022]
[0023] Where resVal is the remaining resource value, resWeight is the resource weight, inflight is the number of tasks currently being processed by the node, and penty is the penalty value calculated by formula two.
[0024] Furthermore, the scheduler also includes a node circuit breaker checking module, used to evaluate whether a node can continue to provide service. The node circuit breaker checking module records the number of errors and decides whether to continue scheduling tasks to that node, reduce invalid network requests, or reduce scheduling latency, or one or more of these factors. The circuit breaker algorithm calculation formula is as follows:
[0025]
[0026] Where k is the tolerance level, decreasing k makes the rate limiting algorithm more aggressive, and more traffic will be rejected, while increasing k makes the rate limiting algorithm more lenient, allowing more traffic; accepts is the number of normal requests in the current time window, and requests is the total number of requests in the current time window;
[0027] When requests increase infinitely while accepts stop increasing, f(a) approaches 1 infinitely, triggering the circuit breaker; conversely, when accepts increase, due to the effect of k, f(a) approaches 0 infinitely, releasing the circuit breaker.
[0028] Furthermore, the scheduler also includes an overload protection module. The overload protection module operates on the computing node side. When a task is submitted to the computing node by the scheduler, the computing node compares the current system resources with the resources required by the task. When the remaining resources are insufficient to execute the task, the overload protection is triggered, and the task is immediately rejected by the scheduler with a resource shortage response, so as to avoid problems such as OOM (Out of Memory) caused by insufficient resources.
[0029] Furthermore, the scheduler also includes a data transmission protocol module, which uses the TCP protocol to construct a Request-Response model and packages ProtoBuf data (Google Protocol Buffer, a structured data storage format) for transmission between the two parties.
[0030] The present invention also provides a dynamic task scheduling method based on computing resources, including: a control module, a computing module, an Etcd, and a MySQL database server; the control module includes multiple schedulers, the computing module includes multiple nodes, and each node includes a node monitoring module and a task processing module;
[0031] The dynamic task scheduling method based on computing resources includes the following steps:
[0032] The scheduler executes tasks based on the task scheduling calculation module submitted by the user. The scheduler completes the election function of the primary scheduler and the backup scheduler based on the high availability and strong data consistency characteristics provided by Etcd, thereby achieving high availability of the scheduler. Under normal service conditions, only the primary scheduler is responsible for scheduling services. When the primary scheduler encounters an anomaly, one of the backup schedulers will immediately detect the anomaly and be elected as the primary scheduler to continue processing services.
[0033] Furthermore, the dynamic task scheduling method based on computing resources also includes a node resource status monitoring step: the monitoring module is responsible for communicating with the scheduler and transmitting the current node resource status to the current scheduler. After the main scheduler sends the task to the monitoring module, the monitoring module is responsible for preparing the data required for the task and starting the task. After the task is completed, the task execution result is returned to the main scheduler.
[0034] Furthermore, the dynamic task scheduling method based on computing resources also includes a task resource requirement assessment step: for different tasks, in order to achieve better scheduling results, the tasks submitted by users are assessed for sensitive resources, and the required amount of resources is calculated. This amount of resources will serve as an important criterion for subsequent node selection.
[0035] Furthermore, the dynamic task scheduling method based on computing resources also includes a node pre-selection step: the scheduler searches for nodes among the currently online nodes whose remaining resources meet the requirements based on the evaluation results of the task resource requirement assessment module; when the node pool is large, the search starting point and step size are determined by random numbers; when the number of nodes that meet the requirements reaches a certain number, the pre-selection can be stopped and the pre-selection results can be passed to the next step.
[0036] Furthermore, the dynamic task scheduling method based on computing resources also includes a load balancing step: the scheduler scores the pre-selected nodes, and the score depends on the various resources required by the task. The product of the resource and its weight is used as a positive parameter; the node penalty value is used as a negative parameter, which is calculated by the number of failures using an exponentially weighted moving average algorithm. The higher the number of failures in a short period of time, the higher the penalty value. Otherwise, the penalty value will gradually decrease to 1 as the task is successfully executed.
[0037] Meanwhile, the number of tasks currently being executed by a node is also included as a negative parameter in the calculation, aiming to balance the load. In addition, to avoid scheduling a large number of tasks to the same node under heavy load, the algorithm first randomly selects two nodes before evaluating the node score, evaluates the scores of the selected nodes, and selects the better node.
[0038] Furthermore, the load balancing step uses the following functions to obtain the weights, penalty values, and node scores:
[0039] The weights are moving average weights, calculated using the following formula (Formula 1):
[0040] w = e (last-now) / lifetime ;
[0041] Where e is a mathematical constant, last is the time when the node was last selected, now is the current time, and lifetime is a custom time period. The time lookup value within this period has a greater impact, and it reaches its half-life after lifetime×ln(2).
[0042] The penalty value is calculated using the following formula (Formula 2):
[0043] penty = old × w + new × (1 - w);
[0044] Where old is the penalty value calculated in the previous calculation, new is the penalty value in the current calculation (1000 for failure, 1 for success), and w is the weight calculated by Formula 1.
[0045] The node scoring formula is as follows:
[0046]
[0047] Where resVal is the remaining resource value, resWeight is the resource weight, inflight is the number of tasks currently being processed by the node, and penty is the penalty value calculated by formula two.
[0048] Furthermore, the dynamic task scheduling method based on computing resources also includes a node circuit breaker check step: the scheduler evaluates whether a node can continue to provide service, records the number of errors, and decides whether to continue scheduling tasks to that node, reduce invalid network requests, or reduce scheduling latency, among one or more of these options; the circuit breaker algorithm calculation formula is as follows:
[0049]
[0050] Where k is the tolerance level, decreasing k makes the rate limiting algorithm more aggressive, and more traffic will be rejected, while increasing k makes the rate limiting algorithm more lenient, allowing more traffic; accepts is the number of normal requests in the current time window, and requests is the total number of requests in the current time window;
[0051] When requests increase infinitely while accepts stop increasing, f(a) approaches 1 infinitely, triggering the circuit breaker; conversely, when accepts increase, due to the effect of k, f(a) approaches 0 infinitely, releasing the circuit breaker.
[0052] Furthermore, the dynamic task scheduling method based on computing resources also includes an overload protection step: when a task is submitted to a computing node by the scheduler, the computing node compares the current system resources with the resources required by the task. When the remaining resources are insufficient to execute the task, overload protection is triggered, and the task is immediately rejected by the scheduler and a resource shortage response is issued.
[0053] To avoid problems such as OOM (Out of Memory) caused by insufficient resources.
[0054] Furthermore, the dynamic task scheduling method based on computing resources adopts the following data transmission protocol: a Request-Response model is constructed using the TCP protocol, and ProtoBuf data (Google Protocol Buffer, a structured data storage format) is packaged for transmission between the two parties.
[0055] The beneficial effects of this technical solution are:
[0056] 1. This solution implements its own scheduling algorithm. For different types of tasks, adaptation can be achieved by modifying the weights of various resources in the node scoring process.
[0057] 2. The solution provides a complete scheduling algorithm, including load balancing, circuit breaking, and overload protection, fully considering various issues in scheduling scenarios. Load balancing ensures that the load across nodes is as level as possible. The circuit breaking mechanism prevents unnecessary data interaction; direct rejection at the scheduler end reduces network I / O and lowers scheduling latency. Overload protection ensures that nodes will not experience unpredictable errors or task execution failures due to resource shortages. These three mechanisms complement each other, forming a complete scheduling algorithm and building a stable scheduling platform.
[0058] 3. The solution provides a high availability architecture. The scheduler is based on the Raft distributed consensus protocol provided by Etcd and forms a master-slave cluster. When the master scheduler fails, the standby scheduler will immediately elect a master scheduler to continue processing.
[0059] 4. The solution eliminates the HTTP protocol and uses TCP in conjunction with Protobuf encoding to achieve efficient communication between the scheduler and the monitoring program, allowing the scheduler to perceive the status of each node more promptly;
[0060] 5. Supports containerized deployment, simplifying operation and maintenance. Attached Figure Description
[0061] Figure 1 This is a schematic diagram of a scheduling method in the prior art;
[0062] Figure 2 This is a schematic diagram of the system architecture of the present invention;
[0063] Figure 3 This is a schematic diagram of the task scheduling process of the present invention. Detailed Implementation
[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0065] like Figure 2 As shown, a dynamic task scheduling device based on computing resources includes: a control module, a computing module, an Etcd, and a MySQL database server; the control module includes multiple schedulers, the computing module includes multiple nodes, and each node includes a node monitoring module and a task processing module.
[0066] The scheduler is used to execute tasks according to the task scheduling calculation module submitted by the user. The scheduler completes the election function of the primary scheduler and the backup scheduler based on the high availability and strong data consistency characteristics provided by Etcd, thereby realizing the high availability of the scheduler. Under normal service conditions, only the primary scheduler is responsible for scheduling services. When the primary scheduler fails, one of the backup schedulers will immediately detect the failure and be elected as the primary scheduler to continue processing services.
[0067] This invention provides a high availability architecture. The scheduler is based on the Raft distributed consensus protocol provided by Etcd and forms a master-slave cluster. When the master scheduler fails, the standby scheduler will be immediately elected as the master scheduler to continue processing.
[0068] In some embodiments, the monitoring module is responsible for communicating with the scheduler, transmitting the current node resource status to the current master scheduler. After the master scheduler distributes the task to the monitoring module, the monitoring module is responsible for preparing the data required for the task and starting the task. After the task is completed, the monitoring module sends the task execution result back to the master scheduler. Here, the master scheduler is not a fixed scheduler, but rather the current master scheduler designated according to the system election rules. The entire system has multiple backup schedulers and one current master scheduler. When the current master scheduler encounters a problem, a new master scheduler is immediately elected from among the backup schedulers to continue executing the relevant task. The original master scheduler then becomes one of the backup schedulers. Therefore, the master scheduler always refers to the current master scheduler that is currently executing the task.
[0069] The scheduler also includes a task resource requirement assessment module. For different tasks, to achieve better scheduling performance, it assesses the sensitive resources required for the user-submitted tasks and calculates the amount of resources needed. This resource quantity will serve as an important criterion for subsequent node selection. The resource requirements for different tasks may vary. Computational resources include CPU, GPU, memory, and video memory, etc. For example, simulation tasks have a higher memory requirement but a lower CPU usage.
[0070] In some embodiments, the scheduler further includes a node pre-selection module, used to find nodes among the currently online nodes whose remaining resources meet the requirements based on the evaluation results of the task resource requirement assessment module; when the node pool is large, a random number is used to determine the search starting point and step size, and the pre-selection can be stopped when the number of nodes that meet the requirements reaches a certain number, and the pre-selection results are passed to the next step. This can further speed up the response.
[0071] In some embodiments, the scheduler further includes a load balancing module for scoring pre-selected nodes. The scoring depends on the various resources required by the task, with the product of the resources and their weights serving as a positive parameter. The node penalty value serves as a negative parameter, calculated from the number of failures using an exponentially weighted moving average algorithm. The higher the number of failures in a short period, the higher the penalty value; otherwise, the penalty value will gradually decrease to 1 as the task is successfully executed.
[0072] Meanwhile, the number of tasks currently being executed by a node is also included as a negative parameter in the calculation, aiming to balance the load. In addition, to avoid scheduling a large number of tasks to the same node under heavy load, the algorithm first randomly selects two nodes before evaluating the node score, evaluates the scores of the selected nodes, and selects the better node.
[0073] In practical applications, the load balancing module can use the following functions to obtain the weight, penalty value, and node score:
[0074] The weights are moving average weights, calculated using the following formula (Formula 1):
[0075] w = e (last-now) / lifetime ;
[0076] Where e is a mathematical constant, last is the time when the node was last selected, now is the current time, and lifetime is a custom time period. The time lookup value within this period has a greater impact, and it reaches its half-life after lifetime×ln(2).
[0077] The penalty value is calculated using the following formula (Formula 2):
[0078] penty = old × w + new × (1 - w);
[0079] Where old is the penalty value calculated in the previous calculation, new is the penalty value in the current calculation (1000 for failure, 1 for success), and w is the weight calculated by Formula 1.
[0080] The node scoring formula is as follows:
[0081]
[0082] Where resVal is the remaining resource value, resWeight is the resource weight, inflight is the number of tasks currently being processed by the node, and penty is the penalty value calculated by formula two.
[0083] Furthermore, the scheduler also includes a node circuit breaker checking module, used to evaluate whether a node can continue to provide service. The node circuit breaker checking module records the number of errors and decides whether to continue scheduling tasks to that node, reduce invalid network requests, or reduce scheduling latency, among other things. The circuit breaker algorithm calculation formula is as follows:
[0084]
[0085] Where k is the tolerance level, decreasing k makes the rate limiting algorithm more aggressive, and more traffic will be rejected, while increasing k makes the rate limiting algorithm more lenient, allowing more traffic; accepts is the number of normal requests in the current time window, and requests is the total number of requests in the current time window;
[0086] When requests increase infinitely while accepts stop increasing, f(a) approaches 1 infinitely, triggering the circuit breaker; conversely, when accepts increase, due to the effect of k, f(a) approaches 0 infinitely, releasing the circuit breaker.
[0087] In some embodiments, the scheduler further includes an overload protection module, which operates on the computing node. When a task is submitted to the computing node by the scheduler, the computing node compares the current system resources with the resources required by the task. When the remaining resources are insufficient to execute the task, the overload protection is triggered, and the task is immediately rejected by the scheduler with a resource shortage response, thus avoiding problems such as OOM (Out of Memory) caused by insufficient resources.
[0088] In some embodiments, the scheduler further includes a data transmission protocol module, which uses the TCP protocol to construct a Request-Response model and packages ProtoBuf data (Google Protocol Buffer, a structured data storage format) for transmission between the two parties.
[0089] The present invention also provides a dynamic task scheduling method based on computing resources, including: a control module, a computing module, an Etcd, and a MySQL database server; the control module includes multiple schedulers, the computing module includes multiple nodes, and each node includes a node monitoring module and a task processing module;
[0090] like Figure 3 As shown, the dynamic task scheduling method based on computing resources includes the following steps:
[0091] The scheduler executes tasks based on the task scheduling calculation module submitted by the user. The scheduler completes the election function of the primary scheduler and the backup scheduler based on the high availability and strong data consistency characteristics provided by Etcd, thereby achieving high availability of the scheduler. Under normal service conditions, only the primary scheduler is responsible for scheduling services. When the primary scheduler encounters an anomaly, one of the backup schedulers will immediately detect the anomaly and be elected as the primary scheduler to continue processing services.
[0092] The dynamic task scheduling method based on computing resources also includes a node resource status monitoring step: the monitoring module is responsible for communicating with the scheduler and transmitting the current node resource status to the current scheduler. After the main scheduler sends the task to the monitoring module, the monitoring module is responsible for preparing the data required for the task and starting the task. After the task is completed, the task execution result is returned to the main scheduler.
[0093] In some embodiments, the dynamic task scheduling method based on computing resources further includes a task resource requirement assessment step: for different tasks, in order to achieve better scheduling results, the tasks submitted by users are assessed for sensitive resources, and the required resource quantity is calculated. This resource quantity will serve as an important criterion for subsequent node selection.
[0094] In some embodiments, the dynamic task scheduling method based on computing resources further includes a node pre-selection step: the scheduler searches for nodes among the currently online nodes whose remaining resources meet the requirements based on the evaluation results of the task resource requirement assessment module; when the node pool is large, the search starting point and step size are determined by using random numbers; when the number of nodes that meet the requirements reaches a certain number, the pre-selection can be stopped, and the pre-selection results are passed to the next step.
[0095] In some embodiments, the dynamic task scheduling method based on computing resources further includes a load balancing step: the scheduler scores the pre-selected nodes, the score depends on the various resources required by the task, and the product of the resources and their weights is used as a positive parameter; the node penalty value is used as a negative parameter, which is calculated by the number of failures using an exponentially weighted moving average algorithm. The higher the number of failures in a short period of time, the higher the penalty value; otherwise, the penalty value will gradually decrease to 1 as the task is successfully executed.
[0096] Meanwhile, the number of tasks currently being executed by a node is also included as a negative parameter in the calculation, aiming to balance the load. In addition, to avoid scheduling a large number of tasks to the same node under heavy load, the algorithm first randomly selects two nodes before evaluating the node score, evaluates the scores of the selected nodes, and selects the better node.
[0097] In some embodiments, the load balancing step uses the following functions to obtain weights, penalty values, and node scores:
[0098] The weights are moving average weights, calculated using the following formula (Formula 1):
[0099] w = e (last-now) / lifetime ;
[0100] Where e is a mathematical constant, last is the time when the node was last selected, now is the current time, and lifetime is a custom time period. The time lookup value within this period has a greater impact, and it reaches its half-life after lifetime×ln(2).
[0101] The penalty value is calculated using the following formula (Formula 2):
[0102] penty = old × w + new × (1 - w);
[0103] Where old is the penalty value calculated in the previous calculation, new is the penalty value in the current calculation (1000 for failure, 1 for success), and w is the weight calculated by Formula 1.
[0104] The node scoring formula is as follows:
[0105]
[0106] Where resVal is the remaining resource value, resWeight is the resource weight, inflight is the number of tasks currently being processed by the node, and penty is the penalty value calculated by formula two. In some embodiments, the dynamic task scheduling method based on computing resources further includes a node circuit breaker check step: the scheduler evaluates whether a node can continue to provide service, records the number of errors, and decides whether to continue scheduling tasks to the node, reduce invalid network requests, or reduce scheduling latency, or one or more of these options; the circuit breaker algorithm calculation formula is as follows:
[0107]
[0108] Where k is the tolerance level, decreasing k makes the rate limiting algorithm more aggressive, and more traffic will be rejected, while increasing k makes the rate limiting algorithm more lenient, allowing more traffic; accepts is the number of normal requests in the current time window, and requests is the total number of requests in the current time window;
[0109] When requests increase infinitely while accepts stop increasing, f(a) approaches 1 infinitely, triggering the circuit breaker; conversely, when accepts increase, due to the effect of k, f(a) approaches 0 infinitely, releasing the circuit breaker.
[0110] In some embodiments, the dynamic task scheduling method based on computing resources further includes an overload protection step: after a task is submitted to a computing node by the scheduler, the computing node compares the current system resources with the resources required by the task. When the remaining resources are insufficient to execute the task, overload protection is triggered, and the task is immediately rejected by the scheduler and a resource shortage response is made to avoid problems such as OOM (Out of Memory) caused by insufficient resources.
[0111] In some embodiments, the dynamic task scheduling method based on computing resources employs the following data transmission protocol: a Request-Response model is constructed using the TCP protocol, and ProtoBuf data (Google Protocol Buffer, a structured data storage format) is packaged for transmission between the two parties.
[0112] The beneficial effects of this technical solution are:
[0113] 1. This solution implements its own scheduling algorithm. For different types of tasks, adaptation can be achieved by modifying the weights of various resources in the node scoring process.
[0114] 2. The solution provides a complete scheduling algorithm, including load balancing, circuit breaking, and overload protection, fully considering various issues in scheduling scenarios. Load balancing ensures that the load across nodes is as level as possible. The circuit breaking mechanism prevents unnecessary data interaction; direct rejection at the scheduler end reduces network I / O and lowers scheduling latency. Overload protection ensures that nodes will not experience unpredictable errors or task execution failures due to resource shortages. These three mechanisms complement each other, forming a complete scheduling algorithm and building a stable scheduling platform.
[0115] 3. The solution provides a high availability architecture. The scheduler is based on the Raft distributed consensus protocol provided by Etcd and forms a master-slave cluster. When the master scheduler fails, the standby scheduler will immediately elect a master scheduler to continue processing.
[0116] 4. The solution eliminates the HTTP protocol and uses TCP in conjunction with Protobuf encoding to achieve efficient communication between the scheduler and the monitoring program, allowing the scheduler to perceive the status of each node more promptly;
[0117] 5. Supports containerized deployment, simplifying operation and maintenance.
[0118] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A dynamic task scheduling device based on computing resources, characterized in that, include: Control module, computing module, Etcd, and MySQL database server; The control module includes multiple schedulers, and the computing module includes multiple nodes, each node including a node monitoring module and a task processing module. The scheduler is used to execute tasks according to the task scheduling calculation module submitted by the user. The scheduler completes the election function of the primary scheduler and the backup scheduler based on the high availability and strong data consistency characteristics provided by Etcd, thereby realizing the high availability of the scheduler. Under normal service conditions, only the primary scheduler is responsible for scheduling services. When the primary scheduler fails, one of the backup schedulers will immediately detect the failure and elect itself as the primary scheduler to continue processing services. The monitoring module is responsible for communicating with the scheduler and transmitting the current node resource status to the current master scheduler. After the master scheduler sends the task to the monitoring module, the monitoring module is responsible for preparing the data required for the task and starting the task. After the task is completed, the task execution result is sent back to the master scheduler. The scheduler also includes a task resource requirement assessment module, which assesses the sensitive resources of user-submitted tasks for different tasks and calculates the required resource quantity. This resource quantity will serve as an important criterion for subsequent node selection. The scheduler also includes a node pre-selection module, which is used to find nodes among the currently online nodes whose remaining resources meet the requirements based on the evaluation results of the task resource requirement assessment module. When the node pool is large, the search starting point and step size are determined by random numbers. When the number of nodes that meet the requirements reaches a certain number, the pre-selection can be stopped and the pre-selection results can be passed to the next step. The scheduler also includes a load balancing module for scoring pre-selected nodes. The scoring depends on the various resources required by the task, and the product of the resources and their weights is used as a positive parameter. The node penalty value is used as a negative parameter, which is calculated by the number of failures using an exponentially weighted moving average algorithm. The higher the number of failures in a short period of time, the higher the penalty value. Otherwise, the penalty value will gradually decrease to 1 as the task is successfully executed. The load balancing module uses the following functions to obtain the weight, penalty value, and node score: The weights are moving average weights, and the calculation formula is as shown in Formula 1: ； Where e is a mathematical constant, last is the time when the node was last selected, now is the current time, and lifetime is a custom time period. The time lookup value within this period has a greater impact, and it reaches its half-life after lifetime × ln(2). The penalty value is calculated using Formula 2: ； Where old is the penalty value calculated in the previous calculation, new is the penalty value in the current calculation (1000 for failure, 1 for success), and w is the weight calculated by Formula 1. The node scoring formula is as follows: ； Where resVal is the remaining resource value, resWeight is the resource weight, inflight is the number of tasks currently being processed by the node, and penty is the penalty value calculated by formula two.
2. The dynamic task scheduling device based on computing resources according to claim 1, characterized in that, The scheduler also includes a node circuit breaker checking module, used to evaluate whether a node can continue to provide service. The node circuit breaker checking module records the number of errors and decides whether to continue scheduling tasks to that node, reduce invalid network requests, or reduce scheduling latency, among other things. The circuit breaker algorithm calculation formula is as follows: ； Where k is the tolerance level, decreasing k makes the rate limiting algorithm more aggressive, and more traffic will be rejected, while increasing k makes the rate limiting algorithm more lenient, allowing more traffic; accepts is the number of normal requests in the current time window, and requests is the number of all requests in the current time window; When requests increase infinitely while accepts stop increasing, f(a) approaches 1 infinitely, triggering the circuit breaker; conversely, when accepts increase, due to the effect of k, f(a) approaches 0 infinitely, releasing the circuit breaker.
3. The dynamic task scheduling device based on computing resources according to claim 2, characterized in that, The scheduler also includes an overload protection module, which operates on the computing node side. When a task is submitted to the computing node by the scheduler, the computing node compares the current system resources with the resources required by the task. When the remaining resources are insufficient to execute the task, the overload protection is triggered, and the task is immediately rejected by the scheduler with a resource shortage response to avoid OOM problems caused by insufficient resources.
4. The dynamic task scheduling device based on computing resources according to any one of claims 1-3, characterized in that, The scheduler also includes a data transmission protocol module, which uses the TCP protocol to build a Request-Response model and packages ProtoBuf data for transmission between the two parties.
5. A dynamic task scheduling method based on computing resources, characterized in that, include: Control module, computing module, Etcd, and MySQL database server; The control module includes multiple schedulers, and the computing module includes multiple nodes, each node including a node monitoring module and a task processing module. The dynamic task scheduling method based on computing resources includes the following steps: The scheduler executes tasks based on the task scheduling calculation module submitted by the user. The scheduler leverages the high availability and strong data consistency features provided by Etcd to elect a primary and backup scheduler, thus achieving high availability. Under normal service conditions, only the primary scheduler is responsible for scheduling services. When the primary scheduler malfunctions, one of the backup schedulers will immediately detect the anomaly and be elected as the primary scheduler to continue processing services. The dynamic task scheduling method based on computing resources further includes a node resource status monitoring step: the monitoring module communicates with the scheduler, transmitting the current node resource status to the current scheduler. After the primary scheduler sends a task to the monitoring module, the monitoring module prepares the necessary data for the task and starts the task. After the task is completed, the task execution result is returned to the primary scheduler. The dynamic task scheduling method based on computing resources also includes a task resource requirement assessment step: for different tasks, the user-submitted tasks are assessed for sensitive resources, and the required resource quantity is calculated. This resource quantity will serve as an important criterion for subsequent node selection. The dynamic task scheduling method based on computing resources also includes a node pre-selection step: the scheduler searches for nodes among the currently online nodes whose remaining resources meet the requirements based on the evaluation results of the task resource requirement assessment module; when the node pool is large, the search starting point and step size are determined by random numbers; when the number of nodes that meet the requirements reaches a certain number, the pre-selection can be stopped and the pre-selection results can be passed to the next step. The dynamic task scheduling method based on computing resources also includes a load balancing step: the scheduler scores the pre-selected nodes, and the score depends on the various resources required by the task. The product of the resources and their weights is used as a positive parameter; the node penalty value is used as a negative parameter, which is calculated by the number of failures using an exponentially weighted moving average algorithm. The higher the number of failures in a short period of time, the higher the penalty value. Otherwise, the penalty value will gradually decrease to 1 as the task is successfully executed. The load balancing step uses the following functions to obtain weights, penalty values, and node scores: The weights are moving average weights, and the calculation formula is as shown in Formula 1: ； Where e is a mathematical constant, last is the time when the node was last selected, now is the current time, and lifetime is a custom time period. The time lookup value within this period has a greater impact, and it reaches its half-life after lifetime × ln(2). The penalty value is calculated using Formula 2: ； Where old is the penalty value calculated in the previous calculation, new is the penalty value for the current calculation (1000 for failure, 1 for success), and w is the weight calculated by formula one. The node scoring formula is as follows: ； Where resVal is the remaining resource value, resWeight is the resource weight, inflight is the number of tasks currently being processed by the node, and penty is the penalty value calculated by formula two.
6. The dynamic task scheduling method based on computing resources according to claim 5, characterized in that, The dynamic task scheduling method based on computing resources also includes a node circuit breaker check step: the scheduler evaluates whether a node can continue to provide service, records the number of errors, and decides whether to continue scheduling tasks to that node, reduce invalid network requests, or reduce scheduling latency, among other things; the circuit breaker algorithm calculation formula is as follows: ； Where k is the tolerance level, decreasing k makes the rate limiting algorithm more aggressive, and more traffic will be rejected, while increasing k makes the rate limiting algorithm more lenient, allowing more traffic; accepts is the number of normal requests in the current time window, and requests is the number of all requests in the current time window; When requests increase infinitely while accepts stop increasing, f(a) approaches 1 infinitely, triggering the circuit breaker; conversely, when accepts increase, due to the effect of k, f(a) approaches 0 infinitely, releasing the circuit breaker.
7. The dynamic task scheduling method based on computing resources according to claim 6, characterized in that, The dynamic task scheduling method based on computing resources also includes an overload protection step: when a task is submitted to a computing node by the scheduler, the computing node compares the current system resources with the resources required by the task. When the remaining resources are insufficient to execute the task, overload protection is triggered, and the task is immediately rejected by the scheduler with a resource shortage response.
8. The dynamic task scheduling method based on computing resources according to any one of claims 5-7, characterized in that, The dynamic task scheduling method based on computing resources adopts the following data transmission protocol: a Request-Response model is constructed using the TCP protocol, and ProtoBuf data is packaged for transmission between the two parties.