Distributed decentralized task scheduling system
Through the distributed decentralized task scheduling system, the modular design and real-time monitoring mechanism are used to solve the scalability and single-point failure problems of the traditional centralized scheduling method, the efficient and reliable allocation and execution of tasks are achieved, and the system is improved robustness and user satisfaction.
Patent Information
- Application Number
- CN202411027912.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The traditional centralized scheduling method faces the problems of scalability and single point of failure in large-scale data centers, resulting in insufficient system reliability and scalability, high maintenance costs and limited processing capabilities.
A distributed decentralized task scheduling system is designed, including task type hierarchical module, node monitoring and computing module, node selection module, load balancing module, fault-tolerant self-healing module and accuracy verification module, and real-time monitoring of node performance and grading Task priority, balanced task allocation, self-recovery and accuracy verification ensure that tasks are executed on nodes with sufficient capabilities and avoid overloading and resource waste.
Improves the robustness and reliability of the system, ensures that tasks dynamically adjust and rebalance loads when node failures or performance deteriorates, reduces waiting time, optimizes resource utilization, enhances the system's adaptability to different workloads, and improves users Trust in system performance and stability.
Smart Images

Figure CN118963946A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of task scheduling, and in particular to a distributed decentralized task scheduling system. Background Art
[0002] With the rapid development of information technology and the growing demand for computing, distributed and decentralized task scheduling systems have become a key technology for processing large-scale distributed computing tasks. These systems play a vital role in many fields such as cloud computing, big data processing, real-time analysis, and grid computing. With the expansion of the scale of data centers and the increase in the complexity of computing tasks, traditional centralized scheduling methods face the challenges of scalability and single point failure. There is an urgent need for a scheduling mechanism that can adapt to dynamic changes and has high fault tolerance and adaptability.
[0003] The patent document with publication number CN115061814A discloses a distributed high-concurrency scheduling system based on decentralized job tasks, including distributed message queues; nodes; nodes including schedulers and executors; memory databases; used as caches; nodes send data to designated distributed message queues for consumption by other nodes, thereby realizing data interaction and communication with other nodes; important data sent is automatically saved in the memory database to improve data persistence efficiency. In this distributed system, the message queue may face the risk of data loss or repeated consumption, especially in the case of network partitions or node failures; although the memory database can provide fast data access and write speeds, its data is mainly stored in volatile memory, which may cause data loss in the event of system failure; the performance and reliability of the system may be highly dependent on network conditions, and network delays or instability may affect the overall performance of the system. Summary of the invention
[0004] To this end, the present invention provides a distributed decentralized task scheduling system to overcome the problems of high maintenance cost and limited processing capacity caused by insufficient system reliability and scalability due to single point failure problems in the prior art.
[0005] To achieve the above objectives, the present invention provides a distributed decentralized task scheduling system, comprising:
[0006] A task type classification module is used to classify tasks according to the task types of several input tasks;
[0007] A node monitoring and calculation module, connected to the task type classification module, is used to monitor the real-time processor usage rate, real-time memory occupancy rate and real-time disk reading speed of each node, and calculate the node processing capacity value according to the monitoring results;
[0008] A node selection module, connected to the task type classification module and the node monitoring and calculation module respectively, for selecting the number of nodes according to the node processing capacity value and the priority of the task;
[0009] A load balancing module, connected to the node selection module and the node monitoring and calculation module respectively, for calculating the task allocation of each node according to the number of selected nodes, the node processing capacity value of each node, and the total task volume, and executing the task according to the calculation result;
[0010] A fault-tolerant self-healing module, connected to the load balancing module and the node selection module respectively, for controlling the node selection module to reselect the number of nodes that have not completed tasks according to the sum of the task allocations actually executed by each node, the total task amount and a preset standard deviation value;
[0011] The accuracy verification module is connected to the node selection module and the fault-tolerant self-healing module respectively, and is used to select whether to adjust the standard deviation value according to the deviation between the accuracy of executing the task and the preset standard accuracy.
[0012] Furthermore, the task type classification module assigns a priority to each task according to the input task type and a preset classification standard, and the priority includes a low priority, a medium priority and a high priority.
[0013] Furthermore, the node monitoring and calculation module standardizes the real-time processor usage, real-time memory occupancy and real-time disk reading speed of each corresponding node, and multiplies the real-time processor usage, real-time memory occupancy and real-time disk reading speed with the corresponding weights to sum up and obtain the node processing capacity value.
[0014] Furthermore, the node selection module selects the number of nodes according to the node processing capability value and the priority of the task, including:
[0015] The node selection module selects a single node according to a node processing capability value that is greater than a preset standard capability value.
[0016] Furthermore, the node selection module calculates the number of selected nodes according to the node processing capability value that is less than or equal to a preset standard capability value.
[0017] Furthermore, the load balancing module calculates the task allocation of each node according to the node processing capacity value of each node, including:
[0018] The load balancing module calculates the proportion of the node processing capacity value of each node according to the node processing capacity value of each node and the sum of the node processing capacity values of all nodes, obtains several task allocation ratios, and rounds down the product of the total task volume and each task allocation ratio to obtain the task allocation volume of each node.
[0019] Furthermore, the fault-tolerant self-healing module controls the node selection module to reselect nodes according to the sum of the task allocations actually executed by each node, the total task amount, and a preset standard deviation value, including:
[0020] The fault-tolerant self-healing module calculates the difference between the sum of the task allocation amounts actually executed by each node and the total task amount, and controls the node selection module to reselect the number of nodes that have not completed the task if the difference is greater than the standard deviation.
[0021] Furthermore, the fault-tolerant self-healing module controls the node selection module to reselect the number of nodes that have not completed the task, including:
[0022] The fault-tolerant self-healing module calculates the total remaining processing capacity of all nodes based on the current remaining processing capacity of each node and the total number of all current nodes, calculates the additional processing capacity requirement for reselecting nodes based on the average processing capacity required for each task, the total remaining processing capacity, and the total number of unfinished tasks, and calculates the number of nodes to be reselected for unfinished tasks based on the average processing capacity required for each task and the additional processing capacity requirement.
[0023] Furthermore, when the difference is less than or equal to the standard deviation, the fault-tolerant self-healing module waits for the current task to be completed and then allocates the remaining unallocated tasks according to the system load and node status.
[0024] Furthermore, when the deviation between the accuracy of executing the task and the preset standard accuracy is greater than or equal to the preset standard deviation, the accuracy verification module reduces and adjusts the standard deviation value according to a preset difference adjustment coefficient.
[0025] Compared with the prior art, the beneficial effect of the present invention lies in that the system first monitors the real-time performance indicators of each node by utilizing the node monitoring and calculation module, and determines the priority of the task in combination with the task type classification module to provide data support for subsequent task allocation. The node selection module selects the node to execute the task according to the processing capacity of the node and the priority of the task. The load balancing module further ensures that the task is evenly distributed among the selected nodes to avoid overload and waste of resources. When the system detects that the task execution deviates from the expectation through the fault-tolerant self-healing module, it will trigger the process of reselecting the node to ensure the reliability of the task and the stability of the system. The accuracy verification module monitors the accuracy of task execution, compares it with the preset standard, and adjusts the standard deviation value when necessary to improve the system's prediction accuracy of future task execution quality. The above modules successfully solve the problems of insufficient system reliability and scalability caused by single point failure problems.
[0026] Furthermore, the system compares the processing capacity of the nodes with the preset standard capacity value and selects multiple nodes with processing capacity values less than or equal to the standard capacity value to perform the task. This selection mechanism ensures that the task is assigned to nodes with sufficient capacity while avoiding overloading those nodes with insufficient capacity.
[0027] Furthermore, by monitoring the difference between the actual execution of tasks and expectations, the system can promptly detect and handle anomalies or failures during execution, thereby ensuring the final completion of the task. When it is found that there is a large deviation between the execution of tasks and expectations, the system will automatically reallocate tasks, which helps the system recover from node failures or performance degradation. The process of reselecting nodes takes into account the current status and remaining processing capacity of each node, which helps to allocate resources more reasonably and avoid resource waste.
[0028] Furthermore, by accurately evaluating the deviation of the system in task execution and intelligently reallocating tasks to nodes with sufficient remaining processing capacity, the robustness and reliability of the distributed task scheduling system are significantly improved. This module ensures that even when some nodes have insufficient performance or failures, the system can dynamically adjust and rebalance the load to ensure task continuity, reduce waiting time, and optimize overall resource utilization. In addition, it enhances the system's adaptability to different workloads by responding to abnormal situations in task execution in a timely manner, thereby improving users' trust in system performance and stability.
[0029] Furthermore, the accuracy verification module detects that the accuracy of task execution is lower than the preset standard, and the deviation exceeds the preset standard deviation. According to the preset difference adjustment coefficient, the module reduces the standard deviation to improve the system's strict control over the quality of task execution. This helps the system to more accurately match the processing capacity of nodes in future task allocation, thereby improving the overall execution accuracy and user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 This is a schematic diagram of the structure of the distributed decentralized task scheduling system of this embodiment;
[0031] Figure 2 A decision logic diagram for selecting the number of nodes for this embodiment;
[0032] Figure 3 A decision logic diagram for reselecting the number of nodes that have not completed tasks in this embodiment;
[0033] Figure 4 This is a decision logic diagram for adjusting the standard deviation value in this embodiment. DETAILED DESCRIPTION
[0034] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0035] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.
[0036] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.
[0037] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0038] See also Figure 1 As shown, it is a schematic diagram of the structure of the distributed decentralized task scheduling system of this embodiment;
[0039] This embodiment provides a distributed decentralized task scheduling system, including:
[0040] A node monitoring and calculation module is connected to a task type classification module for classifying the priorities of tasks into low priority, medium priority and high priority according to the task types of several input tasks, and is used to monitor the real-time processor usage rate, real-time memory occupancy rate and real-time disk reading speed of each node, and calculate the node processing capacity value according to the monitoring results;
[0041] A node selection module, connected to the task type classification module and the node monitoring and calculation module respectively, for selecting a number of nodes according to the node processing capacity value and the priority of the task;
[0042] A load balancing module, connected to the node selection module and the node monitoring and calculation module respectively, for calculating the task allocation of each node according to the number of selected nodes, the node processing capacity value of each node, and the total task volume, and executing the task according to the calculation result;
[0043] A fault-tolerant self-healing module, connected to the load balancing module and the node selection module respectively, for controlling the node selection module to reselect the number of nodes that have not completed tasks according to the sum of the task allocations actually executed by each node, the total task amount and a preset standard deviation value;
[0044] The accuracy verification module is connected to the node selection module and the fault-tolerant self-healing module respectively, and is used to select whether to adjust the standard deviation value according to the deviation between the accuracy of executing the task and the preset standard accuracy.
[0045] The system first uses the node monitoring and calculation module to monitor the real-time performance indicators of each node, and combines the task type classification module to determine the priority of the task, providing data support for subsequent task allocation. The node selection module selects the node to execute the task based on the node's processing power and the priority of the task. The load balancing module further ensures that the tasks are evenly distributed among the selected nodes to avoid overload and waste of resources. When the system detects that the task execution deviates from the expectation through the fault-tolerant self-healing module, it will trigger the process of reselecting nodes to ensure the reliability of the task and the stability of the system. The accuracy verification module monitors the accuracy of task execution, compares it with the preset standard, and adjusts the standard deviation value when necessary to improve the system's prediction accuracy of the quality of future task execution.
[0046] Specifically, the task type classification module assigns a priority to each task according to the input task type and a preset classification standard, and divides the tasks into low priority, medium priority and high priority.
[0047] Specifically, the node monitoring and calculation module standardizes the real-time processor usage, real-time memory occupancy and real-time disk reading speed of each node it monitors, and sums the product of the real-time processor usage and the preset usage weight, the product of the real-time memory occupancy and the preset occupancy weight, and the product of the real-time disk reading speed and the preset reading speed weight to obtain the node processing capacity value.
[0048] Specifically, the node selection module selects several nodes according to the node processing capacity value and the priority of the task and executes the task with a preset task allocation, including:
[0049] If the priority of the task is high priority, and the node processing capacity value is greater than the preset standard capacity value, the node selection module selects a single node to execute the task;
[0050] The processing power of a single node is sufficient to handle high-priority tasks without the need for additional resources. It can respond quickly to high-priority tasks, reduce task waiting time, and improve the system's processing efficiency for critical tasks.
[0051] If the priority of the task is high priority, and the node processing capacity value is less than or equal to the preset standard capacity value, the node selection module calculates the number of nodes according to the node processing capacity value being less than the preset standard capacity value, and selects the number of nodes according to the calculation result to execute the task with the preset task allocation amount;
[0052] A single node cannot independently undertake high-priority tasks, and multiple nodes need to collaborate to meet task requirements. Parallel processing can speed up task completion, improve resource utilization, and ensure timely completion of tasks.
[0053] If the priority of the task is low priority, and the node processing capacity value is greater than the preset standard capacity value, the node selection module selects a single node to execute the task;
[0054] Low-priority tasks have low resource requirements and can be handled by a single high-performance node. The idle resources of the node can be used rationally to avoid resource waste while maintaining the system's responsiveness to high-priority tasks.
[0055] If the priority of the task is low, and the node processing capacity value is less than or equal to the preset standard capacity value, the node selection module waits for tasks of other priorities to be completed before executing.
[0056] When resources are tight, high-priority tasks are prioritized for execution, and low-priority tasks can wait, ensuring that the resource requirements of critical tasks are met, avoiding system overload, and maintaining system stability.
[0057] Specifically, the node monitoring calculation module standardizes the real-time processor usage, real-time memory occupancy and real-time disk reading speed of each node it monitors, and sums the product of the real-time processor usage and the preset usage weight, the product of the real-time memory occupancy and the preset occupancy weight, and the product of the real-time disk reading speed and the preset reading speed weight to obtain the node processing capacity value, wherein, assuming that there is a node A, its real-time CPU usage is 90%, the real-time memory occupancy is 80%, the real-time disk reading speed is 150MB / s, and the set weights are wCPU=0.5, wMemory=0.3, and wDis sk=0.2. Assume that the maximum and minimum values of these indicators are:
[0058] CPU usage: 0%-100%;
[0059] Memory usage: 0%-100%;
[0060] Disk read speed: 50MB / s-300MB / s;
[0061] First, standardize it:
[0062] Normalized CPU usage is (90%-0%) / (100%-0%)=0.9;
[0063] The normalized memory usage is (80%-0%) / (100%-0%)=0.8;
[0064] The normalized disk read speed is (150-50) / (300-50)=0.4;
[0065] Then calculate the processing capacity value of node A: the processing capacity value of node A is (0.5×0.9)+(0.3×0.8)+(0.2×0.4)=0.45+0.24+0.08=0.788;
[0066] In this way, the comprehensive processing capacity value of node A is obtained, and the processing capacity values of other nodes can be calculated similarly, which can then be used by the node selection module to select nodes suitable for executing tasks.
[0067] Please continue reading Figure 2 As shown, it is a decision logic diagram for selecting the number of nodes in this embodiment;
[0068] Specifically, the node selection module selects the number of nodes according to the node processing capability value and the priority of the task, including:
[0069] If the node processing capability value is greater than a preset standard capability value, the node selection module selects a single node.
[0070] Assume that there are three nodes A, B, and C, whose processing capacity values are 75, 85, and 60 respectively, and the standard capacity value is set to 70. According to the above rules, only the processing capacity value of node B is greater than the standard capacity value, so the node selection module will select node B.
[0071] Specifically, the node selection module calculates the number of selected nodes according to the node processing capability value that is less than or equal to the preset standard capability value.
[0072] For example, the preset standard capability value S=70.
[0073] There are 5 nodes, and their processing capacity values are: C1=65, C2=50, C3=70, C4=45, C5=65.
[0074] List the processing power values of all nodes:
[0075] Node 1: C1=65;
[0076] Node 2: C2 = 50;
[0077] Node 3: C3=70;
[0078] Node 4: C4=45;
[0079] Node 5: C5=65;
[0080] Filter nodes whose processing capacity value is less than or equal to the standard capacity value:
[0081] The processing capacity values of Node 1, Node 2, Node 4, and Node 5 are all less than or equal to 70, so these nodes will be considered for task allocation.
[0082] Calculate the number of selected nodes:
[0083] In this example, there are 4 nodes (Node 1, Node 2, Node 4, and Node 5) that meet the criteria.
[0084] The system compares the processing capacity of the nodes with the preset standard capacity value and selects multiple nodes with processing capacity values less than or equal to the standard capacity value to perform the task. This selection mechanism ensures that the task is assigned to nodes with sufficient capacity while avoiding overloading those nodes with insufficient capacity.
[0085] Specifically, the load balancing module calculates the task allocation amount of each node according to the node processing capacity value of each node, including:
[0086] The load balancing module calculates the proportion of the node processing capacity value of each node according to the node processing capacity value of each node and the sum of the node processing capacity values of all nodes, obtains several task allocation ratios, and rounds down the product of the total task volume and each task allocation ratio to obtain the task allocation volume of each node.
[0087] Assumptions:
[0088] Assume that there are three nodes, and their processing capabilities are respectively: E1=100, E2=150, and E3=200.
[0089] The sum of the processing capacity values of all nodes is Etotal=E1+E2+E3=100+150+200=450.
[0090] Assume that the total task volume is Ttotal = 1000 task units.
[0091] Implementation steps:
[0092] Calculate the proportion of each node's processing capacity:
[0093] The proportion of node 1: E1 / Etotal=100 / 450;
[0094] The proportion of node 2: E2 / Etotal=150 / 450;
[0095] The proportion of node 3: E3 / Etotal=200 / 450;
[0096] Calculate the task allocation ratio of each node based on the proportion:
[0097] The task allocation ratio of node 1 is approximately equal to 0.2222;
[0098] The task allocation ratio of node 2 is approximately equal to 0.3333;
[0099] The task allocation ratio of node 3 is approximately equal to 0.4444;
[0100] Calculate the task allocation for each node:
[0101] Task allocation amount T1 of node 1: T1≈222;
[0102] Task allocation amount T2 of node 2: T2≈333;
[0103] Task allocation amount T3 of node 3: T3≈444;
[0104] Round down to get the task allocation for each node:
[0105] Since the task allocation may need to be an integer, the calculation result is rounded down:
[0106] T1 = 222 (already an integer, no need to round down);
[0107] T2 = 333 (already an integer, no need to round down);
[0108] T3 = 444 (already an integer, no need to round down);
[0109] In this embodiment, the load balancing module calculates the task allocation ratio of each node based on the processing capacity value of each node and the sum of the processing capacity values of all nodes. Then, the total task amount is multiplied by these ratios and rounded down to obtain the task allocation amount of each node. Node 1, node 2 and node 3 are allocated 222, 333 and 444 task units respectively. This allocation method ensures that tasks are evenly distributed among the nodes according to their processing capabilities.
[0110] Please continue reading Figure 3 As shown, it is a decision logic diagram for reselecting the number of nodes that have not completed tasks in this embodiment;
[0111] Specifically, the fault-tolerant self-healing module controls the node selection module to reselect nodes according to the sum of the task allocations actually executed by each node, the total task amount, and the preset standard deviation value, including:
[0112] The fault-tolerant self-healing module calculates the difference between the sum of the task allocation amounts actually executed by each node and the total task amount, and controls the node selection module to reselect the number of nodes that have not completed the task if the difference is greater than the standard deviation.
[0113] By monitoring the difference between the actual execution of tasks and expectations, the system can promptly detect and handle anomalies or failures in the execution process, thereby ensuring the final completion of the task. When it is found that there is a large deviation between the execution of tasks and expectations, the system will automatically reallocate tasks, which helps the system recover from node failures or performance degradation. The process of reselecting nodes takes into account the current status and remaining processing capacity of each node, which helps to allocate resources more reasonably and avoid resource waste.
[0114] Specifically, the fault-tolerant self-healing module controls the node selection module to reselect the number of nodes that have not completed the task, including:
[0115] The fault-tolerant self-healing module calculates the total remaining processing capacity of all nodes based on the current remaining processing capacity of each node and the total number of all current nodes, calculates the additional processing capacity requirement for reselecting nodes based on the average processing capacity required for each task, the total remaining processing capacity, and the total number of unfinished tasks, and calculates the number of nodes to be reselected for unfinished tasks based on the average processing capacity required for each task and the additional processing capacity requirement.
[0116] For example, the system has 5 nodes, and the current remaining processing capacities are: Crem1=50, Crem2=40, Crem3=60, Crem4=30, Crem5=50, and the total task volume is Ttota l'=300 task units.
[0117] The total amount of task assignments actually executed by each node is Tactua l=280.
[0118] The default standard deviation value is Sstd=20.
[0119] The average processing power required for each task is Ctask=10 (ie, each task requires 10 units of processing power).
[0120] Calculate the difference Δ:
[0121] The difference Δ=300-280=20.
[0122] Evaluate the difference and standard deviation:
[0123] Since the difference Δ=20 is greater than the standard deviation Sstd=10, the fault-tolerant self-healing module needs to intervene.
[0124] Calculate the total remaining processing capacity Ctota l_rem:
[0125] Ctota l_rem=Crem1+Crem2+Crem3+Crem4+Crem5=50+40+60+30+50=230.
[0126] Calculate the additional processing capacity requirement Cextra_need:
[0127] Cextra_need = Δ × Ctask = 20 × 10 = 200 task units, which is less than the total remaining processing capacity Ctotal_rem;
[0128] Calculate the number of reselected nodes Nreselect:
[0129] Sort the nodes in descending order based on their remaining processing capacity.
[0130] Node 3: Crem3 = 60;
[0131] Node 1: Crem1=50;
[0132] Node 5: Crem5 = 50;
[0133] Node 2: Crem2 = 40;
[0134] Node 4: Crem4=30;
[0135] Select nodes: Start with the node with the highest remaining processing capacity and accumulate their remaining processing capacities one by one until the accumulated value reaches or exceeds Cextra_need.
[0136] Select node 3, remaining Cextra_need=200-60=140.
[0137] Continue to select node 1, and the remaining Cextra_need=140-50=90.
[0138] Continue to select node 5, and the remaining Cextra_need=90-50=40.
[0139] Continue to select node 2, and the remaining Cextra_need=40-40=0.
[0140] At this point, nodes 3, 1, 5, and 2 have been selected, and the sum of their remaining processing capabilities just meets the additional processing capacity requirement Cextra_need.
[0141] The number of reselected nodes Nreselect=4 (nodes 3, 1, 5 and 2 are selected).
[0142] For differences less than or equal to the standard deviation:
[0143] If the difference is less than or equal to the standard deviation, the system will wait for the current task to complete execution and then assign the remaining unassigned tasks based on the system load and node status.
[0144] The robustness and reliability of the distributed task scheduling system are significantly improved by accurately evaluating the deviation of the system in task execution and intelligently reallocating tasks to nodes with sufficient remaining processing capacity. This module ensures that even when some nodes have insufficient performance or failures, the system can dynamically adjust and rebalance the load to ensure task continuity, reduce waiting time, and optimize overall resource utilization. In addition, it enhances the system's adaptability to different workloads by responding to abnormal situations in task execution in a timely manner, thereby improving users' trust in system performance and stability.
[0145] Please continue reading Figure 4 As shown, it is a decision logic diagram for adjusting the standard deviation value in this embodiment;
[0146] The accuracy verification module reduces and adjusts the standard deviation value according to a preset difference adjustment coefficient when the deviation between the accuracy of executing the task and the preset standard accuracy rate is greater than the preset standard deviation.
[0147] The purpose of the accuracy verification module is to monitor the accuracy during the task execution and compare it with the preset standard. If the deviation is found to be outside the acceptable range, the module will adjust the standard deviation value to ensure the accuracy of future task assignments and the overall performance of the system.
[0148] For example, the preset standard accuracy is Accstd=95%.
[0149] The preset standard deviation is Devstd=2%.
[0150] The difference adjustment coefficient is Adjcoef = -0.5 (indicating that the standard deviation value will be reduced by 50% of the deviation value).
[0151] Calculate the accuracy of the task execution:
[0152] The actual accuracy of the task is Accacual = 93%.
[0153] Calculate the accuracy deviation:
[0154] Accuracy deviation Devaccuracy = | Accacual - Accstd | = | 93% - 95% | = 2%.
[0155] The deviation Devaccuracy=2% is equal to the preset standard deviation Devstd=2%.
[0156] Reduce the standard deviation value according to the difference adjustment coefficient: Devstd_new = Devstd + Adjcoef × Devaccuracy.
[0157] Calculation yields Devstd_new = 2% - 0.5 × 2% = 1%.
[0158] The updated standard deviation value is set to 1% for future task assignment and accuracy evaluation.
[0159] The accuracy verification module detects that the accuracy of task execution is lower than the preset standard, and the deviation exceeds the preset standard deviation. According to the preset difference adjustment coefficient, the module reduces the standard deviation to improve the system's strict control over the quality of task execution. This helps the system to more accurately match the processing capacity of nodes in future task allocations, thereby improving overall execution accuracy and user satisfaction.
[0160] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.
[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A distributed decentralized task scheduling system, characterized in that: include: A task type classification module is used to classify tasks according to the task types of several input tasks; A node monitoring and calculation module, connected to the task type classification module, is used to monitor the real-time processor usage rate, real-time memory occupancy rate and real-time disk reading speed of each node, and calculate the node processing capacity value according to the monitoring results; A node selection module, connected to the task type classification module and the node monitoring and calculation module respectively, for selecting the number of nodes according to the node processing capacity value and the priority of the task; A load balancing module, connected to the node selection module and the node monitoring and calculation module respectively, for calculating the task allocation of each node according to the number of selected nodes, the node processing capacity value of each node, and the total task volume, and executing the task according to the calculation result; A fault-tolerant self-healing module, connected to the load balancing module and the node selection module respectively, for controlling the node selection module to reselect the number of nodes that have not completed tasks according to the sum of the task allocations actually executed by each node, the total task amount and a preset standard deviation value; The accuracy verification module is connected to the node selection module and the fault-tolerant self-healing module respectively, and is used to select whether to adjust the standard deviation value according to the deviation between the accuracy of executing the task and the preset standard accuracy.
2. The distributed decentralized task scheduling system according to claim 1, characterized in that: The task type classification module assigns a priority to each task according to the input task type and a preset classification standard, and the priority includes a low priority, a medium priority and a high priority.
3. The distributed decentralized task scheduling system according to claim 2, characterized in that: The node monitoring calculation module normalizes the real-time processor usage, real-time memory occupancy and real-time disk read speed of each corresponding node, and multiplies the real-time processor usage, real-time memory occupancy and real-time disk read speed with the corresponding weights to sum up and obtain the node processing capacity value.
4. The distributed decentralized task scheduling system according to claim 3, characterized in that: The node selection module selects the number of nodes according to the node processing capability value and the priority of the task, including: The node selection module selects a single node according to a node processing capability value that is greater than a preset standard capability value.
5. The distributed decentralized task scheduling system according to claim 4, characterized in that: The node selection module calculates the number of selected nodes according to the node processing capability value that is less than or equal to the preset standard capability value.
6. The distributed decentralized task scheduling system according to claim 5, characterized in that: The load balancing module calculates the task allocation of each node according to the node processing capacity value of each node, including: The load balancing module calculates the proportion of the node processing capacity value of each node according to the node processing capacity value of each node and the sum of the node processing capacity values of all nodes, obtains several task allocation ratios, and rounds down the product of the total task volume and each task allocation ratio to obtain the task allocation volume of each node.
7. The distributed decentralized task scheduling system according to claim 6, characterized in that: The fault-tolerant self-healing module controls the node selection module to reselect nodes according to the sum of the task allocations actually executed by each node, the total task amount, and the preset standard deviation value, including: The fault-tolerant self-healing module calculates the difference between the sum of the task allocation amounts actually executed by each node and the total task amount, and controls the node selection module to reselect the number of nodes that have not completed the task if the difference is greater than the standard deviation.
8. The distributed decentralized task scheduling system according to claim 7, characterized in that: The fault-tolerant self-healing module controls the node selection module to reselect the number of nodes that have not completed the task, including: The fault-tolerant self-healing module calculates the total remaining processing capacity of all nodes based on the current remaining processing capacity of each node and the total number of all current nodes, calculates the additional processing capacity requirement for reselecting nodes based on the average processing capacity required for each task, the total remaining processing capacity, and the total number of unfinished tasks, and calculates the number of nodes to be reselected for unfinished tasks based on the average processing capacity required for each task and the additional processing capacity requirement.
9. The distributed decentralized task scheduling system according to claim 8, characterized in that: When the difference is less than or equal to the standard difference, the fault-tolerant self-healing module waits for the current task to be completed and then allocates the remaining unallocated tasks according to the system load and node status.
10. The distributed decentralized task scheduling system according to claim 9, characterized in that: The accuracy verification module reduces and adjusts the standard deviation value according to a preset difference adjustment coefficient when the deviation between the accuracy of executing the task and the preset standard accuracy rate is greater than or equal to the preset standard deviation.
Citation Information
Patent Citations
Edge computing-based scheduling algorithm and system for local area network
CN112148449A
Large computing task planning method and system under limited resources
CN113886066A
Distributed high-concurrency scheduling system based on decentralized job tasks
CN115061814A
Method and device for determining computing power node
CN116302476A
Distributed computing power scheduling management system and method
CN118132228A
Cited By
Intelligent set top box system based on multi-channel synchronous processing
CN119364072A
