Distributed task scheduling method and device

Abstract

The invention discloses a distributed task scheduling method and device, and belongs to the technical field of distributed task scheduling, and the method comprises the steps: obtaining a current load parameter, a first task label and preset task processing time consumption, and sending a task pulling request signal to a task pool under the condition of judging that the task pool is in an idle state, sending the task to be processed to a task pool to enable the task pool to feed back second task labels of all the tasks to be processed, task estimated load parameters and a task preset timeout time threshold value, and then for each task to be processed, when it is determined that the current task to be processed is a target task which can be currently processed by itself, pulling the target task from the task pool; and finally, processing the target task, and feeding back a processing result to the task pool after the target task is successfully processed. By implementing the method and the device, the problems that the CPU is easy to idle, the overall resource utilization rate is reduced and the task processing efficiency is low due to the adoption of a scheduling mechanism based on an active task pushing mode in the prior art can be solved.

Description

Technical Field

[0001] This invention relates to the field of distributed task scheduling technology, and in particular to a distributed task scheduling method and apparatus. Background Technology

[0002] With the development of digitalization, traditional stand-alone computing is facing unprecedented challenges. To overcome the bottlenecks of stand-alone computing, distributed computing has emerged. Distributed computing significantly improves computing efficiency and system processing capabilities by allocating computing tasks to different computing nodes for parallel processing.

[0003] In modern distributed computing environments, efficient scheduling of cross-node tasks is crucial for improving system performance. The Master-Worker architecture used in existing technologies employs an automated scheduling mechanism based on an active task push model. In this mechanism, the Master (management node) polls the status of the Workers (worker nodes) to process tasks before scheduling and allocating tasks. If the Worker is idle at this time, it will cause CPU idling, resulting in decreased overall resource utilization and low task processing efficiency. Summary of the Invention

[0004] This invention provides a distributed task scheduling method and apparatus, which can solve the problem of existing scheduling mechanisms based on active task push mode, which require the management node to continuously poll the status of the work nodes used to process tasks throughout the entire task allocation process, and then perform task scheduling and allocation according to the status, resulting in CPU idling, reduced overall resource utilization and low task processing efficiency.

[0005] An embodiment of the present invention provides a distributed task scheduling method, applicable to worker nodes, comprising: The system obtains its current load parameters, first task label, and preset task processing time, and determines whether it is in an idle state based on the current load parameters; wherein, the first task label is used to indicate the type of task that the above-mentioned work node is allowed to process; If it determines that it is in an idle state, it sends a task pull request signal to the task pool, so that the task pool will return the second task tag, the estimated load parameters of the task, and the preset timeout threshold of all pending tasks; wherein, the second task tag is used to indicate the task type of the corresponding pending task. For each pending task, based on the above load parameters, first task label, second task label, estimated task load parameters, preset timeout threshold, and preset task processing time, determine whether the current pending task is the target task that it can currently process, and when it is determined that the pending task is the target task, pull the target task from the task pool. The above-mentioned target tasks are processed, and the processing results are fed back to the task pool after successful processing.

[0006] Furthermore, the process of determining whether a system is in an idle state based on its current load parameters includes: Compare the CPU utilization rate in the above load parameters with the preset CPU utilization rate threshold; Compare the memory utilization rate in the above load parameters with the preset memory utilization rate threshold; If the CPU utilization rate does not exceed the preset CPU utilization rate threshold and the memory utilization rate does not exceed the preset memory utilization rate threshold, then it is determined that it is in an idle state; otherwise, it is determined that it is not in an idle state.

[0007] Furthermore, the estimated load parameters for the above tasks include: estimated CPU utilization and estimated memory utilization; The above-mentioned determination of whether the current pending task is a target task that it can currently process, based on the aforementioned load parameters, first task label, second task label, estimated task load parameters, preset task timeout threshold, and preset task processing time, includes: Calculate the sum of the estimated CPU utilization rate and the CPU usage rate to obtain the predicted CPU utilization rate; Calculate the sum of the above estimated memory occupancy rate and the above memory usage rate to obtain the predicted memory usage rate; If the predicted CPU utilization does not exceed the preset CPU utilization threshold, the predicted memory utilization does not exceed the preset memory utilization threshold, the first task label is the same as the second task label, and the preset task processing time does not exceed the preset task timeout threshold, then the current task to be processed is determined to be the target task.

[0008] Furthermore, it also includes: If the target task fails to process, a node failure event signal containing the node parameter information of the working node is sent to the task pool, so that the task pool can reclaim the target task from the working node according to the node parameter information after receiving the node failure event signal.

[0009] Furthermore, it also includes: When the aforementioned working nodes start up, they send the aforementioned node parameter information to the management node so that the management node can record the information. After startup, the aforementioned working node continuously sends a heartbeat event signal containing the aforementioned node parameter information to the management node at preset time intervals. If the management node does not receive the aforementioned heartbeat event signal within the aforementioned preset time interval, it determines that the aforementioned working node has malfunctioned and sends a node malfunction event signal containing the aforementioned node parameter information to the aforementioned task pool. Upon receiving the aforementioned node failure event signal, the task pool reclaims the unprocessed target tasks from the corresponding working node based on the aforementioned node parameter information.

[0010] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments; This invention provides a distributed task scheduling device, comprising: Work nodes and task pools; The aforementioned working node is used to obtain its current load parameters, first task tag, and preset task processing time, and to determine whether it is in an idle state based on the current load parameters; if it determines that it is in an idle state, it sends a task pull request signal to the task pool; wherein, the aforementioned first task tag is used to indicate the type of task that the aforementioned working node is allowed to process. The aforementioned task pool is used to, upon receiving the aforementioned task pull request signal, feed back the second task tag, estimated task load parameters, and preset timeout threshold of all pending tasks to the aforementioned working node; wherein, the aforementioned second task tag is used to indicate the task type of the corresponding pending task. The aforementioned working nodes are also used to determine, for each pending task, whether it is a target task that it can currently process, based on the aforementioned load parameters, first task label, second task label, estimated task load parameters, preset timeout threshold, and preset task processing time. If the pending task is determined to be a target task, the node pulls the target task from the task pool, processes the target task, and feeds back the processing result to the task pool after successful processing.

[0011] Furthermore, the process of determining whether a system is in an idle state based on its current load parameters includes: Compare the CPU utilization rate in the above load parameters with the preset CPU utilization rate threshold; Compare the memory utilization rate in the above load parameters with the preset memory utilization rate threshold; If the CPU utilization rate does not exceed the preset CPU utilization rate threshold and the memory utilization rate does not exceed the preset memory utilization rate threshold, then it is determined that it is in an idle state; otherwise, it is determined that it is not in an idle state.

[0012] Furthermore, the estimated load parameters for the above tasks include: estimated CPU utilization and estimated memory utilization; The above-mentioned determination of whether the current pending task is a target task that it can currently process, based on the aforementioned load parameters, first task label, second task label, estimated task load parameters, preset task timeout threshold, and preset task processing time, includes: Calculate the sum of the estimated CPU utilization rate and the CPU usage rate to obtain the predicted CPU utilization rate; Calculate the sum of the above estimated memory occupancy rate and the above memory usage rate to obtain the predicted memory usage rate; If the predicted CPU utilization does not exceed the preset CPU utilization threshold, the predicted memory utilization does not exceed the preset memory utilization threshold, the first task label is the same as the second task label, and the preset task processing time does not exceed the preset task timeout threshold, then the current task to be processed is determined to be the target task.

[0013] Furthermore, the aforementioned working node is also used to: send a node failure event signal containing the node parameter information of the aforementioned working node to the task pool when the aforementioned target task processing fails. The aforementioned task pool is used to reclaim the aforementioned target task from the aforementioned working node based on the aforementioned node parameter information after receiving the aforementioned node failure event signal.

[0014] Furthermore, it also includes: Management node; The aforementioned working node is also used to send the aforementioned node parameter information to the aforementioned management node at startup; The aforementioned management nodes are used to record the parameter information of the aforementioned nodes; The aforementioned working node is also used to continuously send heartbeat event signals containing the node parameter information to the aforementioned management node at preset time intervals after startup. The aforementioned management node is also used to determine that the aforementioned working node has malfunctioned when it does not receive the aforementioned heartbeat event signal within the aforementioned preset time interval, and to send a node malfunction event signal containing the aforementioned node parameter information to the aforementioned task pool. The aforementioned task pool is also used to reclaim unprocessed target tasks from the corresponding working node based on the aforementioned node parameter information after receiving the aforementioned node failure event signal.

[0015] The embodiments of the present invention have the following beneficial effects: This invention provides a distributed task scheduling method and apparatus. The method includes: acquiring its own current load parameters, a first task tag, and a preset task processing time, and determining whether it is in an idle state based on the current load parameters; wherein the first task tag is used to indicate the type of task that the worker node is allowed to process; if it is determined to be in an idle state, sending a task retrieval request signal to a task pool, so that the task pool can return the second task tags, estimated task load parameters, and preset task timeout thresholds for all pending tasks; wherein the second task tag is used to indicate the task type of the corresponding pending task; then, for each pending task, determining whether the current pending task is a target task that it can currently process based on the load parameters, the first task tag, the second task tag, the estimated task load parameters, the preset task timeout threshold, and the preset task processing time, and if it is determined that the pending task is a target task, retrieving the target task from the task pool; finally, processing the target task and returning the processing result to the task pool after successful processing. Therefore, in this invention, the working node first determines whether it is in an idle state. As long as it is in an idle state, it will actively pull the tasks to be processed. The whole process does not require the management node to continuously poll before allocating tasks, which reduces the problem of CPU idle time and thus improves resource utilization and task processing efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a distributed task scheduling method provided in an embodiment of the present invention.

[0018] Figure 2 This is a flowchart of the operation of a working node provided in an embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of the structure of a distributed task scheduling device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0022] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0025] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0026] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0027] See Figure 1 To address the issue that existing scheduling mechanisms based on proactive task push require the management node to continuously poll the status of worker nodes throughout the task allocation process before scheduling and allocating tasks, this leads to CPU idling, decreased overall resource utilization, and low task processing efficiency. An embodiment of this invention provides a distributed task scheduling method applicable to worker nodes, comprising: Step S101: Obtain the current load parameters, first task tag, and preset task processing time, and determine whether the node is in an idle state based on the current load parameters; wherein, the first task tag is used to indicate the type of task that the above-mentioned work node is allowed to process. Specifically, the aforementioned load parameters refer to the CPU utilization and memory utilization of the worker nodes. Since each worker node handles a fixed type of task (e.g., deep learning tasks, general data processing tasks, and file processing tasks), the first task label indicates the type of task the worker node can handle. The preset task processing time is determined based on the average time taken by the worker node in processing this type of task in the past. For example, if a worker node historically took an average of 20 minutes to process deep learning tasks, then the preset task processing time for that worker node for the "deep learning task" type is 20 minutes.

[0028] In a preferred embodiment, determining whether it is in an idle state based on the current load parameters includes: Compare the CPU utilization rate in the above load parameters with the preset CPU utilization rate threshold; Specifically, the preset CPU utilization threshold is set to 80%.

[0029] Compare the memory utilization rate in the above load parameters with the preset memory utilization rate threshold; Specifically, the aforementioned preset memory usage threshold is set to 70%.

[0030] If the CPU utilization rate does not exceed the preset CPU utilization rate threshold and the memory utilization rate does not exceed the preset memory utilization rate threshold, then it is determined that it is in an idle state; otherwise, it is determined that it is not in an idle state.

[0031] Specifically, if the CPU utilization exceeds the preset CPU utilization threshold, or the memory utilization exceeds the preset memory utilization threshold, it indicates that the local load on the worker node is too high and it is not advisable to add more tasks. It is necessary to wait for the local load to drop to a reasonable range before pulling tasks for processing.

[0032] In this preferred embodiment, the idle state of the working node is determined based on its current load parameters.

[0033] Step S102: When it is determined that it is in an idle state, a task pull request signal is sent to the task pool so that the task pool can return the second task tag, estimated task load parameters and preset timeout threshold of all pending tasks; wherein, the second task tag is used to indicate the task type of the corresponding pending task. Specifically, the second task tag, estimated task load parameters, and preset timeout threshold of all the above-mentioned pending tasks will be uploaded to the task pool when a new task is submitted to the task pool, so that the worker nodes can obtain them.

[0034] Specifically, after a new task is generated, it will be submitted to the task pool. The task pool will store the task ID, priority, task type (i.e., the second task tag mentioned above), task-related parameters, and the task's preset timeout threshold for these new tasks. This information will be uniformly used as a "task availability event" to be triggered when a new task is generated, and then sent to the task pool after being triggered.

[0035] Specifically, when a worker node determines that it is in an idle state, it will trigger a "task pull request event". This event will send the worker node's node ID, load weight (between 0 and 1, the smaller the value, the more idle it is) and capability tag (such as "GPU" or "high memory") to the task pool, and then the task pool will provide feedback.

[0036] Step S103: For each task to be processed, based on the above load parameters, first task label, second task label, estimated task load parameters, preset timeout threshold, and preset task processing time, determine whether the current task to be processed is the target task that it can currently process, and when it is determined that the task to be processed is the target task, pull the target task from the task pool. Specifically, each work node will determine each pending task one by one. Once the current pending task is determined to be the target task, the target task can be retrieved from the task pool.

[0037] Specifically, the process of pulling the target task from the task pool to the corresponding worker node after the target task is determined is also called the "task allocation event". When pulling the target task, the corresponding task ID and task metadata are also pulled at the same time so that the worker node can execute the corresponding task content.

[0038] Preferably, the working nodes actively pull target tasks, achieving adaptive load balancing and improving resource utilization by more than 30%.

[0039] In a preferred embodiment, the above-mentioned task load estimation parameters include: estimated CPU utilization and estimated memory utilization; The above-mentioned determination of whether the current pending task is a target task that it can currently process, based on the aforementioned load parameters, first task label, second task label, estimated task load parameters, preset task timeout threshold, and preset task processing time, includes: Calculate the sum of the estimated CPU utilization rate and the CPU usage rate to obtain the predicted CPU utilization rate; Calculate the sum of the above estimated memory occupancy rate and the above memory usage rate to obtain the predicted memory usage rate; If the predicted CPU utilization does not exceed the preset CPU utilization threshold, the predicted memory utilization does not exceed the preset memory utilization threshold, the first task label is the same as the second task label, and the preset task processing time does not exceed the preset task timeout threshold, then the current task to be processed is determined to be the target task.

[0040] Preferably, by comparing the predicted CPU utilization with a preset CPU utilization threshold, and by comparing the predicted memory utilization with a preset memory utilization threshold, overload of worker nodes can be avoided. By comparing the first task label with the second task label, resource mismatch issues can be avoided. By comparing the preset task processing time with the preset task timeout threshold, resource waste and blocking caused by marking "pull failed" when worker nodes time out can be avoided.

[0041] In this preferred embodiment, based on load parameters, a first task label, a second task label, estimated task load parameters, a preset timeout threshold for the task, and a preset task processing time, it is determined whether the current task to be processed is a target task that the working node itself can currently process.

[0042] Step S104: Process the above target task and, upon successful processing, return the processing result to the task pool.

[0043] Specifically, the above processing results will be sent to the task pool as a "task completion event". The processing results include the task ID of the target task, the execution result of successful execution, the output data obtained after executing the target task, and the execution time.

[0044] Preferably, the above processing results will also be sent to the management node simultaneously.

[0045] In a preferred embodiment, it further includes: If the target task fails to process, a node failure event signal containing the node parameter information of the working node is sent to the task pool, so that the task pool can reclaim the target task from the working node according to the node parameter information after receiving the node failure event signal.

[0046] Specifically, when sending a node failure event signal, the node parameter information carried is the node ID of the worker node. Therefore, the task pool also uses this node ID to reclaim tasks. However, at the same time, the node failure event signal also carries the unfinished target tasks and the failure type of the worker node (such as "crash", "resource overrun", etc.), and sends them to the task pool and management node.

[0047] In this preferred embodiment, when the target task fails to process, a node failure event signal is sent to the task pool to enable the task pool to reclaim the target task.

[0048] In another preferred embodiment, it further includes: When the aforementioned working nodes start up, they send the aforementioned node parameter information to the management node so that the management node can record the information. After startup, the aforementioned working node continuously sends a heartbeat event signal containing the aforementioned node parameter information to the management node at preset time intervals. If the management node does not receive the aforementioned heartbeat event signal within the aforementioned preset time interval, it determines that the aforementioned working node has malfunctioned and sends a node malfunction event signal containing the aforementioned node parameter information to the aforementioned task pool. Upon receiving the aforementioned node failure event signal, the task pool reclaims the unprocessed target tasks from the corresponding working node based on the aforementioned node parameter information.

[0049] Specifically, when a worker node starts up, in addition to sending the node ID from the node parameter information, it also sends the worker node's IP port, first task tag, and hardware configuration information. This information is sent to the management node as a "node registration event" for recording.

[0050] Specifically, in addition to node parameter information, the aforementioned heartbeat event signal also includes the current number of local tasks on the worker node. These data are treated as a single "heartbeat event" and periodically sent to the management node at the aforementioned preset time interval (e.g., 30 seconds). At the same time, the current load parameters of the worker node are sent to the improved Dispatcher.

[0051] Preferably, during the "heartbeat event," the management node only records the "liveness status" of the worker nodes and does not store detailed load data. At the same time, all data within the management node is backed up through distributed storage with multiple replicas. It is stateless, horizontally scalable, and reduces the possibility of becoming a performance bottleneck.

[0052] Preferably, the management node can monitor the worker nodes simply through "heartbeat events," replacing the traditional mechanism of the Master polling the worker status. Furthermore, "heartbeat events" are only sent when the worker node is alive, reducing invalid communication.

[0053] Preferably, by using heartbeat monitoring, when a failure is detected in a worker node, the corresponding unprocessed tasks can be automatically reclaimed, and then other worker nodes can retrieve and execute them again, achieving recovery within seconds.

[0054] In this preferred embodiment, the worker node sends node parameter information to the management node for recording upon startup, and subsequently sends heartbeat event information to the management node periodically to help the management node detect abnormal states of the worker node in a timely manner.

[0055] Preferably, based on the above method flow, this invention establishes a novel task scheduling method for Workers and Masters, transforming the Worker's role from one of traditionally requiring the Master to poll for tasks to one of actively pulling and executing tasks. At the initial stage of a distributed task startup, a Master is started on one node machine, and Workers are started on other nodes. Then, the Workers actively report to the Master (i.e., the aforementioned node registration event). The Master then gains control of all available resources and can monitor the Worker's heartbeat. If a Worker terminates unexpectedly, the Master can react. Workers continuously switch between idle and working states, driving the progress of the entire task. The system is entirely asynchronous message-driven, and Workers respond to message commands based entirely on their current state. An illustrative flowchart of the worker node's operation is shown below. Figure 2 As shown.

[0056] Preferably, based on the above, this solution constructs a network-wide event-driven method based on the Event site. All operations are triggered by events. Seven core event categories are set up: worker node registration event, task availability event, task retrieval request event, task allocation event, task completion event, heartbeat event, and node failure event. Their triggering scenarios and the key information carried by each event, along with their relationship to time subscribers, are shown in the table below: Therefore, the entire task-driven process under the successful execution of the target task is as follows: task submission, followed by the task pool generating a task availability event, then the Worker's improved Dispatcher subscribes and detects local idle time, triggering a task fetch request event, then the task pool allocates tasks and generates a task allocation event, then the Worker executes the target task, sends a task completion event upon completion, and finally the task pool updates its status. The entire process is polling-free; each step is triggered by a preceding event, completely eliminating CPU idling.

[0057] For worker nodes, the management process is as follows: the worker starts up and sends a node registration event. Then the master records the node information. The worker then periodically sends a heartbeat event. The master then records the node's liveness status. If no heartbeat is received within a timeout, a node failure event is triggered. Finally, the task pool reclaims the unfinished tasks.

[0058] Preferably, through the above event-driven design and implementation, multiple events such as task retrieval, execution, and node failure are triggered in an orderly manner, replacing the traditional polling method and solving the problems of CPU idling and resource waste.

[0059] Preferably, the improved BalancingDispatcher (load balancer), the novel Worker-Master interaction, and the event-driven method are integrated to obtain a distributed task scheduling architecture based on the Akka model (i.e., a distributed computing framework design concept based on the "Actor Model"). In this architecture, the improved Dispatcher no longer acts as a central scheduling component but as a distributed routing component. Its core components are the Master module at the management layer, the event station module, and the Worker module at the execution layer. The Master module at the management layer includes a task queue collection, a resource scheduler, a fault monitoring center, and a Worker status table. The event station module includes message routing to implement event distribution and routing, as well as event log storage, and also houses the lock-free concurrency control area at this layer. The Worker module at the execution layer contains multiple Worker nodes that execute the pulled tasks, achieving local task caching and execution. This architecture design solves the problem of traditional architectures relying on centralized cluster scheduling.

[0060] Meanwhile, this invention also achieves high-concurrency scheduling through a lock-free design. Task allocation, retrieval, and fault detection are all triggered by events. There is no shared state between Workers and Masters, supporting asynchronous communication via message queues. Without lock mechanisms, lock contention is avoided, enabling the scheduling of millions of tasks per second, eliminating polling overhead, and saving system resources. Ultimately, this constitutes a novel distributed task scheduling architecture based on event-driven and pull models, achieving completely lock-free, 100% concurrency, and 100% secure flow control. This architecture solves the problems of single point of failure, resource waste, and low efficiency of manual recovery caused by the centralized design of traditional scheduling systems, achieving completely lock-free, 100% concurrent and secure task scheduling, reducing fault recovery time to the second level, and improving resource utilization by more than 40%.

[0061] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments; like Figure 3 As shown, one embodiment of the present invention provides a distributed task scheduling device, comprising: Work nodes and task pools; The aforementioned working node is used to obtain its current load parameters, first task tag, and preset task processing time, and to determine whether it is in an idle state based on the current load parameters; if it determines that it is in an idle state, it sends a task pull request signal to the task pool; wherein, the aforementioned first task tag is used to indicate the type of task that the aforementioned working node is allowed to process. The aforementioned task pool is used to, upon receiving the aforementioned task pull request signal, feed back the second task tag, estimated task load parameters, and preset timeout threshold of all pending tasks to the aforementioned working node; wherein, the aforementioned second task tag is used to indicate the task type of the corresponding pending task. The aforementioned working nodes are also used to determine, for each pending task, whether it is a target task that it can currently process, based on the aforementioned load parameters, first task label, second task label, estimated task load parameters, preset timeout threshold, and preset task processing time. If the pending task is determined to be a target task, the node pulls the target task from the task pool, processes the target task, and feeds back the processing result to the task pool after successful processing.

[0062] In a preferred embodiment, determining whether it is in an idle state based on the current load parameters includes: Compare the CPU utilization rate in the above load parameters with the preset CPU utilization rate threshold; Compare the memory utilization rate in the above load parameters with the preset memory utilization rate threshold; If the CPU utilization rate does not exceed the preset CPU utilization rate threshold and the memory utilization rate does not exceed the preset memory utilization rate threshold, then it is determined that it is in an idle state; otherwise, it is determined that it is not in an idle state.

[0063] In another preferred embodiment, the above-mentioned task load estimation parameters include: estimated CPU utilization and estimated memory utilization; The above-mentioned determination of whether the current pending task is a target task that it can currently process, based on the aforementioned load parameters, first task label, second task label, estimated task load parameters, preset task timeout threshold, and preset task processing time, includes: Calculate the sum of the estimated CPU utilization rate and the CPU usage rate to obtain the predicted CPU utilization rate; Calculate the sum of the above estimated memory occupancy rate and the above memory usage rate to obtain the predicted memory usage rate; If the predicted CPU utilization does not exceed the preset CPU utilization threshold, the predicted memory utilization does not exceed the preset memory utilization threshold, the first task label is the same as the second task label, and the preset task processing time does not exceed the preset task timeout threshold, then the current task to be processed is determined to be the target task.

[0064] In another preferred embodiment, the aforementioned working node is further configured to: send a node failure event signal containing node parameter information of the aforementioned working node to the task pool when the aforementioned target task processing fails. The aforementioned task pool is used to reclaim the aforementioned target task from the aforementioned working node based on the aforementioned node parameter information after receiving the aforementioned node failure event signal.

[0065] In another preferred embodiment, it further includes: Management node; The aforementioned working node is also used to send the aforementioned node parameter information to the aforementioned management node at startup; The aforementioned management nodes are used to record the parameter information of the aforementioned nodes; The aforementioned working node is also used to continuously send heartbeat event signals containing the node parameter information to the aforementioned management node at preset time intervals after startup. The aforementioned management node is also used to determine that the aforementioned working node has malfunctioned when it does not receive the aforementioned heartbeat event signal within the aforementioned preset time interval, and to send a node malfunction event signal containing the aforementioned node parameter information to the aforementioned task pool. The aforementioned task pool is also used to reclaim unprocessed target tasks from the corresponding working node based on the aforementioned node parameter information after receiving the aforementioned node failure event signal.

[0066] It should be noted that the device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without creative effort. The above schematic diagrams are merely examples of a distributed task scheduling device and do not constitute a limitation on a distributed task scheduling device. It may include more or fewer components than illustrated, or combine certain components, or use different components.

[0067] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A distributed task scheduling method, characterized in that, Applicable to working nodes, including: The system obtains its current load parameters, first task tag, and preset task processing time, and determines whether it is in an idle state based on the current load parameters; wherein, the first task tag is used to indicate the type of task that the worker node is allowed to process; When it determines that it is in an idle state, it sends a task pull request signal to the task pool, so that the task pool will return the second task tag, the estimated load parameter of the task, and the preset timeout threshold of all tasks to be processed; wherein, the second task tag is used to indicate the task type of the corresponding task to be processed. For each pending task, based on the load parameters, the first task tag, the second task tag, the estimated load parameters, the preset timeout threshold, and the preset task processing time, it is determined whether the current pending task is the target task that it can currently process. When it is determined that the pending task is the target task, the target task is pulled from the task pool. The target task is processed, and the processing result is fed back to the task pool after successful processing.

2. The distributed task scheduling method according to claim 1, characterized in that, The step of determining whether it is in an idle state based on the current load parameters includes: The CPU utilization rate in the load parameters is compared with a preset CPU utilization rate threshold; The memory utilization rate in the load parameters is compared with a preset memory utilization rate threshold; If the CPU utilization rate does not exceed the preset CPU utilization rate threshold and the memory utilization rate does not exceed the preset memory utilization rate threshold, then it is determined that it is in an idle state; otherwise, it is determined that it is not in an idle state.

3. The distributed task scheduling method according to claim 2, characterized in that, The estimated task load parameters include: estimated CPU utilization and estimated memory utilization; The step of determining whether the current pending task is a target task that it can currently process based on the load parameters, the first task tag, the second task tag, the estimated task load parameters, the preset task timeout threshold, and the preset task processing time includes: The predicted CPU utilization rate is obtained by summing the estimated CPU utilization rate and the CPU usage rate. The predicted memory utilization rate is obtained by summing the estimated memory occupancy rate and the memory usage rate. If the predicted CPU utilization does not exceed the preset CPU utilization threshold, the predicted memory utilization does not exceed the preset memory utilization threshold, the first task label and the second task label are the same, and the preset task processing time does not exceed the preset task timeout threshold, then the current task to be processed is determined as the target task.

4. The distributed task scheduling method according to claim 3, characterized in that, Also includes: If the target task fails to process, a node failure event signal containing the node parameter information of the working node is sent to the task pool, so that the task pool can reclaim the target task from the working node according to the node parameter information after receiving the node failure event signal.

5. A distributed task scheduling method according to claim 4, characterized in that, Also includes: When the working node starts up, it sends the node parameter information to the management node so that the management node can record the information. After startup, the working node continuously sends a heartbeat event signal containing the node parameter information to the management node at preset time intervals, so that if the management node does not receive the heartbeat event signal within the preset time interval, it determines that the working node has failed and sends a node failure event signal containing the node parameter information to the task pool. Upon receiving the node failure event signal, the task pool reclaims the unprocessed target tasks from the corresponding working node based on the node parameter information.

6. A distributed task scheduling device, characterized in that, include: Work nodes and task pools; The worker node is used to obtain its current load parameters, first task tag, and preset task processing time, and to determine whether it is in an idle state based on the current load parameters; if it determines that it is in an idle state, it sends a task pull request signal to the task pool; wherein, the first task tag is used to indicate the type of task that the worker node is allowed to process. The task pool is used to, upon receiving the task pull request signal, feed back the second task tag, estimated task load parameters, and preset timeout threshold of all pending tasks to the working node; wherein, the second task tag is used to indicate the task type of the corresponding pending task. The working node is also used to determine, for each pending task, whether the current pending task is a target task that it can currently process, based on the load parameters, the first task tag, the second task tag, the estimated task load parameters, the preset timeout threshold, and the preset task processing time. When the pending task is determined to be the target task, the node pulls the target task from the task pool, processes the target task, and feeds back the processing result to the task pool after successful processing.

7. A distributed task scheduling device according to claim 6, characterized in that, The step of determining whether it is in an idle state based on the current load parameters includes: The CPU utilization rate in the load parameters is compared with a preset CPU utilization rate threshold; The memory utilization rate in the load parameters is compared with a preset memory utilization rate threshold; If the CPU utilization rate does not exceed the preset CPU utilization rate threshold and the memory utilization rate does not exceed the preset memory utilization rate threshold, then it is determined that it is in an idle state; otherwise, it is determined that it is not in an idle state.

8. A distributed task scheduling device according to claim 7, characterized in that, The estimated task load parameters include: estimated CPU utilization and estimated memory utilization; The step of determining whether the current pending task is a target task that it can currently process based on the load parameters, the first task tag, the second task tag, the estimated task load parameters, the preset task timeout threshold, and the preset task processing time includes: The predicted CPU utilization rate is obtained by summing the estimated CPU utilization rate and the CPU usage rate. The predicted memory utilization rate is obtained by summing the estimated memory occupancy rate and the memory usage rate. If the predicted CPU utilization does not exceed the preset CPU utilization threshold, the predicted memory utilization does not exceed the preset memory utilization threshold, the first task label and the second task label are the same, and the preset task processing time does not exceed the preset task timeout threshold, then the current task to be processed is determined as the target task.

9. A distributed task scheduling device according to claim 8, characterized in that, The working node is also configured to: send a node failure event signal containing the node parameter information of the working node to the task pool when the target task fails to process; The task pool is used to reclaim the target task from the working node according to the node parameter information after receiving the node failure event signal.

10. A distributed task scheduling device according to claim 9, characterized in that, Also includes: Management node; The working node is also used to send the node parameter information to the management node at startup; The management node is used to record the node parameter information; The working node is also used to continuously send heartbeat event signals containing node parameter information to the management node at preset time intervals after startup. The management node is also used to determine that the working node has failed when it does not receive the heartbeat event signal within the preset time interval, and to send a node failure event signal with the node parameter information to the task pool. The task pool is also used to reclaim unprocessed target tasks from the corresponding working node based on the node parameter information after receiving the node failure event signal.

Images