A task scheduling and dynamic scheduling method for multi-machine concurrency
By using task waiting queues and periodic scheduling mechanisms, the problem of inconsistent recovery of jobs and states in a multi-computer environment is solved, realizing unified task scheduling and dynamic balancing in a multi-machine environment, and improving execution stability and efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JILIN UNIVERSITY
- Filing Date
- 2026-04-09
- Publication Date
- 2026-06-09
Smart Images
Figure CN121979645B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of workflow engine and intelligent orchestration technology, and in particular to a method for task orchestration and dynamic scheduling for multi-machine concurrency. Background Technology
[0002] In today's era of booming high-performance computing and large-scale data processing, multi-computer or cluster environments have become the core infrastructure for many research institutions and enterprises to perform complex computing tasks. Users often need to distribute a large number of computing jobs to different execution platforms to meet diverse computing needs. These jobs may involve complex scientific simulations, large-scale data analysis, machine learning model training, etc., and have huge and highly dynamic demands on computing resources.
[0003] To address these needs, various job orchestration and scheduling solutions have been proposed. Some solutions involve manually writing and maintaining multiple sets of submission scripts, customizing job submission processes for different execution platforms to enable job execution across various platforms. However, current job orchestration and scheduling solutions tend to favor execution and monitoring models for single computing resource tasks (i.e., a set of submissions completes script generation, submission, and querying on a designated machine or cluster). This is insufficient when real-world scenarios require simultaneous access to multiple computing resources (e.g., multiple login nodes, multiple clusters, multiple scheduling systems) and dynamic job distribution based on the real-time available capacity of each resource. Other solutions utilize integration tools to encapsulate job submission interfaces across different platforms, providing users with a relatively unified job submission interface. After users submit jobs through this interface, the tool distributes them to the appropriate platforms for execution based on job configurations. Still other solutions focus on monitoring job status by periodically querying job status information on each execution platform and providing feedback to users, allowing them to understand the job execution status promptly.
[0004] However, relying on manually writing and maintaining multiple submission scripts not only increases the user's workload and the probability of errors, but also incurs extremely high costs for script modification and maintenance when the execution platform changes or job requirements are adjusted. Secondly, manually uploading and downloading files and monitoring job status makes the entire job management process inefficient and prone to file transfer errors or untimely job status monitoring due to human error. More importantly, if the main control program experiences an abnormal exit, such as a network outage, crash, or restart, it is difficult to guarantee the consistency and recovery of jobs and their status. This may lead to idle computing resources, wasting valuable computing resources; it may also result in duplicate job submissions, increasing computing costs; and it may even lead to the loss of calculation results, causing irreparable losses to users. Summary of the Invention
[0005] This application provides a method for task orchestration and dynamic scheduling for multi-machine concurrency, in order to solve the technical problem that currently, when dealing with multiple computers, it is often necessary to manually write and maintain multiple sets of submission scripts. When the main control program exits abnormally, it is difficult to restore the job and status in a consistent manner, which can easily lead to idle resources, duplicate submissions or loss of results.
[0006] This application provides a task orchestration and dynamic scheduling method for multi-machine concurrency, applied to a plurality of task machines; the task machine is a computer device for executing tasks to be executed; each task machine has at least one task node, which is an execution unit within the task machine for running the tasks to be executed; the method includes:
[0007] Receive tasks to be executed;
[0008] The tasks to be executed are written into a task waiting queue; each task machine is equipped with a corresponding task waiting queue, which is used to cache the tasks to be executed.
[0009] Refresh the task running status of the tasks to be executed in each task node; the task running status includes: running, completed, and abnormal status.
[0010] Determine the task running status of the task to be executed. If the task running status is the completed status, remove the corresponding executed task from the task node. If the task running status is the abnormal status, rewrite the corresponding abnormal task into the corresponding task node for execution.
[0011] Determine whether the available capacity in the task machine is greater than 0. If so, determine whether there are any idle task nodes in the task machine. If so, determine whether there is any task to be executed in the task waiting queue corresponding to the task machine. If so, write the task to be executed in the task waiting queue into the idle task node. The available capacity represents the number of tasks to be executed that the task machine can currently receive.
[0012] Based on the task to be executed, determine the task output result; the task output result includes: completed, failed, and retryable exception.
[0013] In some embodiments, the method further includes:
[0014] The parameter format in the task to be executed will be standardized.
[0015] The tasks to be executed are uniformly mapped to schedulable objects; the schedulable objects are used to store the task information corresponding to the tasks to be executed and to provide the interface corresponding to the scheduling process; the task information includes: task instructions, input and output file information, shared file dependency information, and execution directory information.
[0016] In some embodiments, after the steps of writing the task to be executed into the task waiting queue and rewriting the corresponding abnormal task into the corresponding task node for execution, the method includes:
[0017] Record the status file in the task waiting queue; the status file is the real-time tasks to be executed stored in the task waiting queue.
[0018] In some embodiments, the step of determining the task running state of the task to be executed includes:
[0019] Obtain the execution evidence of the task to be executed; the execution evidence includes: process presence information, log text signals, completion marker files, and timeout marker information;
[0020] Using the aforementioned operational evidence, the task execution status of the task to be executed is determined.
[0021] In some embodiments, the step of determining the task execution status of the task to be executed using the execution evidence includes:
[0022] Based on the process in-place information and the completion marker file, determine whether the task to be executed has been completed;
[0023] Based on whether there is an abnormal log text signal in the log text signal and whether the timeout mark information exists, determine whether the task running status of the task to be executed is abnormal.
[0024] In some embodiments, the step of rewriting the corresponding abnormal task into the corresponding task node for execution includes:
[0025] Obtain the number of times the abnormal task is executed when the task's running status is the abnormal status;
[0026] If the number of times the abnormal task is executed is less than the preset retry execution threshold, the abnormal task will be rewritten into the corresponding task node for execution.
[0027] In some embodiments, after determining whether the available capacity in the task machine is greater than 0, the method further includes:
[0028] If not, the task to be executed is written into the task waiting queue of the corresponding task machine. When the task machine has available capacity and there are idle task nodes in the task machine, the task to be executed is written into the idle task nodes.
[0029] In some embodiments, the step of writing the task to be executed in the task waiting queue into the task node in the idle state includes:
[0030] Determine the batch parameters for the task nodes with available capacity; the batch parameters include: the number of tasks to be executed in each batch and the dependency information between the tasks to be executed;
[0031] Determine the task information of the tasks to be executed in the task waiting queue; the task information includes: task type and execution priority;
[0032] Based on the batch parameters and the task information, the tasks to be executed are organized into task units and written into the idle task nodes according to the batch parameters.
[0033] In some embodiments, the method further includes:
[0034] Based on the task to be executed, an execution script is generated; the execution script is used to execute the task to be executed.
[0035] Based on the execution script, an execution identifier is generated;
[0036] Based on the execution identifier, the task running status of the task to be executed is changed to running.
[0037] In some embodiments, the step of determining the task output result based on the task to be executed includes:
[0038] If the task to be executed has been completed, then the task output result of the task to be executed is determined to be completed, and the completed task results are summarized in the node where the main control program is located, and the corresponding executed task is removed from the task node;
[0039] If the task to be executed has not been completed, then obtain the number of times the task to be executed has been executed;
[0040] If the number of executions is less than the preset retry execution count threshold, then the task output result of the task to be executed is determined to be a retryable exception;
[0041] If the number of executions is greater than or equal to a preset retry threshold, then the task output result of the task to be executed is determined to be a failure.
[0042] This application provides a task orchestration and dynamic scheduling method for multi-machine concurrency, applied to a plurality of task machines; each task machine is a computer device for executing tasks to be executed; each task machine has at least one task node, which is an execution unit within the task machine for running the tasks to be executed; the method includes: receiving tasks to be executed; writing the tasks to be executed into a task waiting queue; each task machine has a corresponding task waiting queue, which is used to cache the tasks to be executed; refreshing the task running status of the tasks to be executed in each task node; the task running status includes: running, completed, and abnormal; determining the task running status of the task to be executed, and if the task running status is the completed status, removing the corresponding executed task from the task node; If the task running state is the abnormal state, the corresponding abnormal task is rewritten into the corresponding task node for execution; it is determined whether the available capacity in the task machine is greater than 0. If so, it is determined whether there is an idle task node in the task machine; if so, it is determined whether there is a task to be executed in the task waiting queue corresponding to the task machine; if so, the task to be executed in the task waiting queue is written into the idle task node; the available capacity represents the number of tasks to be executed that the task machine can currently receive; based on the tasks to be executed, the task output result is determined; the task output result includes: completed, failed, and retryable exception, so as to achieve a more complete engineering closed loop for multiple computers, which can be uniformly scheduled, dynamically balanced, converged in case of anomalies, and recovered from interruptions, making it more suitable for real production computing environments. Attached Figure Description
[0043] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1 This is a flowchart of the task orchestration and dynamic scheduling method for multi-machine concurrency in this application;
[0045] Figure 2 This is a timing diagram for task orchestration and dynamic scheduling for multi-machine concurrency in this application. Detailed Implementation
[0046] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.
[0047] For example, a high-performance computing (HPC) cluster typically consists of a login node, a scheduling management component, a cluster of compute nodes, and a shared storage system. The login node provides a secure shell protocol login interface and installs the scheduling system's command-line tools, thereby performing operations such as task generation, file preparation, submission control, status monitoring, and result aggregation. The scheduling management component is responsible for queue maintenance, resource allocation strategies, and job lifecycle management. Compute nodes actually host hardware resources such as CPUs / graphics processing units and execute job scripts. Shared storage (e.g., a parallel file system or network file system) is used to store input / output files, logs, and completion marker files. Scheduling systems, such as Slurm (a resource management and scheduling system), play a core role in HPC by allocating resources, providing a job execution framework, and scheduling queued jobs. They manage job execution through mechanisms such as batch script submission and queue querying. Existing mainstream scheduling systems differ significantly in their submission and query interfaces, script header resource declarations, status codes, and output formats. Therefore, cross-platform batch job management typically faces high adaptation costs and operational complexity.
[0048] In fields such as materials simulation, molecular dynamics, machine learning training and inference, and engineering simulation, it is often necessary to complete a large number of independent or weakly dependent computational tasks within the same time window. These tasks are characterized by "large quantity, long single-task time, scattered task directories, complex file flow, and numerous abnormal scenarios." In actual operation, whether a job is "truly completed" is not always equivalent to the scheduler displaying "complete," because the result file may not be fully written, there may be write delays in shared storage, or an additional completion tag file may need to be generated before it can enter the downstream process. At the same time, network fluctuations, unreachable login nodes, abnormal return values from scheduling queries, queue congestion, and computing node failures can cause the job status to be "unknown / abnormal." If there is a lack of a clear state machine, retry boundaries, and a consistent persistent recording mechanism, tasks can easily be lost, repeatedly submitted, or stuck for a long time. Because batch assignments often migrate across platforms (different teams use local servers, departmental clusters, and cloud clusters simultaneously), and the scheduling systems and directory structures of different platforms differ significantly, a more "closed-loop" requirement has gradually emerged in the industry: not only should "submission / query" be unified, but "uploading input / distributing shared files / batch submission / polling monitoring / anomaly recovery / breakpoint resume / result collection" should also be incorporated into the same set of reusable job orchestration methods and system devices.
[0049] Existing technologies tend to favor execution and monitoring models for single computing resource tasks (i.e., a set of submissions completes script generation, submission, and querying on a designated machine / cluster). When real-world scenarios require simultaneous access to multiple computing resources (e.g., multiple login nodes / multiple clusters / various scheduling systems) and dynamic job distribution based on the real-time available capacity of each resource, existing technologies are insufficient.
[0050] To address the aforementioned technical problems of relying on manual writing and maintenance of multiple submission scripts when dealing with multiple computers, which leads to difficulties in consistently restoring jobs and states when the main control program exits abnormally, resulting in resource idleness, duplicate submissions, or result loss, this application provides a task orchestration and dynamic scheduling method for multi-machine concurrency. The following describes the task orchestration and dynamic scheduling method for multi-machine concurrency:
[0051] like Figure 1 The diagram shown is a flowchart of the task orchestration and dynamic scheduling method for multi-machine concurrency in this application.
[0052] This application provides a task orchestration and dynamic scheduling method for multi-machine concurrency, applied to a plurality of task machines; the task machine is a computer device for executing tasks to be executed; each task machine has at least one task node, which is an execution unit within the task machine for running the tasks to be executed; the method includes:
[0053] S100: Receive the task to be executed; the task to be executed includes not only the task object itself, but also constraints such as the running command, input / output file relationship, shared file dependency, and execution directory information.
[0054] S110: Standardize the parameter format in the task to be executed.
[0055] For example, a unified parameter format enables the system to seamlessly accommodate tasks submitted from different platforms and users, regardless of their original format or structure. This significantly improves the system's versatility and flexibility, lowering the barrier to entry for users. Standardized task parameters simplify subsequent processes such as task allocation, script generation, and file distribution. The system no longer needs to write specific processing logic for tasks of different formats, thus improving processing efficiency and stability. A unified parameter format helps reduce task execution failures or system anomalies caused by inconsistent parameters or format errors. Through rigorous validation and standardization, the system can identify and correct potential problems before task submission, ensuring successful task execution. Furthermore, a unified parameter format provides the foundation for dynamic task distribution and load balancing. The system can dynamically allocate standardized tasks to the most suitable nodes for execution based on the real-time status and available capacity of each computing node, thereby achieving efficient resource utilization and rapid task completion.
[0056] S120: Map the tasks to be executed to a unified schedulable object; the schedulable object is used to store the task information corresponding to the tasks to be executed and to provide the interface corresponding to the scheduling process; the task information includes: task instructions, input and output file information, shared file dependency information, and execution directory information; by mapping tasks from different sources and in different formats to internal schedulable objects, problems such as missing fields, inconsistent paths, or unexecutable scripts are avoided in the subsequent allocation stage.
[0057] S200: Write the task to be executed into the task waiting queue; each task machine is equipped with a corresponding task waiting queue, which is used to cache the task to be executed; by first writing the task to be executed into the task waiting queue, and then waiting for the resource conditions to be met to execute it.
[0058] S210: Record the status file in the task waiting queue; the status file is a list of real-time tasks to be executed stored in the task waiting queue. By recording the status file in the task waiting queue, it is ensured that even if the system experiences an interruption shortly after initialization, it can recover to a stable, schedulable state through the status file.
[0059] S300: Refresh the task running status of the tasks to be executed in each task node; the task running status includes: running, completed, and abnormal status; the above steps are prerequisites for subsequent capacity calculation and task deployment, and can avoid duplicate submissions or over-allocation caused by outdated statuses. The task running status also includes: not submitted, resubmitted, and queued; the tasks to be executed in the not submitted status are the tasks waiting to enter the task waiting queue; the tasks to be executed in the resubmitted status are abnormal tasks whose execution count is less than the preset retry execution count threshold; the tasks to be executed in the queued status are the tasks to be executed located in the task waiting queue.
[0060] S400: Determine the task running status of the task to be executed. If the task running status is the completed status, remove the corresponding executed task from the task node. By removing the corresponding executed task from the task node, the node view is brought together in a timely manner to prevent completed tasks (executed tasks) from occupying logical capacity for a long time.
[0061] This application employs a periodically driven hierarchical scheduling mechanism. Upon entering the system, tasks are not executed immediately and unconditionally; instead, they first enter a waiting queue while their status is recorded. The technical value of this step lies in decoupling the "task generation time" and the "resource availability time," avoiding resource contention and submission failures caused by instantaneous high concurrency. After entering the scheduling loop, the system prioritizes processing the status of tasks already running or queued on nodes, including whether they are completed, have encountered an error, have timed out, or have terminated. Only after the existing task status is refreshed and synchronized will the system execute the next round of new task allocation. This sequential constraint significantly reduces misallocation problems caused by inconsistent status. For each computing node, the system first calculates the current available capacity, then extracts task groups from the waiting queue according to batch size, prepares input files and scripts, and triggers submission. Upon successful submission, the task enters the running state and is bound to an execution flag; if submission fails, the system records the failure information and enters the subsequent exception handling branch. Through this "refresh before allocation" closed-loop logic, the system can maintain stable throughput even under resource fluctuation scenarios.
[0062] The step of determining the task running status of the task to be executed includes the following sub-steps:
[0063] S410: Obtain the running evidence of the task to be executed; the running evidence includes: process in-place information, log text signals, completion marker files, and timeout marker information.
[0064] S420: Using the aforementioned operational evidence, determine the task execution status of the task to be executed.
[0065] The step of determining the task execution status of the task to be executed using the execution evidence includes the following sub-steps:
[0066] S421: Determine whether the task to be executed has been completed based on the process in-place information and the completion marker file.
[0067] S422: Determine whether the task running status of the task to be executed is abnormal based on whether there is an abnormal log text signal in the log text signal and whether there is the timeout mark information.
[0068] Specifically, the system (especially the status monitoring submodule) first queries the compute nodes to obtain the process availability information corresponding to the task to be executed. This process availability information indicates whether the task is still running. The system then checks whether a completion marker file corresponding to the task exists in shared storage or the local file system. This completion marker file is typically generated by the task execution script upon successful completion, serving as an explicit indication of task completion. If the process availability information indicates that the process no longer exists (i.e., the task has terminated), and a completion marker file exists, the system determines that the task to be executed has been completed.
[0069] Specifically, the system collects log text signals generated during the execution of the task to be executed. These log text signals record the detailed process of task execution and possible error information. The system analyzes the log text signals, checking for text patterns or keywords indicating abnormal task execution. These abnormal log text signals may include error messages, warning messages, or specific failure patterns. The system also checks for timeout markers. Timeout markers are typically generated by the system when the task execution time exceeds a preset threshold, indicating that the task may be terminated due to prolonged inactivity. If abnormal log text signals or timeout markers are present, the system determines that the task's execution status is abnormal.
[0070] S500: If the task running status is the abnormal status, the corresponding abnormal task is rewritten into the corresponding task node for execution; the failure caused by short-term fluctuations is transformed from a one-time error into a controlled retry process, thereby improving the overall success rate.
[0071] S510: Record the status file in the task waiting queue; the status file is the real-time tasks to be executed stored in the task waiting queue. Solidify the status changes of this round to form a traceable scheduling trajectory.
[0072] The step of rewriting the corresponding abnormal task into the corresponding task node for execution includes the following sub-steps:
[0073] S520: Obtain the number of times the abnormal task is executed when the task running status is the abnormal status.
[0074] S530: If the number of times the abnormal task is executed is less than the preset retry execution number threshold, then the abnormal task is rewritten into the corresponding task node for execution.
[0075] This application proposes an engineering-feasible anomaly handling scheme. For tasks that encounter unknown or erroneous states during operation, the system does not immediately and permanently fail. Instead, it performs controlled retries based on a preset retry count threshold: when the retry count has not reached the upper limit, the task is reset and re-queued, waiting for the next round of capacity to allow execution again; when the retry count reaches the upper limit, the task is marked as failed and exits the automatic retry channel. This mechanism balances availability and controllability, improving the success rate of tasks under short-term failures while avoiding unlimited retries that consume resources. For completed or failed tasks, the system triggers result feedback and state convergence operations to ensure that subsequent result extraction interfaces can accurately return the unclaimed result set, and prevents duplicate result distribution through a "claimed flag." This forms a complete lifecycle management system of "execution, detection, correction, and convergence."
[0076] S600: Determine whether the available capacity in the task machine is greater than 0. If so, determine whether there is an idle task node in the task machine. If so, determine whether there is a task to be executed in the task waiting queue corresponding to the task machine. If so, write the task to be executed in the task waiting queue into the idle task node. The available capacity is represented by the number of tasks to be executed that the task machine can currently receive.
[0077] After determining whether the available capacity in the task machine is greater than 0, the following steps are also included:
[0078] S610: If not, the task to be executed is written into the task waiting queue of the corresponding task machine. When the task machine has available capacity and there are idle task nodes on the task machine, the task to be executed is written into the idle task node. The system first performs a hard constraint judgment based on "whether the node has available capacity", and then performs a supply judgment based on "whether the waiting queue is empty". When both conditions are met simultaneously, the task is issued.
[0079] Specifically, if the available capacity of the task machine is greater than 0, the task to be executed can be submitted to the task machine. Then, it is determined whether there are any idle task nodes in each task machine, and the target task machine with the idle task node is determined. If it exists, the task to be executed in the task waiting queue is submitted to the idle task node in the target task machine for execution. If it does not exist, the task to be executed is written into the task waiting queue of the corresponding task machine. When an idle task node appears in the corresponding task machine, the task to be executed is read from its task waiting queue and automatically submitted to the idle task node for execution, thereby preventing the scenario where some task machines exceed the limit by submitting too many tasks, causing subsequent task submissions to fail.
[0080] The step of writing the tasks to be executed in the task waiting queue into the task nodes in the idle state includes the following sub-steps:
[0081] S620: Determine the batch parameters of the task nodes with available capacity; the batch parameters include: the number of tasks to be executed in each batch and the dependency information between the tasks to be executed.
[0082] S630: Determine the task information of the task to be executed in the task waiting queue; the task information includes: task type and execution priority.
[0083] S640: Based on the batch parameters and the task information, the tasks to be executed are organized into task units and written into idle task nodes according to the batch parameters. The system organizes tasks into task units based on node batch parameters; this design reduces submission overhead and enhances execution consistency within the group.
[0084] The method further includes the following steps:
[0085] S650: Based on the task to be executed, generate an execution script; the execution script is used to execute the task to be executed; the system completes the runtime dependency preparation to ensure that the execution environment matches the task requirements.
[0086] S660: Based on the execution script, generate an execution identifier; the system will submit the return value as an anchor point for subsequent status tracking.
[0087] S670: Based on the execution identifier, change the task running state of the task to be executed to "running". Align the control plane state with the execution plane state to avoid state drift caused by "committed but not marked".
[0088] S700: Based on the task to be executed, determine the task output result; the task output result includes: completed, failed, and retryable exception.
[0089] The step of determining the task output result based on the task to be executed includes the following sub-steps:
[0090] S710: If the task to be executed has been completed, then determine that the task output result of the task to be executed is completed and remove the corresponding executed task from the task node.
[0091] S720: If the task to be executed has not been completed, then obtain the number of times the task to be executed has been executed.
[0092] S730: If the number of executions is less than the preset retry execution count threshold, then the task output result of the task to be executed is determined to be a retryable exception.
[0093] S740: If the number of executions is greater than or equal to the preset retry execution count threshold, then the task output result of the task to be executed is determined to be a failure.
[0094] like Figure 2 The diagram shown is a sequence diagram of task orchestration and dynamic scheduling for multi-machine concurrency in this application. This application adopts a continuous scheduling mechanism rather than a one-time distribution mechanism. Its technical significance lies in the fact that when events such as resource idleness, task addition, task exception, and task completion occur at different times, the system can complete consistency correction and incremental advancement in the next cycle, thereby achieving global stable convergence.
[0095] Specifically, the sequence diagram illustrates the message exchange relationships in actual operation from the perspective of "participant collaboration," emphasizing the sequential dependencies of different entities within the same scheduling cycle. First, the user submits a set of tasks to the scheduling center. Instead of directly pushing the tasks to the execution environment, the scheduling center first writes them into a waiting queue and synchronously records their initial states. This sequential design ensures that tasks are not lost and states can be replayed even in the face of network jitter, process restarts, or sudden shutdowns. Upon entering periodic scheduling, the scheduling center first initiates a state refresh request to the computing nodes. The nodes then query the execution and submission units for operational evidence, which includes at least process presence information, log text signals, completion marker files, or timeout marker information. After the execution and submission units return the state results, the nodes feed back a structured state snapshot to the scheduling center, which updates the global task view accordingly and writes state records when necessary. This sequence of "first collecting execution-side evidence, then updating the control-side state" is a key point in the sequence design, ensuring that control plane decisions are based on the latest verifiable state. Subsequently, the scheduling center queries the compute nodes for available capacity. If the capacity is insufficient or the queue is empty, the current cycle only performs state maintenance and does not deploy new tasks. If the capacity is available and the queue is not empty, the system enters a submission chain: "retrieve task group—issue task and input file—generate script and submit—return execution identifier—write back running status." This submission chain clearly distinguishes between control messages and execution messages: control messages drive the action sequence, while execution messages provide feedback on the submission result and running identity. Both ensure that there is no ambiguity when a task switches from the "waiting state" to the "running state." In the latter half of the cycle, the system enters branch processing based on the latest status: if the task is completed, the compute node triggers result feedback, the scheduling center updates the completion status, and releases the node's logical capacity; if the task is abnormal, the scheduling center triggers retry queuing and updates the retry count and status. Through this branch convergence mechanism, the system can map three types of events—"completion," "failure," and "retryable exception"—to different control actions, avoiding a one-size-fits-all approach to all exceptions. Figure 2 It also implies an important engineering characteristic, namely the idempotency of sequential repetition: even if some queries or writes fail briefly in the same round, the deviation can still be corrected in the next cycle by re-collecting the state and aligning the control plane, thereby improving the robustness and maintainability of the system in complex operating environments.
[0096] This application provides a method for task orchestration and dynamic scheduling for multi-machine concurrency. An "Orchestrator" is set up as the global scheduling hub on the local or remote login node where tasks are submitted. This orchestrator can simultaneously load configurations from multiple machines and establish machine mappings, maintaining a unified queue of pending submissions and a job status database. An independent monitoring thread polls the status of each machine at time intervals and updates the job status. In each round of scheduling, the number of jobs that can be accepted is first calculated based on machine capacity and currently occupied job identifiers. Then, jobs are grouped by batch size, uploaded, and submitted in batches, thereby achieving "capacity awareness and parallel distribution across multiple machines." Meanwhile, this application introduces a clear job state machine and provides executable rollback and resubmission strategies for unknown / abnormal states (such as resetting the job identifier, setting it to resubmission and re-entering the queue, and limiting the maximum number of retries) to avoid job freezing or disordered duplicate submissions. In addition, this application adopts a JSON persistent record with file lock (flock) and latest pointer mechanism to atomically write the global state to the record file and support breakpoint resume recovery, so that the queue to be submitted and the mapping of submitted data on each machine can still be rebuilt after the control process restarts, achieving more stable consistent orchestration and automatic fault tolerance in long-term, large-scale, and multi-platform environments.
[0097] This application provides a task orchestration and dynamic scheduling method for multi-machine concurrency. The core objective is to unify distributed, heterogeneous computing resources with significant load variations into a single scheduling framework, achieving closed-loop control throughout the entire process: task submission, resource matching, batch execution, status tracking, exception retry, result feedback, and breakpoint recovery. The technical approach is as follows: First, external tasks are abstracted and queued using a task access mechanism; second, the status of each computing node is refreshed and capacity is assessed through a periodic scheduling mechanism; third, tasks are grouped and executed according to available node capacity and batch rules; subsequently, status determination, exception identification, and retry control are continuously performed during the execution phase; finally, result feedback and status persistence mechanisms ensure that task results are obtainable, processes are traceable, and the system is recoverable. This technical solution does not rely on a single scheduling backend, is compatible with both local and remote execution scenarios, and can be extended to different job scheduling systems. It has good engineering feasibility and industrial adaptability, thereby achieving automated closed-loop job management across scheduling platforms.
[0098] This application also employs a collaborative mechanism of state persistence and recovery takeover. The system continuously writes to a state log file during critical stages such as task enqueuing, state refreshing, task assignment, and exception retries, using locked writes to ensure record consistency in concurrent scenarios. If the system is interrupted due to external reasons, the recovery takeover submodule can read the most recent state record and automatically reconstruct the task set, waiting queue, and node task view, allowing the scheduling process to continue from near the interruption point rather than starting from scratch. This capability is particularly important for long-cycle computing scenarios, significantly reducing time and computing power losses caused by unexpected interruptions and improving the overall task completion determinism.
[0099] The system mapped by the multi-machine concurrent task orchestration and dynamic scheduling method in this application can be deployed on an orchestration host or login node. The orchestration host includes at least a processor, memory, network interface, and storage medium. The processor executes the orchestration program in the storage medium, the memory is used to cache the current job set, machine mapping relationships, and monitor thread status, the network interface is used to establish SSH (Secure Shell) connections with remote login nodes or clusters, and shared storage or a local file system is used to save input files, output files, log files, completion marker files, and record files.
[0100] Logically, the system includes at least the following submodules: task access, scheduling control, waiting queue, compute node management, execution submission, status monitoring, result feedback, status persistence, and recovery takeover. The task access submodule receives user tasks and performs format validation and parameter normalization. The scheduling control submodule, as the core control plane, drives the cyclic scheduling loop and policy decisions. The waiting queue submodule caches unexecuted tasks and maintains the scheduling order. The compute node management submodule maintains the task view and available capabilities of each node. The execution submission submodule generates scripts, distributes files, and triggers execution. The status monitoring submodule identifies multi-dimensional statuses based on processes, logs, and completion markers. The result feedback submodule recycles output files from the execution side to the result directory. The status persistence submodule atomically writes critical states to disk. The recovery takeover submodule reads historical states and restores the execution context after a system restart. These components are not stacked in parallel but rather form a dynamic linkage relationship of "input, decision, execution, feedback, and re-decision" through a scheduling closed loop.
[0101] For example, compared with existing solutions, this application incorporates multiple machines into the same scheduling loop for unified control. During the initialization phase, this application receives the machine set and constructs a machine list. In the scheduling loop, the status is updated, capacity is calculated, and tasks are assigned to each machine individually. Therefore, it can continuously coordinate the concurrent execution of multiple machines within a single control plane, rather than independent polling at the single-machine or single-job level. Based on this mechanism, the system can dynamically distribute tasks in the waiting queue to different machines according to available machine capacity and batch rules, automatically adjusting the distribution position when the load changes, achieving cross-machine throughput improvement and load balancing. This technical approach directly solves the common problems of resource fragmentation and scheduling disruption in multi-machine scenarios.
[0102] Furthermore, this application demonstrates significantly enhanced reliability in multi-machine scenarios. Each machine has an independent state view and available capacity determination. The scheduling center refreshes the state before allocating tasks, reducing the risks of duplicate submissions and over-allocation under multi-machine concurrency. Simultaneously, the system continuously persists the state at critical scheduling nodes, enabling tasks to resume progress even after machine state fluctuations or process interruptions. This means that compared to existing solutions primarily based on submission and polling, this application possesses a more complete engineering closed loop under multi-machine concurrent call conditions: unified scheduling, dynamic balancing, anomaly convergence, and interrupt recovery are all possible. The technical effects are more verifiable and more suitable for real-world production computing environments.
[0103] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.
Claims
1. A task orchestration and dynamic scheduling method for multi-machine concurrency, applied to a plurality of task machines; wherein the task machine is a computer device for executing tasks to be executed; the task machine is provided with at least one task node, the task node being an execution unit within the task machine for running the tasks to be executed; characterized in that, The method includes: Receive tasks to be executed; The tasks to be executed are written into a task waiting queue; each task machine is equipped with a corresponding task waiting queue, which is used to cache the tasks to be executed. Refresh the task running status of the tasks to be executed in each task node; the task running status includes: running, completed, and abnormal status. Determine the task running status of the task to be executed. If the task running status is the completed status, remove the corresponding executed task from the task node. If the task running status is the abnormal status, rewrite the corresponding abnormal task into the corresponding task node for execution. Determine whether the available capacity in the task machine is greater than 0. If so, determine whether there are any idle task nodes in the task machine. If so, determine whether there is any task to be executed in the task waiting queue corresponding to the task machine. If so, write the task to be executed in the task waiting queue into the idle task node. The available capacity represents the number of tasks to be executed that the task machine can currently receive. Based on the task to be executed, determine the task output result; the task output result includes: completed, failed, and retryable exception.
2. The task orchestration and dynamic scheduling method for multi-machine concurrency according to claim 1, characterized in that, The method further includes: The parameter format in the task to be executed will be standardized. The tasks to be executed are uniformly mapped to schedulable objects; the schedulable objects are used to store the task information corresponding to the tasks to be executed and to provide the interface corresponding to the scheduling process; the task information includes: task instructions, input and output file information, shared file dependency information, and execution directory information.
3. The task orchestration and dynamic scheduling method for multi-machine concurrency according to claim 1, characterized in that, After the steps of writing the task to be executed into the task waiting queue and rewriting the corresponding abnormal task into the corresponding task node for execution, the following steps are included: Record the status file in the task waiting queue; the status file is the real-time tasks to be executed stored in the task waiting queue.
4. The task orchestration and dynamic scheduling method for multi-machine concurrency according to claim 1, characterized in that, The step of determining the task running status of the task to be executed includes: Obtain the execution evidence of the task to be executed; the execution evidence includes: process presence information, log text signals, completion marker files, and timeout marker information; Using the aforementioned operational evidence, the task execution status of the task to be executed is determined.
5. The task orchestration and dynamic scheduling method for multi-machine concurrency according to claim 4, characterized in that, The step of determining the task execution status of the task to be executed using the execution evidence includes: Based on the process in-place information and the completion marker file, determine whether the task to be executed has been completed; Based on whether there is an abnormal log text signal in the log text signal and whether the timeout mark information exists, determine whether the task running status of the task to be executed is abnormal.
6. The task orchestration and dynamic scheduling method for multi-machine concurrency according to claim 1, characterized in that, The step of rewriting the corresponding abnormal task into the corresponding task node for execution includes: Obtain the number of times the abnormal task is executed when the task's running status is the abnormal status; If the number of times the abnormal task is executed is less than the preset retry execution threshold, the abnormal task will be rewritten into the corresponding task node for execution.
7. The task orchestration and dynamic scheduling method for multi-machine concurrency according to claim 1, characterized in that, After determining whether the available capacity in the task machine is greater than 0, the method further includes: If not, the task to be executed is written into the task waiting queue of the corresponding task machine. When the task machine has available capacity and there are idle task nodes in the task machine, the task to be executed is written into the idle task nodes.
8. The task orchestration and dynamic scheduling method for multi-machine concurrency according to claim 1, characterized in that, The step of writing the tasks to be executed in the task waiting queue into the task nodes in the idle state includes: Determine the batch parameters for the task nodes with available capacity; the batch parameters include: the number of tasks to be executed in each batch and the dependency information between the tasks to be executed; Determine the task information of the tasks to be executed in the task waiting queue; the task information includes: task type and execution priority; Based on the batch parameters and the task information, the tasks to be executed are organized into task units and written into the idle task nodes according to the batch parameters.
9. A task orchestration and dynamic scheduling method for multi-machine concurrency according to claim 1, characterized in that, The method further includes: Based on the task to be executed, an execution script is generated; the execution script is used to execute the task to be executed. Based on the execution script, an execution identifier is generated; Based on the execution identifier, the task running status of the task to be executed is changed to running.
10. A task orchestration and dynamic scheduling method for multi-machine concurrency according to claim 1, characterized in that, The step of determining the task output result based on the task to be executed includes: If the task to be executed has been completed, then the task output result of the task to be executed is determined to be completed, and the completed task results are summarized in the node where the main control program is located, and the corresponding executed task is removed from the task node; If the task to be executed has not been completed, then obtain the number of times the task to be executed has been executed; If the number of executions is less than the preset retry execution count threshold, then the task output result of the task to be executed is determined to be a retryable exception; If the number of executions is greater than or equal to a preset retry threshold, then the task output result of the task to be executed is determined to be a failure.