Method and system for arranging multi-process distributed tasks

Abstract

The invention relates to the technical field of distributed computing and task scheduling management, and provides a method and system for multi-process distributed task orchestration, and the method comprises the steps: initializing an extensible distributed task framework deployed based on gRPC, constructing a Server-Worker bidirectional communication channel, and starting Server nodes to monitor and manage the connection state and task scheduling process of the Worker nodes; a task request submitted through the task configuration interface is responded, and a to-be-executed task in the task queue is distributed to the Worker node to be executed according to a preset scheduling strategy; the Server node dynamically updates the task state according to the execution result returned by the Worker node; and after all the tasks are completed, the Server node generates a task execution summary report, and execution result archiving is completed. By introducing gRPC two-way communication, a task life cycle management mechanism, multi-type Worker scheduling support and a flexible configuration system, automatic task scheduling, real-time monitoring of an execution process and failed task retry in a high-concurrency environment are realized.

Description

Technical Field

[0001] This invention relates to the field of distributed computing and task scheduling management technology, and in particular to an orchestration method and system for multi-process distributed tasks. Background Technology

[0002] In the VLSI (Very Large Scale Integration) design flow, parasitic parameter extraction (PEX) is a critical step affecting timing analysis and signal integrity assessment. This process typically relies on numerous high-precision modeling and simulation tasks, which are computationally intensive, highly concurrent, and complex in execution, often requiring parallel execution in a distributed environment. To improve extraction efficiency and resource utilization, task scheduling systems are widely used to distribute multiple tasks across computing nodes or multi-process environments. Existing distributed task scheduling schemes mainly rely on cluster job management systems such as LSF (Load Sharing Facility) and SGE (Sun Grid Engine), which can support scheduling user-submitted computational tasks to multiple nodes for execution. Combined with protocols such as SSH and RSH, they can achieve remote task distribution across hosts. These frameworks are widely used in industry and possess basic task submission and scheduling capabilities. However, they essentially only provide general interface encapsulation and do not offer fine-grained task lifecycle management for high-concurrency, high-reliability task execution scenarios.

[0003] The existing scheduling system has the following limitations: (1) It lacks real-time tracking of task execution status, and users cannot dynamically understand task progress and abnormal situations; (2) It does not have an automatic retry or status recording mechanism for failed tasks, and manual intervention is required; (3) It cannot continuously monitor the health status of worker nodes and lacks the ability to detect disconnection and restart; (4) It is complex to configure, has poor scalability, and is difficult to flexibly adapt to various operating environments and automated calling requirements. With the expansion of VLSI design task scale and the development trend of heterogeneous computing resources, the traditional scheduling model can no longer meet the requirements of modern design tools, such as the sign-off level parasitic extraction tool, for efficient scheduling and reliable execution in terms of stability, monitorability and automation.

[0004] Therefore, there is an urgent need for a more flexible, configurable, multi-process distributed task orchestration method and system with task status awareness and fault tolerance capabilities, which can realize automatic task scheduling, real-time monitoring, failure retry and dynamic resource management to support efficient task execution in high-density computing scenarios such as parasitic parameter extraction. Summary of the Invention

[0005] To address the problems existing in current distributed task scheduling systems, such as the inability to monitor task status in real time, lack of automatic retry mechanisms for failed tasks, inability to continuously perceive worker node status, and complex system configuration and poor scalability, this invention proposes a method and system for multi-process distributed task orchestration. This system enables automatic task scheduling, status awareness, fault tolerance management, and dynamic resource allocation in high-concurrency, high-computational-load distributed environments. To achieve these objectives, this invention utilizes the following technologies: This invention provides an orchestration method for multi-process distributed tasks, comprising the following steps: Step S1: Initialize the scalable distributed task framework based on gRPC deployment, build a Server-Worker bidirectional communication channel, and start the Server node to listen to and manage the connection status of the Worker node and the task scheduling process; Step S2: In response to the task request submitted through the task configuration interface, the Server node parses the task request and builds a task queue. According to the preset scheduling strategy, the tasks to be executed in the task queue are distributed to the established Worker nodes for execution. Step S3: The Server node dynamically updates the task status based on the execution result returned by the Worker node and triggers the fault tolerance mechanism for failed tasks. The fault tolerance mechanism is to re-add the task to the task queue when the number of task failures does not exceed the retry threshold, or to make the task unexecutable when the number of task failures exceeds the retry threshold. Step S4: Once all tasks are completed, the Server node generates a task execution summary report, archives the execution results, and supports dynamic control operations based on task status.

[0006] Further, step S1 includes: Load the task framework's startup configuration file, parse the supported target Worker types and management parameters. The target Worker types include cluster scheduling environments based on LSF or SGE, remote server nodes supporting SSH or RSH protocols, and local multi-process execution environments. The management parameters include preset upper limits for the number of Worker nodes to be created, upper limits for the task framework's runtime, and execution configuration parameters for each Worker. The execution configuration parameters include upper limits for the number of tasks and runtime of Worker nodes. According to the initialization configuration file, start the Server node and listen to the preset gPRC communication port to receive connection requests from the Worker nodes, establish a bidirectional communication channel between the Server node and each Worker node, and register the corresponding connection information. Initialize the task scheduling and management functions of the Server node, and combine them with the management parameters and connection information of the Worker node to build a task scheduling process for task distribution and execution management.

[0007] Further, step S2 includes: Parse the task request and extract the task parameter information, which includes the task execution command, number of executions, target Worker type, task queue, and result return method. The target task object is constructed based on the task parameter information and added to the corresponding task queue. The task state is initialized to the pending scheduling state and is used as a task to be executed. Based on the task parameter information of the target task object, a matching node is selected from the established Worker nodes, the task to be executed is distributed to the matching node for execution, and the task status is updated to running status. At the same time, the task framework also supports dynamic control operations based on task status.

[0008] Furthermore, the task framework also supports dynamic control operations based on task status, including: when a task is in a running state, triggering the termination, pause, resumption, or suspension of the task through external control commands.

[0009] Further, step S3 includes: The server node receives the execution results returned by the worker node and extracts the corresponding task identifier and execution status; The corresponding task object is identified and retrieved based on the task identifier, and the task status is updated to a failure status or a success status according to the execution status. When the task fails, it is determined whether the number of failures exceeds the preset retry threshold. If the number of failures does not exceed the retry threshold, the task will be re-added to the corresponding task queue and reassigned as a task to be executed. Otherwise, if the number of failures has reached or exceeded the number of retries, the task will be considered unexecutable.

[0010] Furthermore, step S3 also includes, When a task is terminated or fails to execute, the task status changes to the failure status. When a task is paused, the task status changes to paused and then changes to running upon receiving a resume command, or to failed upon receiving a terminate command. When a task is suspended, its state changes to suspended. After receiving a resume command, it changes to waiting. When the corresponding Worker node becomes idle, it resumes running. When the task is executed successfully, the task status is updated to a success status.

[0011] Furthermore, in step S4, after all tasks are completed, the Server node generates a task execution summary report and archives the execution results, including: After all tasks have been completed, the Server node summarizes the execution results of each task and generates a corresponding summary report.

[0012] Based on the same inventive concept, this invention provides an orchestration system for multi-process distributed tasks, employing the orchestration method for multi-process distributed tasks as described above, including: The configuration module is used to initialize the scalable distributed task framework based on gRPC deployment, build a bidirectional communication channel between Server and Worker, and start the Server node to listen to and manage the connection status of Worker nodes and the task scheduling process. Management module: Used to respond to task requests submitted through the task configuration interface. The Server node parses the task request and builds a task queue. According to the preset scheduling strategy, the tasks to be executed in the task queue are distributed to the established Worker nodes for execution. The scheduling module is used by the Server node to dynamically update the task status based on the execution results returned by the Worker node, and to trigger the fault tolerance mechanism for failed tasks. The fault tolerance mechanism is to re-add the task to the task queue when the number of failed tasks does not exceed the retry threshold, or to make the task unexecutable when the number of failed tasks exceeds the retry threshold. The results archiving module is used by the Server node to generate a task execution summary report after all tasks are completed, thus archiving the execution results and supporting dynamic control operations based on task status.

[0013] Furthermore, the configuration module includes, The configuration loading and parsing unit is used to load the startup configuration file of the task framework and parse the supported target worker types and management parameters. The target worker types include cluster scheduling environments based on LSF or SGE, remote server nodes supporting SSH or RSH protocols, and local multi-process execution environments. The management parameters include preset upper limit thresholds for the number of worker nodes to be created, upper limit thresholds for the task framework runtime, and execution configuration parameters for each worker. The execution configuration parameters include upper limit thresholds for the number of tasks and upper limit thresholds for runtime of worker nodes. The communication management unit is used to start the Server node and listen to the preset gPRC communication port according to the initialization configuration file. It is used to receive connection requests from Worker nodes, establish bidirectional communication channels between the Server node and each Worker node, and register the corresponding connection information. It also initializes the task scheduling management function of the Server node and, in combination with the management parameters and connection information of the Worker nodes, constructs a task scheduling process for task distribution and execution management.

[0014] Furthermore, the scheduling module also includes, The status control unit is used to trigger the termination, pause, resumption, or suspension of a task via external control commands when the task is in the running state. When a task is terminated or fails to execute, the task status switches to the failure state; when a task is paused, the task status switches to the paused state and switches back to the running state after receiving a resumption command, or switches back to the failure state after receiving a termination command; when a task is suspended, the task status switches to the suspended state and switches to the waiting state after receiving a resumption command, and resumes to the running state when the corresponding Worker node is idle; when a task is executed successfully, the task status is updated to the success state.

[0015] Compared with the prior art, the present invention has at least one of the following beneficial effects: This invention provides an orchestration method and system for multi-process distributed tasks, which can effectively solve the shortcomings of traditional scheduling systems in terms of task status tracking, failure tolerance, resource adaptation and system scalability. By introducing gRPC bidirectional communication, task lifecycle management mechanism, multi-type worker scheduling support and flexible configuration system, it realizes automatic task scheduling, real-time monitoring of execution process, retry of failed tasks and dynamic resource management in high-concurrency environment, and improves the reliability, flexibility and automation level of task execution. It is particularly suitable for computing scenarios with extremely high requirements for scheduling accuracy and resource utilization, such as parasitic parameter extraction.

[0016] (1) First, in terms of task status management, this invention constructs a fine-grained task lifecycle state machine, covering multiple states such as WAITING, RUNNING, FAILD, STOPPED, SUSPENDED, and SUCCEED, and supports control commands such as task termination, suspension, resumption, and retry, realizing dynamic intervention and state visualization tracking of the task execution process, which significantly improves the transparency and controllability of the scheduling system.

[0017] (2) Secondly, regarding task fault tolerance and stability assurance, this invention supports an automatic retry mechanism for failed tasks. Combined with the retry count threshold judgment logic, it can ensure that tasks have a chance to be redistributed in non-fatal failure scenarios. If the retry limit is reached, the task is archived as a failure for subsequent traceability, effectively reducing the need for manual intervention and improving the system's stable operation capability. Regarding resource adaptation and heterogeneous environment compatibility, this invention supports multiple worker node access modes, including cluster scheduling environments based on LSF / SGE, remote servers supporting SSH / RSH protocols, and local multi-process execution environments. It has strong adaptability and high deployability, meeting the diverse needs of different organizations in terms of resource architecture.

[0018] (3) Finally, in terms of system deployment and operation management, this invention provides a startup configuration file mechanism, which allows users to predefine key parameters such as the number of Workers, task runtime, and concurrency limit. Combined with the gRPC communication of the Server node and the status handshake mechanism of the Worker node, the rapid deployment, horizontal expansion, and efficient resource scheduling of the task framework can be realized, effectively supporting the long-term stable operation of large-scale parallel computing tasks. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the steps of the orchestration method for multi-process distributed tasks according to the present invention. Figure 2 This is a flowchart of the Server-Worker collaborative execution process in an embodiment of the present invention; Figure 3 This is a sequence diagram of the task interaction between Server and Worker in an embodiment of the present invention; Figure 4 This is a schematic diagram of task state transition in an embodiment of the present invention; Figure 5 This is a task lifecycle state transition diagram in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0022] First Embodiment With the increasing complexity of integrated circuit design and the widespread deployment of high-concurrency computing tasks in the EDA (Electronic Design Automation) process, task scheduling systems are playing an increasingly crucial role in the entire design automation chain. After in-depth analysis of existing mainstream scheduling frameworks, such as LSF (Load Sharing Facility), SGE (Sun Grid Engine), and traditional multi-threaded / multi-process scheduling mechanisms, the inventors found that existing technologies still have many limitations in terms of task execution state management, resource utilization efficiency, system scalability, and fault tolerance mechanisms.

[0023] Specifically, existing scheduling frameworks often focus on the initial stage of task distribution, lacking state awareness and dynamic management capabilities throughout the entire task execution process. Their resource management strategies are static and rigid, making it difficult to adapt to the dynamic scheduling needs in a heterogeneous computing resource environment. Furthermore, existing solutions often rely on specific platform interfaces, resulting in strong system closure and hindering rapid cross-platform deployment and integration. In addition, existing frameworks lack effective fault tolerance strategies and automatic task recovery mechanisms for operational anomalies such as task execution failures and worker disconnections, severely impacting system stability and execution efficiency. For example: Existing distributed job scheduling frameworks, such as LSF and SGE, while capable of efficiently managing and scheduling large-scale computing tasks and supporting task distribution to multiple nodes or execution via multi-process methods, still have many shortcomings in practical applications. On the one hand, these frameworks typically only provide basic interfaces for users to call, lacking real-time monitoring of task execution status, making it difficult for users to know about task progress or anomalies. On the other hand, they lack dynamic awareness and management capabilities regarding the use of distributed resources, making them difficult to adapt to complex computing environments. Furthermore, current common scheduling methods often employ thread pool mechanisms, allocating tasks to local threads for execution, failing to support remote scheduling across hosts. If collaborative task execution across multiple machines is required, manual calls to the underlying cluster API are often necessary, resulting in high development and maintenance costs. Simultaneously, in many implementations, each worker can only handle a single task, leading to low resource utilization and limited scalability. While some well-known vendors have built their own distributed scheduling frameworks, most are closed within their own product systems, with closed interfaces, making task scheduling capabilities difficult to reuse in other systems, thus limiting their versatility and adaptability.

[0024] Therefore, based on an in-depth analysis of the limitations of existing scheduling frameworks, and with a systematic consideration of task execution status awareness, fault tolerance mechanisms, dynamic resource scheduling, and cross-platform adaptability, the inventors propose an orchestration method for multi-process distributed tasks. The aim is to design a simple, easy-to-use, and highly scalable distributed task distribution framework that supports various execution environments and flexible task scheduling strategies. This framework not only supports task distribution to distributed clusters based on LSF and SGE, but is also compatible with remote servers connected via SSH and RSH protocols, as well as local multi-process environments, demonstrating excellent environment adaptability. The system allows users to customize the creation of a specified number of Worker nodes and automatically assigns tasks to be executed on each Worker. When a Worker completes a task, the system can automatically schedule the remaining tasks to it for continued execution. For tasks that fail, the framework has a built-in fault tolerance mechanism that supports automatic retries. After exceeding a preset number of retries, failed tasks are recorded as unexecutable tasks to generate an execution summary report. Meanwhile, the system provides standard API interfaces and configuration files, supporting external system calls and independent deployment. It also features Worker connection status monitoring and self-recovery capabilities, automatically restarting the required number of Workers when a Worker becomes disconnected, ensuring the continuity of task execution and the overall robustness of the system. The specific implementation method is as follows: This invention provides an orchestration method for multi-process distributed tasks, such as... Figure 1 As shown, the steps include: Step S1: Initialize the scalable distributed task framework deployed based on gRPC, build a Server-Worker bidirectional communication channel, and start the Server node to listen to and manage the connection status of the Worker nodes and the task scheduling process; wherein, Step S1 includes: Load the task framework's startup configuration file, parse the supported target Worker types and management parameters. The target Worker types include cluster scheduling environments based on LSF or SGE, remote server nodes supporting SSH or RSH protocols, and local multi-process execution environments. The management parameters include preset upper limits for the number of Worker nodes to be created, upper limits for the task framework's runtime, and execution configuration parameters for each Worker. The execution configuration parameters include upper limits for the number of tasks and runtime of Worker nodes. According to the initialization configuration file, start the Server node and listen to the preset gPRC communication port to receive connection requests from the Worker nodes, establish a bidirectional communication channel between the Server node and each Worker node, and register the corresponding connection information. Initialize the task scheduling and management functions of the Server node, and combine them with the management parameters and connection information of the Worker node to build a task scheduling process for task distribution and execution management.

[0025] It needs to be specifically explained that, Step S2: In response to a task request submitted through the task configuration interface, the Server node parses the task request and constructs a task queue. Based on a preset scheduling strategy, it distributes the tasks to be executed in the task queue to the established Worker nodes for execution. Step S2 includes: Parse the task request and extract the task parameter information, which includes the task execution command, number of executions, target Worker type, task queue, and result return method. The target task object is constructed based on the task parameter information and added to the corresponding task queue. The task state is initialized to the pending scheduling state and is used as a task to be executed. Based on the task parameter information of the target task object, a matching node is selected from the established Worker nodes, the task to be executed is distributed to the matching node for execution, and the task status is updated to running status. At the same time, the task framework also supports dynamic control operations based on task status.

[0026] Furthermore, the task framework also supports dynamic control operations based on task status, including: when a task is in a running state, triggering the termination, pause, resumption, or suspension of the task through external control commands.

[0027] Step S3: The Server node dynamically updates the task status based on the execution results returned by the Worker node and triggers the fault tolerance mechanism for failed tasks. The fault tolerance mechanism involves re-adding the task to the task queue if the number of task failures does not exceed the retry threshold, or deeming the task unexecutable if the number of task failures exceeds the retry threshold. Step S3 includes: The server node receives the execution results returned by the worker node and extracts the corresponding task identifier and execution status; The corresponding task object is identified and retrieved based on the task identifier, and the task status is updated to a failure status or a success status according to the execution status. When the task fails, it is determined whether the number of failures exceeds the preset retry threshold. If the number of failures does not exceed the retry threshold, the task will be re-added to the corresponding task queue and reassigned as a task to be executed. Otherwise, if the number of failures has reached or exceeded the number of retries, the task will be considered unexecutable.

[0028] Furthermore, step S3 also includes, When a task is terminated or fails to execute, the task status changes to the failure status. When a task is paused, the task status changes to paused and then changes to running upon receiving a resume command, or to failed upon receiving a terminate command. When a task is suspended, its state changes to suspended. After receiving a resume command, it changes to waiting. When the corresponding Worker node becomes idle, it resumes running. When the task is executed successfully, the task status is updated to a success status.

[0029] Step S4: After all tasks are completed, the Server node generates a task execution summary report, archives the execution results, and supports dynamic control operations based on task status. Step S4 includes: After all tasks have been completed, the Server node summarizes the execution results of each task and generates a corresponding summary report.

[0030] Second Embodiment Based on the same inventive concept, this invention provides an orchestration system for multi-process distributed tasks, employing the orchestration method for multi-process distributed tasks as described above, including: The configuration module is used to initialize the scalable distributed task framework based on gRPC deployment, build a bidirectional communication channel between Server and Worker, and start the Server node to listen to and manage the connection status of Worker nodes and the task scheduling process. Management module: Used to respond to task requests submitted through the task configuration interface. The Server node parses the task request and builds a task queue. According to the preset scheduling strategy, the tasks to be executed in the task queue are distributed to the established Worker nodes for execution. The scheduling module is used by the Server node to dynamically update the task status based on the execution results returned by the Worker node, and to trigger the fault tolerance mechanism for failed tasks. The fault tolerance mechanism is to re-add the task to the task queue when the number of failed tasks does not exceed the retry threshold, or to make the task unexecutable when the number of failed tasks exceeds the retry threshold. The results archiving module is used by the Server node to generate a task execution summary report after all tasks are completed, thus archiving the execution results and supporting dynamic control operations based on task status.

[0031] Furthermore, the configuration module includes, The configuration loading and parsing unit is used to load the startup configuration file of the task framework and parse the supported target worker types and management parameters. The target worker types include cluster scheduling environments based on LSF or SGE, remote server nodes supporting SSH or RSH protocols, and local multi-process execution environments. The management parameters include preset upper limit thresholds for the number of worker nodes to be created, upper limit thresholds for the task framework runtime, and execution configuration parameters for each worker. The execution configuration parameters include upper limit thresholds for the number of tasks and upper limit thresholds for runtime of worker nodes. The communication management unit is used to start the Server node and listen to the preset gPRC communication port according to the initialization configuration file. It is used to receive connection requests from Worker nodes, establish bidirectional communication channels between the Server node and each Worker node, and register the corresponding connection information. It also initializes the task scheduling management function of the Server node and, in combination with the management parameters and connection information of the Worker nodes, constructs a task scheduling process for task distribution and execution management.

[0032] Furthermore, the scheduling module also includes, The status control unit is used to trigger the termination, pause, resumption, or suspension of a task via external control commands when the task is in the running state. When a task is terminated or fails to execute, the task status switches to the failure state; when a task is paused, the task status switches to the paused state and switches back to the running state after receiving a resumption command, or switches back to the failure state after receiving a termination command; when a task is suspended, the task status switches to the suspended state and switches to the waiting state after receiving a resumption command, and resumes to the running state when the corresponding Worker node is idle; when a task is executed successfully, the task status is updated to the success state.

[0033] Third Embodiment To further illustrate the technical effects of the present invention, this embodiment combines the interaction process between Server and Worker, the task execution status management mechanism, and the configuration file-driven approach to provide a detailed description of the distributed task orchestration method and system described in the present invention.

[0034] Figure 2 The flowchart for Server-Worker collaborative execution is as follows: Figure 2 As shown, the Server node collaborates with multiple Worker nodes to achieve efficient task distribution and scheduling. The execution process of each Worker node includes the following stages: (1) After the Worker node starts, it automatically sends a connection request to the Server node to try to establish a bidirectional communication channel.

[0035] (2) After the connection is successfully established, the Worker node actively sends a task request to the server node and continues to wait for the Server node to transmit task information.

[0036] (3) Once the Worker node receives the task information, it starts to execute the corresponding task until the task is successfully completed or fails to execute.

[0037] (4) After the Worker node finishes executing its task, it sends the task execution result back to the Server node and continues to request the next task from the server.

[0038] (5) If a new task cannot be obtained for a long time, or if the connection with the server is lost, the Worker node will be automatically released and shut down.

[0039] Corresponding to the worker process described above, the relevant server-side operation process is as follows: After the server starts, it listens on the connection port. After creating a worker node, it waits to establish a connection with the worker node and records the connection information with each worker node. After a successful connection is established, tasks are retrieved from the task queue one by one and assigned to idle Worker nodes. 3) Wait for the Worker node to return the task execution result.

[0040] 4) After receiving the task result from the Worker node, update the task status and pass the next task information to the Worker node.

[0041] 5) After all tasks are completed, generate and archive a task execution summary report, and shut down all Worker nodes and server nodes.

[0042] like Figure 3 , Figure 4 , Figure 5 As shown, in order to improve the stability and fault tolerance of the task scheduling system, this embodiment further introduces a task failure handling and status management mechanism.

[0043] Specifically, during task execution, the Server node continuously receives task execution feedback from each Worker node. When a task fails due to runtime anomalies or resource issues, the Worker node promptly reports the failure status to the Server node. The job manager integrated into the Server node automatically records the failure information and re-adds the task to the execution queue for the next round of scheduling. In this way, the system can automatically retry and redistribute failed tasks, ensuring that the overall execution progress is not interrupted by a single point of failure.

[0044] When a task fails due to runtime anomalies or resource issues, the Worker promptly reports the failure status to the Server. The Server's integrated job manager module automatically records the task's failure information and re-adds the task to the execution queue for the next round of scheduling. In this way, the system can automatically retry and redistribute failed tasks, ensuring that the overall execution progress is not interrupted by a single point of failure.

[0045] A task has eight states and their transition relationships. Normally, a task starts in the READY state and enters the RUNNING state, unless special circumstances arise. Based on the execution result, the job will enter one of three states: PASS, FAILD, or ERROR. PASS indicates successful job execution; ERROR usually indicates a problem that prevented the job from completing successfully; and FAILD typically refers to an unexplained failure during execution.

[0046] If a task fails to execute consecutively more than the preset maximum number of retries (e.g., 3 times), the server will automatically mark the task as unrecoverable and remove it from the queue. At the same time, the system will write relevant information about the task (such as task ID, error log, failure reason, etc.) to a specific error archive file for subsequent debugging, analysis, and manual intervention by operations and maintenance personnel.

[0047] In other words, a task has eight states in the system, including: READY: The task has been created but not yet scheduled; RUNNING: The task has been scheduled to a worker and has started execution; SUCCEED: The task completes normally; FAILD: The task encounters a business exception or runtime error during execution; ERROR: The task cannot be executed due to external dependency failure or system error (such as network disconnection or resource unavailability); RETRY: Tasks in the FAILD or ERROR state will enter the retry process; GIVEUP: After the number of retries is exhausted, the system marks the task as abandoned; STOPPED: The task is terminated by user or system interruption and will no longer participate in subsequent scheduling.

[0048] In the normal process, a task moves from READY to RUNNING. If execution is successful, it transitions to the SUCCEED state. If execution fails (FAILD) or an error occurs (ERROR), the system will transfer the task to the RETRY state for rescheduling based on the configured number of retries. If the task still fails within the allowed number of retries, it will eventually enter the GIVEUP state, meaning the system actively abandons the task. At any stage, the user can interrupt the task execution, causing the task to directly enter the STOPPED state.

[0049] As shown in Figure 5, in this embodiment, the task status management mechanism covers the entire lifecycle of a task, from submission and scheduling to completion or abnormal exit. When a user submits a task, the system first marks the task status as "WAITING," indicating that the task has not yet formally entered the execution scheduling process. After the task is received by the system and added to the task queue, the status changes to "PENDING," indicating that the task has met the conditions for being distributed and is waiting for an available Worker node to acquire the execution right.

[0050] When a task is successfully picked up and started by an idle worker, its status is updated to RUNNING. In the RUNNING state, the system supports various control operations. If the task completes successfully, its status will be set to "SUCCEED". If a failure occurs during execution, but is still within the allowed number of retries, the system will record the failure or error status, such as FAILD or ERROR, and return the task to the scheduling queue for retry. If the preset maximum number of retries is exceeded, the task will be considered a failure, and its failure information will be recorded for subsequent troubleshooting.

[0051] During task execution, users can initiate an interrupt command at any time to intervene in the task's execution status. If the user chooses to pause the current task, the task status will switch to "STOPPED". In the paused state, the task will no longer execute, but the system will retain its current context. If the user subsequently initiates a resume operation, the task status will switch back to RUNNING and continue execution; if a termination operation is initiated, the task will directly enter the FILD state and will not attempt to restart.

[0052] In addition, to support more flexible control requirements, the system introduces a SUSPENDED state. The SUSPENDED state is primarily used to temporarily relinquish computing resources or allow for manual control of task pacing. Suspended tasks will be paused by the system and will not participate in scheduling until the user resumes execution. Upon resumption, the task will first enter a PENDING state, and will only resume running once scheduling resources are available.

[0053] The state transition mechanism adopted in this embodiment, combined with automatic retry, active control and fault-tolerant recovery strategies, comprehensively improves the controllability and stability of the distributed task execution process, ensuring that the system can maintain the continuity and reliability of the task flow even in the context of dynamic resource changes or node anomalies.

[0054] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An orchestration method for multi-process distributed tasks, characterized in that the steps include... include: Step S1: Initialize the scalable distributed task framework based on gRPC deployment, build a Server-Worker bidirectional communication channel, and start the Server node to listen to and manage the connection status of the Worker node and the task scheduling process; Step S2: In response to a task request submitted through the task configuration interface, the Server node parses the task request and constructs a task queue, and distributes the tasks to be executed in the task queue to the Worker nodes with established connections for execution according to a preset scheduling strategy; Step S3: The Server node dynamically updates the task status based on the execution result returned by the Worker node and triggers the fault tolerance mechanism for failed tasks. The fault tolerance mechanism is to re-add the task to the task queue when the number of task failures does not exceed the retry threshold, or to make the task unexecutable when the number of task failures exceeds the retry threshold. Step S4: When all the tasks are completed, the Server node generates a task execution summary report, completes the archiving of the execution results, and supports dynamic control operations based on the task status.

2. The orchestration method for multi-process distributed tasks according to claim 1, characterized in that, Step S1 includes: The startup configuration file of the task framework is loaded, and the supported target Worker types and management parameters are parsed. The target Worker types include cluster scheduling environments based on LSF or SGE, remote server nodes supporting SSH or RSH protocols, and local multi-process execution environments. The management parameters include preset upper limit thresholds for the number of Worker nodes to be created, upper limit thresholds for the running time of the task framework, and execution configuration parameters for each Worker. The execution configuration parameters include upper limit thresholds for the number of tasks and upper limit thresholds for the running time of the Worker node. According to the initialization configuration file, the Server node is started and listens to the preset gPRC communication port to receive connection requests from the Worker nodes, establish the bidirectional communication channel between the Server node and each Worker node, and register the corresponding connection information. The task scheduling and management function of the Server node is initialized. Combined with the management parameters of the Worker node and the connection information, the task scheduling process is constructed for task distribution and execution management. At the same time, the Server node periodically monitors the connection information and active timestamps of each Worker node, and automatically attempts to restart any Worker node that is offline or disconnected.

3. The orchestration method for multi-process distributed tasks according to claim 2, characterized in that, Step S2 includes: The task request is parsed and task parameter information is extracted. The task parameter information includes the task execution command, execution count, target Worker type, task queue, and result return method. Based on the task parameter information, a target task object is constructed and added to the corresponding task queue. The task state is initialized to a pending scheduling state, serving as the task to be executed. Based on the task parameter information of the target task object, a matching node is selected from the established Worker nodes, the task to be executed is distributed to the matching node for execution, and the task status is updated to running status. At the same time, the task framework also supports dynamic control operations based on the task status.

4. The orchestration method for multi-process distributed tasks according to claim 3, characterized in that, The task framework also supports dynamic control operations based on the task status, including: when the task is in the running state, triggering the termination, pause, resumption or suspension of the task through external control commands.

5. The orchestration method for multi-process distributed tasks according to claim 4, characterized in that, Step S3 includes: The Server node receives the execution result returned by the Worker node and extracts the corresponding task identifier and execution status; The task object is identified and retrieved based on the task identifier, and the task status is updated to a failure status or a success status according to the execution status. When the task fails to execute, it is determined whether the number of failures of the task exceeds the preset retry threshold. When the number of failures does not exceed the retry threshold, the task is re-added to the corresponding task queue and allocated as a task to be executed; Otherwise, when the number of failures has reached or exceeded the number of retries, the task will be designated as an unexecutable task.

6. The orchestration method for multi-process distributed tasks according to claim 4, characterized in that, Step S3 also includes, When the task is terminated or fails to execute, the task status switches to the failed status; When the task is paused, the task status switches to the paused state, and switches to the running state after receiving a resume command, or switches to the failed state after receiving a termination command; When a task is suspended, the task status switches to the suspended state. After receiving the resume instruction, it switches to the waiting state and resumes the running state when the corresponding Worker node becomes idle. When the task is executed successfully, the task status is updated to the successful status.

7. The orchestration method for multi-process distributed tasks according to claim 6, characterized in that, In step S4, after all the tasks are completed, the Server node generates a task execution summary report to complete the archiving of the execution results, including: After all the tasks have been completed, the Server node summarizes the execution results of each task and generates the corresponding summary report.

8. An orchestration system for multi-process distributed tasks, employing the orchestration method for multi-process distributed tasks as described in any one of claims 1 to 7, characterized in that, include, The configuration module is used to initialize the scalable distributed task framework based on gRPC deployment, build a bidirectional communication channel between Server and Worker, and start the Server node to listen to and manage the connection status of Worker nodes and the task scheduling process. Management module: Used to respond to task requests submitted through the task configuration interface. The Server node parses the task request and builds a task queue. According to the preset scheduling strategy, the tasks to be executed in the task queue are distributed to the Worker nodes that have established connections for execution. The scheduling module is used by the Server node to dynamically update the task status based on the execution result returned by the Worker node, and to trigger the fault tolerance mechanism for failed tasks. The fault tolerance mechanism is to re-add the task to the task queue when the number of task failures does not exceed the retry threshold, or to make the task unexecutable when the number of task failures exceeds the retry threshold. The results archiving module is used to generate a task execution summary report on the Server node after all the tasks are completed, complete the execution results archiving, and support dynamic control operations based on the task status.

9. The orchestration system for multi-process distributed tasks according to claim 8, characterized in that, The configuration module includes, The configuration loading and parsing unit is used to load the startup configuration file of the task framework and parse the supported target Worker types and management parameters. The target Worker types include cluster scheduling environments based on LSF or SGE, remote server nodes supporting SSH or RSH protocols, and local multi-process execution environments. The management parameters include preset upper limit thresholds for the number of Worker nodes to be created, upper limit thresholds for the running time of the task framework, and execution configuration parameters for each Worker. The execution configuration parameters include upper limit thresholds for the number of tasks and upper limit thresholds for the running time of the Worker node. The communication management unit is used to start the Server node and listen to the preset gPRC communication port according to the initialization configuration file, to receive connection requests from the Worker nodes, to establish the bidirectional communication channel between the Server node and each Worker node, and to register the corresponding connection information; to initialize the task scheduling management function of the Server node, and to construct the task scheduling process for task distribution and execution management by combining the management parameters of the Worker nodes and the connection information; at the same time, the Server node periodically monitors the connection information and active timestamps of each Worker node, and automatically attempts to restart any Worker node that is offline or disconnected.

10. The orchestration system for multi-process distributed tasks according to claim 9, characterized in that, The scheduling module also includes, The status control unit is used to trigger the termination, pause, resumption or suspension of the task through external control commands when the task is in the running state. When the task is terminated or fails to execute, the task status switches to the failure state. When the task is paused, the task status switches to the paused state and switches back to the running state after receiving a resumption command, or switches back to the failure state after receiving a termination command. When a task is suspended, its status switches to the suspended state. After receiving the resume instruction, it switches to the waiting state and resumes the running state when the corresponding Worker node becomes idle. When the task is successfully executed, its status is updated to the success state.

Images