Task execution method and system for heterogeneous execution environment

Abstract

This application relates to a task execution method and system for heterogeneous execution environments, belonging to the field of electronic design automation. The method includes: constructing a unified interface for receiving tasks and a main scheduling thread; encapsulating several heterogeneous execution environments as different implementations of the unified interface; receiving business requests; creating tasks based on the business requests, the business requests including the target execution environment corresponding to the task; adding the task to a task manager; when a task meets preset activation conditions, adding the task to the task queue of the task manager; scheduling the task from the task queue when the preset wake-up conditions of the main scheduling thread are met; inputting the task to the target execution environment corresponding to the task through the unified interface; creating a task execution process in the target execution environment; executing the task through the task execution process; and monitoring the task execution process using a thread pool. The technical effect is: efficient task execution.

Description

Technical Field

[0001] This application relates to the field of electronic design automation, and in particular to a task execution method and system for heterogeneous execution environments. Background Technology

[0002] In the field of EDA (Electronic Design Automation), tasks such as compilation, logic synthesis, and placement and routing for hardware-aided verification systems (including FPGA prototyping, hardware simulation, etc.) typically consume significant computing resources. Existing technologies generally employ two main task execution methods: direct execution via local processes and network execution based on a cluster scheduler. The most commonly used cluster schedulers in EDA include: LSF (Load Sharing Facility) cluster job submission and Slurm (Simple Linux Utility for Resource Management) cluster job submission.

[0003] Most existing EDA tools employ a tightly coupled architecture for task management, meaning that the application's business logic code is deeply bound to the process management code within a specific execution environment (local / LSF / Slurm), directly calling the corresponding command-line tools and application programming interfaces (APIs) for each environment. This design results in three independent mechanisms for local process management, LSF cluster job management, and Slurm cluster job management, each with different interfaces: for example, local processes rely on system-level calls (such as Linux's fork / exec and waitpid, and Windows' process creation functions); LSF requires a command system for submitting, querying, and terminating jobs; and Slurm uses a command system for running, querying queues, and canceling jobs.

[0004] When an application needs to adapt to multiple execution environments simultaneously, it must handle the startup, status query, termination and other operations of different environments through a large number of conditional branches in the business code, resulting in code redundancy and maintenance difficulties. Summary of the Invention

[0005] To address the issue of incompatibility between various task execution environments, this application provides a task execution method and system for heterogeneous execution environments.

[0006] Firstly, this application provides a task execution method and system for heterogeneous execution environments, employing the following technical solution:

[0007] A unified interface and main scheduling thread are constructed for receiving tasks, and several heterogeneous execution environments are encapsulated as different implementations of the unified interface; Receive a business request, and create a task based on the business request, wherein the business request includes the target execution environment corresponding to the task; The task is added to the task manager. When the task meets the preset activation conditions, the task is added to the task queue of the task manager. When the preset wake-up conditions of the main scheduling thread are met, the task is scheduled from the task queue; The task is input into the target execution environment corresponding to the task through the unified interface, and a task execution process is created in the target execution environment. The task is executed through the task execution process, and the thread pool is used for monitoring during the execution of the task.

[0008] Through the above technical solutions, this application constructs a unified interface and encapsulates heterogeneous execution environments into different implementations of this unified interface, thereby completely decoupling the details of the upper-layer business logic from the details of the lower-layer execution environment, significantly reducing code complexity and maintenance costs. At the same time, this application separates the task activation conditions and scheduling strategies from the business logic by cooperating with the task manager for task queuing and scheduling, the main scheduling thread for unified scheduling and distribution, and the thread pool for execution process monitoring, making the flow of tasks more flexible and controllable.

[0009] In one specific implementation, adding the task to the task manager when the task meets the preset activation conditions includes: When the task meets the preset activation conditions, the preset activation conditions include the task dependencies being ready and the resources required for the task execution being prepared. Determine whether the execution parameters in the task are within a preset legal range and whether the task type is not in a preset invalid type library; If the execution parameters are within the preset valid range and the task type is not in the preset invalid type library, then the task will be added to the task manager.

[0010] By setting preset activation conditions that include task dependency readiness and resource preparation completion, and by additionally performing dual checks on the legality range of execution parameters and the validity of task types, strict access filtering is performed before tasks are queued, effectively preventing invalid or illegal tasks from entering the task queue. This ensures the operational stability of the task manager and improves the overall success rate of task execution.

[0011] In one specific implementation, the task queue includes several tasks to be executed, and scheduling the tasks from the task queue when the preset wake-up condition of the main scheduling thread is met includes: When the preset wake-up conditions of the main scheduling thread are met, the preset wake-up conditions include detecting that the task has entered the task queue; Based on the priority information of several tasks to be executed, the order in which the tasks to be executed arrive at the task queue, and the load status of the task manager, a scheduling strategy for several tasks to be executed is formulated. The task is scheduled from the task queue according to the scheduling policy.

[0012] By implementing the above technical solution, the main scheduling thread is awakened when a task is detected entering the queue. The scheduling strategy is formulated by comprehensively considering the priority information of the tasks to be executed, the order in which they arrive in the queue, and the current real-time load of the task manager. This approach can prioritize the response to urgent tasks while ensuring fairness, and dynamically adapt to changes in system load, thereby optimizing the average waiting time of tasks and the utilization rate of system resources.

[0013] In one specific implementation scheme, the step of formulating a scheduling strategy for the tasks to be executed based on the priority information of the tasks to be executed, the order in which the tasks to be executed arrive at the task queue, and the load status of the task manager includes: The system receives several historical task scheduling policies, which include historical task information of several historical tasks, historical load information of the task manager, and historical task scheduling policies formulated according to preset scheduling rules. The historical task information includes historical task priority information and historical task arrival order information. Construct a scheduling strategy model and train the scheduling strategy model using several historical task scheduling strategies; The priority information of several tasks to be executed, the order in which the tasks to be executed arrive at the task queue, and the load status of the task manager are input into the trained scheduling strategy model to obtain the scheduling strategy for several tasks to be executed.

[0014] By using the above technical solution, a scheduling strategy model is trained by collecting historical task scheduling strategies, and the priority, arrival order, and manager load of the current task are input into the trained model to automatically generate a scheduling strategy. This application achieves an adaptive intelligent optimization effect of the scheduling strategy, enabling the system to learn from historical execution experience and dynamically adjust scheduling behavior, thereby continuously improving the scheduling accuracy and execution efficiency in complex and ever-changing scenarios.

[0015] In one specific implementation, executing the task through the task execution process includes: Determine whether the task execution process corresponding to the task has been created successfully; If the task execution process corresponding to the task is created successfully, the task is determined to have started successfully, and the task is executed through the task execution process. If the task execution process corresponding to the task is not created successfully, the task is determined to have failed to start, and an error log and the reason for the failure are recorded.

[0016] By using the above technical solution, and by clearly determining whether the task execution process has been successfully created, and recording error logs and reasons for failure when creation fails, this application achieves the technical effect of clearly distinguishing the task startup status, ensuring that the system can capture and differentiate between successful and failed startup results in a timely manner. At the same time, it provides detailed error tracing evidence for operation and maintenance personnel, greatly facilitating system debugging and rapid repair of abnormal problems.

[0017] In one specific implementation, the monitoring using the thread pool during the execution of the task includes: A worker thread is retrieved from the thread pool, and the process handle of the worker thread is bound to the unique identifier of the task. The worker thread is used for monitoring during the execution of the task.

[0018] By employing the aforementioned technical solution, and by retrieving a worker thread from the thread pool and binding the process handle of that worker thread with the unique identifier of the task for dedicated monitoring, this application achieves a precise one-to-one mapping between monitoring resources and task instances. This ensures that monitoring operations can accurately target the task, while avoiding the frequent creation and destruction of monitoring threads, reducing system overhead, and improving the accuracy and reliability of monitoring data.

[0019] In one specific implementation, the monitoring using the worker thread during the execution of the task includes: The environmental state of the target execution environment is mapped to a process-level state according to a preset state mapping table, wherein the state mapping table includes several environmental states and several process-level states corresponding to the environmental states. The process-level state is converted to a task-level state through the task manager; The worker thread is used to monitor the task-level status to obtain the liveness status of the task execution process; The monitoring using the worker thread during the execution of the task also includes: During the execution of the task, the exit code and exception signals of the task execution process are captured; The monitoring using the worker thread during the execution of the task also includes: During the execution of the task, monitor the running time of the task execution process and determine whether the running time exceeds the preset maximum running time; If the running time exceeds the preset maximum running time, the task execution timeout is marked.

[0020] By using the above technical solution, the heterogeneous environment states are uniformly mapped to standardized process-level states, then converted to task-level states, and worker threads are used to monitor the states to obtain survival information. At the same time, exit codes, abnormal signals and runtime are captured and monitored to determine timeouts. This application achieves the technical effect of shielding the differences in the underlying heterogeneous environment states and forming a unified task state view. It realizes comprehensive perception of various abnormal situations throughout the entire life cycle of task execution, and enhances the robustness and control capabilities of the system.

[0021] In one specific implementation, after the process-level state is converted to a task-level state via the task manager, the following steps are included: Determine whether the task-level status of the task has changed; If the task-level state of the task changes, the task's state information is output by calling a pre-registered callback function.

[0022] By using the above technical solution, and by calling a pre-registered callback function to actively output status information when the task-level status changes, this application achieves the effect of real-time proactive notification of task status changes, enabling the upper-layer business logic or user interface to immediately perceive the task progress and react accordingly, thereby enhancing the real-time responsiveness of the system.

[0023] In one specific implementation, after the task is executed through the task execution process, the method further includes: Determine whether redundant data generated by the task needs to be cleaned up based on the task configuration and execution results. If it is necessary to clean up redundant data generated by the task, delete the redundant data, restore the target execution environment, and release the resources occupied by the task.

[0024] By using the above technical solution, this application achieves automated environment cleaning and resource recycling after task execution by actively determining whether redundant data needs to be cleaned up based on task configuration and execution results, and deleting redundant data, restoring the target execution environment to its initial state, and releasing all resources occupied by the task when necessary. This effectively avoids the risks of data accumulation and resource exhaustion, and ensures the long-term stable operation of the system and the efficient recycling of computing resources.

[0025] Secondly, this application provides a task execution system for heterogeneous execution environments, employing the following technical solution: the system includes: A unified interface construction module is used to construct a unified interface for receiving tasks and a main scheduling thread, and to encapsulate several heterogeneous execution environments into different implementations of the unified interface. The task creation module is used to receive a business request and create a task according to the business request, wherein the business request includes the target execution environment corresponding to the task. The task activation module is used to add the task to the task manager. When the task meets the preset activation conditions, the task is added to the task queue of the task manager. The task scheduling module is used to schedule the task from the task queue when the preset wake-up conditions of the main scheduling thread are met. The task execution preparation module is used to input the task into the target execution environment corresponding to the task through the unified interface, and to create a task execution process in the target execution environment. The task execution module is used to execute the task through the task execution process, and to monitor the execution of the task using the thread pool.

[0026] In summary, this application has the following beneficial technical effects: (1) This application constructs a unified interface and encapsulates heterogeneous execution environments into different implementations of the unified interface, thereby completely decoupling the details of the upper-layer business logic from the details of the lower-layer execution environment, significantly reducing the complexity of the code and maintenance costs; at the same time, this application separates the activation conditions and scheduling strategies of tasks from the business logic by cooperating with the task manager for task queuing and scheduling, the main scheduling thread for unified scheduling and distribution, and the thread pool for monitoring the execution process, making the flow of tasks more flexible and controllable.

[0027] (2) This application retrieves a worker thread from the thread pool and binds the process handle of the worker thread to the unique identifier of the task for dedicated monitoring. By uniformly mapping the states of various heterogeneous environments to standardized process-level states, and then converting them to task-level states, and using worker threads to monitor the states to obtain survival information, this application achieves the technical effect of precise one-to-one mapping between monitoring resources and task instances, ensuring that monitoring operations can accurately act on the target task, reducing system overhead and improving the accuracy and reliability of monitoring data. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the original process execution method.

[0029] Figure 2 This is a flowchart illustrating the LSF cluster job submission process.

[0030] Figure 3This is a flowchart illustrating the job submission process based on a Slurm cluster.

[0031] Figure 4 This application describes the connection relationships between components in a task execution method for heterogeneous execution environments.

[0032] Figure 5 This is a diagram of the HSTask virtual method extension points and lifecycle management.

[0033] Figure 6 This is a flowchart of the Task Manager process.

[0034] Figure 7 This is a flowchart of a task execution method for a heterogeneous execution environment according to an embodiment of this application.

[0035] Figure 8 This is a schematic diagram of the unified task scheduling and process management process.

[0036] Figure 9 It represents the intent of the state mapping.

[0037] Figure 10 This is a structural block diagram of a task execution method for heterogeneous execution environments according to an embodiment of this application.

[0038] Figure reference numerals: 1001, Unified interface construction module; 1002, Task creation module; 1003, Task activation module; 1004, Task scheduling module; 1005, Task execution preparation module; 1006, Task execution module. Detailed Implementation

[0039] The following is in conjunction with the appendix Figures 1-10 This application will be described in further detail.

[0040] This application discloses a task execution method for heterogeneous execution environments, which is used to execute tasks efficiently.

[0041] In the field of EDA (Electronic Design Automation), tasks such as compilation, logic synthesis, and placement and routing for hardware-aided verification systems (including FPGA prototyping, hardware simulation, etc.) typically consume significant computing resources. Existing technologies generally employ two main task execution methods: direct execution via local processes and network execution based on a cluster scheduler. The most commonly used cluster schedulers in EDA include: LSF (Load Sharing Facility) cluster job submission and Slurm (Simple Linux Utility for Resource Management) cluster job submission.

[0042] like Figure 1 The diagram shown is a flowchart illustrating the original process execution method, combined with... Figure 1 To put it another way: In enterprise-level EDA computing environments, applications launch child processes directly on the local machine to execute tasks through the operating system's process creation interface. The main program, acting as the parent process, creates child processes via system calls. The child processes execute computing tasks on the local CPU, while the parent process waits for the child processes to complete by either waiting for them to finish or polling.

[0043] like Figure 2 The diagram shows the LSF cluster job submission process. Figure 2 To put it another way: In enterprise-level EDA computing environments, the IBM LSF distributed job scheduling system is commonly used. Applications submit jobs to the LSF scheduler via job submission commands, and the scheduler allocates compute nodes for execution based on cluster resources. Job status is queried using job submission commands, and job termination is achieved using the terminate job command. The main program interacts with the LSF scheduler through the LSF client, and the scheduler distributes jobs to various compute nodes for execution.

[0044] like Figure 3 The diagram shows the job submission process based on a Slurm cluster, combined with... Figure 3 To put it another way: In an enterprise-level EDA computing environment, applications submit jobs using the "Run Job" or "Batch Submission" command, query job status using the "Query Queue" command, and cancel jobs using the "Cancel Job" command. The main program interacts with the Slurm scheduler through the Slurm command-line client, and the scheduler manages job execution on the compute nodes.

[0045] Most EDA tools currently manage tasks in a tightly coupled manner: that is, the application's business logic code is deeply coupled with the process management code of a specific execution environment (local / LSF / Slurm), directly calling the command-line tools and application programming interfaces (APIs) of the corresponding environment.

[0046] The existing technology has the following drawbacks: First, in existing technologies, the three mechanisms of local process management, LSF cluster job management, and Slurm cluster job management are independent of each other and have completely different interfaces. Local processes use system calls (such as fork / exec and waitpid in Linux, and process creation functions in Windows), LSF uses a command system for submitting jobs / querying jobs / terminating jobs, and Slurm uses a command system for running jobs / querying queues / canceling jobs.

[0047] When an application needs to support multiple execution environments, the startup, status query, and termination operations of different environments must be handled separately in the business logic code through numerous conditional branches, leading to code redundancy and maintenance difficulties. If a new execution environment (such as Kubernetes) needs to be added, branches must be added to all business logic code, violating the open-closed principle.

[0048] Second, local processes typically only have two coarse-grained states: running and exited. LSF defines multiple states such as running, waiting, system suspended, user suspended, pre-suspended, exited, and completed. Slurm defines multiple states such as running, waiting, completing, completed, stopped, and node failed.

[0049] The state definitions of the three environments are incompatible, and applications cannot use a unified state model to describe the task lifecycle. During the task's progress from initiation to completion, the lack of a unified finite state machine to manage state transitions easily leads to errors such as inconsistent states and lost states.

[0050] Third, existing EDA tools typically execute tasks sequentially or manually create threads in the business code to achieve concurrency. Manual thread management lacks unified concurrency control and cannot dynamically limit the number of tasks running simultaneously based on computing resources (number of CPU cores, cluster queue quotas, etc.).

[0051] When the number of tasks is large, the following problems may occur: too many processes are started at the same time in local mode, causing system resources to be exhausted; exceeding the queue job quota in cluster mode, causing submission failure; waiting tasks cannot be scheduled immediately after a task is completed, resulting in idle resources.

[0052] Fourth, in existing technologies, the lifecycle operations of tasks, such as creation, execution, cleanup, and recovery, are scattered throughout the business code, lacking a unified management framework. When a task fails and needs to clean up intermediate artifacts and restore to its initial state for re-execution, a large amount of cleanup and recovery logic needs to be manually written. Furthermore, auxiliary management functions such as task execution time statistics, input change detection, and expiration marking also lack unified implementation.

[0053] In existing technologies, notifications of task state changes are typically implemented using global variables or flags. Observers need to poll to detect state changes, which is inefficient and lacks real-time performance. Furthermore, the notification methods for state changes are inconsistent across different execution environments, making it impossible to achieve a unified observer pattern that automatically triggers callbacks when the task state changes.

[0054] Therefore, this application proposes a task execution method for heterogeneous execution environments, which enables efficient task execution.

[0055] For ease of understanding, the following explains some key terms in this embodiment: EDA: Electronic Design Automation, refers to the technology of using computer-aided design software to complete the design of integrated circuits and electronic systems.

[0056] HAV: Hardware Assisted Verification, uses hardware resources to assist / accelerate the verification of logical functions, including prototype verification and hardware simulation.

[0057] LSF: Load Sharing Facility, a distributed job scheduling system developed by IBM for managing computing resources and scheduling jobs in a cluster environment.

[0058] Emulator: Hardware simulation, which places the part or all of the logic to be verified into a specific hardware system to improve the simulation speed.

[0059] Heterogeneous execution environments: Computing environments with different underlying architectures, operating systems, or resource management mechanisms. For example, these can include local compute nodes, LSF (Load Sharing Facility) based cluster environments, and Slurm (Simple Linux Utility for Resource Management) based cluster environments. These environments may differ in terms of task submission, status querying, and resource allocation.

[0060] Task Manager: A core component whose main function is to centrally manage and coordinate the lifecycle of all tasks in the system. This includes tasks receiving, scheduling, status tracking, and resource allocation. Task Manager aims to provide a unified view and control point to address the complexities of task management in different execution environments.

[0061] Task queue: This is a data structure within the Task Manager used to store tasks awaiting scheduling. Once a task meets preset activation conditions, it is placed in this task queue, waiting for scheduling by the main scheduling thread. This task queue is typically organized based on factors such as task priority and arrival time.

[0062] A thread pool is a mechanism that pre-creates and maintains a set of threads. These threads can be reused to execute multiple tasks, thus avoiding the overhead of frequently creating and destroying threads. In this embodiment, the thread pool is mainly used for monitoring during task execution to ensure stable task operation and exception handling.

[0063] The main scheduling thread is the thread responsible for the core logic of task scheduling in the system. This thread periodically, or when triggered by specific events, selects suitable tasks from the task queue and assigns them to the corresponding target execution environment for execution. The main scheduling thread is the critical control point for task flow.

[0064] A business request is an instruction issued to this method by an external system or user, requesting the execution of a specific task. This business request typically includes information such as the task type, parameters, and the desired target execution environment, serving as the basis for task creation.

[0065] A task is the smallest unit of work that needs to be executed by the system. A task can be a computational job, a data processing flow, or any other operation that requires computing resources to complete. Each task typically includes its own execution logic, required resources, and target execution environment.

[0066] Process creation: refers to the process by which the operating system creates a new process. In Linux, this is achieved through process copying / execution system calls, while in Windows, it is achieved through the process creation interface.

[0067] Startup failed (START_FAILED): The process failed to be created during the task startup process due to insufficient permissions, exhaustion of resources, corruption of executable files, or other reasons.

[0068] Running: The process has been successfully created and is currently executing computational tasks.

[0069] Unified Process Abstraction Interface: Defines a unified virtual interface for process management, shielding the differences between underlying heterogeneous execution environments so that upper-layer business logic does not need to be aware of the specific execution environment.

[0070] Two-layer state machine: A hierarchical state management model consisting of a lower-level process-level state machine and an upper-level task-level state machine, which realizes the mapping from the native state of heterogeneous environments to a unified semantics.

[0071] Condition variables: a thread synchronization mechanism that allows a thread to block and wait when a certain condition is not met, and to be woken up when the condition is met, for the purpose of achieving efficient scheduling notification.

[0072] Concurrent slots: Resource quota counters used to control the number of tasks running simultaneously. Each running task occupies one slot, which is released after the task is completed.

[0073] Callback function: A function registered by the user and automatically called by the system when a specific event (such as a state change) occurs, used to implement event notification in the observer pattern.

[0074] like Figure 4The diagram shows the connection relationships between the various components in this application, explained in detail below: I. Process Abstraction Interface Layer (HSProcess) The Process Abstraction Interface (API) is a pure virtual base class that defines a unified interface for process management. The API is the common parent class of the three concrete implementation classes: the local process implementation, the LSF cluster process implementation, and the Slurm cluster process implementation. The task encapsulation layer holds a reference to the process object through a shared pointer, which points to different implementation classes at runtime based on the configuration.

[0075] II. Local Process Implementation (LocalProcess): The local process implementation inherits from the process abstraction interface and encapsulates the process management capabilities of the local operating system. The main processing flow includes: checking whether the process is in a startable state (state is ready, not terminated, not started, child process pointer is null); searching for the executable file path in the system environment variable path; creating a child process through the process library, and deciding whether to redirect standard output and standard error according to the configuration; verifying the validity of the child process, and setting the startup result flag.

[0076] III. LSF Cluster Process Implementation (LsfProcess): The LSF cluster process implementation inherits from the process abstraction interface and encapsulates the LSF cluster job management capabilities.

[0077] The core components include: job name, job identifier (atomic integer variable), LSF-specific parameter list (such as queue name, resource requirements, etc.), process object of the job submission command itself, and static state mapping table.

[0078] Startup Process: Construct the submit job command parameters (job name parameters + LSF-specific parameters + target command and parameters); execute the submit job command through the process library, redirecting standard output via a pipe; parse the job identifier from the submitted job output; wait for the submit job command itself to exit, confirming the job identifier is valid. Status Query Process: Execute the query job command to query the status of the specified job identifier; parse the line corresponding to the job identifier in the output, extracting the status field; look up the mapped unified process status through a static status mapping table. Termination Process: Execute the terminate job command to terminate the specified job identifier, checking if the output contains termination confirmation information.

[0079] IV. Slurm Cluster Process Implementation (SlurmProcess): The Slurm cluster process implementation inherits from the process abstraction interface and encapsulates the Slurm cluster job management capabilities.

[0080] The core components include: job name, Slurm-specific parameter list, job identifier (atomic integer variable), and static state mapping table.

[0081] Startup Process: Construct the command parameters for running the job (job name parameter + Slurm-specific parameters + target command and parameters); clear the job identifier environment variable (set to empty), supporting nested submission of job subtasks within the main process of an already running job; execute the job command through the process library. Status Query Process: Execute the query queue command to query the status of the specified job name; parse the job identifier and status code in the output; find the mapped unified process status through the static status mapping table. Special Handling: If the status is "Completing" and the process has been marked as terminated, it is mapped to the terminated status. Waiting for Startup Process: Check if the job appears in the queue at fixed time intervals using the polling query queue command, supporting timeout waiting. Termination Process: Execute the cancel job command to cancel by job name.

[0082] V. Task Encapsulation Layer (HSTask): The task encapsulation layer is the core encapsulation class for tasks. Internally, it holds a process abstract interface object, providing task-level state management and lifecycle control. It uses the composite pattern to hold the process abstract interface object, and its lifecycle is managed by the task manager. The task encapsulation layer can be subclassed, and virtual methods can be overridden to implement specific business logic. For example... Figure 5 The diagram shown is a representation of the HSTask virtual method extension points and lifecycle management. Figure 5 The code defines the expansion and callback logic for each key node during task execution. These six methods can be categorized into four types: (1) Task activation and execution preprocessing activeProc (Activation Preprocessing): Called when a task is officially activated, it is used to perform pre-activation preparation work or verification logic.

[0083] (2) Task management and maintenance operation clean (clean up intermediate files): This function is invoked when a cleanup task is executed. It is responsible for deleting temporary or intermediate files generated during the task's execution and releasing storage resources.

[0084] restore (restore initial state): is called when a restore task is executed, and is used to roll back or reset the state of the task to its original initial state.

[0085] update (update and mark as expired): This function is called when an update task is executed. It is responsible for updating the task data and marking the old version or related data as expired.

[0086] (3) Task exit and cleanup taskExitProc (exit post-processing): This function is called after the process associated with the task has completely exited. It is used to perform cleanup tasks after the process ends, such as logging, releasing resources, or notifying external systems.

[0087] (4) Task status determination isNormalExit (determine if the exit was normal): This is a conditional method that is called during state mapping (i.e., the conversion between task state and internal system state). It is used to dynamically determine whether the exit is a normal termination based on logic such as exit code and error message.

[0088] VI. Task Manager (HSTaskManager): The Task Manager is the core scheduler of the entire system, responsible for concurrent task scheduling, status monitoring, and lifecycle management. Figure 6 The diagram shown is a flowchart of the Task Manager process. The process is as follows: (1) Wait for the process to finish: The monitoring thread obtains the child process object, calls the wait-to-finish interface to block and wait for the process to finish executing.

[0089] (2) Get the exit status: After the process ends, get its exit status type, which is divided into normal exit, abnormal exit, terminated, startup failure, etc.

[0090] (3) Determine the task status: If the task fails to start, mark it as a failed start; if it exits normally or is terminated, determine whether it is a normal exit and mark it as completed or failed / terminated respectively; if it exits abnormally, mark it as failed execution.

[0091] (4) Unified Exit Processing: Regardless of the exit status, the following steps will be executed: Set the task end time and call the exit processing function to perform exit cleanup; Decrement the global runtime counter and notify the main scheduling thread through a condition variable; Execute the task status callback function; Monitor thread termination. like Figure 7 As shown, the method includes: S10 constructs a unified interface and main scheduling thread for receiving tasks, and encapsulates several heterogeneous execution environments into different implementations of the unified interface.

[0092] Specifically, the Process Abstraction Interface (PAI) is a pure virtual base class that defines a unified interface for process management. The PAI is the common parent class of the three concrete implementation classes: the local process implementation, the LSF cluster process implementation, and the Slurm cluster process implementation.

[0093] A main scheduling thread is constructed, which is responsible for the core logic of task scheduling. The main scheduling thread periodically or when a specific event is triggered, selects appropriate tasks from the task queue and assigns them to the corresponding target execution environment for execution.

[0094] like Figure 8 The diagram shows the unified task scheduling and process management process. First, the initial configuration of the core components is completed: (1) Initialize Task Manager: Initialize the task queue, which is used to temporarily store tasks waiting to be executed. The task queue is usually designed as a blocking queue or a bounded queue to coordinate the speed difference between producers (submitting tasks) and consumers (worker threads). Initialize a status table, which is used to centrally record and track the real-time status of each task (e.g., pending submission, queued, running, completed, failed, etc.). It typically uses a concurrency-safe Map structure, with the task ID as the key, to store task instances and their status information, providing data support for system monitoring and recovery. Initialize the concurrent slot pool, which means creating a fixed number of slots (e.g., a Semaphore with an initial value of N). Each task must "request" a slot before execution and "release" the slot after execution. When all slots are used up, new tasks can only wait in the queue.

[0095] (2) Initialize the thread pool: Initialize the core thread count, which is the number of threads that remain alive under normal circumstances; initialize the concurrency limit, which is the maximum number of threads that can be handled; configure the number of monitoring threads, selecting a certain number of worker threads from the thread pool as monitoring threads.

[0096] (3) Initialize the main scheduling thread: Set wake-up conditions. The main scheduling thread can enter a sleep state when there are no tasks. Wake-up conditions include new tasks being added to the queue, tasks being completed and releasing concurrent slots, and timed scheduling being triggered. When the wake-up condition is triggered, the main scheduling thread starts working. Define a scheduling strategy to determine the order in which tasks are retrieved from the queue. Common strategies include first-in, first-out (FIFO), priority scheduling (execute tasks according to their priority), and time wheel algorithm (suitable for handling a large number of delayed or timed tasks). The priority can be dynamically changed according to the actual situation.

[0097] (4) Configuration log: Configuring a logging framework (such as Log4j2), setting log levels, output formats, and rolling strategies, and recording task lifecycles, system performance metrics, and exception information are the foundation for monitoring and troubleshooting.

[0098] (5) Exception handling: Establish a global exception handling mechanism. In the worker thread or scheduling thread, set a try-catch block for task execution to catch and log exceptions.

[0099] (6) Resource recycling module: Configure a resource reclamation module to ensure that system resources are released in a timely manner and avoid memory leaks.

[0100] The initialization phase ensures that all scheduling and execution dependencies are ready, providing a stable operating environment for subsequent task execution.

[0101] S20: Receive a business request and create a task based on the business request. The business request includes the target execution environment corresponding to the task.

[0102] Specifically, business requests can be received in various forms, such as via HTTP requests, message queues, or command-line arguments. For example, a business request can be a JSON-formatted data packet containing the task name, the path to the execution script, the location of input data, and an explicitly specified target execution environment (e.g., "LSF cluster").

[0103] Upon receiving the business request, the system parses the information within it and creates an internal task object based on that information. The task object encapsulates all the necessary attributes of the task, including a unique task identifier, execution parameters, dependent resources, priority, timeout configuration, and the target execution environment specified in the business request.

[0104] S30: Add the task to the Task Manager. When a task meets the preset activation conditions, it will be added to the task queue in the Task Manager.

[0105] Specifically, there are preset activation conditions for tasks. The activation conditions can be simply set to activate immediately after the task is successfully created, or to activate when the task's input data is ready, or to activate when the resources required to execute the task are ready, or to activate when the main thread's scheduling window arrives. For example, after a task is created, if all its required input files exist in the specified path, the task is considered to meet the activation conditions.

[0106] After receiving a task, the Task Manager will initially register it and determine whether to add it to the task queue based on the activation conditions, while also establishing an initial record of the task status.

[0107] S40: When the preset wake-up conditions of the main scheduling thread are met, schedule tasks from the task queue.

[0108] Specifically, the wake-up conditions for the main scheduler thread can be preset. These conditions can be set to wake the main scheduler thread once at fixed time intervals (e.g., every 5 seconds), or to be triggered when a new task is added to the task queue, or when a task completes and releases a concurrent slot. For example, the main scheduler thread can be configured to check the task queue every 10 seconds. When the main scheduler thread is woken up, it will retrieve tasks from the task queue and process them.

[0109] S50 inputs tasks into the target execution environment corresponding to the task through a unified interface, and creates a task execution process in the target execution environment.

[0110] Specifically, such as Figure 4 As shown, the task is invoked to the task encapsulation layer. The task encapsulation layer holds a reference to the process object through a shared pointer. At runtime, it points to different implementation classes, i.e., the target execution environment, according to the configuration. Then, it is input into the target execution environment through a unified interface to create the task execution process.

[0111] S60 executes tasks through a task execution process and uses a thread pool for monitoring during task execution.

[0112] Specifically, once the task execution process is successfully created and starts running in the target execution environment, the task manager allocates an idle worker thread from the thread pool. This worker thread is bound to the task execution process and is responsible for continuously querying the process's status. The worker thread can periodically query the target execution environment for the process's running status, for example, by calling the LSF cluster's "bjobs" command to obtain the job status. In this way, the task's execution progress and health status can be monitored in real time.

[0113] This application constructs a unified interface and encapsulates heterogeneous execution environments into different implementations of this unified interface, thereby completely decoupling the details of the upper-layer business logic from the details of the lower-layer execution environment, significantly reducing code complexity and maintenance costs. At the same time, this application separates the task activation conditions and scheduling strategies from the business logic by cooperating with the task manager for task queuing and scheduling, the main scheduling thread for unified scheduling and distribution, and the thread pool for execution process monitoring, making the flow of tasks more flexible and controllable.

[0114] In one embodiment, to efficiently execute a task, the step of adding the task to the task manager when the task meets preset activation conditions can be specifically performed as follows: When a task meets the preset activation conditions, the preset activation conditions include the task dependencies being ready and the resources required for task execution being prepared. Specifically, there are preset activation conditions for tasks. The activation conditions can be simply set to activate immediately after the task is successfully created, or set to the task's input data being ready, or set to the resources required to execute the task being ready, or set to the arrival of the main thread's scheduling window. For example, after a task is created, if all its required input files exist in the specified path, the task is considered to meet the activation conditions. The system determines whether the execution parameters in a task are within a preset valid range and whether the task type is not in a preset invalid type library. Specifically, determining whether the execution parameters are within the preset valid range can be achieved by defining rules for the data type, value range, and format requirements of the parameters. For example, a task's concurrency parameter must be a positive integer and cannot exceed the maximum value set by the system; a file path parameter must conform to specific naming conventions and point to an accessible storage location. Determining whether the task type is in a preset invalid type library is to avoid executing unsupported, deprecated, or potentially risky task types. This can be achieved by maintaining a task type blacklist or whitelist mechanism. For example, certain high-privilege system-level operation task types can be added to the invalid type library, prohibiting ordinary business requests from creating such tasks. If the execution parameters are within the preset valid range and the task type is not in the preset invalid type library, the task will be added to the task manager. Specifically, the task will be officially accepted and added to the task manager only if the execution parameters are valid and the task type is valid.

[0115] By setting preset activation conditions that include task dependencies being ready and resources being prepared, and by additionally performing double checks on the legality of execution parameters and the validity of task types, strict admission filtering is conducted before tasks are enqueued, effectively preventing invalid or illegal tasks from entering the task queue. This ensures the stability of the task manager and improves the overall success rate of task execution.

[0116] In one embodiment, to efficiently execute tasks, when the preset wake-up conditions of the main scheduling thread are met, the step of scheduling tasks from the task queue can be specifically performed as follows: When the preset wake-up conditions of the main scheduler thread are met, these conditions include detecting a task entering the task queue. Specifically, the preset wake-up conditions for the main scheduler thread can be set to wake it up once at fixed time intervals (e.g., every 5 seconds), or triggered when a new task is added to the task queue, or triggered when a task completes and releases a concurrent slot. For example, the main scheduler thread is configured to check the task queue every 10 seconds. When the main scheduler thread is woken up, it will retrieve and process tasks from the task queue. First, a scheduling strategy is formulated based on the priority information of several tasks to be executed, the order in which the tasks arrive at the task queue, and the load of the task manager. Specifically, formulating a scheduling strategy requires comprehensive consideration of multiple factors to achieve optimal task scheduling. Each task can be assigned a priority information, indicating its importance or urgency; higher-priority tasks should generally be scheduled first. The order in which tasks arrive at the task queue, i.e., the time order in which tasks enter the task queue, is typically used to ensure fairness or handle time-sensitive tasks. The load of the task manager refers to the workload currently borne by the system or task manager, such as allocated resources, the number of running tasks, CPU utilization, and memory usage. When scheduling, load must be considered to avoid over-scheduling that could lead to system overload. The scheduling strategy can be based on a preset rule engine, which makes logical judgments and decisions based on the above information. For example, high-priority tasks are scheduled first; if the priorities are the same, they are scheduled in the order of arrival; if the system load is too high, low-priority tasks are suspended.

[0117] Then, tasks are scheduled from the task queue according to the scheduling policy. Specifically, once the scheduling policy is determined, the main scheduling thread selects one or more tasks from the task queue for scheduling based on that policy. Scheduling means removing the selected tasks from the task queue and preparing them for submission to the target execution environment.

[0118] By setting the main scheduling thread to be awakened upon detecting a task entering the queue, and comprehensively considering the priority information of the tasks to be executed, the order in which they arrive in the queue, and the current real-time load of the task manager to formulate a scheduling strategy, the system can prioritize urgent tasks while ensuring fairness, and dynamically adapt to changes in system load, thereby optimizing the average waiting time of tasks and the utilization of system resources. In one embodiment, to efficiently execute tasks, the step of formulating a scheduling strategy for several tasks to be executed based on their priority information, the order in which they arrive at the task queue, and the load status of the task manager can be specifically performed as follows: First, the system receives several historical task scheduling policies. These policies include historical task information, historical load information from the task manager, and scheduling strategies based on preset rules. The historical task information includes priority and arrival order. Specifically, this historical data can be extracted from system logs, which detail the execution status of each task, the system's scheduling decisions, and the system resource usage at the time. Alternatively, a dedicated monitoring module can collect and persistently store scheduling data in real time. These historical task scheduling policies provide valuable foundational data for subsequent model training, reflecting the system's scheduling behavior and effectiveness under different operating conditions. Next, a scheduling strategy model is constructed and trained using several historical task scheduling strategies. Specifically, a mathematical or computational model capable of learning and predicting task scheduling strategies is established. This model can be implemented based on various machine learning algorithms, such as supervised learning models like decision trees, support vector machines, multilayer perceptrons, or recurrent neural networks; alternatively, a reinforcement learning-based model can be constructed to learn and optimize scheduling strategies through continuous interaction with the execution environment. This model provides an intelligent decision-making framework, enabling it to learn and identify effective scheduling patterns from complex historical data. Historical data is used to adjust the internal parameters of the scheduling strategy model, enabling it to accurately map inputs (task information, load) to outputs (scheduling strategy). For supervised learning models, historical task and load information can be used as input features, and historical task scheduling strategies based on preset scheduling rules can be used as training labels. For reinforcement learning models, the states, actions taken, and rewards obtained during historical scheduling processes can be used as training data, trained using algorithms such as policy gradient or Q-learning. Through training, the model acquires the ability to learn from experience, thereby generating more optimized scheduling strategies.

[0119] Finally, the priority information of several tasks to be executed, the order in which these tasks arrived in the task queue, and the load status of the task manager are input into the trained scheduling policy model to obtain scheduling policies for these tasks. Specifically, this involves acquiring various information about the currently scheduled tasks and the system's running status in real time. This includes real-time collection of the priorities of the tasks to be executed, their timestamps of entering the task queue, and the current load status of the task manager, such as CPU utilization, memory usage, and I / O throughput. After appropriate feature engineering, this real-time data is converted into an input format acceptable to the model, providing it with all the contextual information needed for the current decision. Based on the knowledge learned from historical data and combined with the current input real-time task information and system load status, the scheduling policy model outputs a scheduling decision for the currently scheduled tasks.

[0120] By collecting historical task scheduling strategies to train a scheduling strategy model, and inputting the current task's priority, arrival order, and manager load into the trained model to automatically generate a scheduling strategy, this application achieves an adaptive intelligent optimization effect for the scheduling strategy. This enables the system to learn from historical execution experience and dynamically adjust its scheduling behavior, thereby continuously improving the scheduling accuracy and execution efficiency in complex and ever-changing scenarios.

[0121] In one embodiment, to efficiently execute a task, the step of executing the task through a task execution process can be specifically performed as follows: The process of creating and starting an execution process for a selected task includes first loading the executor or script corresponding to the task, then allocating CPU, memory, and I / O resources, creating a process context, and finally attempting to start the process.

[0122] First, it is determined whether the task execution process corresponding to the task has been successfully created. Specifically, after the task is scheduled and is ready to create the task execution process in the target execution environment, a check is first performed to confirm whether the task execution process corresponding to the task has been successfully created. This check can be performed by querying the process management interface provided by the operating system to obtain the creation status of the task execution process. For example, in Linux system, check the return value of the fork() or exec() system call, or use the ps command to query whether the process ID exists. Then, if the task execution process corresponding to the task is created successfully, the task is considered to have started successfully, and the task is executed through the task execution process. Specifically, if the judgment result shows that the task execution process corresponding to the task has been created successfully, the task is explicitly considered to have started successfully, and the task execution process is allowed to continue executing the task. This means that the task has officially entered the execution phase from the preparation phase; after the task execution process is created successfully, it can directly load and run the pre-compiled task code or script, and the task execution process begins to process the task data according to its internal logic. Next, if the task execution process corresponding to the task fails to be created, the task is determined to have failed to start. The error log and the reason for failure are recorded. Specifically, when the task execution process fails to be created, this abnormal situation needs to be identified in time, the task is marked as a startup failure, and the reason for failure is recorded in detail for subsequent troubleshooting, retry, or alarm. By capturing the error code or exception information returned by the operating system when the process creation fails, it is written to the system log file or database along with the task ID, target execution environment information, etc., and the task status is updated to "startup failure".

[0123] By clearly determining whether the task execution process was created successfully, and recording error logs and reasons for failure when creation fails, this application achieves the technical effect of clearly distinguishing the task startup status, ensuring that the system can promptly capture and differentiate between successful and failed startup results, while providing detailed error tracing evidence for operation and maintenance personnel, greatly facilitating system debugging and rapid repair of abnormal problems.

[0124] In one embodiment, to efficiently execute tasks, the step of monitoring using a thread pool during task execution can be specifically performed as follows: A worker thread is retrieved from the thread pool, and its process handle is bound to the unique identifier of the task. The worker thread is then used for monitoring during the task's execution.

[0125] Specifically, a thread pool is a mechanism for managing and reusing threads. It pre-creates a certain number of threads, and when a task needs to be executed, an idle thread is obtained from the thread pool to perform the task. After the task is completed, the thread is returned to the thread pool. A worker thread is a specific thread instance in the thread pool, and its main responsibility is to execute the tasks assigned to it.

[0126] To continuously monitor the task execution process, a dedicated worker thread is retrieved from a pre-built thread pool. This worker thread does not directly execute business logic but focuses solely on monitoring. To ensure the accuracy and specificity of the monitoring, the worker thread binds the process handle of the task it is responsible for monitoring to the unique identifier of that task. Once the binding is complete, the worker thread can use its associated process handle to continuously perform monitoring activities such as status queries and resource usage checks on the target task's execution process.

[0127] By retrieving a worker thread from the thread pool and binding the process handle of that worker thread with the unique identifier of the task for dedicated monitoring, this application achieves a precise one-to-one mapping between monitoring resources and task instances. This ensures that monitoring operations can accurately target the task, while avoiding frequent creation and destruction of monitoring threads, reducing system overhead, and improving the accuracy and reliability of monitoring data.

[0128] In one embodiment, to efficiently execute tasks, the step of monitoring using worker threads during task execution can be specifically performed as follows: First, the environmental state of the target execution environment is mapped to the process-level state according to a preset state mapping table. The state mapping table includes several environmental states and the corresponding process-level states. Specifically, system resource data, i.e. the environmental state of the target execution environment, can be obtained through the API provided by the operating system. The environmental state of the target execution environment refers to the current running status or resource usage of the heterogeneous execution environment in which the task actually runs, such as CPU utilization, memory usage, disk I / O, network bandwidth, process list, system load, etc. The environmental state can provide the underlying information of the task's running context. A predefined state mapping table is a predefined data structure or configuration rule used to transform the raw environment state obtained from the target execution environment into a more general or abstract process-level state. Predefined state mapping tables are used to standardize state representations in different heterogeneous environments, facilitating subsequent unified processing.

[0129] Process-level state is an abstract description of the running status of a task execution process in the target execution environment. It focuses more on the process itself than the raw environment state. Process-level state provides an intermediate layer, transforming low-level environment information into a state directly related to the task execution process. Process-level states include, for example, ready, starting, start failed, pending, suspended, running, normal exit, abnormal exit, terminated, and undefined. Figure 9 The table shown is the state mapping table for LSF and Slurm; Then, the process-level state is converted to the task-level state through the task manager. Specifically, this application designs a two-level state machine with 10 process-level states and 8 task-level states. The process-level states include ready, starting, start failed, pending, suspended, running, normal exit, abnormal exit, terminated, and undefined. Task-level statuses include ready, waiting, starting, start failed, ongoing, finished, terminated, and execute failed.

[0130] The task manager is responsible for further abstracting from process-level states to reflect the overall running status of tasks. It provides a high-level, business-related view of task states for easy understanding by upper-layer applications and users. Possible implementations include: the task manager internally maintains a state transition logic that converts the received process-level states and task configurations into task-level states. For example, a process-level state of pending is converted to a task-level state of waiting; process-level states of suspended and running are converted to a task-level state of ongoing; process-level states of normal exit are converted to a task-level state of finished; and process-level states of abnormal exit and undefined are converted to a task-level state of executed failed.

[0131] Next, worker threads are used to monitor the task-level status to determine the liveness of the task execution process. Specifically, worker threads continuously acquire and analyze the task-level status provided by the task manager to determine whether the task execution process is still alive and running normally, ensuring the continuous availability of the task execution process. Monitoring using worker threads during task execution can also involve the following steps: During task execution, the exit code and exception signals of the task execution process are captured. Specifically, the worker thread can obtain the exit code returned by the process and the signals sent by the operating system in case of exceptions. Its purpose is to provide detailed reasons for task termination, facilitating troubleshooting and error handling. Possible implementation methods include: in Linux / Unix systems, obtaining the exit status and signal information of the child process through system calls such as waitpid(); in Windows systems, obtaining the process's exit code through APIs such as GetExitCodeProcess(). Monitoring using worker threads during task execution can also involve the following steps: During task execution, monitor the running time of the task execution process and determine whether the running time exceeds the preset maximum running time; if the running time exceeds the preset maximum running time, mark the task execution as timed out.

[0132] Specifically, worker threads continuously track the elapsed time from task execution to the current moment, providing a basis for determining whether a task has timed out. Implementation methods include recording a start timestamp when the task starts, and the worker thread periodically obtaining the current timestamp and calculating the difference between it and the start timestamp. The currently monitored task execution time is compared with a pre-set maximum allowed execution time. When the task's execution time exceeds the preset maximum execution time, the system marks the task as having entered a timeout state.

[0133] By uniformly mapping the states of various heterogeneous environments to standardized process-level states, then converting them to task-level states, and using worker threads to monitor the states to obtain survival information, while capturing exit codes, exception signals, and monitoring runtime to determine timeouts, this application achieves the technical effect of shielding the differences in the underlying heterogeneous environment states and forming a unified task state view. It realizes comprehensive perception of various abnormal situations throughout the entire lifecycle of task execution, enhancing the robustness and control capabilities of the system.

[0134] In one embodiment, to execute tasks efficiently, after changing the process-level state to the task-level state via Task Manager in this step, the following steps can also be performed: Determine if the task's task-level state has changed; if the task's task-level state has changed, output the task's state information by calling a pre-registered callback function.

[0135] Specifically, it detects whether the internal state of a task transitions from a stable state to another state during execution, such as from a task-level state of waiting to starting. The transition of task state can be achieved by comparing the current task-level state with the previously recorded task-level state.

[0136] When a task triggers a state update, the system immediately detects this change. The task's state information is output by calling a pre-registered callback function, which is a function invoked when a specific event occurs. This callback function is used to automatically execute predefined logic when the task's state changes, passing the task's latest state information to other system components or external services.

[0137] By determining whether a task's task-level state has changed and using pre-registered callback functions to output the task's status information, this solution enables real-time perception and response to key events in the task lifecycle. This significantly improves the dynamism and automation level of the task management system, avoiding subsequent processing delays caused by lagging status information. When a task's state changes, relevant system modules or external services can immediately obtain the latest status, thereby promptly triggering dependent operations, updating the user interface, handling errors, or scheduling resources, greatly improving the system's response speed and processing efficiency to task execution.

[0138] In one embodiment, to execute tasks efficiently, after executing the task through the task execution process, the following steps may also be performed: Determine whether redundant data generated by the task needs to be cleaned up based on the task configuration and execution results. If it is necessary to clean up redundant data generated by the task, delete the redundant data, restore the target execution environment, and release the resources occupied by the task.

[0139] Specifically, task configuration refers to the various parameters and attributes set for a task when it is created or submitted, such as the task type, input / output paths, required computing resources, and whether post-processing cleanup is needed. Execution results are a crucial basis for determining whether redundant data needs to be cleaned up. For example, if a task fails, all data may need to be retained for debugging; if the task succeeds, some temporary files may need to be cleaned up. Redundant data refers to data generated during task execution but no longer needed after the task is completed, such as temporary files, intermediate calculation results, cached data, and copies of log files.

[0140] The system comprehensively analyzes the task's configuration information, such as whether the task is marked as "temporary" or "intermediate results do not need to be retained," and the actual execution result of the task, such as whether the task was successfully completed or terminated due to an error. For example, scenarios requiring cleanup / recovery include: tasks generating temporary files, caches, modifying environment variables, or changing database transaction states; scenarios not requiring cleanup include: no temporary data, no environment changes, and the task can be terminated directly.

[0141] Based on these assessments, if redundant data is identified, the system will proactively delete this no longer needed data, thereby freeing up storage space. Simultaneously, to ensure the purity and consistency of the target execution environment, the system will also take measures to restore that environment, such as resetting its state or clearing residual configurations. Furthermore, various computing resources used during task execution, such as memory, CPU, and file handles, will be promptly released and returned to the system for use by other tasks. For example, when cleanup / recovery is required, the cleanup process is executed: deleting temporary files and cached data; restoring environment variables and working directories; rolling back uncommitted transactions or intermediate states; and releasing external resources exclusively used by the task (such as file locks and database connections). After cleanup, the process ends. If cleanup / recovery is not required, the cleanup steps are skipped, and the process ends.

[0142] By proactively determining whether redundant data needs to be cleaned up based on task configuration and execution results, and deleting redundant data, restoring the target execution environment to its initial state, and releasing all resources occupied by the task when necessary, this application achieves automated environment cleaning and resource recycling after task execution. This effectively avoids the risks of data accumulation and resource exhaustion, and ensures the long-term stable operation of the system and the efficient recycling of computing resources.

[0143] Furthermore, this application proposes persistent task state and breakpoint resumption, as well as intelligent anomaly detection and self-healing mechanisms. During task execution, its state (such as intermediate results and variables) is periodically saved. If a task is interrupted due to a failure, execution can resume from the most recent breakpoint, rather than starting completely anew. The current solution can detect "startup failures" and can be further extended to intelligently analyze the causes of failures (such as node failures or resource contention) and automatically take corrective measures (such as restarting or migrating the task).

[0144] Specifically, task state persistence refers to saving critical data during task execution (such as intermediate calculation results, variable values, and execution progress markers) to non-volatile storage media to prevent data loss. This can be achieved in various ways, such as periodically serializing the task's memory state into files and storing them in a local file system or distributed file system (such as HDFS or Ceph). When a task is interrupted due to a failure, the system can resume execution from the most recently successfully saved state point without starting from scratch. When a task starts, it first checks if a persistent state is available; if so, it loads the state and begins execution from the corresponding execution point. The task's internal logic also needs to be designed with corresponding recovery points so that after loading the state, completed steps can be skipped, and the unfinished parts can be executed directly.

[0145] The system can automatically identify unexpected behaviors or errors that occur during task execution. This can be based on preset rules, such as marking CPU usage as abnormal when it detects persistently high CPU usage, memory overflow, abnormal network latency, or specific error codes. Alternatively, it can be based on machine learning models. By collecting historical task execution data, anomaly detection models (such as clustering, classification, or time series analysis models) can be trained, and then the execution data of the current task can be analyzed in real time to identify abnormal situations that deviate significantly from the normal pattern.

[0146] By automatically parsing task execution logs and system logs, extracting keywords and error codes, and matching them with predefined failure modes, the system can further diagnose the specific cause of the anomaly upon detection. Based on the analyzed failure cause, the system can automatically execute preset recovery operations. For example, for task interruptions caused by transient failures or temporary resource shortages, the system can attempt to restart the task on the same or a different node.

[0147] Based on the above method, this application also discloses a task execution system for heterogeneous execution environments. For example... Figure 10 The system includes the following modules: The unified interface construction module 1001 is used to construct a unified interface for receiving tasks and a main scheduling thread, and to encapsulate several heterogeneous execution environments into different implementations of the unified interface. The task creation module 1002 is used to receive business requests and create tasks according to the business requests. The business requests include the target execution environment corresponding to the task. The task activation module 1003 is used to add tasks to the task manager. When a task meets the preset activation conditions, it is added to the task queue of the task manager. The task scheduling module 1004 is used to schedule tasks from the task queue when the preset wake-up conditions of the main scheduling thread are met. The task execution preparation module 1005 is used to input tasks into the target execution environment corresponding to the task through a unified interface, and to create a task execution process in the target execution environment. The task execution module 1006 is used to execute tasks through the task execution process and to monitor the task execution process using a thread pool.

[0148] In one embodiment, the task activation module 1003 is specifically used to determine whether the execution parameters in the task are within a preset legal range and whether the task type is not in a preset invalid type library when the task meets preset activation conditions, the preset activation conditions including task dependencies being ready and the resources required for task execution being prepared; whether the execution parameters in the task are within a preset legal range and whether the task type is not in a preset invalid type library; and whether the execution parameters are within a preset legal range and whether the task type is not in a preset invalid type library, then the task is added to the task manager.

[0149] In one embodiment, the task scheduling module 1004 is specifically used to, when the preset wake-up conditions of the main scheduling thread are met, the preset wake-up conditions include detecting that a task has entered the task queue; formulating a scheduling strategy for a number of tasks to be executed based on the priority information of a number of tasks to be executed, the order in which the tasks to be executed arrive at the task queue and the load of the task manager; and scheduling tasks from the task queue according to the scheduling strategy.

[0150] In one embodiment, the task scheduling module 1004 is specifically used to receive several historical task scheduling policies, which include historical task information of several historical tasks, historical load information of the task manager, and historical task scheduling policies formulated according to preset scheduling rules. The historical task information includes historical task priority information and historical task arrival order information. The module constructs a scheduling policy model and trains the scheduling policy model using several historical task scheduling policies. The module inputs the priority information of several tasks to be executed, the order in which several tasks to be executed arrive at the task queue, and the load status of the task manager into the trained scheduling policy model to obtain the scheduling policies for several tasks to be executed.

[0151] In one embodiment, the task execution module 1006 is specifically used to determine whether the task execution process corresponding to the task has been successfully created; if the task execution process corresponding to the task has been successfully created, the task is determined to have started successfully and the task is executed through the task execution process; if the task execution process corresponding to the task has not been successfully created, the task is determined to have started unsuccessfully and the error log and the reason for the failure are recorded.

[0152] In one embodiment, the task execution module 1006 is specifically used to retrieve a worker thread from the thread pool, bind the process handle of the worker thread to the unique identifier of the task, and use the worker thread for monitoring during the execution of the task.

[0153] In one embodiment, the task execution module 1006 is specifically used to map the environmental state of the target execution environment to a process-level state according to a preset state mapping table, wherein the state mapping table includes several environmental states and several process-level states corresponding to the environmental states; convert the process-level state to the task-level state through the task manager; and use worker threads to monitor the task-level state to obtain the survival status of the task execution process. In one embodiment, the task execution module 1006 is specifically used to capture the exit code and exception signal of the task execution process during the execution of the task; In one embodiment, the task execution module 1006 is specifically used to monitor the running time of the task execution process during the execution of the task, and determine whether the running time exceeds a preset maximum running time; if the running time exceeds the preset maximum running time, then mark the task execution as timed out. In one embodiment, the task execution module 1006 is specifically used to determine whether the task-level state of the task has changed; if the task-level state of the task has changed, the task's state information is output by calling a pre-registered callback function.

[0154] In one embodiment, the task execution module 1006 is specifically used to determine whether it is necessary to clean up redundant data generated by the task based on the task configuration and execution result; if it is necessary to clean up redundant data generated by the task, the redundant data is deleted, the target execution environment is restored, and the resources occupied by the task are released.

[0155] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A task execution method for heterogeneous execution environments, characterized in that, The method is based on a task manager and a thread pool, and includes: A unified interface and main scheduling thread are constructed for receiving tasks, and several heterogeneous execution environments are encapsulated as different implementations of the unified interface; Receive a business request, and create a task based on the business request, wherein the business request includes the target execution environment corresponding to the task; The task is added to the task manager. When the task meets the preset activation conditions, the task is added to the task queue of the task manager. When the preset wake-up conditions of the main scheduling thread are met, the task is scheduled from the task queue; The task is input into the target execution environment corresponding to the task through the unified interface, and a task execution process is created in the target execution environment. The task is executed through the task execution process, and the thread pool is used for monitoring during the execution of the task.

2. The method according to claim 1, characterized in that, The step of adding the task to the task manager when the task meets the preset activation conditions includes: When the task meets the preset activation conditions, the preset activation conditions include the task dependencies being ready and the resources required for the task execution being prepared. Determine whether the execution parameters in the task are within a preset legal range and whether the task type is not in a preset invalid type library; If the execution parameters are within the preset valid range and the task type is not in the preset invalid type library, then the task will be added to the task manager.

3. The method according to claim 1, characterized in that, The task queue includes several tasks to be executed, and scheduling the tasks from the task queue when the preset wake-up condition of the main scheduling thread is met includes: When the preset wake-up conditions of the main scheduling thread are met, the preset wake-up conditions include detecting that the task has entered the task queue; Based on the priority information of several tasks to be executed, the order in which the tasks to be executed arrive at the task queue, and the load status of the task manager, a scheduling strategy for several tasks to be executed is formulated. The task is scheduled from the task queue according to the scheduling policy.

4. The method according to claim 3, characterized in that, The step of formulating a scheduling strategy for the tasks to be executed based on the priority information of the tasks to be executed, the order in which the tasks to be executed arrive at the task queue, and the load status of the task manager includes: The system receives several historical task scheduling policies, which include historical task information of several historical tasks, historical load information of the task manager, and historical task scheduling policies formulated according to preset scheduling rules. The historical task information includes historical task priority information and historical task arrival order information. Construct a scheduling strategy model and train the scheduling strategy model using several historical task scheduling strategies; The priority information of several tasks to be executed, the order in which the tasks to be executed arrive at the task queue, and the load status of the task manager are input into the trained scheduling strategy model to obtain the scheduling strategy for several tasks to be executed.

5. The method according to claim 1, characterized in that, Executing the task through the task execution process includes: Determine whether the task execution process corresponding to the task has been created successfully; If the task execution process corresponding to the task is created successfully, the task is determined to have started successfully, and the task is executed through the task execution process. If the task execution process corresponding to the task is not created successfully, the task is determined to have failed to start, and an error log and the reason for the failure are recorded.

6. The method according to claim 1, characterized in that, The monitoring using the thread pool during the execution of the task includes: A worker thread is retrieved from the thread pool, and the process handle of the worker thread is bound to the unique identifier of the task. The worker thread is used for monitoring during the execution of the task.

7. The method according to claim 6, characterized in that, The monitoring using the worker thread during the execution of the task includes: The environmental state of the target execution environment is mapped to a process-level state according to a preset state mapping table, wherein the state mapping table includes several environmental states and several process-level states corresponding to the environmental states. The process-level state is converted to a task-level state through the task manager; The worker thread is used to monitor the task-level status to obtain the liveness status of the task execution process; The monitoring using the worker thread during the execution of the task also includes: During the execution of the task, the exit code and exception signals of the task execution process are captured; The monitoring using the worker thread during the execution of the task also includes: During the execution of the task, monitor the running time of the task execution process and determine whether the running time exceeds the preset maximum running time; If the running time exceeds the preset maximum running time, the task execution timeout is marked.

8. The method according to claim 7, characterized in that, After the process-level state is converted to a task-level state via the task manager, the following steps are included: Determine whether the task-level status of the task has changed; If the task-level state of the task changes, the task's state information is output by calling a pre-registered callback function.

9. The method according to claim 1, characterized in that, After the task is executed through the task execution process, the method further includes: Determine whether redundant data generated by the task needs to be cleaned up based on the task configuration and execution results. If it is necessary to clean up redundant data generated by the task, delete the redundant data, restore the target execution environment, and release the resources occupied by the task.

10. A task execution system for heterogeneous execution environments, characterized in that, The system is based on a task manager and a thread pool, and includes: The unified interface construction module (1001) is used to construct a unified interface for receiving tasks and a main scheduling thread, and to encapsulate several heterogeneous execution environments into different implementations of the unified interface. The task creation module (1002) is used to receive a business request and create a task according to the business request, wherein the business request includes the target execution environment corresponding to the task. The task activation module (1003) is used to add the task to the task manager. When the task meets the preset activation conditions, the task is added to the task queue of the task manager. The task scheduling module (1004) is used to schedule the task from the task queue when the preset wake-up condition of the main scheduling thread is met; The task execution preparation module (1005) is used to input the task into the target execution environment corresponding to the task through the unified interface, and to create a task execution process in the target execution environment. The task execution module (1006) is used to execute the task through the task execution process and to monitor the task execution process using the thread pool.

Images