Distributed task scheduling method and device based on task state switching

Abstract

This invention provides a distributed task scheduling method and apparatus based on task state switching, relating to the field of data processing. The method includes: distributively scheduling multiple tasks according to system service instructions and displaying the task states of the multiple tasks in a graphical user interface; responding to a state switching operation for a target task among the multiple tasks, mapping the target TCB and target key state fields corresponding to the target task to a writable memory region within the process in shared memory according to the state switching operation; modifying the state parameters in the writable memory region within the process online using atomic operations corresponding to memory barriers based on the target TCB and target key state fields; responding to the memory modification event corresponding to the completion of the state parameter modification, continuously polling the executor corresponding to the target task and sensing the state change through notification of the memory modification event, and adjusting the executor's own behavior according to the sensed state change.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a distributed task scheduling method and apparatus based on task state switching. Background Technology

[0002] Currently, existing task scheduling systems primarily employ a centralized scheduling approach. This involves a single scheduling center issuing task commands, with executors passively receiving and executing them, relying on a database to store task configuration information. For example, the Linux-based crontab tool implements simple scheduled tasks, triggering script execution through preset time expressions. However, this approach frequently requires restarting the service or the entire task scheduling system, resulting in low task scheduling efficiency. Summary of the Invention

[0003] The purpose of this invention is to provide a distributed task scheduling method and apparatus based on task state switching, so as to solve the technical problem of low task scheduling efficiency in existing solutions.

[0004] Firstly, this application provides a distributed task scheduling method based on task state switching, the method comprising: Obtain system service instructions, perform distributed scheduling of multiple tasks according to the system service instructions, and display the task status of multiple tasks in the graphical user interface; each task corresponds to a ThreadControl Block (TCB) and key status fields; In response to a state switching operation for a target task among the multiple tasks, the target TCB and target key state fields corresponding to the target task are mapped to a process-writable memory area in shared memory according to the state switching operation. Based on the target TCB and the target key state field, the state parameters in the writable memory area within the process are modified online through atomic operations corresponding to memory barriers. In response to the memory modification event corresponding to the completion of the modification of the status parameters, the executor corresponding to the target task continuously polls and detects the status change through the notification of the memory modification event. Based on the detected status change, the executor adjusts its own behavior so that the online modified status parameters take effect immediately and avoids restarting the service corresponding to the system service instruction and the entire task scheduling system.

[0005] In one possible implementation, the method further includes: Each executor node maintains the parameters of the memory mapping area corresponding to the task state in the memory region, and writes the task snapshot in the memory mapping area into distributed storage. The parameters of the memory-mapped area include at least one of the following: current state, concurrency, and progress offset; the task snapshot includes at least one of the following: task identity ID, state, parameters, last processing position, and rollback point.

[0006] In one possible implementation, after writing the task snapshot in the memory-mapped area to distributed storage, the method further includes: In response to the actuator detecting a failure in the target node, a task transfer instruction is generated; wherein the failure includes system crash and / or network isolation. According to the task transfer instruction, the task scheduler controls a new healthy node other than the target node to take over the task, and the task scheduler reads the most recent valid task snapshot from the distributed storage. The new healthy node is used to load the valid task snapshot into the local memory mapping area, and the executor executes the task based on the task status, task parameters and task progress saved corresponding to the valid task snapshot.

[0007] In one possible implementation, after modifying the state parameters in the memory region online by atomically operating on the corresponding memory barrier based on the target TCB and the target key state field within the writable memory region of the process, the method further includes: In response to the executor detecting a task logic anomaly in the current task, it is determined that a memory mapping write error occurred during the online modification of the state parameters in the memory region; Restore the previously described task snapshot from the local storage to the memory-mapped area, and attempt to continue executing the current task after restoration, so as to achieve a local rollback attempt through the previously described state snapshot stored in the memory-mapped area; In response to the failure of the local rollback attempt, the current node is marked as untrusted for the current task, and the executor is triggered to transfer the current task to another healthy node other than the current node, and retry by loading the previous task snapshot on the other healthy node.

[0008] In one possible implementation, triggering the executor to transfer the current task to a healthy node other than the current node includes: The following formula is used to select a target other healthy node from among the healthy nodes other than the current node, so that the final selected target other healthy node is as close as possible and as idle as possible per unit failure cost: ; in, This represents the task execution failure rate of node n within a preset time period before the current moment; This represents the path distance metric to node n; This represents the current load of node n; This indicates the selection method for other healthy nodes of the target; The executor is triggered to transfer the current task to a healthy node other than the target.

[0009] In one possible implementation, the task corresponds to a parent task and a child task. After all the parent tasks are successful, the child task is triggered through an AND convergence relationship. A directed acyclic graph (DAG) is constructed by resolving the parent-child task dependency relationship between the parent tasks and the child tasks, and a circular dependency check is performed based on the DAG. The distributed scheduling of multiple tasks according to the system service instructions includes: The system service instructions are used to perform distributed scheduling of multiple tasks, and after the executor completes the current task, the state of the task node is updated through the state machine corresponding to the DAG in the task scheduler. When the status of the parent task is updated to success, query all target subtasks in the DAG that have the current task as a prerequisite and all parent tasks corresponding to the target subtasks. In response to the success of all parent tasks corresponding to the target subtask, the target subtask is re-triggered using a sharding strategy. In response to the existence of failures in all parent tasks corresponding to the target subtask and the absence of failed branch processing, the downstream of the target subtask is marked as unexecutable.

[0010] In one possible implementation, the sharding strategy includes using the task scheduler to distribute the target subtask to all executor-corresponding nodes via sharding broadcast based on tags in the task metadata, so as to trigger all nodes to execute the target subtask; The step of rescheduling and triggering the target subtask through a sharding strategy includes: In response to the task scheduler detecting the addition or removal of nodes in the executor cluster, the running target subtask is not interrupted, and a sharding rebalance instruction is generated; In response to the next scheduling of the target subtask, the task scheduler sends the fragment rebalance instruction; In response to each node completing the current sharding, the number of new shards is dynamically adjusted according to the sharding rebalance instruction, and the sharding strategy is redistributed according to the new number of shards to complete the dynamic sharding task.

[0011] Secondly, this application provides a distributed task scheduling device based on task state switching, comprising: The scheduling module is used to obtain system service instructions, perform distributed scheduling of multiple tasks according to the system service instructions, and display the task status of multiple tasks in the graphical user interface; each task corresponds to a status control block (TCB) and key status fields. A mapping module is used to respond to a state switching operation for a target task among the multiple tasks, and to map the target TCB and target key state fields corresponding to the target task to a process-writable memory area in shared memory according to the state switching operation. The modification module is used to modify the state parameters in the writable memory area within the process online by atomically operating on the corresponding memory barriers based on the target TCB and the target key state field. The adjustment module is used to respond to the memory modification event corresponding to the completion of the modification of the status parameters. It continuously polls the executor corresponding to the target task and senses the status change through the notification of the memory modification event. Based on the sensed status change, it adjusts the behavior of the executor so that the online modified status parameters take effect immediately and avoids restarting the service corresponding to the system service instruction and the entire task scheduling system.

[0012] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described in the first aspect above.

[0013] Fourthly, this application also provides a computer-readable storage medium storing computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method described in the first aspect above.

[0014] This application brings the following beneficial effects: This application provides a distributed task scheduling method and apparatus based on task state switching, capable of acquiring system service instructions, performing distributed scheduling of multiple tasks according to the system service instructions, and displaying the task states of the multiple tasks in a graphical user interface; each task corresponds to a state control block (TCB) and a key state field; in response to a state switching operation for a target task among the multiple tasks, the target TCB and target key state field corresponding to the target task are mapped to a writable memory region within the process in shared memory according to the state switching operation; based on the target TCB and target key state field, the state parameters in the writable memory region within the process are modified online through atomic operations corresponding to memory barriers; in response to a memory modification event corresponding to the completion of the state parameter modification, the executor corresponding to the target task continuously polls and detects state changes through notifications of the memory modification event, and adjusts the system according to the detected state changes. The executor's own behavior ensures that the modified status parameters take effect immediately and avoids restarting the service corresponding to the system service instruction and the entire task scheduling system. In this solution, a shared memory mapping mechanism is constructed to directly expose the Task State Control Block (TCB) and key status fields to the writable area inside the process, thereby avoiding the data copying and high latency overhead caused by traditional inter-process communication or file input / output. By using atomic operations in conjunction with memory barriers, data consistency and visibility are ensured in a high-concurrency environment while eliminating the thread blocking and context switching costs caused by heavyweight lock mechanisms. Combined with the executor's polling and event notification awareness mechanism, real-time capture and behavior adjustment of status changes are achieved, completely eliminating the traditional process of restarting services to modify configurations. Thus, with zero downtime and low resource consumption, the effective latency of the scheduling strategy is significantly shortened, the real-time response capability and overall throughput efficiency of the distributed task scheduling system are significantly improved, and the technical problem of low task scheduling efficiency in existing solutions is solved.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating the distributed task scheduling method based on task state switching provided in this application embodiment; Figure 2 Another flowchart illustrating the distributed task scheduling method based on task state switching provided in this application embodiment; Figure 3 Another flowchart illustrating the distributed task scheduling method based on task state switching provided in this application embodiment; Figure 4 Another flowchart illustrating the distributed task scheduling method based on task state switching provided in this application embodiment; Figure 5 An example of a task-dependent DAG graph flow in the distributed task scheduling method based on task state switching provided in the embodiments of this application; Figure 6 A schematic diagram of the structure of a distributed task scheduling device based on task state switching provided in an embodiment of this application; Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this application, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0020] Currently, for configurable task management, existing technical solutions require task creation and modification through XML configuration files or code annotations. For example, in the Spring framework, scheduled tasks are defined using the `@Scheduled` annotation, and modifications require recompilation and deployment to take effect. For basic monitoring solutions, existing technologies record task execution status through log files, requiring manual login to the server to view; some systems provide simple execution result statistics, but lack real-time monitoring and visualization reports. Existing technical solutions have the following significant drawbacks in enterprise-level scenarios: High operational complexity: Task configuration depends on code or configuration files, making management impossible for non-technical personnel. Task creation, modification, and deletion require a development-testing-deployment process, which can take hours or even days. Insufficient high availability: Centralized scheduling solutions have single-point-of-failure risks, and in distributed solutions, executor failure detection latency typically exceeds 30 seconds, with a lack of automatic failover mechanisms, easily leading to task interruptions. Poor dynamic adaptability: After an executor node goes offline or online, the node list must be manually updated and the scheduling service restarted before tasks can be reassigned. Changes to task status (start / stop) do not take effect immediately, with response latency exceeding minutes. Rigid strategy system: Triggering methods only support Cron expressions or fixed intervals, failing to meet complex scenarios such as API triggering and parent-child task linkage. The blocking handling mechanism is simplistic (e.g., only supporting serial execution), easily leading to task backlog during high-concurrency scheduling. Weak dependency management: Only simple linear dependencies are supported (task A executes task B after task A completes). The system suffers from several drawbacks: First, it lacks the ability to configure complex relationships such as parallel dependencies and conditional branches, requiring hard coding in business logic, resulting in high maintenance costs. Second, monitoring and maintenance are inefficient: task execution logs are scattered across various executor nodes, necessitating node-by-node retrieval for troubleshooting. Third, it lacks real-time progress monitoring and visual reports, hindering the rapid identification of performance bottlenecks. Fourth, it has limited technical compatibility: most systems only support a single programming language (such as Java), with insufficient support for scripting tasks like Python and Shell. Fifth, it has poor adaptability to containerized deployments, requiring manual configuration of environment dependencies.

[0021] Based on this, the present application provides a distributed task scheduling method and apparatus based on task state switching, which can solve the technical problems of low task scheduling efficiency in existing solutions.

[0022] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart illustrating a distributed task scheduling method based on task state switching, provided as an embodiment of this application. Figure 1 As shown, the method includes: Step S110: Obtain system service instructions, perform distributed scheduling of multiple tasks according to the system service instructions, and display the task status of multiple tasks in the graphical user interface.

[0024] Each task corresponds to a status control block (TCB) and key status fields.

[0025] In distributed scheduling schemes, practical applications typically employ a multi-node deployment of the scheduling center and executors, allocating tasks using consistent hashing or round-robin algorithms. Some open-source frameworks, such as ZooKeeper, utilize ZooKeeper to elect and coordinate scheduling nodes, ensuring high availability of the scheduling center.

[0026] For scheduling, a centralized design is adopted, supporting multi-node cluster deployment. A database locking mechanism ensures consistency in distributed scheduling and prevents task duplication. Scheduling consistency is ensured through a scheduling center cluster combined with DB locks, and 99.99% availability is achieved through executor failover and elastic scaling, guaranteeing high availability.

[0027] For data security, for example, communication between the scheduling center and the executor uses AES-128 encryption to prevent information leakage. For access control, user management is implemented based on the RBAC model, distinguishing between administrators (full permissions) and ordinary users (permissions assigned at the executor level). Furthermore, a RESTful OpenAPI is provided, supporting integration with multiple languages ​​such as Java and Python; an built-in httpJobHandler enables cross-service task calls. For containerization and AI integration, an official Docker image is provided, supporting Kubernetes deployment; a built-in AI executor integrates with Spring AI, Ollam, and others, supporting AI tasks such as prompt scheduling and model inference.

[0028] In this embodiment, a web interface is provided to support task CRUD operations, i.e., visual operations. Task parameters (name, trigger rules, execution scripts, etc.) are configured via forms, allowing for quick setup within one minute. Furthermore, task dependency configuration based on DAG replaces the traditional hard-coding method, reducing the maintenance cost of complex processes and enabling visual dependency orchestration. Moreover, task progress is visualized, supporting online viewing of rolling logs (refreshed every 500ms by default), achieving real-time monitoring. Email alerts are triggered when a task fails (supporting multi-address group sending), and extended interfaces for SMS, DingTalk, etc., are reserved; alarm thresholds can be configured (e.g., triggering after 3 consecutive failures) to achieve intelligent alarms. Then, real-time statistical data such as task quantity, scheduling frequency, and success rate are generated, providing visual charts of scheduling date distribution and executor load.

[0029] For example, the above tasks correspond to parent tasks and child tasks. After all parent tasks succeed, the child tasks are triggered through an AND convergence relationship. A directed acyclic graph (DAG) is constructed by resolving the parent-child task dependencies and performing circular dependency checks based on the DAG. The above distributed scheduling of multiple tasks according to system service instructions can specifically include the following steps: Multiple tasks are distributed and scheduled according to system service instructions. After the executor completes the current task, the state of the task node is updated through the state machine corresponding to the DAG in the task scheduler. When the state of the parent task is updated to be successful, all target subtasks in the DAG that have the current task as a prerequisite and all parent tasks corresponding to the target subtasks are queried. If all parent tasks corresponding to the target subtask have succeeded, the target subtask is rescheduled and triggered using a sharding strategy; if any parent task corresponding to the target subtask fails and there is no failure branch to handle it, the downstream of the target subtask is marked as unexecutable.

[0030] For executor high availability (HA), the executor cluster deployment supports automatic failover (switching to a healthy node within 10 seconds when a node fails); tasks are automatically reassigned after nodes go online or offline, achieving elastic scaling. For the hybrid registration mechanism, executors automatically register with the scheduling center every 30 seconds, while also supporting manual entry of static addresses to adapt to complex network environments.

[0031] In this embodiment, a multi-dimensional strategy engine module is used to trigger strategies, integrating seven triggering methods: Cron expressions, fixed intervals / delays, API calls, manual triggering, and parent-child task linkage. For exception handling, scheduling expiration offers two processing modes: "ignore" and "immediate compensation," addressing scheduling omissions caused by clock skew. Blocking handling supports three strategies: single-machine serial execution, discarding subsequent schedules, and overwriting previous schedules, which can be dynamically configured according to task type. Timeout and retry: a custom timeout threshold (default 300 seconds) automatically interrupts the task upon timeout; up to five levels of retries are supported, and sharded tasks support retries at the shard granularity.

[0032] Task dependency and orchestration module dependency configuration: Supports parent-child task dependencies (child tasks are automatically triggered after the parent task succeeds, and multiple child tasks are separated by commas), and implements complex dependencies such as parallel convergence and conditional branching through a directed acyclic graph (DAG).

[0033] As an example, a sharding strategy involves using a task scheduler to broadcast a target subtask to all executor nodes via sharding, based on tags in the task metadata, to trigger all nodes to execute the target subtask. The re-scheduling and triggering of the target subtask using the sharding strategy can specifically include the following steps: In response to the task scheduler detecting the addition or removal of nodes in the executor cluster, the running target subtask is not interrupted, and a sharding rebalance instruction is generated; in response to the next scheduling of the target subtask, the task scheduler sends a sharding rebalance instruction; in response to each node completing the current sharding, the new number of shards is dynamically adjusted according to the sharding rebalance instruction, and the sharding strategy is reallocated according to the new number of shards to complete the dynamic sharding task.

[0034] With the sharding mechanism, when the executor cluster is deployed, the "sharding broadcast" strategy can trigger all nodes to execute tasks; dynamic sharding technology automatically adjusts the number of shards as the cluster size expands, improving the efficiency of processing large amounts of data.

[0035] Step S120: In response to a state switching operation for a target task among multiple tasks, the target TCB and target key state fields corresponding to the target task are mapped to a writable memory area within the process in shared memory according to the state switching operation.

[0036] As one possible implementation, dynamic design is used to achieve immediate effect of task status modifications and dynamic awareness of executor nodes through memory mapping and real-time registration mechanisms, with a response latency of less than 1 second.

[0037] Step S130: Based on the target TCB and the target key state field, modify the state parameters in the writable memory area within the process online by using atomic operations to correspond to the memory barrier.

[0038] In one possible implementation, a dynamic development environment with a built-in GLUE Web IDE supports online writing of scripts such as Shell, Python, and NodeJS, with real-time compilation and effectiveness, and supports 30 version rollbacks. As an optional implementation, the status parameters include at least one of the following: startup parameters, stop parameters, termination parameters, concurrency level, timeout threshold, and failure retry interval.

[0039] Step S140: In response to the memory modification event corresponding to the completion of the modification of the status parameters, the executor corresponding to the target task continuously polls and detects the status change through the notification of the memory modification event. Based on the detected status change, the executor adjusts its own behavior so that the online modified status parameters take effect immediately and avoids restarting the service corresponding to the system service instruction and the entire task scheduling system.

[0040] For real-time status management, memory mapping technology enables instant switching of task status (start / stop / terminate), and parameters take effect immediately after online modification without requiring a service restart.

[0041] In this embodiment, a shared memory mapping mechanism is constructed to directly expose the Task State Control Block (TCB) and key state fields to the writable area within the process, thereby avoiding the data copying and high latency overhead caused by traditional inter-process communication or file input / output. Atomic operations combined with memory barriers ensure data consistency and visibility in high-concurrency environments while eliminating thread blocking and context switching costs caused by heavyweight lock mechanisms. Combined with the executor's polling and event notification awareness mechanism, real-time capture and behavior adjustment of state changes are achieved, completely eliminating the traditional process of restarting services after modifying configurations. This significantly shortens the effective latency of scheduling strategies with zero downtime and low resource consumption, significantly improving the real-time response capability and overall throughput efficiency of the distributed task scheduling system, and solving the technical problem of low task scheduling efficiency in existing solutions.

[0042] Through a two-layer distributed architecture of scheduling center and executor, an enterprise-level distributed task scheduling method based on the information technology innovation environment office platform is realized. It integrates cross-language support, containerized deployment and AI task integration to build a complete enterprise-level scheduling ecosystem.

[0043] In this embodiment, the operational threshold is lowered, enabling visualized management of the entire task lifecycle and supporting rapid task configuration for non-technical personnel. Furthermore, system high availability is improved through cluster deployment and automatic failover, ensuring no single point of failure for the scheduling center and executors. Moreover, dynamic adaptability is enhanced, supporting dynamic scaling of executor nodes and real-time modification of task status without service restarts. A comprehensive policy system is provided, offering diverse triggering methods and exception handling mechanisms to meet the needs of complex business scenarios. Dependency management and monitoring capabilities are strengthened, supporting complex task dependency orchestration and real-time log and progress monitoring. Technical compatibility is improved, supporting multi-language task development, containerized deployment, and AI task integration, adapting to enterprise multi-technology stack environments.

[0044] In some embodiments, the method may further include the following steps: Each executor node maintains the parameters of the memory mapping area corresponding to the task state in the memory region, and writes the task snapshot in the memory mapping area to the distributed storage; wherein, the parameters of the memory mapping area include at least one of the current state, concurrency, and progress offset; the task snapshot includes at least one of the task ID, state, parameters, last processing position, and rollback point.

[0045] Through memory mapping, the memory region containing the task state is directly mapped to the virtual address space. When a snapshot needs to be created, the system can directly write the data in this memory region to distributed storage without additional data copying between the user space buffer and the kernel space buffer. This greatly reduces CPU overhead and data transfer latency, thus achieving zero-copy writing.

[0046] The process of writing data from the memory-mapped region to remote distributed storage can be performed asynchronously. This means that after a snapshot is triggered, the computing task can continue processing subsequent data without waiting for the data to be completely written to the storage. This avoids processing pauses caused by snapshot creation, ensures low latency and high throughput for the task, and achieves asynchronous non-blocking.

[0047] In some embodiments, after writing the task snapshot in the memory-mapped area to distributed storage, the method may further include the following steps: In response to the executor detecting a failure in the target node, a task transfer instruction is generated; the failure includes downtime and / or network isolation; according to the task transfer instruction, the task scheduler controls a new healthy node other than the target node to take over the task, and the task scheduler reads the most recent valid task snapshot from the distributed storage; The new health node loads the valid task snapshot into the local memory mapping area, and the executor executes the task based on the saved task status, task parameters and task progress corresponding to the valid task snapshot.

[0048] By detecting the trigger command through the executor and then assigning a new healthy node to the scheduler, manual intervention or complex master election processes are bypassed, enabling tasks to switch to healthy nodes within hundreds of milliseconds to a few seconds, achieving fault transfer within seconds.

[0049] Snapshot-based breakpoint continuation: Read the "most recent valid task snapshot" from distributed storage and load it into the memory mapping area of ​​the new node. This means that the task does not start from scratch, but continues execution from the most recently saved progress point, avoiding a lot of redundant calculations.

[0050] Decoupling of state and execution: Task state, parameters, and progress do not depend on local storage on any single node, but are persisted in distributed storage. Therefore, any healthy node can "plug and play" take over the task without data migration or log recovery.

[0051] The above solution enables tasks in a distributed system to be seamlessly resumed from the nearest breakpoint within seconds when faced with any node failure, thus achieving the most critical fault self-healing and continuous service capabilities in high availability scenarios.

[0052] In some embodiments, after modifying the state parameters in the memory region online by atomically operating on the corresponding memory barrier in the writable memory region within the process based on the target TCB and the target key state field, the method may further include the following steps: In response to the executor detecting a task logic anomaly in the current task, it determines that an error value was written to the memory map during the online modification of the state parameters in the memory region; it restores the memory map region from the previously retained snapshot of the previous task, and attempts to continue executing the current task after restoration, so as to achieve a local rollback attempt through the previously retained snapshot of the state in the memory map. In response to the failure of the local rollback attempt, the current node is marked as untrusted for the current task, and the executor is triggered to transfer the current task to another healthy node other than the current node, and retry by loading the previous task snapshot on the other healthy node.

[0053] In this embodiment, when a task fails to execute due to a logical exception or memory write error, a two-level fault tolerance is implemented, prioritizing rapid self-healing within a node and providing backup for fault transfer between nodes, thereby minimizing task interruption and cross-node migration overhead.

[0054] When a state parameter write error is detected, the current node is not immediately abandoned. Instead, the system restores the previous task snapshot from the local storage to the memory-mapped area and attempts to continue execution. This avoids immediately triggering cross-node task transfer, saving expensive operations such as distributed storage reads, new node loading, and task reallocation. For transient, recoverable logical errors (such as boundary condition writes out of bounds or temporary state inconsistencies), local self-healing can be achieved in milliseconds to seconds.

[0055] Furthermore, the current node is marked as "untrusted" for the task only after a local rollback attempt also fails. The task is then transferred to another healthy node, which loads the previous snapshot and retryes. This avoids wasting resources by repeatedly retries on a node with an unrecoverable problem, and also prevents that node from continuing to handle the task and causing repeated failures. Isolating faulty nodes through the "untrusted" marking improves the overall determinism of the system.

[0056] The above scheme enables the system to prioritize self-healing by rolling back local snapshots. Only when local recovery fails will the task be transferred and the current node isolated. This minimizes the frequency and cost of cross-node task migration while ensuring the final task execution is successful, thus balancing recovery efficiency and system stability.

[0057] In some embodiments, triggering the executor to transfer the current task to a healthy node other than the current node may specifically include the following steps: The following formula is used to select a target healthy node from other healthy nodes besides the current node, so that the final selected target healthy node is as close as possible and as idle as possible per unit failure cost: ; in, This represents the task execution failure rate of node n within a preset time period before the current moment; This represents the path distance metric to node n; This represents the current load of node n; This indicates the selection method for other healthy nodes of the target; The trigger executor will transfer the current task to a healthy node other than the target.

[0058] In this embodiment, usability is improved: Web-based visual operation reduces task deployment time from hours to minutes, allowing non-technical personnel to complete configuration independently, reducing labor costs by 50%. Availability is enhanced: Cluster deployment and automatic failover reduce annual system downtime to less than 52 minutes, meeting financial-grade high availability requirements. Efficiency is optimized: Dynamic scaling capabilities allow task processing throughput to increase linearly with cluster size, improving data processing efficiency by 300% during peak sales periods. Operational efficiency is improved: Real-time monitoring and intelligent alarms reduce troubleshooting time by 60% and reduce workload for operations personnel by 40%. Compatibility is expanded: Multi-language support and containerized deployment adapt to enterprise multi-technology stack environments, shortening system integration cycles from weeks to days. Business boundaries are expanded: AI task integration supports rapid development of intelligent scheduling scenarios (such as LLM-based log analysis and predictive scheduling), expanding enterprise application scenarios.

[0059] Figure 6 A schematic diagram of a distributed task scheduling device based on task state switching is provided. Figure 6 As shown, the distributed task scheduling device 600 based on task state switching includes: The scheduling module 601 is used to obtain system service instructions, perform distributed scheduling of multiple tasks according to the system service instructions, and display the task status of multiple tasks in a graphical user interface; each task corresponds to a status control block (TCB) and a key status field. The mapping module 602 is used to respond to a state switching operation for a target task among the multiple tasks, and to map the target TCB and target key state fields corresponding to the target task to a process-writable memory area in shared memory according to the state switching operation. Modification module 603 is used to modify the state parameters in the writable memory area within the process online by means of atomic operations corresponding to memory barriers, based on the target TCB and the target key state field; The adjustment module 604 is used to respond to the memory modification event corresponding to the completion of the modification of the status parameters, continuously poll the executor corresponding to the target task and detect the status change through the notification of the memory modification event, and adjust the behavior of the executor according to the detected status change, so that the online modified status parameters take effect immediately and avoid restarting the service corresponding to the system service instruction and the entire task scheduling system.

[0060] The distributed task scheduling device based on task state switching provided in this application has the same technical features as the distributed task scheduling method based on task state switching provided in the above embodiments, so it can also solve the same technical problems and achieve the same technical effects.

[0061] An electronic device provided in this application embodiment, such as Figure 7 As shown, the electronic device 700 includes a processor 702 and a memory 701. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the method provided in the above embodiments.

[0062] See Figure 7 The electronic device also includes a bus 703 and a communication interface 704. The processor 702, the communication interface 704 and the memory 701 are connected via the bus 703. The processor 702 is used to execute executable modules, such as computer programs, stored in the memory 701.

[0063] The memory 701 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 704 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0064] Bus 703 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0065] The memory 701 is used to store programs. After receiving an execution instruction, the processor 702 executes the program. The method executed by the apparatus defined by the process disclosed in any of the preceding embodiments of this application can be applied to the processor 702 or implemented by the processor 702.

[0066] The processor 702 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 702 or by instructions in software form. The processor 702 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 701, and processor 702 reads the information from memory 701 and, in conjunction with its hardware, completes the steps of the above method.

[0067] Corresponding to the above-described distributed task scheduling method based on task state switching, this application embodiment also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are invoked and executed by a processor, the computer-executable instructions cause the processor to perform the steps of the above-described distributed task scheduling method based on task state switching.

[0068] The distributed task scheduling device based on task state switching provided in this application embodiment can be specific hardware on a device or software or firmware installed on the device. The device provided in this application embodiment has the same implementation principle and technical effects as the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0069] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0070] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0071] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0073] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the distributed task scheduling method based on task state switching described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0075] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A distributed task scheduling method based on task state switching, characterized in that, The method comprises: acquiring a system service instruction, distributing a plurality of tasks according to the system service instruction, and displaying the task state of the plurality of tasks in a graphical user interface; each of the tasks corresponds to a state control block (TCB) and a key state field; in response to a state switching operation for a target task in the plurality of tasks, mapping the target TCB and the target key state field corresponding to the target task to a memory region inside a process that is writable in shared memory according to the state switching operation; based on the target TCB and the target key state field, modifying the state parameters in the memory region inside the process that is writable in the memory region through atomic operation corresponding to the memory barrier in a manner of online modification; in response to a corresponding memory modification event completing the modification of the state parameters, continuously polling through an executor corresponding to the target task and perceiving a state change through notification of the memory modification event, adjusting the behavior of the executor according to the perceived state change, so that the online modified state parameters take effect immediately and the service corresponding to the system service instruction and the entire task scheduling system are restarted.

2. The method of claim 1, wherein, The method further comprises: maintaining the parameters of the memory mapping area corresponding to the task state in the memory region by each of the corresponding nodes of the executors, and writing the task snapshot in the memory mapping area into distributed storage; wherein the parameters of the memory mapping area include at least one of the current state, concurrency degree, and progress offset; and the task snapshot contains at least one of the task ID, state, parameter, last processing position, and rollback point.

3. The method of claim 2, wherein, After writing the task snapshot in the memory mapping area into the distributed storage, the method further comprises: in response to the executor detecting that a target node fails, triggering the generation of a task transfer instruction; wherein the failure includes a crash and / or network isolation; controlling a new healthy node other than the target node to take over the task through a task scheduler according to the task transfer instruction, and reading the latest valid task snapshot of the task from the distributed storage through the task scheduler; loading the valid task snapshot into the local memory mapping area using the new healthy node, and executing the task based on the task state, task parameters, and task progress corresponding to the valid task snapshot saved by the executor.

4. The method of claim 2, wherein, After modifying the state parameters in the memory region inside the process that is writable in the memory region through atomic operation corresponding to the memory barrier in a manner of online modification based on the target TCB and the target key state field, the method further comprises: in response to the executor detecting that the task logic of the current task is abnormal, determining that an error value is written in the memory mapping during the process of modifying the state parameters in the memory region online; recovering from the last task snapshot locally reserved to the memory mapping area, and attempting to continue executing the current task after recovery, so as to realize local rollback attempt through the last state snapshot reserved in the memory mapping; In response to a failure of the local rollback attempt, the current node marks a state that the current task belongs to an untrusted state, and triggers the executor to transfer the current task to a healthy node other than the current node, and retries by loading a previous task snapshot through the healthy node.

5. The method of claim 4, wherein, The triggering the executor to transfer the current task to a healthy node other than the current node comprises: Selecting a target healthy node from the healthy nodes other than the current node by the following formula, so that the finally selected target healthy node is closer and more idle at a unit failure cost: ； wherein, represents a task execution failure rate of node n in a preset time length before the current time; represents a path distance metric between nodes n; represents a current load of node n; represents a selection manner of the target other healthy node; Triggering the executor to transfer the current task to the target healthy node.

6. The method of claim 1, wherein, The task corresponds to a parent task and a child task, and the child task is triggered through an AND convergence relationship after all the parent tasks are successfully completed; a directed acyclic graph (DAG) is constructed by analyzing the parent-child task dependency relationship between the parent task and the child task, and a circular dependency check is performed based on the DAG; The distributed scheduling of multiple tasks according to the system service instruction comprises: Distributed scheduling of multiple tasks according to the system service instruction and updating task node states through a state machine corresponding to the DAG in the task scheduler after the executor completes the current task; In response to the state of the parent task being updated to success, querying all target child tasks in the DAG that have the current task as a prerequisite dependency and all parent tasks corresponding to the target child tasks; In response to all parent tasks corresponding to the target child tasks being successfully completed, triggering the target child task again through a sharding strategy; In response to the existence of a failure in all parent tasks corresponding to the target child tasks and no failure branch processing, marking the downstream of the target child task as an unexecutable state.

7. The method of claim 6, wherein, The sharding strategy comprises using the task scheduler to issue the target child task to all nodes corresponding to the executor through sharding broadcast based on the markers in the task metadata, so as to trigger all nodes to execute the target child task; The triggering the target child task again through the sharding strategy comprises: In response to the task scheduler detecting an increase or decrease of nodes in the cluster of the executor, not interrupting the running target child task, and generating a sharding rebalance instruction; In response to the next scheduling of the target child task, the task scheduler sends the sharding rebalance instruction; In response to each node completing the current shard, dynamically adjusting a new number of shards according to the sharding rebalance instruction, and reallocating a sharding strategy according to the new number of shards, to complete a dynamic sharding task.

8. A distributed task scheduling device based on task state switching, characterized in that, Comprise: A scheduling module is configured to obtain a system service instruction, distribute multiple tasks according to the system service instruction, and display task states of the multiple tasks in a graphical user interface; each task corresponds to a task control block (TCB) and a key state field. The mapping module is configured to, in response to a state switching operation for a target task in the plurality of tasks, map a target TCB and a target critical state field corresponding to the target task to a memory region inside the process that is writable in the shared memory according to the state switching operation. The modifying module is configured to modify a state parameter in the memory region in a manner of corresponding memory barriers through an atomic operation in the memory region inside the process based on the target TCB and the target critical state field. The adjusting module is configured to, in response to a corresponding memory modification event being completed after the modification of the state parameter, continuously poll through an executor corresponding to the target task and perceive a state change through a notification of the memory modification event, and adjust a self behavior of the executor according to the perceived state change, so that the state parameter modified online is immediately effective and a service corresponding to the system service instruction and a whole task scheduling system are avoided from being restarted.

9. An electronic device comprising a memory, a processor, the memory having stored therein a computer program executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

10. A computer readable storage medium characterized by, The computer readable storage medium stores computer executable instructions, and the computer executable instructions, when called and executed by the processor, cause the processor to execute the method in any one of claims 1 to 7.

Images