Task scheduling method and system based on distributed scheduling framework

Abstract

The application is particularly a task scheduling method and system based on a distributed scheduling framework, relates to the technical field of distributed computing and task scheduling, and comprises: an optimistic scheduling decision module, a resource coordinator and conflict detection module, a conflict resolution arbitration module, and a final consistency state synchronization module.In the application, the lock-free design and local cache mechanism of the optimistic scheduling decision module break the dependence of traditional scheduling on a global real-time consistent state, realize extremely fast decision-making in a high-concurrency scenario, the local cache with time-effectiveness deviation adopts a hierarchical index structure and a dual-mode update mechanism, balances the cache freshness and update overhead, supports the scheduler to complete calculation only by relying on local data, does not need real-time communication across nodes, and has lower decision-making delay.

Description

Technical Field

[0001] This invention relates to the field of distributed computing and task scheduling technology, and in particular to a task scheduling method and system based on a distributed scheduling framework. Background Technology

[0002] With the rapid development of cloud computing, big data and artificial intelligence technologies, the scale of clusters continues to expand, and task scheduling scenarios are characterized by high concurrency, multiple resource types (CPU, memory, GPU, storage, etc.), and strong real-time requirements, which poses a severe challenge to traditional distributed scheduling methods.

[0003] Most existing scheduling systems rely on a globally consistent real-time cluster state for decision-making. The scheduler needs to communicate with the central node or other instances in real time to obtain the latest state, which leads to high decision-making latency and intense lock contention, making it difficult to support the scheduling of millions of concurrent tasks. Throughput becomes the system performance bottleneck.

[0004] Meanwhile, traditional scheduling often adopts coarse-grained resource lock mechanisms, which have a wide range of conflict detection, low efficiency, and simple conflict resolution rules, making them unable to adapt to the multi-dimensional priority requirements of complex business scenarios, further restricting resource utilization and scheduling flexibility.

[0005] Furthermore, traditional state synchronization mechanisms often employ a full synchronization mode, which consumes a lot of network bandwidth, has high synchronization latency, and lacks an effective cache calibration mechanism. This leads to the accumulation of discrepancies between the cache and the actual state, causing scheduling decision errors or frequent conflicts. At the same time, some scheduling methods lack a sound task lifecycle management and kernel-level resource isolation mechanism, resulting in prominent issues such as resource contention between tasks and resource leakage under abnormal conditions. These issues affect system stability and reliability, making it difficult to meet the complex scheduling needs of large-scale mixed business clusters (including online services, offline computing, AI training, etc.).

[0006] Therefore, a task scheduling method and system based on a distributed scheduling framework are proposed to address the aforementioned problems. Summary of the Invention

[0007] The purpose of this invention is to propose a task scheduling method and system based on a distributed scheduling framework in order to solve the above-mentioned problems.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A task scheduling system based on a distributed scheduling framework includes:

[0010] The optimistic scheduling decision module is configured to generate globally unique scheduling proposals and distribute them in batches based on local caching with timeliness bias and a lock-free weighted scoring algorithm.

[0011] The resource coordinator and conflict detection module are configured to be deployed locally on the computing node. They complete the resource pre-allocation, conflict detection and implementation of the proposal through fine-grained resource locks and CAS atomic operations.

[0012] The conflict resolution and arbitration module is configured to adopt a logically centralized and physically distributed architecture. It adjudicates global conflicts through a layered deterministic rule engine and combines reliable compensation instructions with the Raft protocol to handle high-concurrency conflict events while ensuring eventual consistency.

[0013] The eventual consistency state synchronization module is configured to asynchronously and incrementally synchronize global state changes based on distributed ordered logs and publish-subscribe patterns. Through periodic calibration and emergency calibration mechanisms, it ensures that the local caches of each module eventually converge to the true state.

[0014] Preferably, the optimistic scheduling decision module specifically includes:

[0015] Optimistic state caching:

[0016] It adopts a combination of hierarchical index and hash table structure. The top-level index is divided according to resource type, and the bottom-level hash table uses node ID as the key to store node-level resource details.

[0017] A dual-mode update mechanism of scheduled retrieval and incremental push is adopted; an expiration timestamp is attached to the state of each node in the cache. When the cache is not updated, proposals can continue to be generated based on the expired state, but the proposal will be marked with a low confidence label.

[0018] Lock-free proposal algorithm:

[0019] Core logic: Employs a multi-objective weighted scoring algorithm, with inputs being task resource requirements and locally cached node status, and outputs the 1-N target nodes with the highest scores;

[0020] Scoring dimensions and weights: node resource remaining amount, node load rate, task-node affinity, node historical scheduling success rate. Weights can be dynamically configured to adapt to different business scenarios.

[0021] Proposal tagging mechanism:

[0022] Globally unique identifier: The proposal ID is generated by combining the scheduler ID, a local monotonically increasing counter, and a timestamp;

[0023] Logical timestamp: generated based on physical timestamp and counter offset;

[0024] Lock-free guarantee: The algorithm only reads the local cache during execution.

[0025] Preferably, the resource coordinator and conflict detection module specifically include:

[0026] A priority blocking queue is used, and proposals are sorted in reverse logical timestamp order.

[0027] Fine-grained exclusive locks are adopted, and the lock granularity is divided according to resource type and resource unit;

[0028] The lock storage structure uses a hash table and a doubly linked list to implement the lock pool;

[0029] Atomization Attempt and Collision Detection Process:

[0030] After the proposal is dequeued, the resource requirement list in the proposal is first parsed and converted into a lock request list of resource type and resource unit ID;

[0031] A batch lock attempt mechanism is adopted, which attempts to acquire all the required locks at once through the lock manager based on the lock request list.

[0032] Preferably, the method further includes:

[0033] Success path handling:

[0034] Update the local resource ledger immediately;

[0035] Send a proposal confirmation message to the conflict resolution arbitration module;

[0036] Send a successful proposal confirmation to the proposal initiator, along with a snapshot of the node resource ledger;

[0037] Call the startup interface of the local task executor to pass task configuration and resource allocation information;

[0038] Conflict path handling:

[0039] If locking fails, immediately perform a lock rollback operation;

[0040] Obtain the proposal ID currently occupying the target resource, and generate a conflict event by combining it with information about currently failed proposals;

[0041] Send a conflict event to the conflict resolution and arbitration module, and simultaneously send a proposal conflict failure message to the proposal initiator.

[0042] Preferably, the conflict resolution arbitration module specifically includes:

[0043] Conflict queues and collectors:

[0044] A sharded collection and aggregation processing model is adopted, with each arbitrator instance responsible for handling conflict events in its corresponding preset shard; at the same time, a global conflict aggregator is deployed to handle cross-shard related conflicts.

[0045] Each shard corresponds to a lock-free circular queue; the queue adopts a producer-consumer model, with the enqueue thread responsible for receiving conflict events and the dequeue thread responsible for handling the decision-making, ensuring process decoupling.

[0046] Before a conflict event is enqueued, it is deduplicated based on the conflict ID and the hash of the conflict proposal list; at the same time, a timeout is set for the conflict event, and conflicts that are not processed within the timeout will be prioritized for scheduling.

[0047] A hierarchical rule system with main rules and auxiliary rules is adopted. The main rules are used to determine the core priority, and the auxiliary rules are used to solve the draw scenarios that the main rules cannot distinguish.

[0048] All arbitrator instances use a unified rule configuration file, and configuration changes are synchronized through atomic push and canary release mechanisms. The rule engine uses deterministic computation during execution.

[0049] Preferably, the method further includes:

[0050] The winning proposal confirmation instruction is sent to the resource coordinator of the conflicting node, including the conflict ID, the winning proposal ID, and the resource lock confirmation, triggering task execution;

[0051] The failure proposal compensation instruction is sent to the initiator of the failure proposal, including the conflict ID, failure proposal ID, reason for adjudication, compensation strategy, and the latest cluster state snapshot;

[0052] The resource status correction command is sent to the resource coordinator of the conflicting nodes;

[0053] A reliable message queue and a timeout retry mechanism are adopted. After an instruction is sent, the receiver needs to return an acknowledgment. Instructions that are not acknowledged will be retried. An idempotent flag is also added to the instruction.

[0054] The arbitrator cluster uses the Raft consensus protocol to maintain the cluster state;

[0055] Each arbitrator instance maintains an arbitration log;

[0056] When the arbitrator instance fails, the Raft protocol elects a new master instance to take over shard processing.

[0057] Preferably, the eventually consistent state synchronization module specifically includes:

[0058] Segmented log storage is used; event types are categorized into resource allocation events, resource release events, node status events, and task status events.

[0059] After log entries are written to memory, they are asynchronously flushed to distributed storage.

[0060] Based on the publish-subscribe pattern, Kafka is used as the underlying message queue, and each event type corresponds to a topic;

[0061] The incremental part of the synchronization state change;

[0062] The message queue stores events in log index order, and subscribers consume them in sequence.

[0063] Preferably, the method further includes:

[0064] Asynchronous cache update process:

[0065] After receiving the synchronization stream message, the scheduler puts the message into the local update task queue.

[0066] The background update thread pool retrieves tasks from the queue and processes them in log index order;

[0067] During processing, first verify the integrity of the message, and then update the corresponding field in the local cache based on the event type;

[0068] After the update is complete, record the current consumption log index for resuming interrupted downloads.

[0069] Preferably, the method further includes:

[0070] The task executor and lifecycle management module are configured to load the task runtime environment, allocate isolated resources and execute the task after receiving confirmation of resource locking, monitor the task lifecycle status throughout the process, release resources and report the status in a timely manner after the task ends, and complete the scheduling closed loop.

[0071] Task scheduling methods based on distributed scheduling frameworks include:

[0072] Based on local caching with timeliness bias and lock-free weighted scoring algorithm, a globally unique scheduling proposal is generated and distributed to the target node in batches;

[0073] By using fine-grained resource locks and CAS atomic operations, the resource pre-allocation and conflict detection of the proposal are completed atomically. If there is no conflict, the proposal is confirmed to be implemented; if there is a conflict, the conflict event is reported.

[0074] A hierarchical deterministic rule engine is used to adjudicate global conflicts, send the adjudication results and compensation instructions to relevant parties, clarify the winning proposal and guide the failed proposals to retry;

[0075] Based on distributed logs and a publish-subscribe pattern, global state changes are asynchronously and incrementally synchronized, and a calibration mechanism is used to ensure that the caches of each module converge to the real state.

[0076] After receiving confirmation of resource lock, execute the task, monitor the task status throughout, release resources and report the status after the task is completed.

[0077] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0078] 1. This invention breaks the dependence of traditional scheduling on a globally consistent real-time state by using a lock-free design and local caching mechanism in the optimistic scheduling decision module, thus achieving ultra-fast decision-making in high-concurrency scenarios. The local cache with timeliness bias adopts a hierarchical index structure and a dual-mode update mechanism, which balances cache freshness and update overhead while supporting the scheduler to complete calculations based solely on local data, without the need for real-time cross-node communication, resulting in lower decision latency.

[0079] 2. This invention constructs an efficient and reliable conflict resolution mechanism through the collaboration of a hierarchical deterministic rule engine and the Raft protocol, ensuring the uniqueness and consistency of global conflict resolution; the asynchronous incremental state synchronization mode, combined with a periodic and emergency dual calibration mechanism, transmits only the incremental part of the state change, reducing network bandwidth consumption, while keeping the deviation between the cache and the actual state within the required range, avoiding scheduling decision errors. Attached Figure Description

[0080] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0081] Figure 1 This is a system structure diagram of the present invention;

[0082] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0083] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0084] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0085] Example 1

[0086] Its specific implementation method is combined with the appendix Figure 1 and attached Figure 2 Please provide a detailed explanation.

[0087] Appendix Figure 1The block diagram of a task scheduling system based on a distributed scheduling framework provided in this embodiment of the invention shows the connection relationship between the optimistic scheduling decision module and the eventual consistency state synchronization module, and marks the main functional interaction flow of each module.

[0088] Appendix Figure 2 The flowchart of the task scheduling method based on the distributed scheduling framework provided in the embodiments of the present invention illustrates the complete steps from generating a globally unique scheduling proposal to executing the task after receiving resource lock confirmation.

[0089] In this embodiment, it includes:

[0090] The optimistic scheduling decision module is configured to generate globally unique scheduling proposals and distribute them in batches based on local caching with timeliness bias and lock-free weighted scoring algorithm, prioritizing throughput to break the dependence of traditional scheduling on global real-time consistent state.

[0091] Specifically, it includes:

[0092] Functionality: Based on a locally cached cluster state view that allows for timeliness deviations (10-100ms level latency), it quickly generates 1-N scheduling proposals for tasks to be scheduled in a lock-free, single-threaded (or lightweight multi-threaded) manner. Each proposal contains metadata such as task ID, target node ID, resource requirement list, logical timestamp, and cache version number. The entire decision-making process does not communicate in real time with any other scheduler instance, central state storage, or target node, and relies solely on local data to complete the calculation.

[0093] Optimistic state caching:

[0094] Cache data structure: A combination of hierarchical index and hash table structure is adopted. The top-level index is divided according to resource type (CPU, memory, GPU, storage), and the bottom-level hash table uses node ID as the key to store node-level resource details (such as total number of CPU cores, number of used cores, number of remaining cores, total / available memory, GPU model / quantity / availability status, node load threshold, task affinity tag, etc.).

[0095] It adopts a dual-mode update mechanism of scheduled pull and incremental push; the scheduled pull period is configurable (default 50ms), and it pulls the full cluster state from the state synchronizer for cache calibration; the incremental push receives resource change events (such as task start / end, node online / offline) issued in real time by the state synchronizer, and is used to update the key data in the cache in an instant, balancing cache freshness and update overhead.

[0096] Expiration tolerance strategy: An expiration timestamp is attached to the state of each node in the cache. When the cache is not updated, proposals are allowed to continue to be generated based on the expired state, but the proposal will be marked with a low confidence label, which can be used as an auxiliary basis for adjudication in subsequent conflict resolution.

[0097] Lock-free proposal algorithm:

[0098] Core logic: Employs a multi-objective weighted scoring algorithm. The inputs are the task resource requirements (e.g., 2 CPU cores, 4GB memory, 1 GPU) and the node status of the local cache. The output is the 1-N target nodes with the highest scores.

[0099] Scoring dimensions and weights: node resource remaining amount, node load rate, task-node affinity, node historical scheduling success rate. Weights can be dynamically configured to adapt to different business scenarios.

[0100] Proposal tagging mechanism:

[0101] Globally unique identifier: The proposal ID is generated by combining the scheduler ID (8 bytes), the local monotonically increasing counter (8 bytes), and the timestamp (4 bytes) to ensure that the proposal ID is not duplicated in a distributed environment;

[0102] Logical timestamp: generated based on physical timestamp and counter offset. The physical timestamp is accurate to the millisecond level, and the counter offset is used to solve the sorting problem of multiple proposals generated within the same millisecond, ensuring the global order of logical timestamps.

[0103] Cache version tag: Records the cache snapshot version number that the proposal is based on (issued by the state synchronizer and incremented with each full update), which is used to quickly locate whether the conflict is caused by cache expiration or resource contention when a conflict occurs.

[0104] Lock-free guarantee: The algorithm only reads the local cache during execution and does not involve any write operations or cross-thread communication;

[0105] Parallel proposal distribution:

[0106] Distribution mode: Employs non-blocking IO and batch sending mechanism. Each scheduler instance maintains a proposal distribution thread pool (number of threads = number of target nodes / 100, default maximum of 20 threads), and each thread is responsible for pushing proposals to a group of nodes.

[0107] Reliability Guarantee: A one-time confirmation mechanism is added when the proposal is sent. After the resource coordinator of the target node receives the proposal, it must return a receipt confirmation within 1ms. Proposals that do not receive confirmation will be marked as distribution failures and will be automatically retried by the scheduler (maximum number of retries 3 times, with an interval of 10ms) to avoid proposal loss.

[0108] The resource coordinator and conflict detection module are configured to be deployed locally on the compute node. Through fine-grained resource locks and CAS atomic operations, the resource pre-allocation, conflict detection and implementation of the proposal are completed atomically, transforming global conflict detection into efficient local operations.

[0109] Specifically, it includes:

[0110] It receives concurrent proposals from multiple optimistic scheduling decision modules and performs an atomic process of resource pre-allocation, conflict detection, and confirmation for each proposal to ensure that resource allocation on the same node does not conflict. If a conflict is detected, it immediately triggers conflict reporting and resource release to avoid deadlock or resource waste.

[0111] Proposal Receiving Queue:

[0112] Queue type: Priority blocking queue is used, and the queue capacity is configurable (default 1000). Proposals are sorted in reverse logical timestamp (the larger the timestamp, the higher the priority) to ensure that proposals for high-priority tasks are processed first.

[0113] Flow control: When the queue length reaches 80% of the threshold, a flow warning message is sent to the scheduler, and the scheduler suspends sending new proposals to that node; when the queue length is below 50% of the threshold, a flow recovery message is sent, and the scheduler resumes proposal distribution to avoid node overload.

[0114] Local resource lock manager:

[0115] Lock type design: Fine-grained exclusive locks are adopted, and the lock granularity is divided according to resource type and resource unit;

[0116] For example, CPU locks are divided by core ID (such as CPU0, CPU1, ..., CPUn), memory locks are divided by memory block ID (each 4GB is a block), and GPU locks are divided by GPU device ID, ensuring that only proposals competing for the same resource unit will cause a conflict.

[0117] The lock storage structure uses a hash table and a doubly linked list to implement the lock pool. The hash table uses resource type and resource unit ID as keys to store information such as lock status (idle / occupied), occupancy proposal ID, and occupancy time. The doubly linked list is used to quickly traverse all locks of the same resource type and supports batch lock operations.

[0118] Lock operation optimization: Lock acquisition and release adopt CAS (Compare-And-Swap) atomic operations to avoid the context switching overhead of traditional lock mechanisms;

[0119] Atomization Attempt and Collision Detection Process:

[0120] Preprocessing stage: After the proposal is dequeued, the resource requirement list in the proposal is first parsed and converted into a lock request list of resource type and resource unit ID (e.g., CPU lock: CPU2, CPU3; memory lock: MEM1; GPU lock: GPU0).

[0121] Lock checking phase: A batch lock attempt mechanism is adopted, based on the lock request list, and the lock manager attempts to acquire all the required locks at once.

[0122] Implementation logic: Iterate through the lock request list and perform a CAS operation on each lock to attempt to change the state from idle to occupied (proposal ID); if the CAS operation on all locks succeeds, the lock is successfully locked; if the CAS operation on any lock fails (the state is already occupied), the lock fails.

[0123] Success path handling:

[0124] Immediately update the local resource ledger (used resources, remaining resources) to ensure that conflict detection for subsequent proposals is based on the latest status;

[0125] Send a proposal confirmation message to the conflict resolution and arbitration module. The message includes the proposal ID, node ID, resource usage details, logical timestamp, and cache version number.

[0126] Send a successful proposal confirmation to the proposal initiator (optimistic scheduling decision module), along with a snapshot of the node resource ledger;

[0127] Call the startup interface of the local task executor, pass task configuration and resource allocation information, and trigger task loading;

[0128] Conflict path handling:

[0129] If locking fails, immediately perform a lock rollback operation: iterate through the locks that have been successfully acquired and restore their state to idle through a CAS operation to ensure that resources are not leaked;

[0130] Collect conflict information: Obtain the proposal ID currently occupying the target resource (read from the lock's occupying proposal ID field), and generate a conflict event by combining it with the information of the currently failed proposal;

[0131] Conflict event fields: Conflict ID (globally unique), Conflict Node ID, Conflict Resource Type, Unit ID, Conflict Proposal List (including ID, logical timestamp, resource requirement, and cache version number of each proposal), and Conflict Occurrence Timestamp;

[0132] Send the conflict event to the conflict resolution and arbitration module, and at the same time send a proposal conflict failure message to the proposal initiator, along with the reason for the conflict.

[0133] The conflict resolution and arbitration module is configured to adopt a logically centralized and physically distributed architecture. It adjudicates global conflicts through a layered deterministic rule engine and combines reliable compensation instructions with the Raft protocol to efficiently handle high-concurrency conflict events while ensuring eventual consistency.

[0134] Specifically, it includes:

[0135] The system receives conflict events from all node resource coordinators at high concurrency, prioritizes the proposal list in each conflict event, determines the unique winning proposal, sends the decision results and compensation instructions to relevant nodes and schedulers, coordinates the completion of resource state correction and retry of failed proposals, and ensures the eventual consistency of conflict handling.

[0136] Conflict queues and collectors:

[0137] Collector architecture: It adopts a sharded collection and aggregation processing mode. Each arbitrator instance is responsible for handling the conflict events of its corresponding preset shard (sharded by conflict ID hash) to avoid single instance overload; at the same time, a global conflict aggregator is deployed to handle cross-shard related conflicts (such as the same proposal causing conflicts on multiple nodes).

[0138] Conflict queue design: Each shard corresponds to a lock-free circular queue with configurable capacity, supporting high-concurrency enqueueing and dequeueing of conflict events; the queue adopts a producer-consumer model, with the enqueue thread responsible for receiving conflict events and the dequeue thread responsible for processing and adjudicating them, ensuring process decoupling;

[0139] Conflict deduplication: Before a conflict event is enqueued, it is deduplicated based on the conflict ID and the hash of the conflict proposal list to avoid the same conflict being processed repeatedly; at the same time, a timeout is set for the conflict event (default 5ms), and conflicts that are not processed within the timeout will be prioritized for scheduling to ensure real-time performance.

[0140] Rule System: A hierarchical rule system with main rules and auxiliary rules is adopted. The main rules are used to determine the core priority, and the auxiliary rules are used to resolve draw scenarios that the main rules cannot distinguish, ensuring that all conflicts can be resolved with a unique result.

[0141] Example of a main rule:

[0142] Logical timestamp rule: The proposal with the smallest logical timestamp wins (suitable for scenarios sensitive to task start order).

[0143] Task priority rules: The proposal with the highest task priority value wins (task priority is predefined by the business layer, from 0 to 10, with 10 being the highest).

[0144] Resource matching rule: The proposal with the highest matching degree between its resource requirements and the node's remaining resources wins (matching degree = 1 - |node's remaining resources - proposal's required resources| / node's total resources).

[0145] Example of auxiliary rules:

[0146] Cache version number rule: The proposal with the latest cache version number wins (when resolving a tie in the main rule, the proposal generated based on the newest state is given priority).

[0147] Proposal ID hashing rule: The proposal with the smallest proposal ID hash value wins (ensuring a final unique decision and avoiding infinite ties).

[0148] All arbitrator instances use a unified rule configuration file, and configuration changes are synchronized through atomic push and canary release mechanisms to ensure consistency of rules across the entire network. The rule engine uses deterministic computation during execution (the same input will result in the same output) to avoid inconsistent adjudication results due to differences in the execution environment.

[0149] Compensation instruction generator:

[0150] Instruction types and fields:

[0151] The winning proposal confirmation instruction is sent to the resource coordinator of the conflicting node, including the conflict ID, the winning proposal ID, and the resource lock confirmation, triggering task execution;

[0152] The failure proposal compensation instruction is sent to the initiator (scheduler) of the failure proposal, including the conflict ID, failure proposal ID, reason for adjudication, compensation strategy (immediate retry / delayed retry / degraded retry), and the latest cluster state snapshot (used to generate a better proposal during retry).

[0153] The resource status correction command is sent to the resource coordinator of the conflicting node to correct inconsistencies in the resource ledger caused by the conflict (such as releasing temporary resources occupied by failed proposals).

[0154] Command transmission guarantee: A reliable message queue and timeout retry mechanism are adopted. After the command is sent, the receiver needs to return an acknowledgment. Unacknowledged commands will be retried (the maximum number of retries is 5, with an interval of 200 microseconds); at the same time, an idempotent flag is attached to the command to ensure that the receiver will not process it repeatedly when it receives it repeatedly.

[0155] The arbitrator cluster uses the Raft consensus protocol to maintain the cluster state, ensuring that all instances reach a consensus on the conflict resolution results;

[0156] Each arbitrator instance maintains an arbitration log to record the arbitration results of all conflicts, supports log replay during fault recovery, and ensures that the arbitration status is not lost;

[0157] When the arbitrator instance fails, the Raft protocol elects a new master instance to take over shard processing.

[0158] The eventual consistency state synchronization module is configured to asynchronously and incrementally synchronize global state changes based on distributed ordered logs and publish-subscribe pattern. Through periodic calibration and emergency calibration mechanisms, it ensures that the local cache of each module eventually converges to the true state.

[0159] Specifically, it includes:

[0160] The system collects proposal confirmation events from the resource coordinator, task lifecycle events (startup / running / ending / failure) from the task executor, and node status events (online / offline / load anomaly) from the node management module in real time. These events are encapsulated into an ordered state change log and synchronized to all optimistic scheduling decision modules and conflict resolution arbitration modules via asynchronous push, ensuring that each module operates based on a consistent global state. At the same time, it provides log persistence and backtracking capabilities to support state recovery after system failures.

[0161] Status change log:

[0162] Segmented log storage is used, and the size of each log segment is configurable (default 64MB). Log entries include: log index (globally unique), event type, event timestamp, event metadata (such as proposal ID, node ID, resource change details, task status, etc.), and checksum (used for log integrity verification).

[0163] Event type classification:

[0164] Resource allocation event: After the proposal is confirmed, the node's resource occupancy status changes;

[0165] Resource release event: After a task ends or fails, the node resource release status changes;

[0166] Node status events: Node online / offline, load threshold adjustment, resource expansion / contraction;

[0167] Task status events: Task started, running, successful completion, and failed termination.

[0168] After log entries are written to memory, they are asynchronously flushed to distributed storage (such as HDFS or distributed KV storage), with flushing latency controlled within 10ms; log expiration cleanup is also supported (7 days are retained by default) to avoid storage overload.

[0169] Incremental synchronization stream:

[0170] Synchronous stream architecture: Based on the publish-subscribe pattern, using Kafka as the underlying message queue, each event type corresponds to a topic, and the optimistic scheduling decision module and the conflict resolution arbitration module act as subscribers to consume messages corresponding to the topic;

[0171] Incremental synchronization mechanism: Only the incremental part of the state change is synchronized, rather than the full state. For example, in a resource allocation event, only the node ID, resource type, and occupancy are synchronized, rather than the entire node's resource ledger, thus reducing network transmission overhead.

[0172] Order guarantee: The message queue stores events in log index order, and subscribers consume them in order to ensure the consistency of state updates in time sequence; it also supports breakpoint resume, and subscribers record the most recently consumed log index, and continue to consume from that index after the fault is recovered to avoid message loss or duplicate consumption.

[0173] Asynchronous cache update process:

[0174] After receiving the synchronization stream message, the scheduler puts the message into the local update task queue (lockless queue).

[0175] The background update thread pool (4 threads by default) retrieves tasks from the queue and processes them in log index order;

[0176] During processing, the integrity of the message is first verified (by checksum), and then the corresponding fields in the local cache are updated based on the event type (such as the used and remaining resources of the node in the resource allocation event).

[0177] After the update is complete, record the current consumption log index for resuming interrupted downloads;

[0178] A version number verification mechanism is adopted, with each cached entry appended with an update version number (consistent with the log index). During an update, the update is only performed if the log index of the message is greater than the current cache version number, thus avoiding old messages from overwriting the new state. Batch updates are also supported. When a certain number of messages (100 by default) accumulate in the update task queue, the update operation is performed in batches to improve efficiency.

[0179] All schedulers send cached state snapshots to the state synchronizer at preset time intervals (the time interval is configurable). The state synchronizer compares the snapshots with the global real state. If the deviation exceeds a threshold (default 5%), it sends a full synchronization command to the scheduler. The scheduler then pulls the full cluster state again and calibrates the local cache.

[0180] When the conflict resolution arbitration module finds that the conflict rate of a proposal generated by a scheduler is too high (more than 20%), it triggers the state synchronizer to send an emergency calibration command to the scheduler to force an update of the cache and reduce the probability of subsequent conflicts.

[0181] The task executor and lifecycle management module are configured to load the task runtime environment, allocate isolated resources, and execute the task after receiving resource lock confirmation. They also monitor the task lifecycle status throughout the process, promptly release resources and report the status after the task ends, completing the scheduling loop. Specifically, this includes:

[0182] Task loading and resource allocation:

[0183] Runtime environment preparation: Supports both containerized (Docker / K8s) and non-containerized deployment modes. In containerized mode, it receives the container image address and startup parameters from the resource coordinator, calls the local container engine to pull the image and create the container, and binds locked resources (such as limiting the number of CPU cores and memory size through cgroups, and binding a specified GPU device through the GPU scheduling interface); in non-containerized mode, it directly creates processes and allocates resources through system calls.

[0184] Resource isolation guarantee: A kernel-level resource isolation mechanism is adopted to allocate an independent resource namespace (PID, network, mount point) for each task to avoid resource contention or interference between tasks; at the same time, a resource usage limit is set (based on the resource requirements in the proposal). When the resource usage of a task exceeds the limit, an alarm is triggered and resource usage is restricted to prevent node overload.

[0185] Lifecycle monitoring and status reporting:

[0186] Monitoring metrics: Real-time collection of metrics such as CPU utilization, memory usage, GPU utilization, network I / O, disk I / O, task progress (reported by the task itself or based on output log parsing), and error codes. The default collection frequency is 100ms / time.

[0187] State machine design: The task lifecycle includes six states: pending startup, starting, running, success, failure, and termination. State transitions must meet preset rules (e.g., pending startup can only transition to starting, while running can transition to success, failure, or termination) to avoid state chaos.

[0188] Status reporting mechanism:

[0189] Critical status changes (successful startup, run completed, failed, terminated) are reported to the status synchronizer in real time, along with the task ID, node ID, status type, timestamp, and error details (in case of failure).

[0190] Periodic status reporting (default 1 second / time) reports the task running status to the resource coordinator, which is used by the resource coordinator to monitor the task health.

[0191] Reporting reliability: A retry and fallback mechanism is adopted. When a report fails, it will automatically retry (up to 3 times). If the retry fails, the status will be written to the local log and the report will be supplemented after the node recovers.

[0192] Resource release and exception handling:

[0193] Normal release process: When a task successfully ends or is terminated normally, the resource release process consists of three steps: releasing kernel-level resource locks (cgroup, GPU binding, etc.); updating the local resource ledger (reducing the amount of used resources and increasing the amount of remaining resources); and sending a resource release confirmation to the resource coordinator, which triggers the state synchronizer to update the global state.

[0194] Abnormal release process: When a task crashes or a node fails during execution, the resource coordinator triggers passive release: it detects that the task process does not exist (by monitoring the PID); it forcibly releases all resource locks occupied by the task; and it sends a task failure and resource release event to the state synchronizer to ensure that resources are not occupied for a long time.

[0195] Backup guarantee: The node maintains a local resource usage timeout checker, which periodically (default 10 seconds) scans all tasks with allocated resources. If a task has not finished after the preset timeout period (configured by the business layer), it will automatically trigger resource release and task termination to avoid resource leakage.

[0196] The resource coordinator sends start / stop commands to the task executor via local inter-process communication (IPC) to avoid network overhead. The task executor reports the task status to the resource coordinator in real time. The resource coordinator determines whether to allow new proposals to be allocated resources based on the status. If the task failure rate on the node is too high, it will suspend receiving new proposals.

[0197] All status events reported by the task executor must be verified by the resource coordinator (to ensure the authenticity of the status) before being forwarded by the resource coordinator to the state synchronizer, in order to prevent invalid or forged status events from interfering with the global status.

[0198] Example 2

[0199] Please see Figure 2 The task scheduling method based on the distributed scheduling framework includes the following parts:

[0200] Based on local caching with timeliness bias and lock-free weighted scoring algorithm, a globally unique scheduling proposal is generated and distributed to the target node in batches;

[0201] By using fine-grained resource locks and CAS atomic operations, the resource pre-allocation and conflict detection of the proposal are completed atomically. If there is no conflict, the proposal is confirmed to be implemented; if there is a conflict, the conflict event is reported.

[0202] A hierarchical deterministic rule engine is used to adjudicate global conflicts, send the adjudication results and compensation instructions to relevant parties, clarify the winning proposal and guide the failed proposals to retry;

[0203] Based on distributed logs and a publish-subscribe pattern, global state changes are asynchronously and incrementally synchronized, and a calibration mechanism is used to ensure that the caches of each module converge to the real state.

[0204] After receiving confirmation of resource lock, execute the task, monitor the task status throughout, release resources and report the status after the task is completed.

[0205] Example 3: A panoramic view of collaborative workflow:

[0206] Assume two scheduler instances S1 and S2 receive task scheduling requests (T1 and T2) almost simultaneously, and both select node N1 as the target node. The system coordination process is as follows:

[0207] Optimistic proposal generation and distribution:

[0208] S1's local cache shows that N1 has 2 CPU cores and 1 GPU available (cache version number V10), and generates proposal P1 (T1 to N1, 2 CPU cores and 1 GPU, logical timestamp TS=100, proposal ID=S1-0001-1699999999), which is sent to N1's resource coordinator through a parallel distribution thread.

[0209] S2's local cache also shows that N1 has available resources (cache version number V10, no update triggered), and generates proposal P2 (T2 to N1, 2 CPU cores, 1 GPU, logical timestamp TS=102, proposal ID=S2-0001-1699999999), which is synchronously sent to N1's resource coordinator.

[0210] Local atomic conflict detection:

[0211] The resource coordinator of N1 receives P1 (TS=100) and P2 (TS=102) through a priority queue, and processes P1 in reverse order of timestamp.

[0212] P1 processing: The resource requirements are parsed as 2 / 3 CPU cores and 0 GPU cores. A batch lock acquisition is attempted, and all CAS operations are successful (lock status changes from idle to occupied - P1). The local resource ledger is updated (0 CPU cores remaining and 0 GPU cores remaining). A P1 confirmation message is sent to the conflict resolution arbitration module, a success confirmation is sent to S1, and the task executor is invoked to start T1.

[0213] P2 processing: The resource requirement is resolved to be CPU2 / 3 cores and GPU0. When attempting to acquire the lock, the lock status of both CPU2 / 3 cores and GPU0 is occupied -P1, and the CAS operation fails. Immediately roll back the locks that have been attempted to acquire (no locks have been acquired), generate a conflict event (conflict ID=C001, node N1, conflicting resources CPU2 / 3 and GPU0, conflict proposals P1 / TS100 and P2 / TS102), and send it to the conflict resolution arbitration module.

[0214] Global Conflict Resolution Ruling:

[0215] The sharding collector of the conflict resolution arbitration module receives conflict event C001 and shards it to instance A according to the conflict ID hash.

[0216] The rule engine uses the rule with the smallest logical timestamp to win, and P1 (TS=100) wins.

[0217] Send P1 activation confirmation to N1, and send compensation instructions to S2 (P2 conflict failed, compensation strategy: retry immediately, with the latest status snapshot - N1 CPU / GPU is exhausted, N2 has available resources).

[0218] State synchronization and proposal retry:

[0219] The state synchronizer receives the P1 acknowledgment event from N1, generates a state change log (index=10001, event type=resource allocation, node N1, CPU usage +2, GPU usage +1), and pushes it to all schedulers via Kafka;

[0220] S2 receives compensation instructions and status synchronization logs, triggers local cache emergency calibration, and updates the N1 resource status to 0 CPU cores available and 0 GPUs available, and the N2 resource status to 4 CPU cores available and 2 GPUs available.

[0221] S2 regenerates proposal P2' for T2 based on the updated cache (T2 to N2, 2 CPU cores, 1 GPU, logical timestamp TS=103, proposal ID=S2-0002-1699999999), and sends it to the resource coordinator of N2;

[0222] The resource coordinator of N2 processes P2', successfully acquires the resource lock, confirms the proposal and starts T2, the state synchronizer updates the global state, and the process ends.

[0223] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0224] It should be noted that, in this document, the use of relational terms such as "first" and "second" is merely for distinguishing one entity or operation from another, and does not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0225] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0226] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0227] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0228] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0229] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0230] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0231] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A task scheduling system based on a distributed scheduling framework, characterized in that, include: The optimistic scheduling decision module is configured to generate globally unique scheduling proposals and distribute them in batches based on a local cache with timeliness bias and a lock-free weighted scoring algorithm. Specifically, it includes: The local cache with timeliness bias adopts a combination of hierarchical index and hash table structure. The top-level index is divided according to resource type, and the bottom-level hash table stores node-level resource details with node ID as the key. A dual-mode update mechanism of scheduled retrieval and incremental push is adopted; an expiration timestamp is attached to the state of each node in the cache. When the cache is not updated, proposals can continue to be generated based on the expired state, but proposals that continue to be generated will be marked with a low confidence label. The lock-free weighted scoring algorithm adopts a multi-objective weighted scoring algorithm. The input is the task resource requirements and the node status of the local cache, and the output is the 1-N target nodes with the highest scores. The scoring dimensions and weights include the remaining amount of node resources, node load rate, task-node affinity, and historical scheduling success rate of nodes. The weights can be dynamically configured to adapt to different business scenarios. The resource coordinator and conflict detection module are configured to be deployed locally on the computing node. They complete the resource pre-allocation, conflict detection and implementation of the proposal through fine-grained resource locks and CAS atomic operations. The conflict resolution and arbitration module is configured to adopt a logically centralized and physically distributed architecture. It adjudicates global conflicts through a layered deterministic rule engine and combines reliable compensation instructions with the Raft protocol to handle high-concurrency conflict events while ensuring eventual consistency. The eventual consistency state synchronization module is configured to asynchronously and incrementally synchronize global state changes based on distributed ordered logs and publish-subscribe patterns. Through periodic calibration and emergency calibration mechanisms, it ensures that the local caches of each module eventually converge to the true state.

2. The task scheduling system based on a distributed scheduling framework according to claim 1, characterized in that, The optimistic scheduling decision-making module also includes a proposal marking mechanism: The globally unique identifier is generated by combining the scheduler ID, a local monotonically increasing counter, and a timestamp to form the proposal ID. Logical timestamps are generated based on physical timestamps and counter offsets; The algorithm only reads the local cache during execution to ensure lock-free operation.

3. The task scheduling system based on a distributed scheduling framework according to claim 1, characterized in that, The resource coordinator and conflict detection module specifically include: A priority blocking queue is used, and proposals are sorted in reverse logical timestamp order. Fine-grained exclusive locks are adopted, and the lock granularity is divided according to resource type and resource unit; The lock storage structure uses a hash table and a doubly linked list to implement the lock pool; Atomization Attempt and Collision Detection Process: After the proposal is dequeued, the resource requirement list in the proposal is first parsed and converted into a lock request list of resource type and resource unit ID; A batch lock attempt mechanism is adopted, which attempts to acquire all the required locks at once through the lock manager based on the lock request list.

4. The task scheduling system based on a distributed scheduling framework according to claim 3, characterized in that, Also includes: Success path handling: Update the local resource ledger immediately; Send a proposal confirmation message to the conflict resolution arbitration module; Send a successful proposal confirmation to the proposal initiator, along with a snapshot of the node resource ledger; Call the startup interface of the local task executor to pass task configuration and resource allocation information; Conflict path handling: If locking fails, immediately perform a lock rollback operation; Obtain the proposal ID currently occupying the target resource, and generate a conflict event by combining it with information about currently failed proposals; Send a conflict event to the conflict resolution and arbitration module, and simultaneously send a proposal conflict failure message to the proposal initiator.

5. The task scheduling system based on a distributed scheduling framework according to claim 1, characterized in that, The conflict resolution and arbitration module specifically includes: Conflict queues and collectors: A sharded collection and aggregation processing model is adopted, with each arbitrator instance responsible for handling conflict events in its corresponding preset shard; at the same time, a global conflict aggregator is deployed to handle cross-shard related conflicts. Each shard corresponds to a lock-free circular queue; the queue adopts a producer-consumer model, with the enqueue thread responsible for receiving conflict events and the dequeue thread responsible for handling the decision-making, ensuring process decoupling. Before a conflict event is enqueued, it is deduplicated based on the conflict ID and the hash of the conflict proposal list; at the same time, a timeout is set for the conflict event, and conflicts that are not processed within the timeout will be prioritized for scheduling. A hierarchical rule system with main rules and auxiliary rules is adopted. The main rules are used to determine the core priority, and the auxiliary rules are used to solve the draw scenarios that the main rules cannot distinguish. All arbitrator instances use a unified rules configuration file.

6. The task scheduling system based on a distributed scheduling framework according to claim 5, characterized in that, Also includes: The winning proposal confirmation instruction is sent to the resource coordinator of the conflicting node, including the conflict ID, the winning proposal ID, and the resource lock confirmation, triggering task execution; The failure proposal compensation instruction is sent to the initiator of the failure proposal, including the conflict ID, failure proposal ID, reason for adjudication, compensation strategy, and the latest cluster state snapshot; The resource status correction command is sent to the resource coordinator of the conflicting nodes; A reliable message queue and a timeout retry mechanism are adopted. After an instruction is sent, the receiver needs to return an acknowledgment. Instructions that are not acknowledged will be retried. An idempotent flag is also added to the instruction. The arbitrator cluster uses the Raft consensus protocol to maintain the cluster state; Each arbitrator instance maintains an arbitration log; When the arbitrator instance fails, the Raft protocol elects a new master instance to take over shard processing.

7. The task scheduling system based on a distributed scheduling framework according to claim 1, characterized in that, The eventual consistency state synchronization module specifically includes: Segmented log storage is used; event types are categorized into resource allocation events, resource release events, node status events, and task status events. After log entries are written to memory, they are asynchronously flushed to distributed storage. Based on the publish-subscribe pattern, Kafka is used as the underlying message queue, and each event type corresponds to a topic; The incremental part of the synchronization state change; The message queue stores events in log index order, and subscribers consume them in sequence.

8. The task scheduling system based on a distributed scheduling framework according to claim 7, characterized in that, Also includes: Asynchronous cache update process: After receiving the synchronization stream message, the scheduler puts the message into the local update task queue. The background update thread pool retrieves tasks from the queue and processes them in log index order; During processing, first verify the integrity of the message, and then update the corresponding field in the local cache based on the event type; After the update is complete, record the current consumption log index for resuming interrupted downloads.

9. The task scheduling system based on a distributed scheduling framework according to claim 1, characterized in that, Also includes: The task executor and lifecycle management module are configured to load the task runtime environment, allocate isolated resources and execute the task after receiving confirmation of resource locking, monitor the task lifecycle status throughout the process, release resources and report the status in a timely manner after the task ends, and complete the scheduling closed loop.

10. A task scheduling method based on a distributed scheduling framework, and a task scheduling system based on a distributed scheduling framework according to any one of claims 1-9, characterized in that, include: Based on local caching with timeliness bias and lock-free weighted scoring algorithm, a globally unique scheduling proposal is generated and distributed to the target node in batches; By using fine-grained resource locks and CAS atomic operations, the resource pre-allocation and conflict detection of the proposal are completed atomically. If there is no conflict, the proposal is confirmed to be implemented; if there is a conflict, the conflict event is reported. A hierarchical deterministic rule engine is used to adjudicate global conflicts, send the adjudication results and compensation instructions to relevant parties, clarify the winning proposal and guide the failed proposals to retry; Based on distributed logs and a publish-subscribe pattern, global state changes are asynchronously and incrementally synchronized, and a calibration mechanism is used to ensure that the caches of each module converge to the real state. After receiving confirmation of resource lock, execute the task, monitor the task status throughout, release resources and report the status after the task is completed.

Images