Task scheduling method and system based on distributed scheduling framework

Abstract

The invention particularly relates to a task scheduling method and system based on a distributed scheduling framework, and relates to the technical field of distributed computing and task scheduling. The resource coordinator is connected with a conflict detection module; a conflict resolution arbitration module; and a final consistency state synchronization module. According to the invention, the lock-free design of the optimistic scheduling decision module and the local cache mechanism break through the dependence of the traditional scheduling on the global real-time consistent state, and realize the top-speed decision in the high-concurrency scene; the local cache with timeliness deviation adopts a hierarchical index structure and a dual-mode updating mechanism, so that a scheduler is supported to complete calculation only by depending on local data while the cache freshness and the updating overhead are balanced, cross-node real-time communication is not needed, and the decision delay is lower.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed computing and task scheduling, and in particular to a task scheduling method and system based on a distributed scheduling framework. BACKGROUND

[0002] With the rapid development of cloud computing, big data and artificial intelligence technology, the cluster scale continues to expand, and the task scheduling scenario presents the characteristics of high concurrency, multiple resource types (CPU, memory, GPU, storage, etc.), strong real-time performance, etc. The traditional distributed scheduling method faces severe challenges.

[0003] Most of the existing scheduling systems rely on the global real-time consistent cluster state for decision-making, and the scheduler needs to communicate with the center node or other instances in real time to obtain the latest state, resulting in high decision-making delay and intense lock competition, which makes it difficult to support high-concurrency task scheduling of more than one million, and the throughput rate becomes the performance bottleneck of the system; At the same time, the traditional scheduling mostly adopts a coarse-grained resource lock mechanism, which has a wide conflict detection range and low efficiency, and the conflict resolution rule is single, which cannot adapt to the multi-dimensional priority demand in complex business scenarios, further restricting the resource utilization rate and scheduling flexibility.

[0004] In addition, the traditional state synchronization mechanism mostly adopts a full-synchronization mode, which consumes large network bandwidth and has high synchronization delay, and lacks an effective cache calibration mechanism, resulting in the accumulation of cache and real state deviation, causing scheduling decision-making errors or frequent conflicts; At the same time, some scheduling methods lack perfect task life cycle management and kernel-level resource isolation mechanism, and the problems of task resource preemption and resource leakage under abnormal conditions are prominent, which affects the system stability and reliability, and it is difficult to meet the complex scheduling demand of large-scale mixed business clusters (including online services, offline computing, AI training, etc.).

[0005] Therefore, the task scheduling method and system based on the distributed scheduling framework are proposed to solve the above-mentioned problems. SUMMARY

[0006] The purpose of the present application is to solve the above-mentioned problems by proposing a task scheduling method and system based on a distributed scheduling framework.

[0007] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions: The task scheduling system based on the distributed scheduling framework comprises: An optimistic scheduling decision module configured to generate a globally unique scheduling proposal and batch distribute based on a local cache with time bias and a lock-free weighted scoring algorithm; A resource coordinator and a conflict detection module configured to be deployed locally on a computing node, and complete resource pre-occupation, conflict detection and landing of the proposal through fine-grained resource locks and CAS atomic operations. The conflict resolution arbitration module is configured to adopt a logically centralized and physically distributed architecture, resolve global conflicts through a hierarchical deterministic rule engine, combine reliable compensation instructions with the Raft protocol, and process high-concurrency conflict events while ensuring eventual consistency. The final consistency state synchronization module is configured to synchronize global state changes in an asynchronous incremental manner based on a distributed ordered log and a publish-subscribe mode, and ensure that the local caches of each module eventually converge to the true state through regular calibration and emergency calibration mechanisms.

[0008] Preferably, the optimistic scheduling decision module specifically includes: Optimistic state cache: The hierarchical index and hash table combination structure is adopted, the top-level index is divided according to resource types, and the bottom-level hash table takes node ID as the key to store node-level resource details; The dual-mode update mechanism of regular pull and incremental push is adopted; an expiration timestamp is added to each node state in the cache, and when the cache does not trigger an update, it is allowed to continue generating proposals based on expired states, but the proposals will be marked with a low confidence label; Lock-free proposal algorithm: Core logic: a multi-objective weighted scoring algorithm is adopted, with task resource requirements and local cached node states as inputs, and 1-N target nodes with the highest scores as outputs; Scoring dimensions and weights: node resource remaining amount, node load rate, task and node affinity, and node historical scheduling success rate, with dynamic configuration of weights to adapt to different business scenarios; Proposal marking mechanism: Global unique identifier: proposal ID is generated by combining scheduler ID, local monotonic incrementing counter, and timestamp; Logical timestamp: generated based on physical timestamp and counter offset; Lock-free guarantee: only read local cache during algorithm execution.

[0009] Preferably, the resource coordinator and conflict detection module specifically includes: Priority blocking queue is adopted, and proposals are sorted in reverse order of logical timestamp; Fine-grained exclusive lock is adopted, with lock granularity divided according to resource type and resource unit; The lock storage structure adopts a lock pool implemented by hash table and double-linked list; Atomic attempt and conflict detection process: After the proposal is dequeued, the resource requirement list in the proposal is first parsed and converted into a lock request list of resource type and resource unit ID; Batch lock attempt mechanism is adopted, based on the lock request list, all required locks are acquired through the lock manager at one time.

[0010] Preferably, the method further includes: Success path handling: Update the local resource ledger immediately; Send a proposal confirmation message to the conflict resolution arbitration module; Send a successful proposal confirmation to the proposal initiator, along with a snapshot of the node resource ledger; Call the startup interface of the local task executor to pass task configuration and resource allocation information; Conflict path handling: If locking fails, immediately perform a lock rollback operation; Obtain the proposal ID currently occupying the target resource, and generate a conflict event by combining it with information about currently failed proposals; Send a conflict event to the conflict resolution and arbitration module, and simultaneously send a proposal conflict failure message to the proposal initiator.

[0011] Preferably, the conflict resolution arbitration module specifically includes: Conflict queues and collectors: A sharded collection and aggregation processing model is adopted, with each arbitrator instance responsible for handling conflict events in its corresponding preset shard; at the same time, a global conflict aggregator is deployed to handle cross-shard related conflicts. Each shard corresponds to a lock-free circular queue; the queue adopts a producer-consumer model, with the enqueue thread responsible for receiving conflict events and the dequeue thread responsible for handling the decision-making, ensuring process decoupling. Before a conflict event is enqueued, it is deduplicated based on the conflict ID and the hash of the conflict proposal list; at the same time, a timeout is set for the conflict event, and conflicts that are not processed within the timeout will be prioritized for scheduling. A hierarchical rule system with main rules and auxiliary rules is adopted. The main rules are used to determine the core priority, and the auxiliary rules are used to solve the tie scenarios that the main rules cannot distinguish. All arbitrator instances use a unified rule configuration file, and configuration changes are synchronized through atomic push and canary release mechanisms. The rule engine uses deterministic computation during execution.

[0012] Preferably, the method further includes: The winning proposal confirmation instruction is sent to the resource coordinator of the conflicting node, including the conflict ID, the winning proposal ID, and the resource lock confirmation, triggering task execution; The failure proposal compensation instruction is sent to the initiator of the failure proposal, including the conflict ID, failure proposal ID, reason for adjudication, compensation strategy, and the latest cluster state snapshot; The resource status correction command is sent to the resource coordinator of the conflicting nodes; Adopt reliable message queue, timeout retry mechanism, instruction sending needs to receive confirmation, unconfirmed instruction will retry; At the same time, the instruction is attached with idempotent identification; The arbitrator cluster adopts Raft consensus protocol to maintain the cluster state; Each arbitrator instance maintains a decision log; When the arbitrator instance fails, a new master instance is elected by the Raft protocol to take over the shard processing.

[0013] Preferably, the final consistent state synchronization module specifically comprises: Adopt segmented log storage; Event type classification includes resource allocation event, resource release event, node state event, task state event; After writing log entries into memory, asynchronously flush to distributed storage; Based on the publish-subscribe mode, adopt Kafka as the underlying message queue, and each event type corresponds to a topic; Synchronize the incremental part of state change; Message queue stores events in log index order, and subscriber consumes in order.

[0014] Preferably, the still comprises: Cache asynchronous update process: After the scheduler receives the synchronization flow message, put the message into the local update task queue; The background update thread pool takes out the task from the queue and processes it in log index order; When processing, first verify the integrity of the message, and then update the corresponding field of the local cache based on the event type; After updating, record the current log index of consumption, which is used for breakpoint continuation.

[0015] Preferably, the still comprises: The task executor and the life cycle management module are configured to receive resource lock confirmation, load task running environment, allocate isolated resources and execute tasks, monitor task life cycle state throughout the process, release resources in time after task end and report state, complete scheduling closed loop.

[0016] The task scheduling method based on distributed scheduling framework comprises: Based on the local cache with time bias and the lock-free weighted scoring algorithm, generate a globally unique scheduling proposal and distribute it to the target node in batches; Through fine-grained resource lock and CAS atomic operation, atomically complete resource preemption and conflict detection of the proposal, confirm landing without conflict, and report conflict event with conflict; The layered deterministic rule engine is used for resolving global conflicts, sending the resolution result and compensation instruction to relevant parties, and explicitly winning the proposal and guiding the retry of the failed proposal. Based on the distributed log and the publish-subscribe mode, the global state change is synchronously increased in an asynchronous mode, and the calibration mechanism is used to ensure that the caches of the modules converge to the real state. After receiving the resource locking confirmation, the task is executed, the task state is monitored throughout the process, and the resource is released and the state is reported after the task is completed.

[0017] In summary, due to the adoption of the technical solutions, the application has the following beneficial effects: 1. The local cache with time-effectiveness deviation adopts a layered index structure and a double-mode update mechanism to balance the cache freshness and update overhead, support the scheduler to complete the calculation only by relying on the local data, and avoid real-time communication across nodes, so that the decision delay is lower.

[0018] 2. The layered deterministic rule engine and the Raft protocol are cooperated to build an efficient and reliable conflict resolution mechanism, so that the uniqueness and consistency of the global conflict resolution are ensured. BRIEF DESCRIPTION OF DRAWINGS

[0019] In the following description of exemplary embodiments in conjunction with the drawings, more details, features and advantages of the application are disclosed, and in the drawings: Figure 1 The system structure diagram of the application is shown in the figure; Figure 2 The method flowchart of the application is shown in the figure. DETAILED DESCRIPTION

[0020] Several embodiments of the application will be described in detail below with reference to the accompanying drawings so as to enable those skilled in the art to implement the application. The application can be embodied in many different forms and purposes and should not be limited to the embodiments described herein. These embodiments are provided to make the application comprehensive and complete, and to fully convey the scope of the application to those skilled in the art. The embodiments do not limit the application.

[0021] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0022] Embodiment 1 The detailed description is presented in conjunction with the Figure 1 and the accompanying drawings Figure 2 are incorporated in the detailed description.

[0023] The accompanying drawings Figure 1 is provided for the task scheduling system structure based on the distributed scheduling framework, wherein the connection relationship between the optimistic scheduling decision module and the final consistency state synchronization module is shown, and the main function interaction process of each module is marked.

[0024] The accompanying drawings Figure 2 is provided for the task scheduling method based on the distributed scheduling framework, which shows the complete steps from generating a globally unique scheduling proposal to receiving a resource locking confirmation to execute the task.

[0025] In this embodiment, it includes: The optimistic scheduling decision module is configured to generate a globally unique scheduling proposal in batches based on the local cache with time bias and the lock-free weighted scoring algorithm, and to break the dependence of traditional scheduling on global real-time consistent state in priority of throughput; Specifically, it includes: Function: Based on the local cache, the cluster state view allowing time bias (10-100ms delay) is generated in a lock-free, single-thread (or lightweight multi-thread) manner for the to-be-scheduled task to quickly generate 1-N scheduling proposals (the proposal contains task ID, target node ID, resource requirement list, logical timestamp, cache version number, etc. Metadata), and the entire decision-making process does not communicate with any other scheduler instance, central state storage or target node in real time, but only relies on local data for calculation.

[0026] Optimistic state cache: Cache data structure: hierarchical index and hash table combination structure, top-level index is divided according to resource type (CPU, memory, GPU, storage), and bottom-level hash table takes node ID as key to store node-level resource details (such as total CPU core number, used core number, remaining core number, total memory / available memory, GPU model / quantity / available state, node load threshold, task affinity label, etc.); Adopting a dual-mode update mechanism of timing pulling and incremental pushing; the timing pulling period is configurable (default 50 ms), and the full cluster state is pulled from the state synchronizer for cache calibration; the incremental pushing receives real-time resource change events (such as task start / end, node online / offline) issued by the state synchronizer to update the key data in the cache in real time, balancing cache freshness and update overhead; Expiry tolerance strategy: an expiry timestamp is attached to each node state in the cache, allowing the proposal to be generated based on the expired state when the cache does not trigger an update, but the proposal will be marked with a low confidence label, which can be used as an auxiliary basis for conflict resolution; Lock-free proposal algorithm: Core logic: a multi-objective weighted scoring algorithm is adopted, with task resource requirements (such as CPU 2 cores, memory 4 GB, GPU 1) and local cached node states as inputs, and the top 1-N target nodes with the highest scores as outputs; Scoring dimensions and weights: node resource remaining, node load rate, task and node affinity, node historical scheduling success rate, and weights support dynamic configuration to adapt to different business scenarios; Proposal marking mechanism: Global unique identification: proposal ID is generated by combining scheduler ID (8 bytes), local monotonic increasing counter (8 bytes), and timestamp (4 bytes), ensuring that proposal ID is not repeated in a distributed environment; Logical timestamp: generated based on physical timestamp and counter offset, with physical timestamp accurate to millisecond level, and counter offset used to solve the sorting problem of multiple proposals generated in the same millisecond, ensuring the global order of logical timestamp.

[0027] Cache version marking: records the cache snapshot version number (issued by the state synchronizer and incremented each time full update) relied on when generating the proposal, which is used to quickly locate whether the conflict is caused by cache expiration or resource competition when a conflict occurs.

[0028] Lock-free guarantee: only read local cache during algorithm execution, without any write operation or cross-thread communication; Parallel proposal distribution: Distribution mode: using non-blocking IO and batch sending mechanism, each scheduler instance maintains a proposal distribution thread pool (thread number = target node number / 100, default maximum 20 threads), and each thread is responsible for pushing proposals to a group of nodes; Reliability guarantee: one-time confirmation mechanism is attached when sending proposals, and the target node's resource coordinator needs to return a reception confirmation within 1 ms after receiving the proposal, and the proposal that does not receive the confirmation will be marked as a distribution failure, which will be automatically retried by the scheduler (with a maximum of 3 retries and an interval of 10 ms), avoiding proposal loss.

[0029] Resource coordinator and conflict detection module, configured to be deployed locally at the computing node, to atomically complete the resource pre-occupation, conflict detection and landing of the proposal through fine-grained resource locks and CAS atomic operations, to convert global conflict detection into efficient local operations; Specifically, it comprises: Receiving concurrent proposals from multiple optimistic scheduling decision modules, performing an atomic process of resource pre-occupation, conflict detection and confirmation landing for each proposal to ensure that resource allocation on the same node does not conflict; if a conflict is detected, immediately trigger conflict reporting and resource release to avoid deadlock or resource waste.

[0030] Proposal receiving queue: Queue type: priority blocking queue, queue capacity is configurable (default 1000), proposals are sorted in reverse order of logical timestamp (the larger the timestamp, the higher the priority), ensuring that high-priority task proposals are processed first; Traffic control: when the queue length reaches 80% of the threshold, send a traffic warning message to the scheduler, and the scheduler pauses sending new proposals to the node; when the queue length is less than 50% of the threshold, send a traffic recovery message, and the scheduler resumes proposal distribution to avoid node overload; Local resource lock manager: Lock type design: fine-grained exclusive lock, divided by resource type and resource unit; For example, CPU locks are divided by core ID (such as CPU0, CPU1, …, CPUn), memory locks are divided by memory block ID (each 4GB as a block), and GPU locks are divided by GPU device ID, ensuring that only proposals competing for the same resource unit will conflict.

[0031] The lock storage structure uses a lock pool implemented by a hash table and a doubly linked list. The hash table uses resource type and resource unit ID as keys to store lock state (free / occupied), occupied proposal ID, and occupation time information. The doubly linked list is used for fast traversal of all locks of the same resource type, supporting batch lock operations.

[0032] Lock operation optimization: lock acquisition and release use CAS (Compare-And-Swap) atomic operations to avoid the context switching overhead of traditional lock mechanisms; Atomic attempt and conflict detection process: Preprocessing stage: after the proposal is dequeued, first parse the resource requirement list in the proposal and convert it to a lock request list of resource type and resource unit ID (such as CPU locks: CPU2, CPU3; memory locks: MEM1; GPU locks: GPU0); Lock check stage: using batch lock attempt mechanism, based on the lock request list, try to acquire all required locks through the lock manager at one time: Implementation logic: traverse the lock request list, perform CAS operation on each lock to try to change the state from idle to occupied (proposal ID); if all CAS operations are successful, the locking is successful; if any CAS operation fails (the state is already occupied), the locking fails; Success path processing: Immediately update the local resource account (used resource amount, remaining resource amount) to ensure that the conflict detection of subsequent proposals is based on the latest state; Send a proposal confirmation message to the conflict resolution arbitration module, which contains the proposal ID, node ID, resource occupation details, logical timestamp, and cache version number; Send a proposal success confirmation to the proposal initiator (optimistic scheduling decision module) with a snapshot of the node resource account; Call the start interface of the local task executor, pass in the task configuration and resource allocation information, and trigger task loading; Conflict path processing: If the locking fails, immediately perform a lock rollback operation: traverse the successfully acquired locks and restore their state to idle through CAS operation to ensure that resources are not leaked; Collect conflict information: get the proposal ID that occupies the target resource (read from the occupied proposal ID field of the lock), combine the information of the current failed proposal, and generate a conflict event; Conflict event fields: conflict ID (globally unique), conflict node ID, conflict resource type, unit ID, conflict proposal list (containing the ID, logical timestamp, resource requirement, and cache version number of each proposal), conflict timestamp; Send the conflict event to the conflict resolution arbitration module, and send a proposal conflict failure message to the proposal initiator with the conflict reason.

[0033] The conflict resolution arbitration module is configured to use a logically centralized and physically distributed architecture to resolve global conflicts through a hierarchical deterministic rule engine, combined with reliable compensation instructions and the Raft protocol, to efficiently handle high-concurrency conflict events while ensuring eventual consistency; Specifically includes: High concurrency receives conflict events from all node resource coordinators, prioritizes the proposal list in each conflict event to determine the only winning proposal, and sends the resolution results and compensation instructions to related nodes and schedulers to coordinate resource state modification and failed proposal retry, ensuring the eventual consistency of conflict processing.

[0034] Conflict queue and collector: Collector architecture: adopt sharding collection, aggregation processing mode, each arbitrator instance is responsible for processing the conflict events of the corresponding preset shard (sharded by conflict ID hash), avoiding single instance overload; At the same time, deploy a global conflict aggregator to handle cross-shard associated conflicts (such as a proposal triggering conflicts in multiple nodes); Conflict queue design: Each shard corresponds to a lock-free ring queue, and the queue capacity can be configured to support high concurrency of conflict event enqueue and dequeue; The queue uses a producer-consumer model, where the enqueue thread is responsible for receiving conflict events, and the dequeue thread is responsible for processing arbitration to ensure process decoupling; Conflict deduplication: Before conflict event enqueue, deduplicate based on conflict ID and conflict proposal list hash to avoid repeated processing of the same conflict; At the same time, set a timeout time (default 5ms) for the conflict event, and conflicts that are not processed within the timeout will be prioritized for scheduling to ensure real-time performance; Rule system: Adopt a hierarchical rule system with primary rules and auxiliary rules, primary rules for determining core priority, and auxiliary rules for resolving flat scenarios where primary rules cannot distinguish, ensuring that all conflicts can have a unique arbitration result; Primary rule example: Logical timestamp rule: The proposal with the smallest logical timestamp wins (suitable for scenarios sensitive to task start order); Task priority rule: The proposal with the highest task priority value wins (task priority is pre-defined by the business layer, 0-10 levels, with level 10 being the highest); Resource matching degree rule: The proposal with the highest matching degree of resource demand and node remaining resources wins (matching degree = 1 - |node remaining resources - proposal demand resources| / node total resources).

[0035] Auxiliary rule example: Cache version number rule: The proposal with the latest cache version number wins (to resolve primary rule flat scenarios, prefer proposals generated based on newer states); Proposal ID hash rule: The proposal with the smallest proposal ID hash value wins (to ensure a final unique arbitration and avoid infinite flat scenarios).

[0036] All arbitrator instances use a unified rule configuration file, and configuration changes are synchronized through atomic pushing and gray-scale effective mechanisms to ensure consistent rules across the network; The rule engine uses deterministic calculation (the same input will always produce the same output) to avoid inconsistent arbitration results due to differences in execution environment.

[0037] Compensation instruction generator: Instruction type and field: Winning proposal confirmation instruction, sent to the resource coordinator of the conflict node, containing conflict ID, winning proposal ID, resource lock confirmation, triggering task execution; Failure proposal compensation instruction, sent to the initiator of the failure proposal (scheduler), contains conflict ID, failure proposal ID, judgment reason, compensation strategy (immediate retry / delayed retry / degraded retry), the latest cluster state snapshot (used to generate a better proposal when retrying); Resource state correction instruction, sent to the resource coordinator of the conflict node, used to correct the inconsistency of the resource account caused by the conflict (such as releasing the temporary resources occupied by the failure proposal); Instruction transmission guarantee: reliable message queue, timeout retry mechanism, instruction sending needs to receive confirmation, unconfirmed instructions will be retried (retry limit 5 times, interval 200 microseconds); At the same time, the instruction is attached with an idempotent identifier to ensure that the receiver will not be processed repeatedly when receiving repeatedly; The arbitrator cluster uses Raft consensus protocol to maintain the cluster state, ensuring that all instances reach an agreement on the conflict judgment result; Each arbitrator instance maintains a judgment log to record the judgment results of all conflicts, supports log playback when fault recovery, and ensures that the judgment state is not lost; When the arbitrator instance fails, a new master instance is elected by the Raft protocol to take over the shard processing.

[0038] The final consistent state synchronization module is configured based on distributed ordered log and publish-subscribe mode, and asynchronously increments the global state change. Through regular calibration and emergency calibration mechanism, it ensures that the local cache of each module converges to the real state; Specifically, it includes: Real-time collection of proposal confirmation events from resource coordinators, task lifecycle events (start / run / end / failure) from task executors, and node state events (online / offline / load exception) from node management modules. These events are encapsulated as ordered state change logs and synchronized to all optimistic scheduling decision modules and conflict resolution arbitration modules through asynchronous push mode, ensuring that each module works based on consistent global state. At the same time, it provides log persistence and rollback capabilities to support state recovery after system failure.

[0039] State change log: Segmented log storage is used, and the size of each segment log can be configured (default 64MB). Log entries include: log index (globally unique), event type, event timestamp, event metadata (such as proposal ID, node ID, resource change details, task state, etc.), checksum (for log integrity check); Event type classification: Resource allocation event: node resource occupation state change after proposal confirmation; Resource release event: node resource release state change after task end or failure; Node status events: node online / offline, load threshold adjustment, resource scaling up / down; Task status events: task start, running, successful end, failure termination.

[0040] After the log entry is written to memory, it is asynchronously flushed to distributed storage (such as HDFS, distributed K-V storage) with a flush delay of less than 10ms; at the same time, log expiration cleaning is supported (default retention period is 7 days) to avoid storage overload; Incremental synchronization stream: Synchronization stream architecture: based on the publish-subscribe mode, using Kafka as the underlying message queue, each event type corresponds to a topic (Topic), and the optimistic scheduling decision module and conflict resolution arbitration module act as subscribers to consume messages from the corresponding topic; Incremental synchronization mechanism: only the incremental part of the state change is synchronized, not the full state, for example, the resource allocation event only synchronizes the node ID, resource type, and occupancy, not the entire node resource account, reducing network transmission overhead.

[0041] Sequence guarantee: the message queue stores events in log index order, and the subscriber consumes them in order to ensure the timing consistency of state updates; at the same time, it supports breakpoint resuming, the subscriber records the latest consumed log index, and after failure recovery, it continues to consume from the index to avoid message loss or repeated consumption.

[0042] Cache asynchronous update process: The scheduler receives the synchronization stream message and puts the message into the local update task queue (non-lock queue); The background update thread pool (default 4 threads) takes tasks from the queue and processes them in log index order; Before processing, the message integrity is verified (through checksum), and then the corresponding field in the local cache is updated based on the event type (such as updating the used resource amount and remaining resource amount of the node for the resource allocation event); After the update is completed, the current log index for consumption is recorded for breakpoint resuming; Version number verification mechanism is used, each cache entry is attached with an update version number (consistent with the log index), and only when the log index of the message is greater than the current cache version number, the update is performed, avoiding old messages overwriting new states; at the same time, batch update is supported, when a certain number (default 100) of messages are accumulated in the update task queue, batch update operation is performed to improve efficiency; All schedulers send cache state snapshots to the state synchronizer at a preset time interval (time interval configurable), the state synchronizer compares the snapshot with the global real state, and if the deviation exceeds the threshold (default 5%), it sends a full synchronization instruction to the scheduler, which re-pulls the full cluster state to calibrate the local cache; When the conflict resolution arbitration module finds that the proposal conflict rate generated by a certain scheduler is too high (more than 20%), it triggers the state synchronizer to send an emergency calibration instruction to the scheduler, forcing the cache to be updated and reducing the subsequent conflict probability.

[0043] The task executor and the life cycle management module are configured to receive resource lock confirmation, load task running environment, allocate isolated resources, and execute tasks, monitor the task life cycle state throughout the process, release resources in time after the task ends and report the state, complete the scheduling closed loop, and specifically include: Task loading and resource allocation: Environment preparation: Support both containerization (Docker / K8s) and non-containerization deployment modes. In containerization mode, receive the container image address and start parameters passed by the resource coordinator, call the local container engine to pull the image and create a container, and bind the locked resources (such as limiting the number of CPU cores and memory size through cgroup, and binding specified GPU devices through GPU scheduling interface); In non-containerization mode, create a process directly and allocate resources through system calls.

[0044] Resource isolation guarantee: Use kernel-level resource isolation mechanisms to allocate independent resource namespaces (PID, network, mount point) for each task to avoid resource preemption or interference between tasks; At the same time, set the upper limit of resource usage (based on the resource requirements in the proposal), and when the task resource usage exceeds the upper limit, trigger an alarm and limit resource usage to prevent node overload.

[0045] Life cycle monitoring and state reporting: Monitoring indicators: Real-time collection of CPU utilization, memory usage, GPU usage, network IO, disk IO, task progress (reported by the task or based on output log analysis), error code, etc. The collection frequency is 100ms by default.

[0046] State machine design: The task life cycle includes 6 states: standby, starting, running, success, failure, and termination. The state transition needs to meet the preset rules (such as standby can only be converted to starting, and running can be converted to success, failure, or termination) to avoid state confusion.

[0047] State reporting mechanism: Key state changes (start success, run end, failure, termination) are reported to the state synchronizer in real time, with task ID, node ID, state type, timestamp, and error details (when failed); Periodic state reporting (default 1 second / second), report task running state to resource coordinator for resource coordinator to monitor task health; Reporting reliability: Adopt retry and bottom mechanism, automatically retry when reporting fails (up to 3 times), and write the state to the local log if the retry fails. When the node recovers, the report will be supplemented.

[0048] Resource release and exception handling: Normal release process: When the task is successfully completed or terminated normally, perform the resource release trilogy: release the kernel-level resource lock (cgroup, GPU binding, etc.); update the local resource account (reduce the used resource amount and increase the remaining resource amount); send a resource release confirmation to the resource coordinator to trigger the state synchronizer to update the global state.

[0049] Abnormal release process: When the task crashes or the node fails during execution, trigger passive release by the resource coordinator: detect the absence of the task process (through PID monitoring); forcibly release all resource locks occupied by the task; send a task failure and resource release event to the state synchronizer to ensure that resources are not occupied for a long time.

[0050] Bottom protection: The node locally maintains a resource occupation timeout checker, which periodically (default 10 seconds) scans all tasks allocated resources. If a task exceeds the preset timeout (configured by the business layer) and still does not end, automatically trigger resource release and task termination to avoid resource leakage.

[0051] The resource coordinator sends start / terminate instructions to the task executor through local inter-process communication (IPC) to avoid network overhead; the task executor feeds back the task status to the resource coordinator in real time, and the resource coordinator determines whether to allow new proposals to allocate resources based on the status. If the task failure rate on the node is too high, new proposals are temporarily suspended.

[0052] All status events reported by the task executor need to be verified by the resource coordinator (to ensure the authenticity of the status) and then forwarded to the state synchronizer by the resource coordinator to avoid interference with the global state by invalid or fake status events.

[0053] Embodiment 2 Please refer to Figure 2 , a task scheduling method based on a distributed scheduling framework, including the following parts: Based on the local cache with time bias and the lock-free weighted scoring algorithm, generate a globally unique scheduling proposal and batch distribute it to the target node; Through fine-grained resource locks and CAS atomic operations, atomically complete resource pre-occupation and conflict detection of the proposal. If there is no conflict, confirm the landing; if there is a conflict, report the conflict event; Adopt a hierarchical deterministic rule engine to resolve global conflicts, send the resolution results and compensation instructions to the relevant parties, and clearly identify the winning proposal and guide the failed proposal to retry; Based on distributed log and publish-subscribe mode, asynchronously incremental synchronization of global state changes, and through calibration mechanism to ensure that each module cache converges to the real state; Receive resource lock confirmation, execute tasks, monitor task status throughout the process, release resources and report status after task completion.

[0054] Example 3, collaborative workflow panoramic example: Suppose two scheduler instances S1 and S2 receive task scheduling requests (T1 and T2) almost simultaneously, and both select node N1 as the target node. The system collaborative process is as follows: Optimistic proposal generation and distribution: S1's local cache shows that N1 has CPU2 cores and GPU1 available (cache version number V10), generates proposal P1 (T1 to N1, CPU2 cores, GPU1, logical timestamp TS=100, proposal ID=S1-0001-1699999999), and sends it to N1's resource coordinator through parallel distribution threads; S2's local cache also shows that N1 has available resources (cache version number V10, no update triggered), generates proposal P2 (T2 to N1, CPU2 cores, GPU1, logical timestamp TS=102, proposal ID=S2-0001-1699999999), and synchronously sends it to N1's resource coordinator; Local atomic conflict detection: N1's resource coordinator receives P1 (TS=100) and P2 (TS=102) through a priority queue, and processes P1 in reverse order of timestamp priority; Processing P1: parse resource requirements as CPU2 / 3 cores and GPU0, try to acquire locks in batches, and CAS operations are all successful (lock state changes from idle to occupied-P1), update local resource account (CPU remaining 0 cores, GPU remaining 0), send P1 confirmation message to conflict resolution arbitration module, send successful confirmation to S1, and call task executor to start T1; Processing P2: parse resource requirements as CPU2 / 3 cores and GPU0, try to acquire locks, and the lock states of CPU2 / 3 cores and GPU0 are all occupied-P1, CAS operation fails; immediately roll back the locks that have been tried to acquire (no locks have been acquired), generate conflict event (conflict ID=C001, node N1, conflict resources CPU2 / 3, GPU0, conflict proposals P1 / TS100, P2 / TS102), and send it to the conflict resolution arbitration module; Global conflict resolution: The shard collector of the conflict resolution arbitration module receives conflict event C001, and hashes it to instance A according to conflict ID; The rule engine adopts the logical timestamp minimum win main rule to rule that P1 (TS=100) wins; Send P1 validation confirmation to N1 and compensation instruction (P2 conflict failure, compensation strategy: immediate retry, with the latest state snapshot-N1 CPU / GPU has been used up, N2 has available resources) to S2; State synchronization and proposal retry: The state synchronizer receives the P1 confirmation event of N1, generates a state change log (index=10001, event type=resource allocation, node N1, CPU occupation+2, GPU occupation+1), and pushes it to all schedulers through Kafka; S2 receives the compensation instruction and the state synchronization log, triggers local cache emergency calibration, updates the resource state of N1 to CPU 0 core available, GPU 0 available, and the resource state of N2 to CPU 4 core available, GPU 2 available; S2 regenerates the proposal P2' (T2 to N2, CPU 2 core, GPU 1, logical timestamp TS=103, proposal ID=S2-0002-1699999999) based on the updated cache for T2, and sends it to the resource coordinator of N2; The resource coordinator of N2 processes P2', successfully acquires the resource lock, confirms the proposal and starts T2, the state synchronizer updates the global state, and the process ends.

[0055] The above only describes some exemplary embodiments of the present application in a descriptive manner, and it is needless to say that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present application. Therefore, the above figures and description are illustrative in nature and should not be understood as limiting the scope of protection of the claims of the present application.

[0056] It should be noted that in this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by an utterance "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus including the element.

[0057] It should be understood that the size of the sequence number of the above processes does not mean the order of execution in various embodiments of the present application, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0058] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0059] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0060] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment according to actual needs.

[0061] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0062] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0063] The above only describes certain exemplary embodiments of the present application by way of illustration, and it is needless to say that those skilled in the art can modify the described embodiments in various ways without deviating from the spirit and scope of the present application. Therefore, the above figures and descriptions are illustrative in nature and should not be understood as limiting the scope of protection of the claims of the present application.

Claims

1. A task scheduling system based on a distributed scheduling framework, characterized in that, Comprise: Optimistic scheduling decision module, configured to generate globally unique scheduling proposal and batch distribute based on local cache with time bias and lock-free weighted scoring algorithm; Resource coordinator and conflict detection module, configured to be deployed locally on computing nodes, complete resource pre-occupation, conflict detection and landing of the proposal through fine-grained resource locks and CAS atomic operations; Conflict resolution arbitration module, configured to adopt a logically centralized and physically distributed architecture, resolve global conflicts through a hierarchical deterministic rule engine, and process high-concurrency conflict events while ensuring eventual consistency by combining reliable compensation instructions and Raft protocol; Eventual consistency state synchronization module, configured to synchronize global state changes asynchronously based on distributed ordered log and publish-subscribe mode, and ensure that local caches of each module eventually converge to the true state through regular calibration and emergency calibration mechanisms. 2.The task scheduling system based on the distributed scheduling framework of claim 1, wherein, The optimistic scheduling decision module specifically comprises: Optimistic state cache: Adopting a hierarchical index and hash table combination structure, the top-level index is divided by resource type, and the bottom-level hash table uses node ID as the key to store node-level resource details; Adopting a dual-mode update mechanism of regular pull and incremental push; an expiration timestamp is added to each node state in the cache, allowing the generation of proposals based on expired states when the cache is not updated, but the proposals will be marked with a low confidence label; Lock-free proposal algorithm: Core logic: adopt a multi-objective weighted scoring algorithm, input task resource requirements and local cache node state, output 1-N target nodes with the highest score; Scoring dimensions and weights: node resource remaining, node load rate, task and node affinity, node historical scheduling success rate, weight supports dynamic configuration to adapt to different business scenarios; Proposal marking mechanism: Global unique identifier: generate proposal ID by combining scheduler ID, local monotonic increment counter and timestamp; Logical timestamp: generated based on physical timestamp and counter offset; Lock-free guarantee: only read local cache during algorithm execution. 3.The task scheduling system based on the distributed scheduling framework of claim 1, wherein, The resource coordinator and conflict detection module specifically comprises: Adopting a priority blocking queue, proposals are sorted in reverse order of logical timestamp; Adopting fine-grained exclusive locks, lock granularity is divided by resource type and resource unit; Lock storage structure adopts lock pool implemented by hash table and double-linked list; Atomic attempt and conflict detection process: After the proposal is dequeued, first parse the resource requirement list in the proposal, and convert it to a lock request list of resource type and resource unit ID; Adopting batch lock attempt mechanism, based on the lock request list, try to acquire all required locks through the lock manager at one time. 4.The task scheduling system based on the distributed scheduling framework of claim 3, wherein, Further comprising: Success path processing: Immediately update local resource account book; Send proposal confirmation message to conflict resolution arbitration module; Send proposal success confirmation to proposal initiator, with node resource account book snapshot; Call the start interface of local task executor, passing task configuration and resource allocation information; Conflict path processing: If locking fails, immediately perform lock rollback operation; Get the proposal ID of the target resource currently occupied, combine it with the information of the current failed proposal to generate a conflict event; Send the conflict event to the conflict resolution arbitration module, and send the proposal conflict failure message to the proposal initiator. 5.The task scheduling system based on distributed scheduling framework of claim 1, wherein, The conflict resolution arbitration module specifically comprises: The conflict queue and collector: Adopting a sharding collection and aggregation processing mode, each arbitration instance is responsible for processing conflict events of the corresponding preset shard; a global conflict aggregator is deployed at the same time to process cross-shard associated conflicts; Each shard corresponds to a lock-free ring queue; the queue adopts a producer-consumer model, the enqueue thread is responsible for receiving conflict events, and the dequeue thread is responsible for processing the ruling, ensuring process decoupling; Before the conflict event is enqueued, deduplication is performed based on the conflict ID and conflict proposal list hash; at the same time, a timeout time is set for the conflict event, and the conflict that is not processed within the timeout time will be prioritized for scheduling; A hierarchical rule system with primary rules and auxiliary rules is adopted, the primary rules are used to determine the core priority, and the auxiliary rules are used to solve the tie scenario that the primary rules cannot distinguish; All arbitration instances adopt a unified rule configuration file. 6.The task scheduling system based on the distributed scheduling framework of claim 5, wherein, Further comprising: The winning proposal confirmation instruction is sent to the resource coordinator of the conflict node, including the conflict ID, the winning proposal ID, the resource locking confirmation, and triggers the task execution; The failed proposal compensation instruction is sent to the initiator of the failed proposal, including the conflict ID, the failed proposal ID, the ruling reason, the compensation strategy, and the latest cluster state snapshot; The resource state modification instruction is sent to the resource coordinator of the conflict node; A reliable message queue and a timeout retry mechanism are adopted, the receiver needs to return an acknowledgement after receiving the instruction, and the unacknowledged instruction will be retried; at the same time, an idempotent identifier is attached to the instruction; The arbitrator cluster adopts the Raft consensus protocol to maintain the cluster state; Each arbitration instance maintains a ruling log; When the arbitration instance fails, a new master instance is elected by the Raft protocol to take over the shard processing. 7.The task scheduling system based on the distributed scheduling framework of claim 1, wherein, The eventually consistent state synchronization module specifically comprises: Adopting segmented log storage; event type classification includes resource allocation events, resource release events, node state events, and task state events; After the log entry is written into the memory, it is asynchronously flushed to the distributed storage; Based on the publish-subscribe mode, Kafka is used as the underlying message queue, and each event type corresponds to a topic; Synchronize the incremental part of the state change; The message queue stores events in log index order, and the subscriber consumes them in order. 8.The task scheduling system based on the distributed scheduling framework of claim 7, wherein, Further comprising: Asynchronous cache update process: After the scheduler receives the synchronization flow message, the message is put into the local update task queue; The background update thread pool takes tasks from the queue and processes them in log index order; When processing, the integrity of the message is first verified, and then the corresponding field of the local cache is updated based on the event type; After the update is completed, the current consumed log index is recorded for breakpoint continuation. 9.The task scheduling system based on the distributed scheduling framework of claim 1, wherein, Further comprising: The task executor and life cycle management module is configured to receive resource locking confirmation, load task execution environment, allocate isolated resources, and execute tasks, monitor the task life cycle state throughout the process, release resources in time after the task ends, and report the state to complete the scheduling closed loop.

10. The task scheduling method based on the distributed scheduling framework according to any one of claims 1-9, characterized in that, Including: Based on the local cache with time-biased deviation and the lock-free weighted scoring algorithm, a globally unique scheduling proposal is generated and batch distributed to the target node; Through fine-grained resource locks and CAS atomic operations, the atomicization completes the resource pre-occupation and conflict detection of the proposal. If there is no conflict, it is confirmed to land; if there is a conflict, it reports a conflict event. A hierarchical deterministic rule engine is used to resolve global conflicts, send resolution results and compensation instructions to relevant parties, and clearly identify the winning proposal and guide the retry of failed proposals. Based on distributed logs and publish-subscribe mode, the global state changes are synchronized incrementally asynchronously, and the calibration mechanism is used to ensure that the module cache converges to the real state. After receiving the resource lock confirmation, the task is executed, the task status is monitored throughout the process, and the resources are released and the status is reported after the task is completed.

Images