A cloud platform-based distributed task scheduling method and system

Abstract

The application discloses a kind of distributed task scheduling method and system based on cloud platform, it is related to task scheduling technical field;Including the following steps: the full-dimension state data of all nodes of cloud platform is collected, unified resource view is constructed, task is collected and parsed, the data characteristics and directed acyclic graph of current submitted task are obtained, and three-level identification system is constructed;Its technical key points are: the present application is combined with cloud platform distributed architecture and scheduling decision, integrates cloud node and edge node resource to construct global resource pool, realizes resource centralized management and distributed call, breaks through single node resource limit, supports large-scale, high-concurrent task parallel execution, simultaneously, it can cope with mixed attribute task scheduling core demand and abnormal scene, obtains optimal scheduling scheme and closed-loop control mechanism, realizes seamless link and efficient cooperation in each link, realizes cloud and edge node collaborative scheduling, improves scheduling efficiency, resource utilization and system expansibility, with good use prospect.

Description

Technical Field

[0001] This invention relates to the field of task scheduling technology, specifically to a distributed task scheduling method and system based on a cloud platform. Background Technology

[0002] Task scheduling refers to the complete process of overall planning, sorting, allocation, and execution control of various tasks in a computing system based on pre-set rules, task characteristics, and resource status. Its core objective is to rationally allocate system resources, improve task execution efficiency, reduce execution costs, and ensure that all tasks are completed efficiently within constraints. The core elements of task scheduling include task queues, scheduling rules, resource pools, and execution monitoring. It is widely used in various computing scenarios, ranging from task sorting in single-machine systems to multi-task collaborative execution in cloud platforms and cluster systems.

[0003] A distributed system is a collaborative computing system consisting of multiple geographically dispersed but logically interconnected independent computing nodes connected by network links. Each node has independent hardware resources and software environment, enabling it to independently execute partial tasks. At the same time, it uses standardized communication protocols to achieve data interaction and resource sharing between nodes. The system is characterized by decentralized deployment and centralized management. Compared with single-machine systems, distributed systems can break the resource limitations of a single node, improve the overall computing power and storage capacity of the system through parallel processing of nodes, and have advantages such as high availability, scalability, and fault tolerance. Its core objective is to decompose complex overall tasks into subtasks that can be executed in parallel and distributed to different nodes to complete collaboratively. At the same time, it shields the complexity of communication and synchronization caused by the dispersion of nodes, ensuring that the overall system operates efficiently and stably to meet the needs of large-scale, high-concurrency, and high-reliability computing.

[0004] The invention patent with patent authorization announcement number CN111290846B describes a distributed task scheduling method, which includes: setting n unprocessed task storage areas in a cache, with each unprocessed task in the unprocessed task storage area being processed by a processing module; storing the unprocessed tasks in the unprocessed task storage areas respectively; reading the tasks in the unprocessed task storage areas respectively into memory; and running the unprocessed tasks in the memory, wherein n is the number of sub-cycles into which the task processing cycle is divided, and n is a natural number greater than 2. Compared with the prior art, the distributed task scheduling method provided by this application avoids task delays or queuing caused by a single processing module processing all tasks by classifying tasks and having multiple processing modules process different categories of tasks.

[0005] The solution described in the aforementioned patent has a fundamental flaw, specifically manifested as follows: The task classification method is simplistic and rigid, only supporting classification by processing time. It does not consider task data characteristics, dependencies, and priority differences, and cannot adapt to the scheduling requirements of mixed attribute tasks. This can easily lead to tasks with the same time but different priorities and complexities competing for resources, affecting the execution efficiency of core tasks. In addition, it does not dynamically adjust task allocation based on the real-time resource status of nodes. When tasks are concentrated in a certain storage area and the corresponding node is overloaded, it is impossible to achieve balanced resource allocation, and there is still a risk of local task delays.

[0006] In summary, existing distributed task scheduling methods do not meet market demands. Therefore, we propose a distributed task scheduling method and system based on a cloud platform. Summary of the Invention

[0007] To achieve the above objectives, the present invention provides the following technical solution: A distributed task scheduling method based on a cloud platform includes the following steps: Collect all-dimensional status data of all nodes on the cloud platform, build a unified resource view, collect and parse tasks, obtain the data characteristics and directed acyclic graph of the currently submitted tasks, and build a three-level identification system; The legality of task execution is verified. After the verification is successful, the priority of the task and its subordinate fragments and subtasks is calculated based on the three-level identification system and the response ratio priority scheduling algorithm to form a priority sorting table. The collaborative management strategy is then executed to build a task status system. Based on the dual-objective optimization scheme and dynamic adaptation adjustment scheme, the unified resource view, data characteristics, directed acyclic graph, priority sorting table and task status system are analyzed to generate the optimal partitioning scheme. An improved artificial immune algorithm is used to process the optimal partitioning scheme and unified resource view, and the execution order and node allocation of subtasks within the partitions are adjusted to obtain the optimal execution scheme of subtasks. The optimal partitioning scheme and the optimal execution scheme of subtasks are analyzed to obtain a comprehensive scheduling scheme. The system executes a comprehensive scheduling plan and collects execution data in real time. It then visualizes the data in conjunction with the task status system, triggers fault analysis strategies to conduct fault analysis, early warning, and handling, summarizes global results, generates a total task execution report, and archives the data.

[0008] Preferably, the full-dimensional status data covers the hardware status collected by hardware monitoring tools and the network status monitored by the full network link monitoring system. The hardware status includes CPU utilization, memory usage, remaining storage space, disk I / O rate, graphics processor computing load, and node runtime; the network status includes link bandwidth, transmission latency, packet loss rate, and link stability.

[0009] Preferably, the data features of the currently submitted task are collected and parsed, including: Extract full-dimensional information about the task-related data from the original task information and accompanying documentation submitted by the task submitter. This full-dimensional information includes the total amount of data, the granularity of data that can be divided, the storage location, the data type, the fragmentation association rules, the output format and storage address of the processing results, and the data verification rules. Obtain the task flow description provided by the task submitter, and extract the data related to subtask dependencies. The data related to subtask dependencies includes subtask name, computing power / storage requirements, list of pre- and post-subtasks, execution order requirements, time constraints, and number of retries. By using the dependency feature parsing logic, each subtask is regarded as a node in the graph. With the principle of pointing the previous subtask to the next subtask, directed connections between nodes are drawn to obtain a directed acyclic graph. Based on a directed acyclic graph, a unique identifier is assigned to each task, slice, and subtask to construct a three-level identification system.

[0010] Preferably, when using the response ratio priority scheduling algorithm to calculate the priority of a task and its subordinate fragments and subtasks, the response ratio of the task, each fragment, and each subtask is calculated. The response ratio is calculated as (waiting time + estimated execution time) / estimated execution time. Here, the waiting time is the cumulative time from task submission to the current moment, and the estimated execution time is the expected execution time calculated by a linear weighted algorithm based on the data size, subtask complexity, and real-time resource status. The tasks are arranged in descending order of response ratio to obtain a priority sorting table.

[0011] Preferably, the collaborative management strategy includes: Organize data characteristics, data related to subtask dependencies, and priority sorting tables; break down the overall task into several pieces and initially define the scope of each piece. Verify and process cross-shard dependencies, construct cross-shard dependency associations using the unique identifier of each shard, generate a cross-shard dependency mapping table, set the shard execution order, set the execution trigger condition of the subsequent shard to the completion status of the preceding shard, and plan the optimal cross-shard data transmission link. Based on the distributed lock mechanism of remote dictionary service, the system handles the competition and conflict between multiple shards and subtasks of the same resource. Resources are allocated according to the result of the priority sorting table, and a waiting timeout threshold is set. If the waiting time exceeds the waiting timeout threshold and the resource is not acquired, the resource reallocation is triggered, resulting in a three-level hierarchical structure of task, shard, and subtask.

[0012] Preferably, based on a dual-objective optimization scheme and a dynamic adaptation and adjustment scheme, the analysis of the unified resource view, data characteristics, directed acyclic graph, and priority ranking table includes: Integrate a unified resource view, data features, directed acyclic graph, priority sorting table, and task status system to build a basic dataset; The basic dataset is processed based on a dual-objective optimization scheme. The optimization objective and constraints are set, the weight coefficients of slice execution time, cross-slice dependency latency, and resource load imbalance are determined, and an optimization function is constructed with the aim of minimizing the scheduling optimization objective value. The optimization function is transformed by the Kuhn-Tak condition. Combining the data's divisibility, subtask complexity, and real-time resource status, a linear programming algorithm is used to calculate the ideal data volume for each partition, the subtask allocation scheme, and the preliminary allocation scheme for the target node. Based on a dynamic adaptation and adjustment scheme, combined with network status and node load, highly dependent shards are merged, and shards with excessively large data volume deviations are split or merged. The initial allocation scheme of target nodes is optimized to match the computing power and storage characteristics of shards and nodes, reserve redundant resources, and generate the optimal sharding scheme.

[0013] Preferably, the generated integrated scheduling scheme includes: Extract the dependencies of subtasks within each shard in the optimal sharding scheme and the full-dimensional state data of nodes in the unified resource view, and set the core parameters and iteration rules of the improved artificial immune algorithm; An improved artificial immune algorithm is used to optimize the execution order and node allocation of subtasks within the slice in the subtask allocation scheme through affinity calculation and clonal mutation iteration, and output the optimal execution scheme of the subtasks. The system calculates the load imbalance of nodes in real time, builds a dependency triggering mechanism, associates feedback data of sharding and subtask execution status, establishes a collaborative management and control framework, and outputs global collaborative scheduling rules. Based on the collaborative management and control framework, it handles resource contention and load imbalance issues, dynamically adjusts the scheduling scheme of unexecuted shards and subtasks, updates scheduling parameters synchronously, and outputs resource scheduling schemes. The optimal sharding scheme, the optimal execution scheme for subtasks, the resource scheduling scheme, and the global collaborative scheduling rules are combined to form a comprehensive scheduling scheme.

[0014] Preferably, a comprehensive scheduling scheme is implemented, and execution data is collected in real time and visualized in conjunction with the task status system, including: Organize and summarize the overall scheduling plan, the real-time resource status of nodes, and the network status; Based on the resource scheduling scheme, each shard and subtask is assigned to the corresponding node, the node execution environment adaptation work is started, the corresponding computing power and storage resources are allocated to each shard and subtask, and the node resource occupancy status is updated. The shards and subtasks are executed in parallel. Each node executes the subtasks within the shard according to the optimal execution plan of the subtask, verifies the execution results of the subtasks synchronously, collects the status data and data transmission logs during the execution process in real time, and feeds them back to the global control node. Handle subtask timeouts, execution failures, and exceptions according to global collaborative scheduling rules; Each node synchronously reports the shard execution results, integrates the execution data of all subtasks and shards, verifies whether the overall execution results meet the task objectives, updates the final task status in conjunction with the task status system, generates a task execution report, and provides a visual display in conjunction with the task status system.

[0015] Preferably, the fault analysis strategies for conducting fault analysis, early warning, and handling include: The real-time resource status of the acquisition nodes, the execution status of the shards and subtasks, and the cross-shard data transmission status constitute a dynamic data report; The collected data is verified in real time, and the preset thresholds are compared with the normal operation standards to identify whether there are node resource overloads, subtask timeouts or execution failures, or cross-shard transmission anomalies. If it does not exist, no action will be taken; if it exists, it will be considered abnormal, an alarm will be triggered, and the alarm information will be pushed to the control node. The control node combines dynamic data reports and a three-level hierarchical structure to locate faulty nodes, faulty segments and subtasks, distinguish fault types, and activate corresponding fault tolerance strategies. Execute the corresponding fault tolerance strategy, monitor the fault tolerance process synchronously, verify the fault tolerance effect, trace back the status data and execution logs related to the fault, record the fault handling details, and generate a fault tolerance report; Continuously monitor the running status of each node after the fault tolerance strategy is implemented, confirm that the anomaly has been cleared, and improve the dynamic data report and fault tolerance report; The task execution report, dynamic data report, and fault tolerance report are summarized to generate a total task execution report.

[0016] Preferably, a distributed task scheduling system based on a cloud platform includes a basic parsing module, a task analysis module, a solution analysis module, a solution improvement module, and a solution execution module; Basic parsing module: Collects full-dimensional status data of all nodes on the cloud platform, builds a unified resource view, collects and parses tasks, obtains the data characteristics and directed acyclic graph of the currently submitted tasks, and builds a three-level identification system; Task Analysis Module: Verifies the legality of task execution. After successful verification, it uses a response ratio priority scheduling algorithm based on a three-level identification system to calculate the priority of the task and its subordinate fragments and subtasks, forms a priority ranking table, executes collaborative management strategies, and constructs a task status system. Solution Analysis Module: Based on the dual-objective optimization scheme and dynamic adaptation adjustment scheme, it analyzes the unified resource view, data characteristics, directed acyclic graph, priority sorting table and task status system to generate the optimal sharding scheme; Solution Improvement Module: An improved artificial immune algorithm is used to process the optimal sharding scheme and unified resource view, adjust the execution order and node allocation of subtasks within the shards, obtain the optimal execution scheme of subtasks, analyze the optimal sharding scheme and the optimal execution scheme of subtasks, and obtain a comprehensive scheduling scheme. Solution Execution Module: Executes the comprehensive scheduling plan and collects execution data in real time. It visualizes the data in conjunction with the task status system, triggers fault analysis strategies to carry out fault analysis, early warning and handling, summarizes global results, generates a total task execution report, and archives the data.

[0017] This invention provides a distributed task scheduling method and system based on a cloud platform, which has the following beneficial effects: This invention employs a method combining task parsing and priority ranking. It collects and analyzes multi-dimensional feature data of tasks while introducing response ratio-priority scheduling logic. By comprehensively considering the urgency, execution complexity, and resource requirements of tasks, it ranks tasks hierarchically. This method can efficiently handle the needs of mixed-attribute tasks in terms of feature differences and priority differentiation, resulting in a three-level hierarchical structure and priority allocation table. It effectively solves the core problem of existing technologies that only classify tasks based on processing time without considering task priority and feature differences. This achieves refined task classification and precise priority control, avoiding disorderly competition for resources among tasks with different priorities and complexities. It ensures that core and urgent tasks are allocated resources and executed first, significantly improving the execution efficiency and on-time completion rate of core tasks.

[0018] This invention combines real-time node resource awareness with linear programming optimization. It collects node resource data in real time and synchronizes the collected data to the global management node to clarify the resources required for tasks. Based on the resource ledger and task requirements, it uses a linear programming optimization algorithm to iteratively calculate the relationship between tasks and nodes to obtain the optimal sharding scheme for tasks and nodes. According to the changes in node resource status, it adjusts the allocation scheme of unexecuted tasks in real time. This solves the problems of rigid resource allocation, binding of processing modules and storage areas, and unbalanced node load in the prior art, and achieves the maximization of resource utilization efficiency and dynamic balance of node load. It effectively reduces the risk of task delays caused by excessive local node load, while reserving redundant resources to enhance the flexibility and fault tolerance of resource allocation.

[0019] This invention combines directed acyclic graphs (DAGs) with collaborative scheduling. DAGs, addressing the cross-shard dependency characteristics of mixed-attribute tasks, intuitively present the cross-shard dependency logic while simultaneously verifying and eliminating potential circular dependencies. By combining real-time node resource status and network transmission data, a cross-shard data processing method is formulated, efficiently handling the dependencies, data transmission requirements, and differences in node communication states of cross-shard tasks. This results in a comprehensive cross-shard collaborative execution scheme and data encryption transmission strategy, clearly defining the execution sequence, data interaction path, and anomaly handling methods for each shard. This solves the problems of weak cross-shard collaboration capabilities, inability to handle multi-dependency tasks, and narrow adaptability in existing technologies. It achieves smooth collaboration across nodes and shards, effectively simplifying the management of complex dependencies, ensuring the security and real-time performance of cross-shard data interaction, and successfully adapting to scheduling scenarios for complex, multi-dependency, mixed-attribute tasks.

[0020] This invention combines multi-dimensional real-time monitoring with dynamic fault tolerance technology to monitor node operation, task execution, data transmission, and resource allocation status. It collects abnormal data and triggers alarms, monitors abnormal data and alarms, and accurately locates faults by combining task priorities, matches the optimal fault tolerance strategy, records the entire fault handling process, and backtracks task logs and status data to ensure traceability. The two technologies work together to efficiently handle anomalies and faults in task execution, obtain accurate location, fault tolerance solutions, and processing records, and solve the problems of imperfect fault tolerance and inaccurate location in existing technologies. It achieves rapid fault location, handling, and status backtracking, builds a fault tolerance closed loop, improves system stability and reliability, reduces the impact of faults, and ensures the smooth progress of tasks in abnormal scenarios.

[0021] This invention combines the distributed architecture and scheduling decisions of a cloud platform, integrating cloud and edge node resources to build a global resource pool. This enables centralized resource management and distributed invocation, breaking through the limitations of single-node resources and supporting the parallel execution of large-scale, high-concurrency tasks. Simultaneously, standardized communication protocols facilitate data interaction and collaborative linkage between nodes and modules, eliminating the complexity of communication and synchronization. The scheduling decision constructs a full-process intelligent scheduling framework, comprehensively handling the scheduling needs of mixed-attribute tasks, achieving intelligent control and optimization across all stages. The fusion of these two technologies addresses the core requirements and abnormal scenarios of mixed-attribute task scheduling, obtaining the optimal scheduling scheme and closed-loop control mechanism. This achieves seamless connection and efficient collaboration across all stages, not only solving the problems of narrow scenario adaptability and poor scalability in existing technologies, but also enabling collaborative scheduling between cloud and edge nodes, improving scheduling efficiency, resource utilization, and system scalability, demonstrating promising application prospects. Attached Figure Description

[0022] Figure 1 This is a flowchart of a distributed task scheduling method based on a cloud platform according to the present invention; Figure 2This is a structural block diagram of a distributed task scheduling system based on a cloud platform according to the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Example 1: Please see Figure 1 This embodiment provides a distributed task scheduling method based on a cloud platform, including the following steps: S1. Collect all-dimensional status data of all nodes on the cloud platform, build a unified resource view, collect and parse tasks, obtain the data characteristics and directed acyclic graph of the currently submitted tasks, and build a three-level identification system. The comprehensive status data covers hardware status collected by hardware monitoring tools and network status monitored by the network link monitoring system. Hardware status includes CPU utilization, memory usage, remaining storage space, disk I / O rate, GPU computing load, and node runtime; network status includes link bandwidth, transmission latency, packet loss rate, and link stability.

[0025] The data collection frequency is dynamically adjusted based on the node type. Cloud nodes collect data once every ten seconds by default, while edge nodes collect data once every thirty seconds by default. This ensures the real-time nature of information while avoiding excessive consumption of node resources. The above collection frequency is the optimal value obtained based on node resource consumption tests.

[0026] The data adopts a dual-protocol collaboration mechanism, using a multi-protocol boundary gateway protocol to achieve network-wide announcement of computing resource information, and adopts a publish / subscribe model of remote dictionary service stream message queue to achieve real-time push of resource status changes.

[0027] During use, analysis can be conducted based on historical data trends to predict potential problems such as node overload and link congestion in advance, and resource early warning information can be generated.

[0028] The data is collected and parsed to obtain the following characteristics of the currently submitted task: Extract comprehensive information about the task-related data from the original task information and accompanying documentation submitted by the task submitter. This comprehensive information covers the total amount of data, the granularity of data segmentation, storage location, data type, fragmentation association rules, output format and storage address of processing results, and data validation rules. The data segmentability and fragmentation relationships are highlighted. Obtain the task flow description provided by the task submitter, and extract data related to subtask dependencies. This data includes subtask name, computing power / storage requirements, list of preceding / following subtasks, execution order requirements, time constraints, and number of retries. Ensure that subtask dependencies are complete and free of loops. By using the dependency feature parsing logic, each subtask is regarded as a node in the graph. With the principle of pointing the previous subtask to the next subtask, directed connections between nodes are drawn to obtain a directed acyclic graph. Based on a directed acyclic graph, a unique identifier is assigned to each task, slice, and subtask to construct a three-level identification system.

[0029] The three-level identification system assigns a unique identifier to each task, fragment, and subtask, constructs a three-level association mapping table, and stores it in a distributed cache. This provides a unique identifier for subsequent scheduling, monitoring, and fault location. The identifier allocation rules refer to the existing distributed system unique identifier allocation standards.

[0030] S2. Verify the legality of task execution. After the verification is successful, calculate the priority of the task and its subordinate fragments and subtasks based on the three-level identification system and the response ratio priority scheduling algorithm to form a priority sorting table. Then, execute the collaborative management strategy and build a task status system. Verifying the legality of task execution includes checking the standardization of task format, data access permissions, and the reasonableness of time constraints. If the verification fails, an error message is sent to the task submitter, who can then readjust the task and resubmit it.

[0031] After successful verification, the task file and related data are stored in the distributed task database for easy subsequent processing.

[0032] When using the response ratio priority scheduling algorithm to calculate the priority of a task and its subordinate fragments and subtasks, the response ratio of the task, each fragment, and each subtask is calculated. The response ratio is calculated as (waiting time + estimated execution time) / estimated execution time. Here, the waiting time is the cumulative time from task submission to the current moment, and the estimated execution time is the expected execution time calculated by a linear weighted algorithm based on the data size, subtask complexity, and real-time resource status. The tasks are arranged in descending order of response ratio to obtain a priority ranking table.

[0033] The implementation of collaborative management strategies includes organizing data characteristics, data related to subtask dependencies, and priority sorting tables, breaking down the overall task into several segments, each segment containing a set of subtasks with clear directed acyclic graph dependencies, and initially defining the segment scope. Verify and process cross-shard dependencies, construct cross-shard dependency associations using the unique identifier of each shard, generate a cross-shard dependency mapping table, set the shard execution order, set the execution trigger condition of the subsequent shard to the completion status of the preceding shard, and plan the optimal cross-shard data transmission link. If there are subtask dependencies between different shards, a cross-shard dependency association is constructed using the shard's unique identifier, thereby generating a cross-shard dependency mapping table, setting the execution order of the shards, binding the execution triggering condition of the subsequent shard to the completion status of the preceding shard, and planning the optimal cross-shard data transmission link. Based on the real-time network status, the link with the shortest latency and the strongest stability is selected to reduce the latency of cross-shard data transmission.

[0034] Based on the distributed lock mechanism of the remote dictionary service, the system handles the competition and conflict between multiple shards and subtasks of the same resource. Resources are allocated according to the result of the priority sorting table. High-priority tasks can directly acquire resources and execute them, while low-priority tasks enter the waiting queue. When tasks have the same priority, they are sorted according to their submission time. A waiting timeout threshold is set. If the waiting time exceeds the waiting timeout threshold and the resource is not acquired, resource reallocation is triggered. According to the above allocation method, a three-level hierarchical structure of tasks, shards, and subtasks is obtained.

[0035] The task status system includes eight states: pending scheduling, processing in segments, subtasks in execution, subtask dependencies waiting, execution successful, execution failed, retrying, and archived. The initial state is pending scheduling. When the state changes, it will be synchronized to the distributed task database within 5 seconds to facilitate subsequent archiving and optimization.

[0036] S3. Based on the dual-objective optimization scheme and dynamic adaptation adjustment scheme, the unified resource view, data characteristics, directed acyclic graph, priority sorting table and task status system are analyzed to generate the optimal sharding scheme. Based on a dual-objective optimization scheme and a dynamic adaptation and adjustment scheme, the analysis of the unified resource view, data characteristics, directed acyclic graph, and priority ranking table includes: Integrate a unified resource view, data features, directed acyclic graph, priority sorting table, and task status system to build a basic dataset; The basic dataset is processed based on a dual-objective optimization scheme. The optimization objective and constraints are set, the weight coefficients of slice execution time, cross-slice dependency latency, and resource load imbalance are determined, and an optimization function is constructed with the aim of minimizing the scheduling optimization objective value. The dual-objective optimization scheme aims to maximize the efficiency of sharded parallelism and ensure the smooth execution of subtask dependencies. The constructed function is: Optimization Target Value = Shard Execution Time Weighting Coefficient × Average Execution Time of All Shards + Cross-Shard Dependency Latency Weighting Coefficient × Sum of Cross-Shard Dependency Latencies + Resource Load Imbalance Weighting Coefficient × Resource Load Imbalance. A smaller optimization target value indicates a more optimized scheme. The shard execution time weighting coefficient is set to 0.5, the cross-shard dependency latency weighting coefficient to 0.3, and the resource load imbalance weighting coefficient to 0.2. The constraints are: the sum of all shard data volumes equals the total amount of original task data; the shard size is between the minimum and optimal sharding granularity; the data volume of a single shard must be greater than or equal to the minimum sharding granularity; the current node load rate must be less than or equal to the rated load rate; and the number of cross-shard dependencies is less than a pre-set threshold.

[0037] By transforming the optimization function through the Kuhn-Tak condition, the objective function is transformed into a solvable form. Combining the data's separable granularity, subtask complexity, and real-time resource status, a linear programming algorithm is used to calculate the ideal data volume of each partition, the subtask allocation scheme, and the initial allocation scheme of the target nodes, thereby ensuring that the subtask dependencies within the partition are clear, the execution complexity is balanced, and the node load is initially balanced. Based on a dynamic adaptation and adjustment scheme, combined with network status and node load, highly dependent shards are merged, and shards with excessively large data volume deviations are split or merged. The initial allocation scheme of target nodes is optimized to match the computing power and storage characteristics of shards and nodes, reserve redundant resources, and generate the optimal sharding scheme.

[0038] For example, high-performance nodes are prioritized for data / computation-intensive shards; low-load nodes are prioritized for shards with complex dependencies; low-load nodes are those with a load of less than 50%; edge nodes are prioritized for shards closer to the data source. At the same time, 10% of node resources are reserved as redundant resources to cope with node failures or sudden task situations.

[0039] S4. An improved artificial immune algorithm is used to process the optimal sharding scheme and unified resource view, adjust the execution order and node allocation of subtasks within the shards, obtain the optimal execution scheme of subtasks, analyze the optimal sharding scheme and the optimal execution scheme of subtasks, and obtain a comprehensive scheduling scheme. The core of the improved artificial immune algorithm is the calculation of the affinity function, specifically: Scheduling scheme affinity = Execution time adaptation coefficient weight × Execution time adaptation coefficient + Transmission delay adaptation coefficient weight × Transmission delay adaptation coefficient + Resource utilization coefficient weight × Resource utilization coefficient. The higher the value of the scheduling scheme affinity, the better the scheme. Among them, the execution time adaptation coefficient weight is set to 0.4, the transmission delay adaptation coefficient weight is set to 0.3, and the resource utilization coefficient weight is set to 0.3.

[0040] The execution time adaptation coefficient is an indicator for evaluating the adaptability of subtask execution time. The calculation formula is: Execution time adaptation coefficient = (maximum time for node to process subtask - current node processing time + 1) / (sum of differences in processing time of all nodes + 1), where the maximum time for node to process subtask refers to the longest estimated time for all nodes to process subtask, the current node processing time refers to the estimated time for the current node to process subtask, and the sum of differences in processing time of all nodes is the cumulative value of the difference between the processing time of each node and the average processing time.

[0041] The transmission delay adaptation coefficient is an indicator for evaluating the latency adaptability of subtask data transmission. The calculation formula is: Transmission delay adaptation coefficient = (Maximum transmission delay between subtask and preceding subtask node - Current node transmission delay + 1) / (Sum of transmission delay differences of all nodes + 1), where the maximum transmission delay between subtask and preceding subtask node refers to the longest delay for transmitting subtask data between all nodes, the current node transmission delay refers to the transmission delay between the current node and the preceding subtask node, and the sum of transmission delay differences of all nodes is the cumulative value of the difference between the transmission delay of each node and the average transmission delay.

[0042] The resource utilization rate coefficient is an indicator for evaluating the resource utilization rate of a node. The calculation formula is: Resource utilization rate coefficient = Node idle resources / Node rated resources. The node idle resources refer to the total amount of resources that the node is currently not occupied, and the node rated resources refer to the maximum resource carrying capacity of the node.

[0043] The constraints are as follows: the values ​​of the execution time adaptation coefficient, transmission delay adaptation coefficient, and resource utilization coefficient are all between 0 and 1; the sum of the processing time differences of all nodes and the sum of the transmission delay differences of all nodes are greater than or equal to 0; the idle resource amount of a node is not higher than the rated resource amount of the node; the sum of the three weight coefficients is equal to 1; the scheduling scheme must strictly conform to the subtask dependency relationship of the directed acyclic graph.

[0044] The generated integrated scheduling scheme includes: Extract the dependencies of subtasks within each shard in the optimal sharding scheme and the full-dimensional state data of nodes in the unified resource view, and set the core parameters and iteration rules of the improved artificial immune algorithm; An improved artificial immune algorithm is used to optimize the execution order and node allocation of subtasks within the slice in the subtask allocation scheme through affinity calculation and clonal mutation iteration, and output the optimal execution scheme of the subtasks. Specifically, a three-dimensional mapping array of subtasks, execution order, and nodes is used to encode the scheduling scheme. The length of this array is equal to the total number of subtasks within the shard. Each element in the array contains a unique subtask identifier, an execution order number, and a unique target node identifier. Based on the hierarchical priority of subtasks in the directed acyclic graph and combined with the real-time resource status of the shard allocation nodes, an initial scheduling scheme population is generated. The population size is set to the number of subtasks multiplied by 5. Scheduling schemes that violate dependency constraints are eliminated. The affinity value of each scheduling scheme is calculated using the above formula. The top 30% of high-quality schemes and the bottom 30% of low-quality schemes are selected based on their affinity values. High-affinity schemes are cloned. Low-affinity schemes are subjected to a dependency-preserving mutation mechanism. Without disrupting the subtask dependencies, the execution order of subtasks or node allocation is replaced by a sliding window. Random mutation factors are introduced to increase population diversity. The cloning and mutation parameters are derived from algorithm optimization test data. Affinity calculation, cloning, and mutation operations are repeated until a preset number of iterations is reached or the affinity is greater than or equal to 0.9, at which point the iteration stops. The system calculates the load imbalance of nodes in real time, builds a dependency triggering mechanism, associates feedback data of sharding and subtask execution status, establishes a collaborative management and control framework, and outputs global collaborative scheduling rules. The node load imbalance is mainly calculated by the node load deviation. The node load imbalance is equal to the absolute value of the difference between the current load rate of the node and the average load rate of all nodes in the network, divided by the average load rate of all nodes. The larger the node load imbalance, the greater the node load deviation. The current node load rate is the ratio of the node's current resource usage to its rated resource usage. The average load rate of all nodes in the network is the arithmetic mean of the current load rates of all nodes.

[0045] The constraints are: the average load rate of all nodes is greater than 0; the current load rate of a node is between 0 and 1; the load imbalance threshold is 0.3; and the maximum load threshold of a node is 0.8.

[0046] Based on the collaborative management and control framework, it handles resource contention and load imbalance issues, dynamically adjusts the scheduling scheme of unexecuted shards and subtasks, updates scheduling parameters synchronously, and outputs resource scheduling schemes. The execution status of shards and subtasks is monitored in real time. Once a current subtask is completed, its completion status is immediately pushed to the corresponding node to trigger the execution of subsequent subtasks. Once all subtasks of a current shard have been completed, the execution of subsequent shards is triggered, and the execution status in the cross-shard dependency mapping table is updated synchronously. If a current subtask or shard fails to execute, the subsequent subtask or shard is triggered to enter a waiting state, and execution will continue after the fault is resolved.

[0047] The global load imbalance of each node is calculated in real time. When the global load imbalance of a node is greater than 0.3 or the current load rate of the node is greater than 0.8, the load adjustment mechanism is triggered to migrate the subtasks that have not yet been executed on the node to nodes with a load rate of less than 50%. During the migration process, the subtask dependencies are strictly maintained, and the task status and resource allocation information are updated synchronously. After the migration is completed, the node load is recalculated until load balancing is achieved.

[0048] Dynamic adjustments are made every fixed period, taking into account real-time resource status changes, task execution progress deviations, and task priority adjustments, to re-evaluate the scheduling scheme of shards and subtasks and dynamically adjust the unexecuted parts. During the adjustment process, priority is given to ensuring the execution of core subtasks and high-priority tasks to reduce the impact of adjustments on the overall task progress.

[0049] The optimal sharding scheme, the optimal execution scheme for subtasks, the resource scheduling scheme, and the global collaborative scheduling rules are combined to form a comprehensive scheduling scheme.

[0050] S5. Execute the comprehensive scheduling plan and collect execution data in real time. Combine the task status system for visualization display, trigger the fault analysis strategy to carry out fault analysis, early warning and handling work, summarize the global results, generate a total task execution report and archive it.

[0051] The system executes a comprehensive scheduling plan and collects execution data in real time, then visualizes the data in conjunction with the task status system, including: Organize a comprehensive scheduling plan and obtain the real-time resource status and network status of nodes; During use, the system monitors the running status of nodes in real time. If a node malfunctions, it is immediately marked as faulty and the corresponding fault handling process is triggered. When a node needs to be deregistered, any unexecuted tasks on the node are migrated first, and then the deregistration process is completed to avoid task interruption. Once the faulty node recovers, it needs to be re-registered and its status verified. After the verification is passed, the normal scheduling arrangement is restored. In addition, the system supports group management of nodes. Nodes can be grouped according to business scenarios and node types, which facilitates targeted allocation of shards and operation and maintenance management.

[0052] Based on the resource scheduling scheme, each shard and subtask is assigned to the corresponding node, the node execution environment adaptation work is started, the corresponding computing power and storage resources are allocated to each shard and subtask, and the node resource occupancy status is updated. When a subtask is executed, the optimal execution environment is selected based on the characteristics of the subtask to ensure execution efficiency. The optimal execution environment is one where the node is in normal condition, the node's resources match the sharding requirements, and the subtask requirements are met.

[0053] The shards and subtasks are executed in parallel. Each node executes the subtasks within the shard according to the optimal execution plan of the subtask, verifies the execution results of the subtasks synchronously, collects the status data and data transmission logs during the execution process in real time, and feeds them back to the global control node. After the subtask is completed, a result verification code is generated. This verification code is used to verify the integrity of the result. If the verification fails, a fault tolerance strategy is triggered.

[0054] After all subtasks within a slice are completed, the execution results of each subtask are merged and stored in a specified location according to a preset output format; a slice execution report is generated, which includes execution time, resource usage, result verification information, etc., and the slice execution report is fed back to the global control node.

[0055] When performing cross-shard data synchronization, once the preceding shard has finished executing, the sharding results will be synchronized to the execution nodes of the subsequent shards using a standardized interface based on the network communication framework and an advanced encryption standard. During the synchronization process, transmission logs need to be recorded. Once the data synchronization is complete and the verification is successful, the execution of the subsequent shard will be triggered, and the execution status in the cross-shard dependency mapping table will be updated synchronously. If data synchronization fails, a retransmission mechanism will be triggered; if retransmission fails, the corresponding fault tolerance strategy will be triggered.

[0056] Once all shards have been executed, it is necessary to perform global result summarization and archiving preparations, generate a task execution master report, synchronize it to the task database and user terminals, notify users that the task has been completed, and update the overall task status to execution success.

[0057] Handle subtask timeouts, execution failures, and exceptions according to global collaborative scheduling rules; Each node synchronously reports the shard execution results, integrates the execution data of all subtasks and shards, verifies whether the overall execution results meet the task objectives, updates the final task status in conjunction with the task status system, generates a task execution report, and provides a visual display in conjunction with the task status system.

[0058] Triggering fault analysis strategies to conduct fault analysis, early warning, and handling work includes: The real-time resource status of the acquisition nodes, the execution status of the shards and subtasks, and the cross-shard data transmission status constitute a dynamic data report; Specifically, it covers various indicators such as the load of all nodes, the overall execution progress of tasks, the processing status of each shard or subtask, network transmission latency, and resource utilization.

[0059] The collected data is verified in real time, and the preset thresholds are compared with the normal operation standards to identify whether there are node resource overloads, subtask timeouts or execution failures, or cross-shard transmission anomalies. The preset thresholds include node load threshold, subtask execution latency threshold, task failure retry threshold, node failure threshold, and network latency threshold.

[0060] By using line charts, bar charts, directed acyclic graphs, heatmaps, and other formats, monitoring indicators can be presented intuitively, enabling maintenance personnel to quickly grasp the system's operating status and task execution.

[0061] It can filter and query based on conditions such as task unique identifier, fragment unique identifier, subtask unique identifier, and time range, and can quickly locate the execution status and resource usage of a specific task, fragment, or subtask.

[0062] If it does not exist, no action will be taken; if it exists, it will be considered abnormal, an alarm will be triggered, and the alarm information will be pushed to the control node. The control node combines dynamic data reports and a three-level hierarchical structure to locate faulty nodes, faulty segments and subtasks, distinguish fault types, and activate corresponding fault tolerance strategies. Execute the corresponding fault tolerance strategy, monitor the fault tolerance process synchronously, verify the fault tolerance effect, trace back the status data and execution logs related to the fault, record the fault handling details, and generate a fault tolerance report; Continuously monitor the running status of each node after the fault tolerance strategy is implemented, confirm that the anomaly has been cleared, and improve the dynamic data report and fault tolerance report; The task execution report, dynamic data report, and fault tolerance report are summarized to generate a total task execution report.

[0063] If a subtask fails or times out, a retry mechanism will be automatically triggered, with a maximum of three retries. If the retry count reaches the task failure retry threshold and the task still fails, an alarm will be triggered and synchronized with the operations and maintenance personnel. If a shard fails to execute, a rescheduling process will be automatically triggered, reallocating nodes and re-executing the shard. Simultaneously, the results of completed subtasks within the shard will be retained to avoid duplicate execution, thus saving resources and time. When a node fails, the system will automatically migrate any unexecuted subtasks and shards on that node to a healthy node and re-execute them. The system will also update the scheduling scheme and task status to ensure uninterrupted task execution. If shard data is lost or corrupted during transmission, the system will automatically retransmit the data and perform verification. If execution results are lost, the system will automatically re-execute the relevant subtasks to ensure data integrity. If data verification fails, the system will trigger an alarm and indicate the specific cause of the failure. If a temporary failure occurs in the scheduling core layer, the system will automatically switch to a backup scheduling node and retain the current scheduling status. After the failure is resolved, the system will seamlessly resume operation without affecting task execution.

[0064] Implementing archiving facilitates regular review and optimization. During the review, it is necessary to collect key indicators such as task execution efficiency, resource utilization, failure rate, and cross-shard dependency latency, compare the execution performance of different scheduling schemes, and analyze the rationality of the scheduling schemes and algorithm parameters. For the problems found during the review, the parameters such as function weight, affinity function weight, load threshold, and sharding granularity in the bi-objective optimization scheme should be optimized. The scheduling algorithm should be iteratively upgraded to improve its adaptability and execution efficiency. The parameters, algorithms, and templates after the review and optimization should be updated simultaneously to ensure that the optimization effect is implemented in subsequent task scheduling.

[0065] Example 2: like Figure 2 As shown, a distributed task scheduling system based on a cloud platform includes a basic parsing module, a task analysis module, a scheme analysis module, a scheme improvement module, and a scheme execution module. Basic parsing module: Collects full-dimensional status data of all nodes on the cloud platform, builds a unified resource view, collects and parses tasks, obtains the data characteristics and directed acyclic graph of the currently submitted tasks, and builds a three-level identification system; Task Analysis Module: Verifies the legality of task execution. After successful verification, it uses a response ratio priority scheduling algorithm based on a three-level identification system to calculate the priority of the task and its subordinate fragments and subtasks, forms a priority ranking table, executes collaborative management strategies, and constructs a task status system. Solution Analysis Module: Based on the dual-objective optimization scheme and dynamic adaptation adjustment scheme, it analyzes the unified resource view, data characteristics, directed acyclic graph, priority sorting table and task status system to generate the optimal sharding scheme; Solution Improvement Module: An improved artificial immune algorithm is used to process the optimal sharding scheme and unified resource view, adjust the execution order and node allocation of subtasks within the shards, obtain the optimal execution scheme of subtasks, analyze the optimal sharding scheme and the optimal execution scheme of subtasks, and obtain a comprehensive scheduling scheme. Solution Execution Module: Executes the comprehensive scheduling plan and collects execution data in real time. It visualizes the data in conjunction with the task status system, triggers fault analysis strategies to carry out fault analysis, early warning and handling, summarizes global results, generates a total task execution report, and archives the data.

[0066] Example 3: A cloud-based distributed task scheduling system can be integrated into an infrastructure layer, a core business layer, and an auxiliary support layer. The infrastructure layer, as the underlying support of the entire system, provides basic resources such as computing, storage, and networking to ensure the normal operation of each upper-layer module. This layer includes a computing resource pool composed of virtual machines and containers, adapted to multi-language execution environments, and providing computing power support for task execution. The storage resource pool adopts a distributed storage method to store various types of data such as task data, execution logs, and archived information. The network support module provides long-connection communication capabilities to ensure low-latency transmission of instructions and data, and supports cross-module and cross-node collaboration.

[0067] The core business layer is responsible for completing the entire process of mixed-attribute tasks, from perception, preprocessing, scheduling to execution and archiving. It integrates the perception and acquisition layer, task preprocessing layer, scheduling decision layer, execution coordination layer, and archiving optimization layer from the original layered architecture, covering perception and acquisition modules, task preprocessing modules, scheduling decision-making modules, execution coordination modules, and archiving optimization modules. The perception and acquisition module consists of a global resource perception unit and a hybrid attribute task perception unit, responsible for collecting resource and task feature data; the task preprocessing module includes a task receiving and verification unit, a priority sorting unit, a fragmentation and dependency management unit, and a state control unit, responsible for task preprocessing and basic control; the scheduling decision module includes a fragmentation scheduling unit, a subtask scheduling unit, and a global collaborative scheduling unit, responsible for generating the optimal scheduling scheme; the execution collaboration module includes a node management unit, a task execution unit, a cross-fragment collaboration unit, and a communication collaboration unit, responsible for the implementation of the scheduling scheme; the archiving and optimization module includes a task archiving unit and a review optimization unit, responsible for the iterative optimization of task completion archiving and scheduling capabilities, corresponding to the content of Examples 1 and 2.

[0068] The auxiliary support layer provides full-process monitoring, fault handling, and user interaction support for the core business layer, ensuring system stability and controllability. It integrates the monitoring fault tolerance layer and user interaction entry point from the original layered architecture. The auxiliary support layer includes monitoring fault tolerance modules and user interaction entry points. The monitoring fault tolerance modules include a visual monitoring unit, a log alarm unit, and a fault tolerance unit, which are responsible for full-process monitoring, log recording, anomaly alarms, and fault handling. The user interaction entry point provides user operation entry points such as task submission, parameter configuration, status viewing, and alarm reception, realizing interactive linkage between the system and users.

[0069] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0070] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0071] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A distributed task scheduling method based on a cloud platform, characterized in that, Includes the following steps: Collect all-dimensional status data of all nodes on the cloud platform, build a unified resource view, collect and parse tasks, obtain the data characteristics and directed acyclic graph of the currently submitted tasks, and build a three-level identification system; The legality of task execution is verified. After the verification is successful, the priority of the task and its subordinate fragments and subtasks is calculated based on the three-level identification system and the response ratio priority scheduling algorithm to form a priority sorting table. The collaborative management strategy is then executed to build a task status system. Based on the dual-objective optimization scheme and dynamic adaptation adjustment scheme, the unified resource view, data characteristics, directed acyclic graph, priority sorting table and task status system are analyzed to generate the optimal partitioning scheme. An improved artificial immune algorithm is used to process the optimal partitioning scheme and unified resource view, and the execution order and node allocation of subtasks within the partitions are adjusted to obtain the optimal execution scheme of subtasks. The optimal partitioning scheme and the optimal execution scheme of subtasks are analyzed to obtain a comprehensive scheduling scheme. The system executes a comprehensive scheduling plan and collects execution data in real time. It then visualizes the data in conjunction with the task status system, triggers fault analysis strategies to conduct fault analysis, early warning, and handling, summarizes global results, generates a total task execution report, and archives the data.

2. The distributed task scheduling method based on a cloud platform according to claim 1, characterized in that: The comprehensive status data covers hardware status collected by hardware monitoring tools and network status monitored by the network link monitoring system. Hardware status includes CPU utilization, memory usage, remaining storage space, disk I / O rate, GPU computing load, and node runtime; network status includes link bandwidth, transmission latency, packet loss rate, and link stability.

3. The distributed task scheduling method based on a cloud platform according to claim 2, characterized in that: The data is collected and parsed to obtain the following data characteristics of the currently submitted task: Extract full-dimensional information about the task-related data from the original task information and accompanying documentation submitted by the task submitter. This full-dimensional information includes the total amount of data, the granularity of data that can be divided, the storage location, the data type, the fragmentation association rules, the output format and storage address of the processing results, and the data verification rules. Obtain the task flow description provided by the task submitter, and extract the data related to subtask dependencies. The data related to subtask dependencies includes subtask name, computing power / storage requirements, list of pre- and post-subtasks, execution order requirements, time constraints, and number of retries. By using the dependency feature parsing logic, each subtask is regarded as a node in the graph. With the principle of pointing the previous subtask to the next subtask, directed connections between nodes are drawn to obtain a directed acyclic graph. Based on a directed acyclic graph, a unique identifier is assigned to each task, slice, and subtask to construct a three-level identification system.

4. The distributed task scheduling method based on a cloud platform according to claim 1, characterized in that: When using the response ratio priority scheduling algorithm to calculate the priority of a task and its subordinate fragments and subtasks, the response ratio of the task, each fragment, and each subtask is calculated. The response ratio is calculated as (waiting time + estimated execution time) / estimated execution time. Here, the waiting time is the cumulative time from task submission to the current moment, and the estimated execution time is the expected execution time calculated by a linear weighted algorithm based on the data size, subtask complexity, and real-time resource status. The tasks are arranged in descending order of response ratio to obtain a priority ranking table.

5. The distributed task scheduling method based on a cloud platform according to claim 4, characterized in that: Implementing collaborative management strategies includes: Organize data characteristics, data related to subtask dependencies, and priority sorting tables; break down the overall task into several pieces and initially define the scope of each piece. Verify and process cross-shard dependencies, construct cross-shard dependency associations using the unique identifier of each shard, generate a cross-shard dependency mapping table, set the shard execution order, set the execution trigger condition of the subsequent shard to the completion status of the preceding shard, and plan the optimal cross-shard data transmission link. Based on the distributed lock mechanism of remote dictionary service, the system handles the competition and conflict between multiple shards and subtasks of the same resource. Resources are allocated according to the result of the priority sorting table, and a waiting timeout threshold is set. If the waiting time exceeds the waiting timeout threshold and the resource is not acquired, the resource reallocation is triggered, resulting in a three-level hierarchical structure of task, shard, and subtask.

6. The distributed task scheduling method based on a cloud platform according to claim 5, characterized in that: Based on a dual-objective optimization scheme and a dynamic adaptation and adjustment scheme, the analysis of the unified resource view, data characteristics, directed acyclic graph, and priority ranking table includes: Integrate a unified resource view, data features, directed acyclic graph, priority sorting table, and task status system to build a basic dataset; The basic dataset is processed based on a dual-objective optimization scheme. The optimization objective and constraints are set, the weight coefficients of slice execution time, cross-slice dependency latency, and resource load imbalance are determined, and an optimization function is constructed with the aim of minimizing the scheduling optimization objective value. The optimization function is transformed by the Kuhn-Tak condition. Combining the data's divisibility, subtask complexity, and real-time resource status, a linear programming algorithm is used to calculate the ideal data volume for each partition, the subtask allocation scheme, and the preliminary allocation scheme for the target node. Based on a dynamic adaptation and adjustment scheme, combined with network status and node load, highly dependent shards are merged, and shards with excessively large data volume deviations are split or merged. The initial allocation scheme of target nodes is optimized to match the computing power and storage characteristics of shards and nodes, reserve redundant resources, and generate the optimal sharding scheme.

7. A distributed task scheduling method based on a cloud platform according to claim 6, characterized in that: The generated integrated scheduling scheme includes: Extract the dependencies of subtasks within each shard in the optimal sharding scheme and the full-dimensional state data of nodes in the unified resource view, and set the core parameters and iteration rules of the improved artificial immune algorithm; An improved artificial immune algorithm is used to optimize the execution order and node allocation of subtasks within the slice in the subtask allocation scheme through affinity calculation and clonal mutation iteration, and output the optimal execution scheme of the subtasks. The system calculates the load imbalance of nodes in real time, builds a dependency triggering mechanism, associates feedback data of sharding and subtask execution status, establishes a collaborative management and control framework, and outputs global collaborative scheduling rules. Based on the collaborative management and control framework, it handles resource contention and load imbalance issues, dynamically adjusts the scheduling scheme of unexecuted shards and subtasks, updates scheduling parameters synchronously, and outputs resource scheduling schemes. The optimal sharding scheme, the optimal execution scheme for subtasks, the resource scheduling scheme, and the global collaborative scheduling rules are combined to form a comprehensive scheduling scheme.

8. The distributed task scheduling method based on a cloud platform according to claim 7, characterized in that: The system executes a comprehensive scheduling plan and collects execution data in real time, then visualizes the data in conjunction with the task status system, including: Organize and summarize the overall scheduling plan, the real-time resource status of nodes, and the network status; Based on the resource scheduling scheme, each shard and subtask is assigned to the corresponding node, the node execution environment adaptation work is started, the corresponding computing power and storage resources are allocated to each shard and subtask, and the node resource occupancy status is updated. The shards and subtasks are executed in parallel. Each node executes the subtasks within the shard according to the optimal execution plan of the subtask, verifies the execution results of the subtasks synchronously, collects the status data and data transmission logs during the execution process in real time, and feeds them back to the global control node. Handle subtask timeouts, execution failures, and exceptions according to global collaborative scheduling rules; Each node synchronously reports the shard execution results, integrates the execution data of all subtasks and shards, verifies whether the overall execution results meet the task objectives, updates the final task status in conjunction with the task status system, generates a task execution report, and provides a visual display in conjunction with the task status system.

9. A distributed task scheduling method based on a cloud platform according to claim 8, characterized in that: Triggering fault analysis strategies to conduct fault analysis, early warning, and handling work includes: The real-time resource status of the acquisition nodes, the execution status of the shards and subtasks, and the cross-shard data transmission status constitute a dynamic data report; The collected data is verified in real time, and the preset thresholds are compared with the normal operation standards to identify whether there are node resource overloads, subtask timeouts or execution failures, or cross-shard transmission anomalies. If it does not exist, no action will be taken; if it exists, it will be considered abnormal, an alarm will be triggered, and the alarm information will be pushed to the control node. The control node combines dynamic data reports and a three-level hierarchical structure to locate faulty nodes, faulty segments and subtasks, distinguish fault types, and activate corresponding fault tolerance strategies. Execute the corresponding fault tolerance strategy, monitor the fault tolerance process synchronously, verify the fault tolerance effect, trace back the status data and execution logs related to the fault, record the fault handling details, and generate a fault tolerance report; Continuously monitor the running status of each node after the fault tolerance strategy is implemented, confirm that the anomaly has been cleared, and improve the dynamic data report and fault tolerance report; The task execution report, dynamic data report, and fault tolerance report are summarized to generate a total task execution report.

10. A distributed task scheduling system based on a cloud platform, characterized in that, include: Basic parsing module: Collects full-dimensional status data of all nodes on the cloud platform, builds a unified resource view, collects and parses tasks, obtains the data characteristics and directed acyclic graph of the currently submitted tasks, and builds a three-level identification system; Task Analysis Module: Verifies the legality of task execution. After successful verification, it uses a response ratio priority scheduling algorithm based on a three-level identification system to calculate the priority of the task and its subordinate fragments and subtasks, forms a priority ranking table, executes collaborative management strategies, and constructs a task status system. Solution Analysis Module: Based on the dual-objective optimization scheme and dynamic adaptation adjustment scheme, it analyzes the unified resource view, data characteristics, directed acyclic graph, priority sorting table and task status system to generate the optimal sharding scheme; Solution Improvement Module: An improved artificial immune algorithm is used to process the optimal sharding scheme and unified resource view, adjust the execution order and node allocation of subtasks within the shards, obtain the optimal execution scheme of subtasks, analyze the optimal sharding scheme and the optimal execution scheme of subtasks, and obtain a comprehensive scheduling scheme. Solution Execution Module: Executes the comprehensive scheduling plan and collects execution data in real time. It visualizes the data in conjunction with the task status system, triggers fault analysis strategies to carry out fault analysis, early warning and handling, summarizes global results, generates a total task execution report, and archives the data.

Images