Computation-storage flow joint scheduling optimization method and system based on DDQN and heuristic strategy

Abstract

The present invention relates to a computation-storage flow joint scheduling optimization method and system based on DDQN and heuristic strategies, belonging to the technical field of distributed computing and intelligent scheduling, and includes: (1) computation flow and storage flow definition; (2) workflow scheduling and execution: when a workflow is triggered, the scheduler evaluates the executable status of each task one by one according to the dependencies between tasks and the scheduling strategy, and submits the executable tasks to the task executor; the task executor is responsible for executing these tasks on the working nodes, ensuring that the tasks are completed in the predetermined order and priority; (3) DAG structure, queue information and running number collection; (4) task running time and resource usage prediction based on historical running data; (5) scheduling optimization based on DDQN and heuristic strategies; (6) scheduling information acquisition and storage. The present invention significantly improves the task execution efficiency and resource utilization of the system.

Description

Technical Field

[0001] The present application belongs to the field of distributed computing and intelligent scheduling technology, and specifically relates to a computing-storage flow joint scheduling optimization method and system based on DDQN and heuristic strategies. Background Art

[0002] In large-scale distributed computing environments, task scheduling is central to efficient system operation. Its performance directly determines computing efficiency, resource utilization, and quality of service assurance. With the explosive growth of data and the continuous increase in computing density, scheduling systems are facing challenges such as high concurrency of task requests, complex dependencies, and frequent resource status fluctuations. In this context, traditional static scheduling strategies, lacking the ability to perceive and respond to dynamic environmental changes, are unable to meet the dual requirements of high performance and high resilience required by modern systems.

[0003] Taking Airflow as an example, while it supports flexible DAG workflow orchestration and task dependency management, its scheduling strategy is still primarily based on static rules, making it difficult to dynamically perceive changes in resource load and task execution status. This makes it difficult to achieve optimal scheduling performance in complex environments. Therefore, introducing an adaptive intelligent scheduling mechanism is a key path to improving system resource scheduling efficiency and task execution quality, and is a core direction for supporting efficient task scheduling in heterogeneous resource environments in the future.

[0004] Especially in the new data-centric computing paradigm, factors such as data location, storage media, access paths, and transmission latency are becoming key variables affecting scheduling efficiency. The optimization goals of scheduling systems are also shifting from traditional computing resource-oriented approaches to dynamic perception and intelligent modeling of data flows throughout their lifecycle. Data is no longer just an input for task execution; it is also the core driver of scheduling decisions. Its state changes directly impact the rationality of scheduling and overall system performance.

[0005] Airflow's scheduling mechanism is primarily based on static policies and lacks the ability to adapt to dynamic changes in the operating environment. This makes it difficult to meet the high-performance and high-elasticity scheduling requirements of large-scale distributed systems. Specific issues are mainly reflected in the following three aspects:

[0006] 1. Lack of dynamic perception: Airflow's existing scheduling mechanism is unable to perceive changes in resource load, the evolution of task dependency structures, and real-time fluctuations in the scheduling environment. This results in low resource utilization and high task latency in high-concurrency, multi-task scenarios, affecting overall scheduling efficiency.

[0007] 2. Data and Computing Disconnect: Traditional scheduling strategies generally treat data as static resources, ignoring key attributes such as its location, access frequency, storage media, and migration costs. This makes it difficult to achieve coordination between data flow and computing scheduling. This disconnect can easily lead to data access bottlenecks and scheduling imbalances in data-driven tasks, severely hampering system performance.

[0008] 3. Single state modeling dimension: Some existing reinforcement learning-based scheduling methods only use shallow features such as task length and CPU usage in state representation, ignoring complex dependencies between tasks, critical path structure, and resource topology information. This results in weak policy generalization capabilities and low decision-making accuracy, making it difficult to adapt to heterogeneous resource environments and dynamic load scenarios. Summary of the Invention

[0009] In response to the shortcomings of the existing technology, this application proposes a computation-storage flow joint scheduling optimization method based on DDQN and heuristic strategies.

[0010] In this application, the traditional workflow is decoupled and explicitly divided into computational and storage flows to achieve collaborative optimization of task execution and data management. Simultaneously, the DDQN state representation and action space are optimized, integrating task dependency structures, resource status, and historical scheduling records to achieve deep modeling and perception of the environment. Ultimately, by integrating DDQN with heuristic strategies, combining Q-value evaluation with rule guidance, the optimal queue is dynamically selected for each task, achieving the goals of task load balancing and minimizing execution time.

[0011] This method jointly models computational and storage flows, optimizes state representation and action space, and comprehensively considers task dependency structures, queue resource status, and historical scheduling information. Based on this, it makes scheduling decisions based on DDQN and heuristic strategies. By integrating Q-value evaluation with heuristic rules, it selects the optimal execution queue for each task, achieving task load balancing and minimizing execution time.

[0012] This application also proposes a computing-storage flow joint scheduling optimization system based on DDQN and heuristic strategies.

[0013] The technical solution of this application is:

[0014] The computation-storage flow joint scheduling optimization method based on DDQN and heuristic strategy includes:

[0015] (1) Definition of computation flow and storage flow;

[0016] (2) Workflow scheduling and execution: When a workflow is triggered, the scheduler evaluates the executable status of each task based on the dependencies between tasks and the scheduling strategy, and submits the executable tasks to the task executor; the task executor is responsible for executing these tasks on the work nodes, ensuring that the tasks are completed in the predetermined order and priority.

[0017] (3) DAG structure, queue information and run number collection;

[0018] (4) Task running time and resource usage prediction based on historical running data;

[0019] (5) Scheduling optimization based on DDQN and heuristic strategies;

[0020] (6) Obtaining and saving scheduling information.

[0021] According to the preferred embodiment of the present application, in step (1), the calculation flow and storage flow definition include:

[0022] First, the DAG model is used to uniformly model the entire task process. In the DAG, each node represents an independent task, and the directed edges between nodes represent the dependencies between tasks.

[0023] Based on the functional attributes and resource usage types of tasks, tasks are logically divided into two sub-flows:

[0024] Computational flow: includes the execution of operator logic and the scheduling and allocation of computing resources;

[0025] Storage flow: including the storage, reading, transmission and management of data that tasks depend on;

[0026] All task nodes are declared and parameterized through Python script application;

[0027] The computation flow and storage flow implement a linkage trigger mechanism through TriggerDagRunOperator: when the computation task is completed, the corresponding storage task is automatically triggered to ensure the timing connection of the processing flow; at the same time, ExternalTaskSensor is introduced to achieve cross-flow dependency synchronization: the computation flow task monitors the status changes of specific storage tasks to determine their start time.

[0028] According to the preferred embodiment of the present application, in step (2), workflow scheduling and execution specifically include:

[0029] When an Airflow workflow is triggered, the scheduler evaluates the status of each task based on the task dependencies and scheduling policies defined in the DAG. The scheduler checks which tasks meet the dependency conditions and determines whether the scheduling time window has been reached. Eligible tasks are marked as schedulable and placed in the task queue, awaiting processing by the task executor.

[0030] The executor obtains tasks from the task queue and assigns the tasks to the corresponding worker nodes or containers for execution;

[0031] During the execution process, the task status is updated in real time. If a task fails, the scheduler reschedules the task according to the task's retry strategy. Throughout the entire process, the scheduler ensures that the tasks in the DAG are executed strictly according to the defined dependency order and priority order until all tasks are completed, thus completing the entire workflow scheduling and execution loop.

[0032] According to the preferred embodiment of the present application, in step (3), the DAG structure, queue information and operation data collection include:

[0033] In terms of DAG structure, the following information is collected and organized for each task node: dag_id, task_id, whether it is a critical path task, whether it belongs to a compute flow task, whether it is a TriggerDagRunOperator task that triggers a storage flow, the target DAG it triggers, whether it is an ExternalTaskSensor type task, the list of DAGs and tasks it depends on, and the overall DAG adjacency table structure between the compute flow and the storage flow; dag_id refers to the DAG id, and task_id refers to the task id;

[0034] In terms of task running data, i.e., running data, the running status of historical tasks is recorded, including: task_id, dag_id, run_id, job_id, pid, avg_cpu, min_cpu, max_cpu, avg_mem, min_mem, max_mem, rss, file_size, and duration;

[0035] run_id refers to the DAG instance run ID, job_id refers to the task run ID generated by the scheduler, pid refers to the task process ID, avg_cpu refers to the average CPU usage, min_cpu refers to the minimum CPU usage, max_cpu refers to the maximum CPU usage, avg_mem refers to the average memory usage, min_mem refers to the minimum memory usage, max_mem refers to the maximum memory usage, rss refers to the resident memory set size, file_size refers to the file size, and duration refers to the task execution duration;

[0036] In terms of queue resources, i.e. queue information, each execution queue is collected, including: queue_id, queue_code, currently available CPU, memory, storage, and I / O bandwidth, to build a real-time mapping relationship between tasks and execution resources;

[0037] queue_id indicates the queue name, queue_code indicates the queue number;

[0038] Save the collected DAG structure and queue resources to a JSON file, and save the task running data to the airflow_task_stats table in the MySQL database.

[0039] According to the preferred embodiment of the present application, in step (4), the task running time and resource usage prediction based on historical running data includes:

[0040] (4-1) Set up the basic environment for system operation, including the model storage path and operation log configuration; obtain historical task data from the airflow_task_stats table in the MySQL database, including the following fields: task_id, dag_id, run_id, avg_cpu, avg_mem, rss, file_size, and duration;

[0041] Perform data preprocessing on historical data, including type conversion and missing value filling. Convert historical task data to numeric type and save it in MySQL database, as shown in formula (1). Fill missing values ​​with 0, as shown in formula (2):

[0042] X clean =pd.to_numeric(X raw ,errors='coerce') (1);

[0043] X fill =X clean *fillna(0) (2);

[0044] Among them, X raw represents the original input data, X clean Indicates the converted numeric data, and illegal values ​​are converted to NaN; pd.to_numeric() indicates the function provided by Pandas, which is used to convert data into numeric types; errors = 'coerce' indicates that when the data cannot be converted into a numeric value, it is forcibly converted to NaN, that is, missing values; X fill Indicates the value after filling the missing values. fillna(0) means filling all missing values ​​with 0.

[0045] (4-2) Using the XGBoost regression model, three models are trained independently for each task, and three XGBoost regression models are deployed independently for each task_id to predict: running time, CPU usage, and memory usage, as shown in formula (3). The generated regression model is persisted and saved as a pkl file;

[0046]

[0047] Among them, f time () represents the trained XGBoost regression model used to predict the running time of the task; f cpu () represents the trained XGBoost regression model for predicting CPU usage; f mem () represents the trained XGBoost regression model for predicting memory usage; represents f time () predicts the running time of the task, represents f cpu () predicted CPU usage, represents f mem () predicted memory usage;

[0048] The Optuna framework is used for Bayesian optimization to search for the optimal hyperparameter combination. The goal is to minimize the mean absolute error and minimize the square error when training the XGBoost regression model, as shown in formula (4):

[0049]

[0050] Among them, y i represents the true value of the i-th sample, The predicted value of the i-th sample, α represents the L1 regularization coefficient, λ represents the L2 regularization coefficient, γ represents the minimum loss drop threshold for controlling tree splitting, T represents the number of leaf nodes, w represents the leaf node weight vector; ‖‖1 represents the L1 norm, that is, the sum of the absolute values ​​of all leaf node weights; Represents the square of the L2 norm, that is, the sum of the squares of the weights of all leaf nodes; represents the total loss function;

[0051] (4-3) Load the latest prediction model according to the model storage path. The data in the airflow_task_stats table of the MySQL database is divided into datasets by task_id. The input features are [avg_cpu, avg_mem, rss, file_size], and the output targets are duration, avg_cpu, and rss.

[0052] The latest record prediction for each task is shown in formula (5):

[0053]

[0054] Among them, X new represents the latest task feature vector, Indicates the predicted value, which is time, CPU or memory; f XGBoost () represents the trained XGBoost regression model function;

[0055] (4-4) Scan the configuration file directory and write the predicted values ​​into the predicted_time, predicted_cpu, and predicted_memory fields of the JSON file.

[0056] According to the preferred embodiment of the present application, in step (5), scheduling optimization based on DDQN and heuristic strategy includes:

[0057] (5-1) Construct the four-dimensional state tensor S, as shown in formula (6):

[0058]

[0059] Among them, B represents the batch size, N t Indicates the total number of tasks, N q represents the total number of queues, D represents the feature dimension, and D is D task +D queue ;

[0060] Task Feature D task 9+N t , as shown in formula (7):

[0061]

[0062] Among them, w represents the task weight, q assigned Indicates the assigned queue number. Indicates whether it is a critical path task. Indicates whether it is a computational flow task. Indicates whether it is a trigger task. Indicates whether it is a sensor task. represents the predicted running time, CPU, memory, Represents one-hot encoding of dependent tasks;

[0063] Queue characteristics: D queue , as shown in formula (8):

[0064]

[0065] Among them, cavail Indicates the available CPU resources of the queue, c total Indicates the total CPU resources of the queue, m avail Indicates the available memory of the queue, m total Indicates the total queue memory, s avail Indicates the available storage for the queue, s totals Indicates the total storage of the queue, n tasks Indicates the number of tasks assigned to the queue, t end Indicates the end timestamp of the current task in the queue;

[0066] (5-2) Perform multiple rounds of Episode training until the termination condition is met; each Episode includes the following steps:

[0067] Environment reset: restore the simulation environment to its initial state and clear previous interaction history data;

[0068] Mission completion judgment: Check whether the current Episode has completed all preset mission objectives. If the mission has been completed, skip the subsequent steps and go directly to the next Episode;

[0069] (5-3) In the current Episode, perform the following operations according to the time step:

[0070] Available action detection: Obtain a set of legal actions based on the current state of the environment;

[0071] Obtain legal actions from the legal action set and verify the key function V(a) of the legality of the action, as shown in formula (9):

[0072]

[0073] Among them, a represents the task t i The specific operation of t i Indicates the current task to be scheduled, deps(t i ) represents the dependency set, t d Indicates dependent tasks, status(t d ) indicates the task status;

[0074] According to the mixed epsilon-greedy and heuristic rule a, as shown in formula (10), the optimal action is selected;

[0075]

[0076] Among them, ∈ represents the exploration rate, θ online Indicates online network parameters, represents the set of legal queues for task i;

[0077] j represents an action in the legal queue set, Q represents the Q-value function in reinforcement learning, and Heuristic represents the heuristic strategy function;

[0078] The heuristic rule prioritizes assigning tasks to queues with high current load but low demand matching to avoid resource overload, as shown in formula (11):

[0079]

[0080] For each queue j, calculate its load score LoadScore j , as shown in formula (12):

[0081]

[0082] Indicates the CPU importance weight, Indicates the memory importance weight;

[0083] CPU usage As shown in formula (13):

[0084]

[0085] availablecpu j Indicates the number of CPU cores currently available in queue j, C total Indicates the total CPU quota of queue j;

[0086] Memory usage As shown in formula (14):

[0087]

[0088] availablememory j Indicates the current remaining available memory in queue j, M total Indicates the total memory quota of queue j;

[0089] The pros and cons of scheduling actions are comprehensively evaluated by multi-objective weighting, as shown in formula (15):

[0090] R=wb alance *rb alance +w taskbalance *r taskbalance +w dependenc y*r dependency +w local *r local -λ·penalty (15);

[0091] Among them, w balance Indicates the weight of resource load balancing score; rbalance Indicates the resource load balancing score; w taskbalance represents the weight of the task quantity balance score; r taskbalance represents the task quantity balance score; w dependency Represents the weight of the dependent completion reward; r dependency Indicates dependency completion reward; w local represents the weight of the local utilization reward; r local represents the local utilization reward; λ represents the penalty coefficient of the global difference penalty; penalty represents the global difference penalty;

[0092] The specific scoring items are as follows:

[0093] Load balancing score balance Indicates the balance of CPU / memory utilization of each queue, as shown in formula (16):

[0094]

[0095] Among them, u c Indicates the CPU utilization of each queue, u m represents the memory utilization of each queue, and σ(·) is the standard deviation, which measures the volatility of resource utilization;

[0096] Task quantity balance score r taskbalance It represents the balance of the number of tasks in each queue, as shown in formula (17):

[0097]

[0098] Among them, n tasks Each represents the number of tasks in the queue, σ(n tasks ) represents the standard deviation of the number of tasks, max(n tasks ) represents the maximum number of tasks in all queues;

[0099] Dependency Completion Reward dependency represents the penalty for the number of unfinished dependent tasks, as shown in formula (18):

[0100]

[0101] Among them, t d Indicates the dependent tasks of the current task. Represents an indicator function, the value is 1 if the dependent task is not completed, otherwise it is 0;

[0102] Local utilization reward r local represents the CPU / memory utilization of the current queue, as shown in formula (19):

[0103]

[0104] in, Indicates the CPU utilization of the current queue j, Indicates the memory utilization of the current queue j;

[0105] The global difference penalty is penalty, which represents the difference between the maximum and minimum CPU utilization of each queue, as shown in formula (20):

[0106] penalty=max(u c )-min(u c ) (20);

[0107] Among them, max(u c ) represents the maximum CPU utilization of all queues, min(u c ) represents the minimum CPU utilization among all queues;

[0108] (5-4) Through the continuous interaction process, experience data is continuously collected and saved in the experience playback buffer. During each task scheduling process, an interaction sample in the form of a five-tuple is recorded, as shown in formula (21):

[0109] (state,action,reward,next_state,done) (21);

[0110] Among them, state represents the state of the current scheduling environment, action represents the scheduling decision made by the agent in the current state, reward represents the immediate feedback brought by the current scheduling action, next_state represents the next state of the environment after the action is executed, and done represents whether the current scheduling process has ended, which is a Boolean value;

[0111] By continuously executing scheduling actions, five-tuples are gradually collected and stored in the experience replay pool;

[0112] When the number of samples reaches a certain scale, they are sampled to form a training set for optimizing the Q-value function.

[0113] Further preferably, the scheduling strategy model includes an online network Q online and the target network Q target ; Online Network Q online and the target network Q target , the parameter update formula is shown in formula (22):

[0114] θ target ←τθ online +(1-τ)θ target (twenty two);

[0115] Among them, θ online represents the online network parameters, which are responsible for real-time updates and training through experience replay; θ target represents the target network parameters, which are periodically synchronized from the online network and used to stabilize the Q value calculation; τ represents the synchronization coefficient, τ = 1;

[0116] The target Q value is calculated as shown in formula (23):

[0117]

[0118] Where γ represents the discount factor, s′ represents the next state, which is the new state obtained by the environment after executing the action; r represents the immediate reward, which is calculated by the reward function;

[0119] Online Network Q online and the target network Q target The neural network structure is the same as that of , and the neural network includes the first fully connected layer, the first activation function, the second fully connected layer, the second activation function and the third fully connected layer; as shown in formula (24):

[0120]

[0121] in, Represents the input feature vector dimension, which is determined by the task feature D task and queue characteristics D queue Splicing composition; FC 128 Represents the first fully connected layer, which contains 128 neurons, performs linear transformation on the input features, and the output is Then, the activation function is expressed by ReLU, the linear unit is modified, and the nonlinear expression capability is introduced. The output is still Then comes the second fully connected layer FC 64 , contains 64 neurons, further extracts high-order features, and the output is And then activated by ReLU again; finally, the feature is mapped into a scalar through the output layer That is, the Q value, which represents the expected reward of the current task-queue pair in a given state.

[0122] Further preferred, the loss function of the batch training of the scheduling strategy model is The mean square error is used, as shown in formula (25):

[0123]

[0124] Where B represents the batch size, the number of state-action pairs sampled in a single training; s b Represents the state vector of the b-th sample; i b represents the task index of the bth sample; jb Indicates the queue index of the b-th sample; Q online Indicates the Q value predicted by the online network, input s b Post-task i b and queue j b prediction reward; Q target Indicates the Q value calculated by the target network.

[0125] According to the preferred embodiment of the present application, in step (6), obtaining and storing the scheduling information includes:

[0126] Each workflow scheduling optimization result is obtained through step (5); at the same time, the workflow scheduling optimization result is saved in a specific table of the MySQL database, and the specific table includes three columns: "dag_id", "task_id" and "queue_id".

[0127] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the above-mentioned computing-storage flow joint scheduling optimization method based on DDQN and heuristic strategy.

[0128] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned computing-storage flow joint scheduling optimization method based on DDQN and heuristic strategy.

[0129] The computation-storage flow joint scheduling optimization system based on DDQN and heuristic strategies includes:

[0130] The calculation flow and storage flow definition module is configured to: define the calculation flow and storage flow;

[0131] The workflow scheduling and execution module is configured as follows: when a workflow is triggered, the scheduler evaluates the executable status of each task based on the dependencies between tasks and the scheduling policy, and submits the executable tasks to the task executor; the task executor is responsible for executing these tasks on the worker nodes, ensuring that the tasks are completed in the predetermined order and priority.

[0132] The data collection module is configured to: collect DAG structure, queue information and run count;

[0133] The prediction module is configured to: predict task running time and resource usage based on historical running data;

[0134] The scheduling optimization module is configured to: perform scheduling optimization based on DDQN and heuristic strategies;

[0135] The information acquisition and storage module is configured to: acquire and store scheduling information.

[0136] In general, the above technical solutions conceived by this application can achieve the following beneficial effects compared with the existing technology:

[0137] To address the above challenges, this application proposes a method for decoupling and modeling "computation flow" and "storage flow" to structurally reconstruct the traditional hybrid scheduling process. On this basis, an intelligent task scheduling method based on "dual-flow collaboration" is further designed. This method integrates the Double DQN reinforcement learning algorithm and heuristic strategies, and through the joint perception and collaborative scheduling of computation flow and storage flow, it achieves precise matching of tasks between computing power, data and storage resources, thereby significantly improving the system's task execution efficiency and resource utilization.

[0138] 1. Refine task scheduling: By separating and modeling the computation flow and storage flow and co-scheduling them, we can optimize the task execution logic and data access path respectively, achieving efficient coupling and allocation of tasks between computing resources and data resources, thereby effectively improving the accuracy of task scheduling.

[0139] 2. Integrate heuristic strategies with reinforcement learning to improve strategy quality: Introducing a heuristic action guidance mechanism into the ε-greedy strategy, combining task dependencies, resource status, and scheduling history information, reduces ineffective exploration and accelerates strategy convergence.

[0140] 3. State space modeling is more expressive: The state space not only includes the current characteristics of the task, resource requirements, and queue status, but also integrates the historical scheduling information of the task and the characteristics of the critical path, reflecting the scheduling environment more comprehensively and improving the generalization and perception capabilities of the model.

[0141] 4. Multi-objective scheduling optimization: Based on the core goal of minimizing the execution time (Makespan), we further consider system load balancing, energy consumption control, and storage resource utilization to achieve multi-objective joint optimization and improve the overall system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0142] Figure 1 Flowchart of the computation-storage flow joint scheduling optimization method based on DDQN and heuristic strategy.

[0143] Figure 2 Schematic diagram of computing flow and storage flow.

[0144] Figure 3 Gather schematics for queue information and run data.

[0145] Figure 4 Schematic diagram of the running time and resource prediction process.

[0146] Figure 5 Schematic diagram of the scheduling optimization process based on DDQN and heuristic strategy. DETAILED DESCRIPTION

[0147] In order to make the purpose, technical solutions and advantages of this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for explaining this application and are not intended to limit this application. In addition, the technical features involved in the various embodiments of this application described below may be combined with each other as long as they do not conflict with each other.

[0148] Explanation of terms:

[0149] 1. Apache Airflow: Apache Airflow is an open-source workflow scheduling and management platform used to define and orchestrate task execution processes using directed acyclic graphs (DAGs). It supports scheduled task scheduling, dependency control, and retry upon failure. It offers excellent scalability and a visual interface, making it widely used in automated processes such as data processing, ETL, and machine learning. Airflow integrates with various computing and storage systems, making it suitable for building complex, monitorable, and maintainable data-driven workflows.

[0150] 2.Double DQN: Double DQN is also known as DDQN. Double DQN (Double Deep Q Network) is an improved deep reinforcement learning algorithm used to solve the problem of traditional DQN's policy instability caused by over-estimation of action value (Q value) during training. Double DQN introduces a dual network structure, using the main network to select actions and the target network to evaluate actions, thereby effectively reducing the Q-value estimation bias and improving learning stability and decision-making accuracy. DDQN (Double DeepQ Network) is an improved deep reinforcement learning algorithm used to solve the problem of traditional DQN's policy instability caused by over-estimation of action value (Q value) during training. DDQN introduces a dual network structure, using the main network to select actions and the target network to evaluate actions, thereby effectively reducing the Q-value estimation bias and improving learning stability and decision-making accuracy.

[0151] 3. CeleryExecutor: CeleryExecutor is a distributed task executor provided by Apache Airflow, implemented based on the Celery framework. It asynchronously distributes tasks to multiple independent worker nodes, making it suitable for large-scale, multi-task scheduling scenarios. When used in conjunction with messaging middleware (such as Redis), CeleryExecutor can significantly improve Airflow's scalability and throughput, making it a key component in building highly available scheduling systems.

[0152] 4. Scheduler: A core component of Apache Airflow, the Scheduler regularly parses DAG files, determines whether tasks meet scheduling requirements, and submits eligible tasks to an executor (such as CeleryExecutor). It runs continuously to ensure that tasks are triggered and executed according to scheduled times or dependencies. It serves as the control center for the entire automated workflow scheduling and is crucial to the orderly execution of tasks and the stable operation of the system.

[0153] 5. Webserver: The webserver is a visual interface component provided by Apache Airflow, built on Flask. It displays DAG status, task execution status, log information, and scheduling configuration. Through the web interface, users can intuitively manage workflows, including triggering tasks, modifying parameters, viewing dependency graphs, and monitoring runtime status. It is a key entry point for task orchestration and operations management using Airflow.

[0154] 6. Flower: Flower is a visual monitoring tool for Apache Airflow when using CeleryExecutor. It provides real-time visibility into task execution status, queue information, and worker performance. It provides intuitive distributed task monitoring through a web UI, helping to improve the observability and operational efficiency of the scheduling system.

[0155] 7. Worker: A Worker is the compute node that actually executes tasks in Apache Airflow. It is typically enabled when using a distributed executor such as CeleryExecutor. A Worker receives tasks from the Scheduler from a message queue and independently executes them according to the defined logic.

[0156] 8. Redis: Redis (Remote Dictionary Server) is an open source, high-performance key-value database that supports multiple data structures, such as strings, hashes, lists, sets, and ordered sets. It is often used in scenarios such as caching, message queues, and real-time data processing.

[0157] 9. TriggerDagRunOperator: This operator in Airflow is used to trigger the execution of one DAG within another. It is often used to achieve cross-workflow collaboration and process decoupling. It supports dynamically starting specified sub-DAGs during task execution and can pass runtime configuration via the `conf` parameter. It is suitable for scenarios such as triggering model training after data processing and modular orchestration. When used with the ExternalTaskSensor, it can also achieve cross-DAG dependency synchronization and execution control.

[0158] 10. ExternalTaskSensor: This sensor is used in Airflow for synchronizing inter-DAG task dependencies. It monitors the status of a task in another DAG and waits until it meets certain conditions (such as successful completion) before continuing the current task. It is suitable for scenarios where multiple DAGs have logical dependencies, enabling orderly connection and scheduling coordination between them. It is a crucial tool for building complex distributed workflows.

[0159] 11. airflow_task_stats table: used to record resource usage information of Airflow tasks, including task identifiers (such as dag_id, task_id, run_id, job_id, pid), average, minimum, and maximum values ​​of CPU and memory (avg_cpu, min_cpu, max_cpu, avg_mem, min_mem, max_mem), and process-related information such as vsz, rss, tty, and stat, to facilitate subsequent resource analysis and scheduling optimization.

[0160] 12. XGBoost Regression Model: This is an ensemble learning method based on gradient boosting. It offers high efficiency, accuracy, and resistance to overfitting, making it suitable for regression tasks involving structured data. It fits residuals by gradually building a decision tree, features a built-in regularization mechanism, supports automatic handling of missing values, and optimizes parallel computing performance.

[0161] Example 1

[0162] Computation-storage flow joint scheduling optimization method based on DDQN and heuristic strategy, such as Figure 1 Shown, including:

[0163] (1) Definition of computation flow and storage flow;

[0164] (2) Workflow scheduling and execution: When a workflow is triggered, the scheduler evaluates the executable status of each task based on the dependencies between tasks and the scheduling strategy, and submits the executable tasks to the task executor; the task executor is responsible for executing these tasks on the work nodes, ensuring that the tasks are completed in the predetermined order and priority.

[0165] (3) DAG structure, queue information and run number collection;

[0166] (4) Task running time and resource usage prediction based on historical running data;

[0167] (5) Scheduling optimization based on DDQN and heuristic strategies;

[0168] (6) Obtaining and saving scheduling information.

[0169] Example 2

[0170] The difference between the computing-storage flow joint scheduling optimization method based on DDQN and heuristic strategy described in Example 1 is that:

[0171] In step (1), the calculation flow and storage flow are defined as follows: Figure 2 Shown, including:

[0172] First, the DAG (Directed Acyclic Graph) model is used to uniformly model the entire task process. In a DAG, each node represents an independent task, and directed edges between nodes represent inter-task dependencies. That is, a task cannot begin until all upstream tasks have successfully completed. Due to its acyclic graph structure, DAG inherently avoids cyclic dependencies, ensuring logical closure and terminability of task scheduling.

[0173] Based on the functional attributes and resource usage types of tasks, tasks are logically divided into two sub-flows:

[0174] Compute Flow: This includes the execution of operator logic and the scheduling and allocation of computing resources. Operator logic execution includes running Python scripts, executing Shell commands, triggering SQL queries, and calling APIs. Computing resource scheduling distributes tasks to corresponding execution queues on different machines.

[0175] Storage Flow: includes the storage, reading, transmission, and management of the data that the task depends on;

[0176] All task nodes are declared and parameterized through Python script applications; all task nodes (Task) are explicitly declared and parameterized through DAG files in Python scripts. Each DAG file is essentially a Python script, and developers define the structure, task content, and scheduling logic of the entire workflow by writing code. In the script, tasks are usually represented by instances of the Operator class, such as PythonOperator, BashOperator, HttpOperator, etc. Each task node implements fine-grained control over task behavior through configuration parameters (such as task_id, executable function, dependency settings, number of retries, resource requirements, etc.), thereby ensuring the consistency and maintainability of the task structure on a global scale. This centralized configuration method facilitates unified management, version control, and flexible evolution of task scheduling strategies.

[0177] In terms of task flow control, the compute and storage flows implement a linkage triggering mechanism through TriggerDagRunOperator: when a compute task completes, the corresponding storage task is automatically triggered, ensuring the sequential flow of the processing. Furthermore, ExternalTaskSensor is introduced to achieve cross-flow dependency synchronization: the compute flow task monitors the status changes of specific storage tasks to determine when to start them, and vice versa, thus ensuring the scheduling consistency and dependency integrity of the compute and storage flows.

[0178] like Figure 2 As shown in the figure, in this distributed system architecture, the computation flow and storage flow work together to achieve bidirectional decoupling and dynamic linkage between tasks and data. The computation flow starts from the "Start" node and triggers the storage flow through the TriggerDagRunOperator, forming three parallel paths: C1 listens to the storage flow S1 and connects to C4 after completion. C4 monitors the status of S3 and converges to C6. C2 and C3 simultaneously listen to the storage flow S2 and converge to C5 after completion. Finally, the results of C4 and C5 are processed uniformly in C6.

[0179] The storage flow also originates from the Start node and branches into S1 and S2: S1 monitors compute nodes C1 and C2, while S2 monitors C2 and C3. Both flows are aggregated through ExternalTaskSensor to form S3, which dynamically monitors the status of C4 and C5.

[0180] This architecture achieves efficient decoupling and collaborative driving of computing flow and storage flow through a two-way perception mechanism of "computation depends on storage status, and storage tracks computing progress", thereby supporting high-performance scheduling and real-time data status synchronization of large-scale distributed tasks.

[0181] In step (2), workflow scheduling and execution; specifically including:

[0182] When an Airflow workflow (DAG) is triggered, the scheduler evaluates the status of each task based on the task dependencies and scheduling policies defined in the DAG. The scheduler checks which tasks meet the dependency conditions (such as whether the upstream task has completed successfully) and determines whether the scheduling time window has been reached. Tasks that meet the conditions are marked as schedulable (queued) and then placed in the task queue, waiting to be processed by the task executor.

[0183] The type of task executor (such as LocalExecutor, CeleryExecutor, or KubernetesExecutor) determines how tasks are distributed and executed. The executor retrieves tasks from the task queue and assigns them to the corresponding worker nodes or containers. Each task is executed in an isolated process or environment, ensuring resource isolation and task independence.

[0184] During execution, tasks update their status (e.g., running, success, failed, etc.) in real time. The scheduler monitors these status changes and updates the database. If a task fails, the scheduler reschedules it based on the task's retry policy (e.g., number of retries and retry interval). Throughout this process, the scheduler ensures that tasks in the DAG are executed strictly according to the defined dependency and priority order until all tasks are completed, thus completing the entire workflow scheduling and execution loop.

[0185] In step (3), DAG structure, queue information and running data are collected, such as Figure 3 Shown, including:

[0186] Build the multi-dimensional data foundation required for intelligent decision-making in task scheduling, including DAG structure modeling, dynamic collection of task execution data, and queue resource status perception; construct the task scheduling state space through unified modeling; and provide high-dimensional, interpretable input features for subsequent scheduling optimization strategies based on deep reinforcement learning.

[0187] In terms of DAG structure, the following information is collected and organized for each task node: dag_id, task_id, whether it is a critical path task, whether it belongs to a computing flow task, whether it is a TriggerDagRunOperator task that triggers the storage flow, the target DAG it triggers, whether it is an ExternalTaskSensor type task, the DAG and task list it depends on, and the DAG adjacency table structure between the overall computing flow and storage flow; so as to comprehensively describe the dependency topology and triggering logic between tasks; dag_id refers to the DAG id, and task_id refers to the task id; the above DAG structure and task node information are mainly obtained automatically by parsing Airflow DAG files, combining task type judgment (such as TriggerDagRunOperator, ExternalTaskSensor), task attribute extraction (such as dag_id, task_id, upstream_list, etc.), and building based on the adjacency matrix and completing cross-DAG dependencies.

[0188] In terms of task running data, i.e., running data, the running status of historical tasks is recorded, including: task_id, dag_id, run_id, job_id, pid, avg_cpu, min_cpu, max_cpu, avg_mem, min_mem, max_mem, rss, file_size, and duration;

[0189] run_id refers to the DAG instance run ID, job_id refers to the task run ID generated by the scheduler, pid refers to the task process ID, avg_cpu refers to the average CPU usage, min_cpu refers to the minimum CPU usage, max_cpu refers to the maximum CPU usage, avg_mem refers to the average memory usage, min_mem refers to the minimum memory usage, max_mem refers to the maximum memory usage, rss refers to the resident memory set size, file_size refers to the file size, and duration refers to the task execution duration. The above running data is read from the airflow_task_stats table and can be used to construct a resource consumption profile for the task.

[0190] In terms of queue resources, i.e. queue information, each execution queue is collected, including: queue_id, queue_code, currently available CPU, memory (kB), storage (kB), and I / O bandwidth (kB), to build a real-time mapping relationship between tasks and execution resources;

[0191] queue_id represents the queue name, and queue_code represents the queue number. The above queue resource information is obtained by connecting to each Celery Worker host through SSH, calling standard Linux commands to obtain CPU idle rate, available memory, available disk space, and I / O bandwidth in real time. Combined with the Celery queue name, this information is uniformly encapsulated into a scheduling queue object, providing data support for scheduling optimization and resource-aware decision-making.

[0192] Save the collected DAG structure and queue resources to a JSON file, and save the task running data to the airflow_task_stats table in the MySQL database.

[0193] In step (4), task running time and resource usage are predicted based on historical running data, such as Figure 4 Shown, including:

[0194] (4-1) Set up the basic environment for system operation, including the model storage path and operation log configuration; obtain historical task data from the airflow_task_stats table in the MySQL database, including the following fields: task_id, dag_id, run_id, avg_cpu, avg_mem, rss, file_size, and duration;

[0195] Perform data preprocessing on historical data, including type conversion and missing value filling. Convert historical task data to numeric type and save it in MySQL database, as shown in formula (1). Fill missing values ​​with 0, as shown in formula (2):

[0196] X clean =pd.to_numeric(X raw ,errors='coerce') (1);

[0197] X fill =X clean *fillna(0) (2);

[0198] Among them, X raw Represents the original input data (may contain non-numeric fields), X clean Indicates the converted numeric data, and illegal values ​​are converted to NaN; pd.to_numeric() indicates the function provided by Pandas, which is used to convert data into numeric types; errors = 'coerce' indicates that when the data cannot be converted into a numeric value, it is forcibly converted to NaN, that is, missing values; X fill Indicates the value after filling the missing values. fillna(0) means filling all missing values ​​(NaN) with 0;

[0199] (4-2) Using the XGBoost regression model, three models are trained independently for each task, and three XGBoost regression models are deployed independently for each task_id to predict: running time (duration), CPU usage (avg_cpu), and memory usage (rss), as shown in formula (3). The generated regression model is persisted and saved as a pkl file;

[0200]

[0201] Among them, f time () represents the trained XGBoost regression model used to predict the running time of the task; fc cpu () represents the trained XGBoost regression model for predicting CPU usage; f mem () represents the trained XGBoost regression model for predicting memory usage; represents f time () predicts the running time of the task, represents f cpu () predicted CPU usage, represents f mem () predicted memory usage;

[0202] The Optuna framework is used for Bayesian optimization to search for the optimal hyperparameter combination. The goal is to minimize the mean absolute error (MAE). When training the XGBoost regression model, the square error is minimized, as shown in formula (4):

[0203]

[0204] Among them, y i represents the true value of the i-th sample, The predicted value of the i-th sample, α represents the L1 regularization coefficient (controls sparsity), λ represents the L2 regularization coefficient (prevents overfitting), γ represents the minimum loss drop threshold for controlling tree splitting, T represents the number of leaf nodes, w represents the leaf node weight vector; ‖‖1 represents the L1 norm, that is, the sum of the absolute values ​​of all leaf node weights; Represents the square of the L2 norm, that is, the sum of the squares of the weights of all leaf nodes; Represents the total loss function (objective function); it consists of prediction error and model complexity;

[0205] (4-3) Load the latest prediction model according to the model storage path. The data in the airflow_task_stats table in the MySQL database is divided into datasets by task_id to ensure task independence. The input features are [avg_cpu, avg_mem, rss, file_size], and the output targets are duration, avg_cpu, and rss.

[0206] The latest record (maximum value of run_id) of each task is predicted as shown in formula (5):

[0207]

[0208] Among them, X new represents the latest task feature vector, Indicates the predicted value, which is time, CPU or memory; f XGBoost () represents the trained XGBoost regression model function;

[0209] (4-4) Scan the configuration file directory (config_dir) and write the predicted values ​​into the predicted_time, predicted_cpu, and predicted_memory fields of the JSON file. As shown in Table 1:

[0210] Table 1 Example of predicted values;

[0211]

[0212] In step (5), scheduling optimization is performed based on DDQN and heuristic strategies, such as Figure 5 Shown, including:

[0213] (5-1) Create an experience replay buffer and set an upper limit for storage capacity to store experience samples of agent interactions.

[0214] Build a dual network architecture, including:

[0215] Policy network (online network): updated in real time, used to estimate the Q value of each action in the current state;

[0216] Target Network (Target Network): Delayed synchronization, periodic update of policy network parameters, and improved training stability.

[0217] Set the initial exploration rate and configure its decay strategy.

[0218] Construct the four-dimensional state tensor S, as shown in formula (6):

[0219]

[0220] Among them, B represents the batch size, N t Indicates the total number of tasks, N q represents the total number of queues, D represents the feature dimension (task feature + queue feature), and D is D task +D queue ;

[0221] Task Feature D task 9+N t , as shown in formula (7):

[0222]

[0223] Among them, w represents the task weight (priority), q assigned Indicates the assigned queue number. Indicates whether it is a critical path task (0 / 1), Indicates whether it is a computing flow task (0 / 1), Indicates whether it is a trigger task (0 / 1), Indicates whether it is a sensor task (0 / 1), represents the predicted running time, CPU, memory, Represents one-hot encoding of dependent tasks;

[0224] Queue characteristics: D queue , as shown in formula (8):

[0225]

[0226] Among them, c avail Indicates the available CPU resources of the queue, c total Indicates the total CPU resources of the queue, m avail Indicates the available memory of the queue (KB), m total Indicates the total queue memory (KB), s avail Indicates the available storage of the queue (KB), s totals Indicates the total queue storage (KB), n tasks Indicates the number of tasks assigned to the queue, t end Indicates the end timestamp of the current task in the queue;

[0227] (5-2) Perform multiple rounds of Episode training until the termination condition is met; each Episode includes the following steps:

[0228] Environment reset: restores the simulation environment to its initial state and clears previous interaction history data; ensuring the independence and fairness of training.

[0229] Mission completion judgment: Check whether the current Episode has completed all preset mission objectives. If the mission has been completed, skip the subsequent steps and go directly to the next Episode;

[0230] (5-3) In the current Episode, perform the following operations according to the time step:

[0231] Available action detection: Obtain a set of legal actions based on the current state of the environment; avoid selecting illegal actions.

[0232] Obtain legal actions from the legal action set and verify the key function V(a) of the legality of the action, as shown in formula (9):

[0233]

[0234] Among them, a represents the task t i Specific operations (such as assigning to a queue, starting execution, etc.), t i Indicates the current task to be scheduled, deps(t i ) represents the dependency set, t d Indicates dependent tasks, status(t d ) indicates the task status;

[0235] According to the mixed epsilon-greedy and heuristic rule a, as shown in formula (10), the optimal action is selected;

[0236]

[0237] Among them, ∈ represents the exploration rate (decaying with the number of training rounds), θ online Indicates online network parameters, represents the legal queue set of task i; j represents an action in the legal queue set, Q represents the Q value function in reinforcement learning, and Heuristic represents the heuristic policy function;

[0238] The heuristic rule prioritizes assigning tasks to queues with high current load but low demand matching to avoid resource overload, as shown in formula (11):

[0239]

[0240] For each queue j, calculate its load score LoadScore j , as shown in formula (12):

[0241] Indicates the CPU importance weight, which is 0.5. Indicates the memory importance weight; it is 0.5.

[0242] CPU usage As shown in formula (13):

[0243]

[0244] availablecpu j Indicates the number of CPU cores currently available in queue j, C total Indicates the total CPU quota of queue j;

[0245] Memory usage As shown in formula (14):

[0246]

[0247] availablememory j Indicates the current remaining available memory in queue j, M total Indicates the total memory quota of queue j;

[0248] The pros and cons of scheduling actions are comprehensively evaluated by multi-objective weighting, as shown in formula (15):

[0249] R=wb alance *rb alance +w taskbalance *r taskbalance +w dependenc y*r dependency +w local *r local -λ·penalty (15);

[0250] Among them, w balance Indicates the weight of resource load balancing score; r balance Indicates the resource load balancing score; w taskbalance represents the weight of the task quantity balance score; r taskbalance represents the task quantity balance score; w dependency Represents the weight of the dependent completion reward; r dependency Indicates dependency completion reward; w local represents the weight of the local utilization reward; r local represents the local utilization reward; λ represents the penalty coefficient of the global difference penalty; penalty represents the global difference penalty;

[0251] The specific scoring items are as follows:

[0252] Load balancing score r balance Indicates the balance of CPU / memory utilization of each queue, as shown in formula (16):

[0253]

[0254] Among them, u c Indicates the CPU utilization of each queue, u m represents the memory utilization of each queue, and σ(·) is the standard deviation, which measures the volatility of resource utilization;

[0255] Task quantity balance score r taskbalance It represents the balance of the number of tasks in each queue, as shown in formula (17):

[0256]

[0257] Among them, n tasks Each represents the number of tasks in the queue, σ(n tasks ) represents the standard deviation of the number of tasks, max(n tasks ) represents the maximum number of tasks in all queues;

[0258] Dependency Completion Reward dependency represents the penalty for the number of unfinished dependent tasks, as shown in formula (18):

[0259]

[0260] Among them, t d Indicates the dependent tasks of the current task. Represents an indicator function, the value is 1 if the dependent task is not completed, otherwise it is 0;

[0261] Local utilization reward r local represents the CPU / memory utilization of the current queue, as shown in formula (19):

[0262]

[0263] in, Indicates the CPU utilization of the current queue j, Indicates the memory utilization of the current queue j;

[0264] The global difference penalty is penalty, which represents the difference between the maximum and minimum CPU utilization of each queue, as shown in formula (20):

[0265] penalty=max(u c )-min(u c ) (20);

[0266] Among them, max(u c ) represents the maximum CPU utilization of all queues, min(u c ) represents the minimum CPU utilization among all queues;

[0267] After executing the action state, the environment returns the next state next_state and the immediate reward reward. The experience sample quintuple (state, action, reward, next_state, done) is stored in the experience replay buffer for subsequent training.

[0268] (5-4) In order to train an efficient task scheduling strategy, it is necessary to continuously collect experience data through a continuous interaction process and save it in the experience playback buffer. In each task scheduling process, a five-tuple interaction sample is recorded, as shown in formula (21):

[0269] (state,action,reward,next_state,done) (21);

[0270] Among them, state represents the state of the current scheduling environment, action represents the scheduling decision made by the agent in the current state, reward represents the immediate feedback brought by the current scheduling action, next_state represents the next state of the environment after the action is executed, and done represents whether the current scheduling process has ended, which is a Boolean value;

[0271] By continuously executing scheduling actions (task → queue allocation), five-tuples are gradually collected and stored in the experience replay pool (Replay Buffer);

[0272] When the number of samples reaches a certain size (forming a batch), samples are taken from it to form a training set (the batch size is set to BATCH_SIZE=256) for optimizing the Q-value function.

[0273] The scheduling strategy model is trained using the training set, and the Q-value function is optimized to gradually approach the optimal task scheduling strategy, thereby achieving efficient task allocation and maximizing resource utilization in a multi-queue environment.

[0274] The scheduling strategy model includes the online network Q online and the target network Q target ; Online Network Q online and the target network Q target , the parameter update formula is shown in formula (22):

[0275] θ target ←τθ online +(1-τ)θ target (twenty two);

[0276] Among them, θ online Represents the online network parameters, which are responsible for real-time update and training through experience replay; q targetrepresents the target network parameters, which are periodically synchronized from the online network and used to stabilize the Q value calculation; τ represents the synchronization coefficient, τ = 1; periodic hard update;

[0277] The target Q value is calculated as shown in formula (23):

[0278]

[0279] Where γ is the discount factor, which controls the decay weight of future rewards. s′ is the next state, which is the new state of the environment after executing the action; r is the immediate reward, which is calculated by the reward function.

[0280] Online Network Q online and the target network Q target The neural network structure is the same as that of , and the neural network includes the first fully connected layer, the first activation function, the second fully connected layer, the second activation function and the third fully connected layer; as shown in formula (24):

[0281]

[0282] in, Represents the input feature vector dimension, which is determined by the task feature D task and queue characteristics D queue Splicing composition; FC 128 Represents the first fully connected layer, which contains 128 neurons, performs linear transformation on the input features, and the output is Represents the intermediate dimension, the first hidden layer output dimension (128 dimensions). Then, ReLU is used to represent the activation function, the rectified linear unit, and the nonlinear expression capability is introduced. The output is still Then comes the second fully connected layer FC 64 , contains 64 neurons, further extracts high-order features, and the output is Represents the intermediate dimension, the second hidden layer output dimension (64 dimensions); and is activated again by ReLU; finally, the feature is mapped to a scalar through the output layer (fully connected layer) That is, the Q value, which represents the expected reward of the current task-queue pair in a given state.

[0283] Loss function for batch training of scheduling policy model The mean square error is used, as shown in formula (25):

[0284]

[0285] Where B represents the batch size, the number of state-action pairs sampled in a single training; s b Represents the state vector of the b-th sample; i brepresents the task index of the bth sample; j b Indicates the queue index of the b-th sample; Q online Indicates the Q value predicted by the online network, input s b Post-task i b and queue j b prediction reward; Q target Indicates the Q value calculated by the target network.

[0286] The core output of the scheduling strategy model is the Q-value function Q online (s b ,i b ,j b ), which is used to evaluate the expected reward if task i is assigned to queue j in the current state s. Based on the Q-value function, the agent can adopt an action selection strategy to generate a specific scheduling decision.

[0287] In step (6), the scheduling information is obtained and saved, including:

[0288] The optimization results of each workflow scheduling are obtained through step (5); at the same time, the optimization results of the workflow scheduling are saved in a specific table of the MySQL database. The specific table includes three columns: "dag_id", "task_id" and "queue_id", as shown in Table 2.

[0289] Table 2 Scheduling optimization results;

[0290] queue01 [′c2′,′wait_c12′,′s2′,′c5′,′s3′,′c7′,"c8′] queue02 [′start_computer_cs′,′c3′,′s1′,′wait_s2′,′wait_c14′,′s4′,′wait_s4′,′s5′] queue03 [′trigger_storage_ss′,′start_storage_ss′,′wait_c23′,′c4′,′wait_s3′,′wait_c67′] queue04 [′c1′,′wait_s1′,′wait_c35′,′c6′,′wait_s5′]

[0291] Example 3

[0292] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the computing-storage flow joint scheduling optimization method based on DDQN and heuristic strategy described in embodiment 1 or 2 are implemented.

[0293] Example 4

[0294] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the computation-storage flow joint scheduling optimization method based on DDQN and heuristic strategy described in embodiment 1 or 2.

[0295] Example 5

[0296] The computation-storage flow joint scheduling optimization system based on DDQN and heuristic strategies includes:

[0297] The calculation flow and storage flow definition module is configured to: define the calculation flow and storage flow;

[0298] The workflow scheduling and execution module is configured as follows: when a workflow is triggered, the scheduler evaluates the executable status of each task based on the dependencies between tasks and the scheduling policy, and submits the executable tasks to the task executor; the task executor is responsible for executing these tasks on the worker nodes, ensuring that the tasks are completed in the predetermined order and priority.

[0299] The data collection module is configured to: collect DAG structure, queue information and run count;

[0300] The prediction module is configured to: predict task running time and resource usage based on historical running data;

[0301] The scheduling optimization module is configured to: perform scheduling optimization based on DDQN and heuristic strategies;

[0302] The information acquisition and storage module is configured to: acquire and store scheduling information.

Claims

1. A computation-storage flow joint scheduling optimization method based on DDQN and heuristic strategy, characterized by: include: (1) Definition of computation flow and storage flow; (2) Workflow scheduling and execution: When a workflow is triggered, the scheduler evaluates the executable status of each task based on the dependencies between tasks and the scheduling strategy, and submits the executable tasks to the task executor; The task executor is responsible for executing these tasks on the worker nodes, ensuring that the tasks are completed in the predetermined order and priority; (3) DAG structure, queue information and run number collection; (4) Task running time and resource usage prediction based on historical running data; (5) Scheduling optimization based on DDQN and heuristic strategies; (6) Obtaining and saving scheduling information; In step (5), scheduling optimization based on DDQN and heuristic strategy includes: (5-1) Construct the four-dimensional state tensor S, as shown in formula (6): Among them, B represents the batch size, N t Indicates the total number of tasks, N q represents the total number of queues, D represents the feature dimension, and D is D task +D queue ; Task Feature D task 9+N t , as shown in formula (7): Among them, w represents the task weight, q assigned Indicates the assigned queue number. Indicates whether it is a critical path task. Indicates whether it is a computational flow task. Indicates whether it is a trigger task. Indicates whether it is a sensor task. represents the predicted running time, CPU, memory, Represents one-hot encoding of dependent tasks; Queue characteristics: D queue , as shown in formula (8): Among them, c avail Indicates the available CPU resources of the queue, c total Indicates the total CPU resources of the queue, m avail Indicates the available memory of the queue, m total Indicates the total queue memory, s avail Indicates the available storage for the queue, s totals Indicates the total storage of the queue, n tasks Indicates the number of tasks assigned to the queue, t end Indicates the end timestamp of the current task in the queue; (5-2) Perform multiple rounds of Episode training until the termination condition is met; each Episode includes the following steps: Environment reset: restore the simulation environment to its initial state and clear previous interaction history data; Mission completion judgment: Check whether the current Episode has completed all preset mission objectives. If the mission has been completed, skip the subsequent steps and go directly to the next Episode; (5-3) In the current Episode, perform the following operations according to the time step: Available action detection: Obtain a set of legal actions based on the current state of the environment; Obtain legal actions from the legal action set and verify the key function V(a) of the legality of the action, as shown in formula (9): Among them, a represents the task t i The specific operation of t i Indicates the current task to be scheduled, deps(t i ) represents the dependency set, t d Indicates dependent tasks, status(t d ) indicates the task status; According to the mixed epsilon-greedy and heuristic rule a, as shown in formula (10), the optimal action is selected; Among them, ∈ represents the exploration rate, θ online Indicates online network parameters, represents the set of legal queues for task i; j represents an action in the legal queue set, Q represents the Q-value function in reinforcement learning, and Heuristic represents the heuristic strategy function; The heuristic rule prioritizes assigning tasks to queues with high current load but low demand matching to avoid resource overload, as shown in formula (11): For each queue j, calculate its load score LoadScore j , as shown in formula (12): Indicates the CPU importance weight, Indicates the memory importance weight; CPU usage As shown in formula (13): availablecpu j Indicates the number of CPU cores currently available in queue j, C total Indicates the total CPU quota of queue j; Memory usage As shown in formula (14): availablememory j Indicates the current remaining available memory in queue j, M total Indicates the total memory quota of queue j; The pros and cons of scheduling actions are comprehensively evaluated by multi-objective weighting, as shown in formula (15): R=w balance *r balance +w taskbalance *r taskbalance +w dependency *r dependency +w local *r local -λ·penalty (15); Among them, w balance Indicates the weight of resource load balancing score; r balance Indicates the resource load balancing score; w taskbalance represents the weight of the task quantity balance score; r taskbalance represents the task quantity balance score; w dependency Represents the weight of the dependent completion reward; r dependency Indicates dependency completion reward; w local represents the weight of the local utilization reward; r local represents the local utilization reward; λ represents the penalty coefficient of the global difference penalty; penalty represents the global difference penalty; The specific scoring items are as follows: Load balancing score r balance Indicates the balance of CPU / memory utilization of each queue, as shown in formula (16): Among them, u c Indicates the CPU utilization of each queue, u m represents the memory utilization of each queue, and σ(·) is the standard deviation, which measures the volatility of resource utilization; Task quantity balance score r taskbalance It represents the balance of the number of tasks in each queue, as shown in formula (17): Among them, n tasks Each represents the number of tasks in the queue, σ(n tasks ) represents the standard deviation of the number of tasks, max(n tasks ) represents the maximum number of tasks in all queues; Dependency Completion Reward dependency represents the penalty for the number of unfinished dependent tasks, as shown in formula (18): Among them, t d Indicates the dependent tasks of the current task. Represents an indicator function, the value is 1 if the dependent task is not completed, otherwise it is 0; Local utilization reward r local represents the CPU / memory utilization of the current queue, as shown in formula (19): in, Indicates the CPU utilization of the current queue j, Indicates the memory utilization of the current queue j; The global difference penalty is penalty, which represents the difference between the maximum and minimum CPU utilization of each queue, as shown in formula (20): penalty=max(u c )-min(u c ) (20); Among them, max(u c ) represents the maximum CPU utilization of all queues, min(u c ) represents the minimum CPU utilization among all queues; (5-4) Through the continuous interaction process, experience data is continuously collected and saved in the experience playback buffer. During each task scheduling process, an interaction sample in the form of a five-tuple is recorded, as shown in formula (21): (state,action,reward,next_state,done) (21); Among them, state represents the state of the current scheduling environment, action represents the scheduling decision made by the agent in the current state, reward represents the immediate feedback brought by the current scheduling action, next_state represents the next state of the environment after the action is executed, and done represents whether the current scheduling process has ended, which is a Boolean value; By continuously executing scheduling actions, five-tuples are gradually collected and stored in the experience replay pool; When the number of samples reaches a certain scale, they are sampled to form a training set for optimizing the Q-value function.

2. The computing-storage flow joint scheduling optimization method based on DDQN and heuristic strategy according to claim 1 is characterized in that: In step (1), the calculation flow and storage flow definition include: First, the DAG model is used to uniformly model the entire task process. In the DAG, each node represents an independent task, and the directed edges between nodes represent the dependencies between tasks. Based on the functional attributes and resource usage types of tasks, tasks are logically divided into two sub-flows: Computational flow: includes the execution of operator logic and the scheduling and allocation of computing resources; Storage flow: including the storage, reading, transmission and management of data that tasks depend on; All task nodes are declared and parameterized through Python scripting artifacts; The computation flow and storage flow implement a linkage trigger mechanism through TriggerDagRunOperator: when the computation task is completed, the corresponding storage task is automatically triggered to ensure the timing connection of the processing flow; at the same time, ExternalTaskSensor is introduced to achieve cross-flow dependency synchronization: the computation flow task monitors the status changes of specific storage tasks to determine their start time.

3. The computing-storage flow joint scheduling optimization method based on DDQN and heuristic strategy according to claim 1 is characterized in that: In step (2), workflow scheduling and execution; specifically including: When an Airflow workflow is triggered, the scheduler evaluates the status of each task based on the task dependencies and scheduling policies defined in the DAG. The scheduler checks which tasks meet the dependency conditions and determines whether the scheduling time window has been reached. Eligible tasks are marked as schedulable and placed in the task queue, awaiting processing by the task executor. The executor obtains tasks from the task queue and assigns them to the corresponding worker node or container for execution; During the execution process, the task status is updated in real time. If a task fails, the scheduler reschedules the task according to the task's retry strategy. Throughout the entire process, the scheduler ensures that the tasks in the DAG are executed strictly according to the defined dependency order and priority order until all tasks are completed, thus completing the entire workflow scheduling and execution loop.

4. The computing-storage flow joint scheduling optimization method based on DDQN and heuristic strategy according to claim 1 is characterized in that: In step (3), DAG structure, queue information and running data collection include: In terms of DAG structure, the following information is collected and organized for each task node: dag_id, task_id, whether it is a critical path task, whether it belongs to a compute flow task, whether it is a TriggerDagRunOperator task that triggers a storage flow, the target DAG it triggers, whether it is an ExternalTaskSensor type task, the list of DAGs and tasks it depends on, and the overall DAG adjacency table structure between the compute flow and the storage flow; dag_id refers to the DAG id, and task_id refers to the task id; In terms of task running data, i.e., running data, the running status of historical tasks is recorded, including: task_id, dag_id, run_id, job_id, pid, avg_cpu, min_cpu, max_cpu, avg_mem, min_mem, max_mem, rss, file_size, and duration; run_id refers to the DAG instance run ID, job_id refers to the task run ID generated by the scheduler, pid refers to the task process ID, avg_cpu refers to the average CPU usage, min_cpu refers to the minimum CPU usage, max_cpu refers to the maximum CPU usage, avg_mem refers to the average memory usage, min_mem refers to the minimum memory usage, max_mem refers to the maximum memory usage, rss refers to the resident memory set size, file_size refers to the file size, and duration refers to the task execution duration; In terms of queue resources, i.e. queue information, each execution queue is collected, including: queue_id, queue_code, currently available CPU, memory, storage, and I / O bandwidth, to build a real-time mapping relationship between tasks and execution resources; queue_id indicates the queue name, queue_code indicates the queue number; Save the collected DAG structure and queue resources to a JSON file, and save the task running data to the airflow_task_stats table in the MySQL database.

5. The computing-storage flow joint scheduling optimization method based on DDQN and heuristic strategy according to claim 1 is characterized in that: In step (4), task running time and resource usage prediction based on historical running data includes: (4-1) Set up the basic environment for system operation, including the model storage path and operation log configuration; obtain historical task data from the airflow_task_stats table in the MySQL database, including the following fields: task_id, dag_id, run_id, avg_cpu, avg_mem, rss, file_size, and duration; Perform data preprocessing on historical data, including type conversion and missing value filling. Convert historical task data to numeric type and save it in MySQL database, as shown in formula (1). Fill missing values ​​with 0, as shown in formula (2): X clean =pd.to_numeric(X raw ,errors=’coerce’) (1); X fill =X clean *fillna(0) (2); Among them, X raw represents the original input data, X clean Indicates the converted numeric data, and illegal values ​​are converted to NaN; pd.to_numeric() indicates the function provided by Pandas, which is used to convert data to numeric types; errors = 'coera' indicates that when the data cannot be converted to a numeric value, it is forced to be converted to NaN, that is, missing values; X fill Indicates the value after filling the missing values. fillna(0) means filling all missing values ​​with 0. (4-2) Using the XGBoost regression model, three models are trained independently for each task, and three XGBoost regression models are deployed independently for each task_id to predict: running time, CPU usage, and memory usage, as shown in formula (3). The generated regression model is persisted and saved as a pkl file; Among them, f time () represents the trained XGBoost regression model used to predict the running time of the task; f cpu () represents the trained XGBoost regression model for predicting CPU usage; f mem () represents the trained XGBoost regression model for predicting memory usage; represents f time () predicts the running time of the task, represents f cpu () predicted CPU usage, represents f mem () predicted memory usage; The Optuna framework is used for Bayesian optimization to search for the optimal hyperparameter combination. The goal is to minimize the mean absolute error and minimize the square error when training the XGBoost regression model, as shown in formula (4): Among them, y i represents the true value of the i-th sample, The predicted value of the i-th sample, α represents the L1 regularization coefficient, λ represents the L2 regularization coefficient, γ represents the minimum loss drop threshold for controlling tree splitting, T represents the number of leaf nodes, w represents the leaf node weight vector; ‖‖1 represents the L1 norm, that is, the sum of the absolute values ​​of all leaf node weights; Represents the square of the L2 norm, that is, the sum of the squares of the weights of all leaf nodes; represents the total loss function; (4-3) Load the latest prediction model according to the model storage path. The data in the airflow_task_stats table of the MySQL database is divided into datasets by task_id. The input features are [avg_cpu, avg_mem, rss, file_size], and the output targets are duration, avg_cpu, and rss. The latest record prediction for each task is shown in formula (5): Among them, X new represents the latest task feature vector, Indicates the predicted value, which is time, CPU or memory; f XGB”st () represents the trained XGBoost regression model function; (4-4) Scan the configuration file directory and write the predicted values ​​into the predicted_time, predicted_cpu, and predicted_memory fields of the JSON file.

6. The computing-storage flow joint scheduling optimization method based on DDQN and heuristic strategy according to claim 1 is characterized in that: The scheduling strategy model includes the online network Q online and the target network Q target ; Online Network Q online and the target network Q target , the parameter update formula is shown in formula (22): i target ←tth online +(1-τ)θ target (22); Among them, θ online represents the online network parameters, which are responsible for real-time updates and training through experience replay; θ target represents the target network parameters, which are periodically synchronized from the online network and used to stabilize the Q value calculation; τ represents the synchronization coefficient, τ = 1; The target Q value is calculated as shown in formula (23): Where γ represents the discount factor, s′ represents the next state, which is the new state obtained by the environment after executing the action; r represents the immediate reward, which is calculated by the reward function; Online Network Q online and the target network Q target The neural network structure is the same as that of , and the neural network includes the first fully connected layer, the first activation function, the second fully connected layer, the second activation function and the third fully connected layer; as shown in formula (24): in, Represents the input feature vector dimension, which is determined by the task feature D task and queue characteristics D queue Splicing composition; FC 128 Represents the first fully connected layer, which contains 128 neurons, performs linear transformation on the input features, and the output is Then, the activation function is expressed by ReLU, the linear unit is modified, and the nonlinear expression capability is introduced. The output is still Then comes the second fully connected layer FC 64 , contains 64 neurons, further extracts high-order features, and the output is And then activated by ReLU again; finally, the feature is mapped into a scalar through the output layer That is, the Q value, which represents the expected reward of the current task-queue pair in a given state.

7. The computing-storage flow joint scheduling optimization method based on DDQN and heuristic strategy according to claim 1 is characterized in that: Loss function for batch training of scheduling policy model The mean square error is used, as shown in formula (25): Where B represents the batch size, the number of state-action pairs sampled in a single training; s b Represents the state vector of the b-th sample; i b represents the task index of the bth sample; j b Indicates the queue index of the b-th sample; Q online Indicates the Q value predicted by the online network, input s b Post-task i b and queue j b prediction reward; Q target Indicates the Q value calculated by the target network.

8. The computing-storage flow joint scheduling optimization method based on DDQN and heuristic strategy according to any one of claims 1 to 7, characterized in that: In step (6), the scheduling information is obtained and saved, including: Each workflow scheduling optimization result is obtained through step (5); at the same time, the workflow scheduling optimization result is saved in a specific table of the MySQL database, and the specific table includes three columns: "dag_id", "task_id" and "queue_id".

9. A computing-storage flow joint scheduling optimization system based on DDQN and heuristic strategy, characterized by: include: The calculation flow and storage flow definition module is configured to: define the calculation flow and storage flow; The workflow scheduling and execution module is configured as follows: when a workflow is triggered, the scheduler evaluates the executable status of each task one by one based on the dependencies between tasks and the scheduling strategy, and submits the executable tasks to the task executor; The task executor is responsible for executing these tasks on the worker nodes, ensuring that the tasks are completed in the predetermined order and priority; The data collection module is configured to: collect DAG structure, queue information and run count; The prediction module is configured to: predict task running time and resource usage based on historical running data; The scheduling optimization module is configured to: perform scheduling optimization based on DDQN and heuristic strategies; The information acquisition and storage module is configured to: acquire and store scheduling information; Scheduling optimization based on DDQN and heuristic strategies, including: (5-1) Construct the four-dimensional state tensor S, as shown in formula (6): Among them, B represents the batch size, N t Indicates the total number of tasks, N q represents the total number of queues, D represents the feature dimension, and D is D task +D queue ; Task Feature D task 9+N t , as shown in formula (7): Among them, w represents the task weight, q assigned Indicates the assigned queue number. Indicates whether it is a critical path task. Indicates whether it is a computational flow task. Indicates whether it is a trigger task. Indicates whether it is a sensor task. represents the predicted running time, CPU, memory, Represents one-hot encoding of dependent tasks; Queue characteristics: D queue , as shown in formula (8): Among them, c avail Indicates the available CPU resources of the queue, c total Indicates the total CPU resources of the queue, m avail Indicates the available memory of the queue, m total Indicates the total queue memory, s avail Indicates the available storage for the queue, s totals Indicates the total storage of the queue, n tasks Indicates the number of tasks assigned to the queue, t end Indicates the end timestamp of the current task in the queue; (5-2) Perform multiple rounds of Episode training until the termination condition is met; each Episode includes the following steps: Environment reset: restore the simulation environment to its initial state and clear previous interaction history data; Mission completion judgment: Check whether the current Episode has completed all preset mission objectives. If the mission has been completed, skip the subsequent steps and go directly to the next Episode; (5-3) In the current Episode, perform the following operations according to the time step: Available action detection: Obtain a set of legal actions based on the current state of the environment; Obtain legal actions from the legal action set and verify the key function V(a) of the legality of the action, as shown in formula (9): Among them, a represents the task t i The specific operation of t i Indicates the current task to be scheduled, deps(t i ) represents the dependency set, t d Indicates dependent tasks, status(t d ) indicates the task status; According to the mixed epsilon-greedy and heuristic rule a, as shown in formula (10), the optimal action is selected; Among them, ∈ represents the exploration rate, θ online Indicates online network parameters, represents the set of legal queues for task i; j represents an action in the legal queue set, Q represents the Q-value function in reinforcement learning, and Heuristic represents the heuristic strategy function; The heuristic rule prioritizes assigning tasks to queues with high current load but low demand matching to avoid resource overload, as shown in formula (11): For each queue j, calculate its load score LoadScore j , as shown in formula (12): Indicates the CPU importance weight, Indicates the memory importance weight; CPU usage As shown in formula (13): availablecpu j Indicates the number of CPU cores currently available in queue j, C total Indicates the total CPU quota of queue j; Memory usage As shown in formula (14): availablememory j Indicates the current remaining available memory in queue j, M total Indicates the total memory quota of queue j; The pros and cons of scheduling actions are comprehensively evaluated by multi-objective weighting, as shown in formula (15): R=w balance *r balance +w taskbalance *r taskbalance +w dependencu *r dependency +w local *r local -λ·penalty (15); Among them, w balance Indicates the weight of resource load balancing score; r balance Indicates the resource load balancing score; w taskbalance represents the weight of the task quantity balance score; r taskbalance represents the task quantity balance score; w dependency Represents the weight of the dependent completion reward; r dependency Indicates dependency completion reward; w local represents the weight of the local utilization reward; r local represents the local utilization reward; λ represents the penalty coefficient of the global difference penalty; penalty represents the global difference penalty; The specific scoring items are as follows: Load balancing score r balance Indicates the balance of CPU / memory utilization of each queue, as shown in formula (16): Among them, y c Indicates the CPU utilization of each queue, u m represents the memory utilization of each queue, and σ(·) is the standard deviation, which measures the volatility of resource utilization; Task quantity balance score r taskbalance It represents the balance of the number of tasks in each queue, as shown in formula (17): Among them, n tasks Each represents the number of tasks in the queue, σ(n tasks ) represents the standard deviation of the number of tasks, max(n tasks ) represents the maximum number of tasks in all queues; Dependency Completion Reward dependency represents the penalty for the number of unfinished dependent tasks, as shown in formula (18): Among them, t d Indicates the dependent tasks of the current task. Represents an indicator function, the value is 1 if the dependent task is not completed, otherwise it is 0; Local utilization reward r local represents the CPU / memory utilization of the current queue, as shown in formula (19): in, Indicates the CPU utilization of the current queue j, Indicates the memory utilization of the current queue j; The global difference penalty is penalty, which represents the difference between the maximum and minimum CPU utilization of each queue, as shown in formula (20): penalty=max(u c )-min(u c ) (20); Among them, max(u c ) represents the maximum CPU utilization of all queues, min(u c ) represents the minimum CPU utilization among all queues; (5-4) Through the continuous interaction process, experience data is continuously collected and saved in the experience playback buffer. During each task scheduling process, an interaction sample in the form of a five-tuple is recorded, as shown in formula (21): (state,action,reward,next_state,done) (21); Among them, state represents the state of the current scheduling environment, action represents the scheduling decision made by the agent in the current state, reward represents the immediate feedback brought by the current scheduling action, next_state represents the next state of the environment after the action is executed, and done represents whether the current scheduling process has ended, which is a Boolean value; By continuously executing scheduling actions, five-tuples are gradually collected and stored in the experience replay pool; When the number of samples reaches a certain scale, they are sampled to form a training set for optimizing the Q-value function.

Images