Rigid constraint parallel task scheduling method based on deep Q network
By using an improved DQN end-to-end learning framework, which utilizes a state encoder and a Q-value evaluation network, the problem of balancing efficiency and quality in large-scale parallel task scheduling is solved, achieving efficient scheduling decisions and shortening task completion time.
Patent Information
- Application Number
- CN202510940395.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-09
- Publication Date
- 2025-10-28
AI Technical Summary
Existing technologies struggle to balance solution efficiency and result quality in large-scale parallel task scheduling, and traditional DRL architectures are ill-suited to the dynamic action space problem.
An improved DQN-based end-to-end learning framework is adopted, which generates state context vectors and evaluates the value of candidate actions through a state encoder and a Q-value evaluation network. Reinforcement learning is used for training to optimize the scheduling strategy.
It achieves efficient and high-quality scheduling decisions in large-scale parallel task scheduling, shortening the total task scheduling completion time.
Smart Images
Figure CN120849045A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer technology, and specifically relates to a rigid constraint parallel task scheduling method based on deep Q-value function networks. Background Technology
[0002] With the continuous improvement of computing power, massively parallel task scheduling, as an important means to improve the overall efficiency of computing systems, has long been a research focus in the field of computer science. In modern multi-core processors and cloud computing environments, the rigidly constrained parallel task scheduling problem has become one of the core research problems. Its goal is to efficiently allocate a large number of parallel tasks to multiple processors while meeting resource constraints, in order to optimize specific performance indicators, such as minimizing the maximum completion time. This problem exists widely in practical scenarios such as industrial manufacturing, port logistics, labor allocation, and cloud computing resource scheduling, and is usually characterized by a large number of tasks, complex dependencies, and strict response time requirements.
[0003] The rigidly constrained parallel task scheduling problem has been proven to be NP-hard. When faced with large-scale scheduling instances, traditional optimization methods, such as exact algorithms, although theoretically able to find the optimal solution, suffer from computational complexity that increases exponentially with the problem size, resulting in excessively long solution times and failing to meet the real-time requirements of practical applications.
[0004] To strike a balance between solution quality and computational efficiency, metaheuristic methods, such as genetic algorithms, tabu search, and particle swarm optimization, are widely used. These methods find high-quality solutions through iterative search, but their computation remains time-consuming when the problem size is large, especially in environments with frequently changing tasks. Another type of method is rule-based scheduling, such as priority scheduling rules (PDR), such as First-Come, First-Served (FCFS) and Longest Processing Time First (LPT). PDR is used because of its simplicity and fast computation speed, but its decisions are based on local information and lack global optimization capabilities, often making it difficult to obtain high-quality scheduling solutions, and its performance is highly sensitive to specific rules.
[0005] In recent years, Deep Reinforcement Learning (DRL), as a method with autonomous learning and rapid inference capabilities, has gradually become a powerful tool for solving combinatorial optimization problems. DRL methods model the scheduling process as a Markov Decision Process (MDP), learning scheduling policies through neural networks and achieving millisecond-level decision-making during the inference phase. However, a core challenge in parallel task scheduling problems is that the action space (i.e., the set of schedulable tasks) is dynamically changing; once a task is scheduled, it cannot be selected again. Traditional algorithms such as Deep Q-Networks (DQN) typically assume a fixed action space, making them difficult to apply directly to such problems.
[0006] Therefore, there is an urgent need to propose a novel rigid-constrained parallel task scheduling method based on DRL to overcome the limitations of existing methods in balancing solution efficiency and result quality when dealing with large-scale scheduling problems, while also addressing the difficulty of adapting traditional DRL architectures to dynamic action spaces. To address these challenges, this invention proposes an improved end-to-end learning framework based on DQN, aiming to minimize the total completion time in task scheduling. Summary of the Invention
[0007] The purpose of this invention is to provide a rigid constraint parallel task scheduling method based on deep Q-networks. The entire process is optimized through end-to-end training, enabling the Q-value evaluation network to learn accurate value judgment capabilities.
[0008] The technical solution adopted is:
[0009] In this invention, scheduling parallel tasks with rigid constraints is considered a sequential decision-making process, iteratively taking scheduling actions and selecting appropriate tasks for scheduling in each state until all tasks are scheduled. The workflow of the proposed method is as follows: Figure 1 As shown in the diagram, at each decision-making moment, the system takes the current task state and machine state as input and feeds them into the state encoder for processing. The state encoder encodes the input information to generate a state context vector containing global scheduling information. Subsequently, this state context information is combined with all currently available scheduling actions to generate a feature representation for each candidate action. These candidate action features are then input into the Q-value evaluation module, which evaluates the value of each candidate action through a neural network and outputs the corresponding Q-value. Based on this, the system selects the action with the highest Q-value, i.e., selects a task for scheduling. The above process is repeated until all tasks are scheduled, realizing an end-to-end scheduling decision optimization process.
[0010] The key to this technical solution lies in:
[0011] Apply the state encoder model to generate the system state context information at the current moment.
[0012] A Q-value evaluation network is applied to evaluate the features of candidate actions.
[0013] The system is trained using reinforcement learning-based methods to meet the comprehensive decision-making requirements of task ranking and machine allocation.
[0014] By conducting numerous "decision-reward-learning" iterations in a simulated environment and optimizing network parameters using an experience replay mechanism, the agent can learn an efficient and high-quality end-to-end scheduling strategy. Attached Figure Description
[0015] Figure 1This is a flowchart of the present invention.
[0016] Figure 2 This is a flowchart of the algorithm of the present invention.
[0017] Figure 3 This is a structural diagram of a state encoder.
[0018] Figure 4 The structure diagram of the network used for Q-value evaluation. Detailed Implementation
[0019] A rigidly constrained parallel task scheduling method based on deep Q-networks mainly includes the following steps:
[0020] Algorithm Design:
[0021] 1. Algorithm Idea
[0022] Algorithm flowchart as follows Figure 2 The algorithm flowchart shows the following: First, the scheduling process is modeled as an MDP decision process, where the system state is concisely defined as a composite feature vector containing processor available time information and aggregated remaining task information. To address the challenge of the constantly changing candidate task set in scheduling, this invention innovatively decomposes Q-value calculation into two stages: In the first stage, a state encoder compresses the current system state into an information-dense state context vector to capture the essence of the scheduling environment; in the second stage, an independent Q-value evaluation network receives this context vector and, combined with the characteristics of each candidate task, independently and sequentially evaluates the long-term value of that specific "state-action" pair. During decision-making, the agent uses this architecture to estimate all currently feasible scheduling tasks and selects the task with the highest Q-value for execution. After each decision, the system generates an immediate reward based on the impact on the maximum completion time and transitions to the next state. Finally, through numerous "decision-reward-learning" iterations in a simulated environment and by optimizing network parameters using an experience replay mechanism, the agent can learn an efficient and high-quality end-to-end scheduling strategy.
[0023] 2. Markov Process
[0024] To transform the parallel task scheduling problem into a form that can be handled by DQN, a Markov Decision Process (MDP) is used to model the problem. By rationally designing the state space, action space, reward function, and state transition mechanism, the reinforcement learning agent can gradually learn and optimize the scheduling strategy through interaction with the scheduling environment. The Markov Decision Process consists of five parts: a set of states, a set of actions, state transition probabilities, a reward function, and a discount factor.
[0025] State: The constructed state information mainly includes two parts: processor state and task state. Processor state: A vector corresponding to the number of processors represents the earliest available time for each processor. To avoid interference from the processor numbering order in the reinforcement learning model, the vector corresponding to the number of processors is sorted in ascending order before input, retaining only processor availability information, thereby enhancing the model's generalization ability. Task state: To prevent the state dimension from expanding rapidly with the number of tasks, an aggregated form is used to represent the overall characteristics of the set of currently unscheduled tasks. Specifically, this includes: the number of unscheduled tasks, the total processing time of all unscheduled tasks, and the total number of processors required for all unscheduled tasks. This design effectively reduces the dimensionality of the state space while preserving the representativeness of task information.
[0026] Action: At each scheduling decision moment, the agent selects a task from the currently available set of tasks for scheduling, constituting an action. The set of actions is dynamically changing and affected by the current system state. Whether a task can be scheduled depends on whether the number of processors required by the task is less than or equal to the number of idle processors in the current system. When all tasks are temporarily unschedulable, the agent will execute an implicit "wait" action, that is, skip the current decision moment, and the system time will automatically advance to the earliest moment when the next processor is released, before proceeding to the next round of scheduling.
[0027] Rewards: To guide the agent in learning better scheduling strategies, the reward mechanism is designed based on changes in the total system completion time. After each task scheduling, the agent receives an immediate reward based on the impact of the scheduling decision on the completion time. If the scheduling operation results in a small increase or even a decrease in the system's completion time, a higher reward is given; conversely, if the scheduling leads to a significant increase in completion time, a penalty is imposed. This reward mechanism effectively motivates the agent to learn in a direction that shortens the overall scheduling time, accelerates convergence, and improves the quality of the final scheduling scheme.
[0028] State Transition: When an agent chooses to perform a scheduling action (i.e., selects a task from candidate tasks for allocation) at a certain decision moment, the system state changes accordingly, entering the next decision stage. This state transition process mainly includes the following two updates: First, the processor state is updated. The selected scheduled task will occupy a certain number of processors, and the available time of its corresponding processors will be adjusted according to the processing time of the task. Specifically, the earliest available time of each occupied processor will be updated to the current system time plus the execution time of the task. Second, the task state is updated. The scheduled task will be removed from the set of tasks to be scheduled. At the same time, the total number of unfinished tasks, the total processing time of all unfinished tasks, and the total number of processors required for the remaining tasks will also be reduced accordingly. The entire scheduling process is organized in a complete round. Each round begins in the initial stage where the system has not yet performed any scheduling operations and all tasks are in the state of waiting to be scheduled. It continues to make several scheduling decisions until all tasks are successfully allocated to processors and executed, at which point the round ends.
[0029] 3. Model Interpretation
[0030] Figure 3The structure diagram of the state encoder is introduced. In this invention, the state encoder plays the role of the core feature extractor, and its structural design aims to efficiently encode the raw, high-dimensional system state information into an information-dense context vector. The internal structure of this encoder is preferably a multilayer perceptron, and the detailed data processing flow is as follows: For the input system state vector (composed of sorted processor states and aggregated task information), the input system state vector is first fed into the input layer of the state encoder. Subsequently, the data flows through one or more hidden layers for layer-by-layer, non-linear feature transformation. In each hidden layer, the input feature data first undergoes a linear transformation, which maps the features to a higher or lower dimensional space through a weight matrix. Next, the transformed result is processed by a non-linear activation function (ReLU). This step aims to break the limitations of linear mapping, enabling the network to learn and represent more complex feature relationships. The feature vector output from one hidden layer becomes the input to the next hidden layer, repeating the above "linear transformation + non-linear activation" process. After deep processing through all hidden layers, the final feature vector is fed into an output linear layer. This output linear layer, without any additional non-linear activation function, serves to integrate and adjust the dimensions of the deep features extracted from the last hidden layer, generating a fixed-dimensional state context vector. This output context vector encapsulates all the decision-making-valuable information extracted from the original state and is passed to the subsequent Q-value evaluation network module in a stable and compact form. The entire process is optimized through end-to-end training, enabling the state encoder to automatically learn the most effective state representation.
[0031] Figure 4The structure diagram of the Q-value evaluation network is introduced. The Q-value evaluation network is the second core component of the deep Q-network architecture proposed in this invention. Its design aims to accurately evaluate the long-term value (i.e., Q-value) of performing a specific candidate action based on the global environmental information provided by the state encoder. The internal structure of this evaluation network is preferably a multilayer perceptron. The detailed data processing flow of the multilayer perceptron is as follows: For the input composite feature vector (composed of the state context vector from the state encoder and the feature vector of the single candidate task to be evaluated), this composite feature vector is first fed into the input layer of the Q-value evaluation network. This concatenation step is crucial, as it combines global environmental information with local action information, providing a complete input for accurate evaluation. Subsequently, the data flows through one or more hidden layers for layer-by-layer, non-linear feature transformation. In each hidden layer, the input feature data first undergoes a linear transformation, which maps the features to another feature space through a weight matrix. Then, the transformed result is processed by a ReLU, a step designed to break the limitations of linear mapping, enabling the network to learn and represent more complex value relationships between states and actions. The feature vector output from one hidden layer is used as the input to the next hidden layer, repeating the "linear transformation + non-linear activation" process. After deep processing through all hidden layers, the final feature vector is fed into an output linear layer. This output linear layer does not have an additional non-linear activation function and has only one neuron. Its function is to integrate the final deep features and output a scalar value, which is the desired Q-value. This output Q-value represents the expected cumulative reward of performing a specific candidate action in the current state and is passed to the final decision-making module. By repeating this evaluation process for all candidate tasks, the agent obtains a list of Q-values and selects the optimal action accordingly. The entire process is optimized through end-to-end training, enabling the Q-value evaluation network to learn accurate value judgment capabilities.
[0032] Implementation Cases
[0033] 1. Dataset
[0034] This invention tested five datasets (A to E), covering task scheduling instances from small to large scale, aiming to comprehensively evaluate the accuracy, efficiency, and scalability of the proposed method under different complexities. The small-scale environment included datasets A (3–5 processors, 5–10 tasks, 180 instances), B (3–5 processors, 11–20 tasks, 300 instances), and C (6–10 processors, 20–40 tasks, 150 instances), totaling 630 instances. The large-scale environment included datasets D (15–30 processors, 60–150 tasks, 40 instances) and E (30–50 processors, 250–500 tasks, 30 instances), totaling 70 instances. The five datasets, totaling 700 instances, fully cover various scheduling scenarios, validating the adaptability and robustness of the proposed method under different scales and complexities.
[0035] 2. Parameter settings
[0036] The DQN agent consists of a state encoder and a Q-value evaluation head, both of which are multilayer perceptrons with 128 hidden layers and ReLU activation function. During training, the Adam optimizer (learning rate set to 1×10⁻²) is used with a smoothed L1 loss function and a discount factor of 0.99. An ε-greedy strategy is employed to balance exploration and exploitation, with ε decreasing linearly from 1.0 to 0.1. Experience is stored in a 10,000-sample replay pool, with 32 samples sampled per update. The target network synchronizes with the main network every 10 epochs. The total number of training epochs is set to 1000, and an early stopping mechanism is introduced: training is terminated early if the average completion time does not decrease significantly within 100 consecutive epochs. All hyperparameters are tuned on dataset A and maintained consistent across all other tests.
[0037] 3. Experimental Results
[0038] Table 1 (Results) shows the final results obtained using each dataset, i.e., the maximum completion time.
[0039] Instance size DQN model Dataset A 223.75 Dataset B 427.34 Dataset C 741.43 Dataset D 2458.10 Dataset E 8716.20
[0040] The entire process is optimized through end-to-end training, enabling the Q-value evaluation network to learn accurate value judgment capabilities.
Claims
1. A rigidly constrained parallel task scheduling method based on deep Q-networks, characterized in that... Includes the following steps: First, the scheduling process is modeled as an MDP decision process, where the system state is defined as a composite feature vector containing processor available time information and aggregated remaining task information. The Q-value calculation is decomposed into two stages: In the first stage, a state encoder compresses the current system state into an information-dense state context vector to capture the essence of the scheduling environment; In the second stage, an independent Q-value evaluation network receives this context vector and, in conjunction with the characteristics of each candidate task, independently and one by one evaluates the long-term value of the specific "state-action" pair. When making a decision, the agent evaluates all currently feasible scheduling tasks and selects the task with the highest Q value to execute. After each decision, the system generates an immediate reward based on the impact on the maximum completion time and moves to the next state. Ultimately, through numerous "decision-reward-learning" iterations in a simulated environment and by optimizing network parameters using an experience replay mechanism, the agent is able to learn an efficient and high-quality end-to-end scheduling strategy.
2. The rigid constraint parallel task scheduling method based on deep Q-networks according to claim 1, characterized in that... Includes the following steps: The parallel task scheduling problem is transformed into a form that can be handled by DQN, and MDP is used to model the parallel task scheduling problem. By reasonably designing the state space, action space, reward function and state transition mechanism, the reinforcement learning agent can gradually learn and optimize the scheduling strategy in the process of interacting with the scheduling environment. A Markov decision process includes a set of states, a set of actions, state transition probabilities, a reward function, and a discount factor.
3. The rigid constraint parallel task scheduling method based on deep Q-networks according to claim 2, characterized in that... Includes the following steps: State: The constructed state information includes two parts: processor state and task state. Processor state: A vector corresponding to the number of processors is used to represent the earliest available time of each processor. The vector corresponding to the number of processors is sorted in ascending order before input, retaining only the processor availability information. Task state: To prevent the state dimension from expanding rapidly with the increase of the number of tasks, an aggregate form is used to represent the overall characteristics of the set of tasks that have not yet been scheduled. Specifically, this includes: the number of unscheduled tasks, the total processing time for all unscheduled tasks, and the total number of processors required for all unscheduled tasks.
4. The rigid constraint parallel task scheduling method based on deep Q-networks according to claim 2, characterized in that... Includes the following steps: Action: At each scheduling decision moment, the agent selects a task from the currently available set of tasks for scheduling, which constitutes an action.
5. A rigid constraint parallel task scheduling method based on deep Q-networks according to claim 2, characterized in that... Includes the following steps: Rewards: After each task scheduling, the agent will receive an immediate reward based on the impact of the scheduling decision on the completion time; if the scheduling operation results in a small increase or even a decrease in the system's completion time, a higher reward will be given; conversely, if the scheduling leads to a significant increase in the completion time, a penalty will be given.
6. A rigid constraint parallel task scheduling method based on a deep Q-network according to claim 2, characterized in that... Includes the following steps: State transition: When an agent chooses to execute a certain scheduling action at a certain decision moment, the system state will change accordingly and enter the next decision stage; the state transition process includes the following two updates: First, the processor state is updated; the selected scheduled task will occupy a certain number of processors, and the corresponding processor available time will be adjusted according to the processing time of the selected scheduled task. Secondly, there is the updating of task status; The scheduled task will be removed from the set of tasks to be scheduled; at the same time, the total number of incomplete tasks, the total processing time of all incomplete tasks, and the total number of processors required for the remaining tasks will also be reduced accordingly. The entire scheduling process is organized in a complete round; each round begins at the initial stage where the system has not yet performed any scheduling operations and all tasks are in a state of waiting to be scheduled, and continues to make several scheduling decisions until all tasks are successfully assigned to processors and executed, at which point the round ends.
7. A rigidly constrained parallel task scheduling method based on a deep Q-network according to claim 1, characterized in that... Includes the following steps: The internal structure of the state encoder is preferably a multilayer perceptron. The detailed data processing flow of the encoder is as follows: For the input system state vector, the input system state vector is first fed into the input layer of the state encoder; then, the data flows through one or more hidden layers for layer-by-layer, non-linear feature transformation; in each hidden layer, the input feature data first undergoes a linear transformation, which maps the features to a higher or lower dimension space through a weight matrix; then, the transformed result is processed by a non-linear activation function, and the feature vector output from one hidden layer is used as the input of the next hidden layer, repeating the above "linear transformation + non-linear activation" process; after deep processing through all hidden layers, the final feature vector is fed into an output linear layer; this output linear layer does not have a non-linear activation function attached, and generates a state context vector with a fixed dimension, which is passed to the subsequent Q-value evaluation network module.
8. A rigidly constrained parallel task scheduling method based on a deep Q-network according to claim 1, characterized in that... Includes the following steps: The internal structure of the Q-value evaluation network is a multilayer perceptron. The detailed data processing flow of the multilayer perceptron is as follows: The input is first fed into the input layer of the Q-value evaluation network, and then the data flows through one or more hidden layers for layer-by-layer, non-linear feature transformation. In each hidden layer, the input feature data first undergoes a linear transformation, which maps the features to another feature space through a weight matrix. Then, the transformed result is processed by a ReLU, and the feature vector output from one hidden layer becomes the input of the next hidden layer, repeating the above "linear transformation + non-linear activation" process. After deep processing through all hidden layers, the final feature vector is fed into an output linear layer. This output linear layer does not have an additional non-linear activation function, and the number of neurons in the output linear layer is 1. The role of the output linear layer is to integrate the final deep features and output a scalar value, which is the Q-value to be obtained. The Q-value of this output represents the expected cumulative reward that can be obtained by performing the specific candidate action in the current state, and is passed to the final decision-making module. By repeating this evaluation process for all candidate tasks, the agent can obtain a list of Q-values and select the optimal action accordingly.