A hierarchical dynamic scheduling method based on the policy optimization DDQN algorithm
By constructing a batching and scheduling agent based on the policy optimization DDQN algorithm, and combining Markov decision process and self-attention mechanism, the complex dynamic disturbance and multi-objective optimization problem in the scheduling problem of reentrant hybrid flow shop with batch processor is solved, achieving efficient and accurate scheduling optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHWEST JIAOTONG UNIV
- Filing Date
- 2025-01-15
- Publication Date
- 2026-05-26
AI Technical Summary
Existing dynamic scheduling methods suffer from low efficiency, inability to perform global optimization, and inability to handle complex dynamic disturbances in reentrant hybrid flow shop scheduling problems with batch processors. In particular, in production scenarios such as semiconductor manufacturing and cold drawing of steel pipes, the scheduling challenges caused by multiple batch processing stages and the arrival of new workpieces have not been effectively solved.
A hierarchical dynamic scheduling method based on the policy optimization DDQN algorithm is adopted to construct a batching agent and a scheduling agent. Combined with the Markov decision process model, the state, reward and action are defined. Through heuristic variable threshold batching strategy and self-attention mechanism, the batching and scheduling of workpieces are optimized to achieve collaborative scheduling with multi-stage batch processing and reentrancy characteristics.
It improves scheduling efficiency and accuracy, can more accurately reflect dynamic changes in the workshop, optimizes the dynamics of workpiece batching and the collaborative capabilities of intelligent agents, solves multi-objective optimization problems that are difficult to handle with traditional methods, and improves machine utilization and workpiece completion time.
Smart Images
Figure CN119987302B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of hybrid flow workshop scheduling optimization, and particularly relates to a hierarchical dynamic scheduling method based on the strategy optimization DDQN algorithm. Background Technology
[0002] The reentrant hybrid pipelined room scheduling problem with batch processing machines is a classic extension of the hybrid pipelined room problem, more closely related to manufacturing scenarios such as semiconductor manufacturing, blade coating, and steel pipe cold drawing. In recent years, it has also attracted attention due to the simultaneous consideration of batch processing and reentrancy constraints. Current research only considers problems with a single batch processing stage, but production scenarios often involve multiple batch processing stages. For example, oxidation and diffusion processes in semiconductor manufacturing are both batch processing stages, and not necessarily continuous. Therefore, it is necessary to study reentrant hybrid pipelined room scheduling problems with multiple batch processing stages. Furthermore, the actual reentrant hybrid pipelined room scheduling problem with batch processing machines involves various dynamic perturbations, such as the arrival of new workpieces. Therefore, the problem is extended to a dynamic reentrant hybrid pipelined room scheduling problem with batch processing machines.
[0003] Current common dynamic scheduling methods in workshops include priority rules, metaheuristic methods, and deep reinforcement learning. Priority rules and metaheuristic methods each have some shortcomings in solving dynamic scheduling problems. Priority rules are highly efficient but lack re-optimization and global optimization capabilities; metaheuristic optimization takes a long time; deep reinforcement learning can learn to control the interaction between the decision model and the scheduling environment, making decisions quickly while ensuring both current and long-term scheduling performance. In particular, the DDQN series of deep reinforcement learning methods, with its double-Q network structure and experience replay mechanism, can effectively solve problems such as high-dimensional state space, complex action selection, and high real-time requirements in workshop dynamic scheduling problems. Summary of the Invention
[0004] To overcome the shortcomings of existing technologies in solving the dynamic scheduling problem of reentrant hybrid production lines with batch processors, this invention provides a hierarchical dynamic scheduling method based on the strategy optimization DDQN algorithm.
[0005] This invention discloses a hierarchical dynamic scheduling method based on the policy optimization DDQN algorithm, used to solve the dynamic scheduling problem of a reentrant hybrid flow shop with a batch processor, specifically as follows:
[0006] A. Description and assumptions of the scheduling problem.
[0007] There are n initially arriving and randomly arriving workpieces of Q types. Each workpiece goes through one stage according to its process route, which includes |BS| batch processing stages, 1≤|BS|≤1; each stage has m... s Taiwan uncorrelated parallel machine, ms ≥1, the batch processing stage contains batch processing machines that can process a limited number of workpieces at the same time, not exceeding the machine capacity, while the single processing stage contains single processing machines for single-piece processing; each workpiece can choose one machine in the stage for processing, there is no priority constraint between workpieces, but there is a sequential constraint between the processing steps of each workpiece, workpieces may need to be processed repeatedly at one station, and workpieces are allowed to skip certain stages of processing at different access times.
[0008] B. Objective function and constraints.
[0009] Objective function:
[0010] Minimizing completion time, minimizing weighted total delay, and maximizing average machine utilization are expressed as follows:
[0011]
[0012] Wherein, equation (1) represents the completion time of all workpieces, j represents the workpiece index, and o j C represents the final process for workpiece j. j,oj Let w represent the completion time of workpiece j. Equation (2) represents the weighted total delay time of all workpieces. j D represents the urgency level of each workpiece j. j represents the due date of each workpiece j. Equation (3) represents the average machine utilization rate, o represents the process index of the workpiece, s is the stage index, k is the machine index, and m s M is the number of machines in stage s. s P is the set of machines in stage s. j,o,s,k Y represents the processing time of operation o for workpiece j on machine k in stage s. j,o,s,k Whether process o, representing workpiece j, is processed on machine k in stage s; if yes, Y j,o,s,k =1, otherwise Y j,o,s,k =0.
[0013] Constraints:
[0014]
[0015] Σ j,j'∈b Z j,j' =1 (11)
[0016]
[0017]
[0018] Equation (4) indicates that all workpieces j must arrive before processing can begin, S j,1A represents the start time of process 1 for workpiece j. j This represents the arrival time of workpiece j. Equation (5) indicates that each process of each workpiece can only be processed on one machine in one stage. Equation (6) indicates that in a single processing stage, each machine can only process one workpiece at a time, S. j,o X represents the start time of process o for workpiece j. jo,j’o’,sk Whether process o of workpiece j is processed before workpiece o' of workpiece j'. If so, X jo,j’o’,sk =1, otherwise X jo,j’o’,sk =0, N is a sufficiently large positive number, O s This represents the set of operations that need to be processed in stage s. Equation (7) represents the sequencing constraint between operations. For two operations of the same job with sequence constraints, the latter must be processed after the former has been processed. Equation (8) represents the sequencing constraint between workpiece j cycles. Accessing a workpiece in the later cycle must be processed after the previous cycle has been processed. j,r C represents the start time of the r-th cycle of workpiece j. j,r This represents the end processing time of the r-th cycle for workpiece j. Equation (9) indicates that batch processing is performed before processing to ensure that each job is assigned to only one batch during batch processing, where b represents the batch index, B s,k α represents the set of batches processed on machine k in stage s. j,b This indicates whether workpiece j is assigned to batch b. If so, α j,b =1, otherwise α j,b =0. Equation (10) indicates that a batch can only be processed on one batch processing machine, BS represents the set of batch processing stages, β b,s,k This indicates whether batch b is processed on machine k in stage s; if so, β. b,s,k =1, otherwise, β b,s,k =0. Equation (11) indicates that only workpieces j belonging to the same process type can form a batch b, Z j,j' This indicates whether workpiece j and workpiece j' belong to the same process type. If they do, then Z. j,j' =1, otherwise Z j,j' =0. Equation (12) represents the capacity limit of the batch processing machine, ensuring that the total number of jobs in the batch processed by the machine does not exceed its maximum capacity, V s,k S represents the processing capacity of machine k in stage s. Equation (13-14) represents the sequence priority constraint, ensuring that different batches processed by the same batch processing machine follow the order of processing. b,k C represents the start time of batch b on machine k. b,k γ represents the end processing time of batch b on machine k. b,b' Indicates whether batch b is processed before batch b'. If so, γb,b' =1; otherwise γ b,b' =0.
[0019] C. The hierarchical dynamic scheduling method based on strategy optimization DDQN (HSDDQN) is used to solve the dynamic scheduling problem of reentrant hybrid flow shop.
[0020] First, a hierarchical structure based on the policy optimization DDQN algorithm is proposed, which constructs a batching agent (BA) and a scheduling agent (SA) to solve the work batching and scheduling subproblems in the dynamic scheduling problem of a reentrant hybrid flow shop with batch processing machine, respectively. The batching agent BA decides whether to execute a batch of jobs, while the scheduling agent SA determines the priority order of jobs or batches and the machine selection.
[0021] Secondly, based on the two intelligent agents mentioned above, a Markov decision process model suitable for the dynamic scheduling problem of reentrant hybrid flow shop with batch processing machine is designed. It combines multi-stage batch processing and reentrancy characteristics, defines state, reward and action, and combines heuristic variable threshold batching strategy (VTBP) with dynamic job batching.
[0022] Markov decision process model:
[0023] (1) Status:
[0024] Considering the stage-based processing characteristics of a hybrid assembly line workshop, the features of each agent are designed based on the processing stage at the decision-making time point. Six state features are extracted as inputs for the Business Intelligence (BA), specifically:
[0025]
[0026] Twelve state features are extracted as input to the SA, specifically:
[0027]
[0028]
[0029] (2) Action:
[0030] A heuristic variable threshold batching strategy (VTBP) is used to construct the action space of the batching agent. The VTBP strategy is as follows: First, the process type of the job is determined; then, the batch capacity threshold is determined; next, the jobs are sorted according to certain rules; finally, a batch is formed by selecting a threshold number of jobs. During scheduling, all jobs in the processing stage buffer are sorted in ascending order according to their arrival time at that stage and classified by job type. Then, the average waiting time for each type of job is calculated; the type q with the longest average waiting time is selected for batch processing. Next, the threshold V of the current batch needs to be determined. b Let Os,q,t This represents the set of job operations of type q that need to be batch processed in stage s but have not yet reached the current time t, in BN. s,q Indicates the type and number of jobs in the current stage s; if O s,q,t If BN ≠ 0, it means that there are still batch processing operations for jobs that have not been completed in the current stage; in this case, if BN s,q >|O s,q,t |, then V b =min(BN) s,q V s,k If BN s,q ≤|O s,q,t |, then V b =0; otherwise, if O s,q,t =0, then V b =min(BN) s,q V s,k ).
[0031] The batching scheme constructed based on different batching rules directly affects the optimization results. When batch processing jobs within the buffer, the consistency of processing time is given priority, that is, jobs with similar processing times are grouped into the same batch, thereby reducing wasted processing time. In addition, in order to minimize total latency, completion time and reduce the waiting time of jobs within the buffer, three batching rules are proposed to balance processing time consistency with other time-based performance metrics. A weighted batch quality index is also introduced. This index takes into account the consistency of delivery time and processing time of operations within the buffer zone, as shown in Equation (15):
[0032]
[0033] In the formula, buffer s q represents the set of jobs in stage s buffer during the batch scheduling decision process. j =q j' This indicates that only jobs belonging to the same job family can be grouped for batch processing. θ is the weighting coefficient, and P... j,o,s,k D represents the processing time required for the current batch operation of job j on machine k. j c represents the delivery time of task j. j A represents the remaining shortest processing time for task j. s,j Indicates the arrival time of task j in stage s; batch quality index The smaller the value, the higher the priority for batch processing of the job; jobs are processed according to... Sort in ascending order to determine the batch sequence.
[0034] The action space of BA is as follows:
[0035]
[0036]
[0037] The scheduling point is set to machine idle time; the SA's action space includes job selection and machine selection; in the batch processing stage, SA schedules batches already formed by BA; in the individual processing stage, the job selection rules and machine selection rules are as follows:
[0038]
[0039] (3) Rewards:
[0040] The total weighted delay of the workpiece is subdivided into each scheduling decision of the decision model BA. After the decision model BA executes each step, an immediate reward is generated to reflect the impact of the action. The immediate reward of BA at time t is shown in Equation (16). In order to make the two objectives of minimizing the maximum completion time and maximizing machine utilization reflected in the actions of the same decision model, the difference between the average machine utilization and the difference in completion time between each scheduling step is used as the immediate reward of the decision model SA. The immediate reward of SA at time t is shown in Equation (17). In order to ensure that the guidance of the average machine utilization and the difference in completion time between each scheduling step on the behavior of the decision model is of the same order of magnitude, α1 and α2 are introduced.
[0041]
[0042] r SA =α1[U ave (t')-U ave (t)]-α2[C step (t')-C step (t)] (17)
[0043] in, This represents the number of jobs in the job pool at time t in stage p. Z represents the number of workpieces in batch b, and t′ represents the next decision time. If the due date of a workpiece is less than the next decision time, Z is 1; otherwise, Z is 0. If a workpiece completes its current process at the next decision time, t′ represents the next decision time. m It equals the completion time of that process for that workpiece; otherwise, t m It equals the next decision time t′.
[0044] Finally, to further enhance the effectiveness and generalization ability of the hierarchical dynamic scheduling method, two improvement strategies are proposed:
[0045] a) Combining a mask-based action selection strategy with an ε-greedy strategy to limit the action space of each agent at different decision points and improve the efficiency of collaborative decision-making.
[0046] b) Use a soft-start network update strategy for offline training to accelerate learning and convergence.
[0047] The beneficial technical effects of this invention are as follows:
[0048] 1. This invention addresses the dynamic scheduling problem in reentrant hybrid production lines with batch processors, constructing a dynamic scheduling model aimed at improving scheduling efficiency by minimizing completion time, total delay, and maximizing average machine utilization. The model fully considers the arrival of new orders, the constraints of incompatible workpiece families, and the capacity characteristics of batch processors, building a comprehensive scheduling framework. Furthermore, considering the complexity and dynamic nature of the problem, a corresponding Markov decision model is designed. The model defines the agent's state, action space, and reward mechanism, and achieves an efficient scheduling decision-making process by constructing a highly adaptive model network.
[0049] 2. The dynamic scheduling model and Markov decision model proposed in this invention can more accurately reflect the dynamic changes in the workshop environment, especially under complex conditions such as the arrival of new orders and incompatibility with workpiece families, enabling more precise scheduling optimization. Secondly, the dual-depth Q-network based on the self-attention mechanism can effectively handle complex scheduling problems, significantly improving the efficiency and accuracy of scheduling decisions through cooperation and interaction among agents. Finally, through heuristic variable threshold batching strategies and agent improvement strategies, not only is the dynamic nature of workpiece batching enhanced, but also the collaborative ability, generalization ability, and decision accuracy of agents are strengthened, thereby further optimizing system performance and solving multi-objective optimization problems that traditional scheduling methods struggle to address. Attached Figure Description
[0050] Figure 1 This invention presents the principle architecture of the hierarchical dynamic scheduling method based on strategy optimization DDQN.
[0051] Figure 2 This refers to the structure of an intelligent agent network. Detailed Implementation
[0052] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0053] This invention discloses a hierarchical dynamic scheduling method based on the policy optimization DDQN algorithm, used to solve the dynamic scheduling problem of a reentrant hybrid flow shop with a batch processor, specifically as follows:
[0054] A. Description and assumptions of the scheduling problem.
[0055] There are n initially arriving and randomly arriving workpieces of Q types. Each workpiece goes through one stage according to its process route, which includes |BS| batch processing stages, 1≤|BS|≤1; each stage has m... s Taiwan uncorrelated parallel machine, m s ≥1, the batch processing stage contains batch processing machines that can process a limited number of workpieces at the same time, not exceeding the machine capacity, while the single processing stage contains single processing machines for single-piece processing; each workpiece can choose one machine in the stage for processing, there is no priority constraint between workpieces, but there is a sequential constraint between the processing steps of each workpiece, workpieces may need to be processed repeatedly at one station, and workpieces are allowed to skip certain stages of processing at different access times.
[0056] B. Objective function and constraints.
[0057] Objective function:
[0058] Minimizing completion time, minimizing weighted total delay, and maximizing average machine utilization are expressed as follows:
[0059]
[0060] Wherein, equation (1) represents the completion time of all workpieces, j represents the workpiece index, and o j C represents the final process for workpiece j. j,oj Let w represent the completion time of workpiece j. Equation (2) represents the weighted total delay time of all workpieces. j D represents the urgency level of each workpiece j. j represents the due date of each workpiece j. Equation (3) represents the average machine utilization rate, o represents the process index of the workpiece, s is the stage index, k is the machine index, and m s M is the number of machines in stage s. s P is the set of machines in stage s. j,o,s,k Y represents the processing time of operation o for workpiece j on machine k in stage s. j,o,s,k Whether process o, representing workpiece j, is processed on machine k in stage s; if yes, Y j,o,s,k =1, otherwise Y j,o,s,k =0.
[0061] Constraints:
[0062]
[0063] ∑ j,j'∈b Z j,j' =1(11)
[0064]
[0065]
[0066] Equation (4) indicates that all workpieces j must arrive before processing can begin, S j,1 A represents the start time of process 1 for workpiece j. j This represents the arrival time of workpiece j. Equation (5) indicates that each process of each workpiece can only be processed on one machine in one stage. Equation (6) indicates that in a single processing stage, each machine can only process one workpiece at a time, S. j,o X represents the start time of process o for workpiece j. jo,j’o’,sk Whether process o of workpiece j is processed before workpiece o' of workpiece j'. If so, X jo,j’o’,sk =1, otherwise X jo,j’o’,sk =0, N is a sufficiently large positive number, O s This represents the set of operations that need to be processed in stage s. Equation (7) represents the sequencing constraint between operations. For two operations of the same job with sequence constraints, the latter must be processed after the former has been processed. Equation (8) represents the sequencing constraint between workpiece j cycles. Accessing a workpiece in the later cycle must be processed after the previous cycle has been processed. j,r C represents the start time of the r-th cycle of workpiece j. j,r This represents the end processing time of the r-th cycle for workpiece j. Equation (9) indicates that batch processing is performed before processing to ensure that each job is assigned to only one batch during batch processing, where b represents the batch index, B s,k α represents the set of batches processed on machine k in stage s. j,b This indicates whether workpiece j is assigned to batch b. If so, α j,b =1, otherwise α j,b =0. Equation (10) indicates that a batch can only be processed on one batch processing machine, BS represents the set of batch processing stages, β b,s,k This indicates whether batch b is processed on machine k in stage s; if so, β. b,s,k =1, otherwise, β b,s,k =0. Equation (11) indicates that only workpieces j belonging to the same process type can form a batch b, Z j,j' This indicates whether workpiece j and workpiece j' belong to the same process type. If they do, then Z. j,j' =1, otherwise Z j,j' =0. Equation (12) represents the capacity limit of the batch processing machine, ensuring that the total number of jobs in the batch processed by the machine does not exceed its maximum capacity, V s,k S represents the processing capacity of machine k in stage s. Equation (13-14) represents the sequence priority constraint, ensuring that different batches processed by the same batch processing machine follow the order of processing. b,kC represents the start time of batch b on machine k. b,k γ represents the end processing time of batch b on machine k. b,b' Indicates whether batch b is processed before batch b'. If so, γ b,b' =1; otherwise γ b,b' =0.
[0067] C. The hierarchical dynamic scheduling method based on policy optimization DDQN (HSDDQN) solves the dynamic scheduling problem of reentrant hybrid flowrooms. The principle of the hierarchical dynamic scheduling method based on policy optimization DDQN is as follows: Figure 1 As shown.
[0068] First, a hierarchical structure based on the policy optimization DDQN algorithm is proposed, which constructs a batching agent (BA) and a scheduling agent (SA) to solve the work batching and scheduling subproblems in the dynamic scheduling problem of a reentrant hybrid flow shop with batch processing machine, respectively. The batching agent BA decides whether to execute a batch of jobs, while the scheduling agent SA determines the priority order of jobs or batches and the machine selection.
[0069] Secondly, based on the two intelligent agents mentioned above, a Markov decision process model suitable for the dynamic scheduling problem of reentrant hybrid flow shop with batch processing machine is designed. It combines multi-stage batch processing and reentrancy characteristics, defines state, reward and action, and combines heuristic variable threshold batching strategy (VTBP) with dynamic job batching.
[0070] Markov decision process model:
[0071] (1) Status:
[0072] Considering the stage processing characteristics of the hybrid assembly line workshop, the features of each agent are designed according to the processing stage at the decision point. Six state features are extracted as inputs to the BA. A detailed description of each state feature is shown in Table 1.
[0073] Table 1 BA State Design
[0074]
[0075] Twelve state features are extracted as input to SA, and a detailed description of each state feature is shown in Table 2.
[0076] Table 2 SA State Design
[0077]
[0078]
[0079] (2) Action:
[0080] A heuristic variable threshold batching strategy (VTBP) is used to construct the action space of the batching agent. The VTBP strategy is as follows: First, the process type of the job is determined; then, the batch capacity threshold is determined; next, the jobs are sorted according to certain rules; finally, a batch is formed by selecting a threshold number of jobs. During scheduling, all jobs in the processing stage buffer are sorted in ascending order according to their arrival time at that stage and classified by job type. Then, the average waiting time for each type of job is calculated; the type q with the longest average waiting time is selected for batch processing. Next, the threshold V of the current batch needs to be determined. b Let O s,q,t This represents the set of job operations of type q that need to be batch processed in stage s but have not yet reached the current time t, in BN. s,q Indicates the type and number of jobs in the current stage s; if O s,q,t If BN ≠ 0, it means that there are still batch processing operations for jobs that have not been completed in the current stage; in this case, if BN s,q >|O s,q,t |, then V b =min(BN) s,q V s,k If BN s,q ≤|O s,q,t |, then V b =0; otherwise, if O s,q,t =0, then V b =min(BN) s,q V s,k ).
[0081] The batching scheme constructed based on different batching rules directly affects the optimization results. When batch processing jobs within the buffer, the consistency of processing time is given priority, that is, jobs with similar processing times are grouped into the same batch, thereby reducing wasted processing time. In addition, in order to minimize total latency, completion time and reduce the waiting time of jobs within the buffer, three batching rules are proposed to balance processing time consistency with other time-based performance metrics. A weighted batch quality index is also introduced. The index takes into account the consistency of delivery time and processing time of operations within the buffer zone, as shown in Equation (15).
[0082] Decision-makers can adjust the weighting coefficients to align them with their preferred optimization objectives.
[0083]
[0084] In the formula, buffer s q represents the set of jobs in stage s buffer during the batch scheduling decision process. j =q j'This indicates that only jobs belonging to the same job family can be grouped for batch processing. θ is the weighting coefficient, and P... j,o,s,k D represents the processing time required for the current batch operation of job j on machine k. j c represents the delivery time of task j. j A represents the remaining shortest processing time for task j. s,j Indicates the arrival time of task j in stage s; batch quality index The smaller the value, the higher the priority for batch processing of the job; jobs are processed according to... The order is determined by ascending sorting. The action space of the BA is shown in Table 3.
[0085] Table 3BA's Action Space
[0086]
[0087] When V b When the value is 0, BA selects the "None" action, and the job will continue to wait;
[0088] The scheduling point is set to machine idle time; the action space of SA includes job selection and machine selection; in the batch processing stage, SA scheduling is based on batches already formed by BA. In the individual processing stage, the job selection rules and machine selection rules are shown in Table 4.
[0089] Table 4 SA's Action Space
[0090]
[0091] (3) Rewards:
[0092] The total weighted delay of the workpiece is subdivided into each scheduling decision of the decision model BA. After the decision model BA executes each step, an immediate reward is generated to reflect the impact of the action. The immediate reward of BA at time t is shown in Equation (16). In order to make the two objectives of minimizing the maximum completion time and maximizing machine utilization reflected in the actions of the same decision model, the difference between the average machine utilization and the difference in completion time between each scheduling step is used as the immediate reward of the decision model SA. The immediate reward of SA at time t is shown in Equation (17). In order to ensure that the guidance of the average machine utilization and the difference in completion time between each scheduling step on the behavior of the decision model is of the same order of magnitude, α1 and α2 are introduced.
[0093]
[0094] r SA =α1[U ave (t')-U ave (t)]-α2[C step (t')-C step(t)] (17)
[0095] in, This represents the number of jobs in the job pool at time t in stage p. Z represents the number of workpieces in batch b, and t′ represents the next decision time. If the due date of a workpiece is less than the next decision time, Z is 1; otherwise, Z is 0. If a workpiece completes its current process at the next decision time, t′ represents the next decision time. m It equals the completion time of that process for that workpiece; otherwise, t m It equals the next decision time t′.
[0096] Finally, to further enhance the effectiveness and generalization ability of the hierarchical dynamic scheduling method, two improvement strategies are proposed:
[0097] a) Combining a mask-based action selection strategy with an ε-greedy strategy to limit the action space of each agent at different decision points and improve the efficiency of collaborative decision-making.
[0098] In this problem, the Business Analytics (BA) needs to determine the processing mode for the current stage and select actions from different sets of available actions based on the mode. When the processing stage is in batch processing mode, the BA can select appropriate job batching rules. However, in a single processing stage, batching rules cannot be included in the set of available actions. To reduce the dimensionality of the action space in complex environments and help the decision model learn task knowledge more effectively and efficiently, a masking mechanism is adopted. This mechanism divides the overall action space into multiple regions, thereby forming specific action spaces for different states.
[0099] The ultimate learning objective of the decision-making agent is the distribution of action scores. Simply preventing the agent from selecting inappropriate actions does not accelerate the convergence process. Therefore, a masking policy is typically added to the value function vector before action selection. The masked values are then fed into the neural network for training, combined with the ε-greedy policy from the DDQN algorithm. The action constraint selection method combining the masking mechanism and the ε-greedy policy is shown in Algorithm 1.
[0100]
[0101]
[0102] b) Use a soft-start network update strategy for offline training to accelerate learning and convergence.
[0103] This invention proposes a soft-start target network update strategy. This strategy employs a gradual update process, with a high update frequency in the early stages of training, gradually decreasing as training progresses. The soft-start strategy adjusts the target network update frequency according to the training progress, more flexibly adapting to the training needs at different stages, thereby accelerating agent convergence and reducing agent instability. The pseudocode for the soft-start strategy is shown in Algorithm 2, where C = 20, C... f =200, δ=random(0.75,1).
[0104]
[0105] Network structure:
[0106] Fully connected networks are widely used in scheduling problems due to their ability to flexibly model complex environments and tasks. However, traditional network architectures often suffer from bloated structures and a tendency to get trapped in local optima, especially when the input feature dimensionality is high. Therefore, this paper introduces a self-attention mechanism, enabling the model to dynamically adjust its attention distribution, enhancing the flexibility and efficiency of information processing in complex environments. This paper optimizes the network structure by combining a self-attention mechanism layer with a fully connected layer that gradually reduces the number of neurons, thereby improving the efficiency and accuracy of the HSDDQN algorithm. The network structure of each agent uses ReLU as the activation function, such as... Figure 2 As shown.
[0107] Example:
[0108] 1. Experimental Design:
[0109] This experiment was implemented using Python and ran on a computer equipped with an AMD Ryzen 7 5800H@3.20GHz CPU, 64GB of RAM, and an NVIDIA GeForce RTX 3070 graphics card. Since there are no existing benchmarks for the dynamic scheduling problem of reentrant hybrid pipeline workshops with batch processing machines, this invention constructs a series of datasets to support the validation of HSDDQN and its improvements. The initial workshop has multiple jobs, after which new jobs arrive according to a Poisson distribution, and the interval between the arrival of two consecutive new jobs follows an exponential distribution “exp(1 / λ)”. The training and testing benchmarks used in this invention were randomly generated based on the parameters listed in Table 5, where “randi” and “randf” represent uniform distributions of integers and real numbers, respectively, and the total number of machines M is obtained by summing the number of machines in each stage. The parameter settings for the training process are listed in Table 6.
[0110] Table 5 Rules for Generating Training and Test Cases
[0111]
[0112] Table 6 Hyperparameters during training
[0113]
[0114]
[0115] The example uses two performance metrics, generation distance (GD) and inverse generation distance (IGD), to evaluate the quality of the solution obtained under Pareto optimality. GD and IGD are defined as follows:
[0116]
[0117] Where, d i,A,P Let represent the minimum Euclidean distance from the i-th solution in set P to solutions in set A. In the numerical experiments of this invention, all Pareto optimal solutions obtained through multiple runs of HSDDQN and other comparative methods are merged, and then the set of non-dominated solutions is extracted as an approximation of the true Pareto front P. This method ensures that P can be considered as an approximation of the true Pareto front, sufficient for performance comparison. To eliminate the influence of different dimensions, the objective function values are normalized.
[0118] 2. Comparison of Algorithm Results
[0119] (1) Comparison results with dominance rules
[0120] To verify the effectiveness and superiority of the proposed HSDDQN algorithm, this invention compares its performance with known scheduling rules at a specific scale. The study selects the four most effective combinations of six job ordering rules and two machine selection rules as benchmark rules. Furthermore, to confirm whether the proposed HSDDQN has learned a feasible rule selection strategy, this invention also compares it using a random strategy, i.e., randomly selecting the job ordering rule and machine selection rule at each rescheduling point. Each method is run 20 times on each test instance. The performance metrics of the final Pareto optimal front obtained by comparing the methods are shown in Tables 7 and 8, where the best results are highlighted in bold.
[0121] Table 7. GD values of HSDDQN compared to the rules
[0122]
[0123]
[0124] Table 8 IGD values of HSDDQN compared to the rules
[0125]
[0126]
[0127] Compared to random action selection strategies, the HSDDQN proposed in this invention achieves better results on all metrics for almost all instances. This proves that HSDDQN has learned an efficient strategy for selecting feasible objectives and scheduling rules at each rescheduling point. Furthermore, compared to other known scheduling rules, HSDDQN achieves best results on GD and IGD metrics on 75% of instances and also achieves good results on 68.75% of instances. This indicates that the solution obtained by HSDDQN is closest in convergence to the true Pareto optimal frontier.
[0128] (2) Comparison results with deep reinforcement learning
[0129] To evaluate the convergence performance and learning ability of the proposed HSDDQN, this invention compares it with four dynamic deep reinforcement learning scheduling methods: HMAPPO, THDQN, MA-IDDQN, and HRLDDQN. The comparison is conducted under consistent state, action, and reward function designs. The aim is to demonstrate the effectiveness of the proposed algorithm and environment settings. All deep reinforcement learning algorithms were trained using the same test cases. To evaluate the generalization ability of the trained HSDDQN on large-scale problems compared to other scheduling methods, we introduced new jobs of varying sizes and increased the frequency of job entry into the production system. This resulted in higher work-in-process levels, increased preemption of production equipment, and more complex production scheduling scenarios. The parameter settings were consistent with those listed in Table 5. Experiments were repeated 20 times for each test instance. Tables 9-10 show the performance metrics of each method, highlighting the best results.
[0130] Table 9. GD values of HSDDQN compared to other deep reinforcement learning methods.
[0131]
[0132]
[0133] Table 10 IGD values of HSDDQN compared to other deep reinforcement learning methods
[0134]
[0135]
[0136] The results show that HSDDQN outperforms other methods in GD values in 85.4% of cases and in IGD values in 69% of cases, indicating that HSDDQN achieves the best trade-off among the three research objectives.
[0137] (3) Comparison results of workpiece batching strategies
[0138] To verify the effectiveness of the proposed Variable Threshold Batch Processing (VTBP) strategy, it is compared with common fixed capacity threshold strategies. Common fixed capacity threshold strategies can be divided into fixed capacity threshold strategy (FCTBP) and fixed time threshold strategy (FTTBP). Two equipment capacity thresholds are set—70% and 100%, denoted as FCTBP1 and FCTBP2; two time thresholds are set—1*PT and 2*PT, denoted as FTTBP1 and FTTBP2 (PT is the average time of all workpieces in the same batch processing operation).
[0139] Tables 11 and 12 respectively present the GD and IGD indices of the four strategies on nine representative cases.
[0140] Table 11 Comparison of Average GD Indicators for Strategies
[0141]
[0142]
[0143] Table 12 Comparison of Average IGD Indicators for Strategies
[0144]
[0145] The experimental results show that the VTBP strategy proposed in this invention outperforms the FCTBP and FTTBP strategies by an average of 40.69% and 50.76% in GD index, respectively; and outperforms the FCTBP and FTTBP strategies by an average of 23.72% and 44.39% in IGD index, respectively. This indicates that the solutions obtained by the VTBP strategy have advantages over other strategies in terms of diversity and convergence.
Claims
1. A hierarchical dynamic scheduling method based on the policy optimization DDQN algorithm, characterized in that, This problem is used to solve the dynamic scheduling problem of a reentrant hybrid flow shop with a batch processing machine, specifically: A. Description and assumptions of the scheduling problem; Q types of n initial arrival and random arrival jobs, each job according to its process route through l stages, including |BS| batch processing stages, 1≤|BS|≤l; each stage exists m s unrelated parallel machines, m s ≥1, the batch processing machines contained in the batch processing stage can process no more than a limited number of jobs at the same time, and the single processing machines contained in the single processing stage are single-piece processing; each process job can select a machine in the stage for processing, and there is no priority constraint between the processing order of jobs, but there is a processing order constraint between each process of the job, the job may need to be processed repeatedly in a station multiple times, and the job is allowed to skip the processing of some stages at different access times; B. Objective function and constraints; Objective function: Minimizing completion time, minimizing weighted total delay, and maximizing average machine utilization are expressed as follows: ； Wherein, equation (1) represents the completion time of all workpieces, j represents the workpiece index, and o j C represents the final process for workpiece j. j,oj w represents the completion time of workpiece j; equation (2) represents the weighted total delay time of all workpieces. j D represents the urgency level of each workpiece j. j represents the due date of each workpiece j; equation (3) represents the average machine utilization rate, o represents the process index of the workpiece, s is the stage index, k is the machine index, and m s M is the number of machines in stage s. s P is the set of machines in stage s. j,o,s,k Y represents the processing time of operation o for workpiece j on machine k in stage s. j,o,s,k Whether process o of workpiece j is processed on machine k in stage s, if yes, Y j,o,s,k =1, otherwise Y j,o,s,k =0; Constraints: ； Equation (4) indicates that all workpieces j must arrive before processing can begin, and S j,1 A represents the start time of process 1 for workpiece j. j S represents the arrival time of workpiece j; Equation (5) indicates that each process of each workpiece can only be processed on one machine in one stage; Equation (6) indicates that in a single processing stage, each machine can only process one workpiece at a time, S j,o X represents the start time of process o for workpiece j. jo,j’o’,sk This indicates whether process o of workpiece j is processed before process o' of workpiece j'. If so, X jo,j’o’,sk =1, otherwise X jo,j’o’,sk =0, N is a sufficiently large positive number, O s S represents the set of operations that need to be processed in stage s; Equation (7) represents the ordering constraint between operations. For two operations of the same job with sequence constraints, the latter must be processed after the former has been processed; Equation (8) represents the ordering constraint between workpieces j cycles. Accessing a workpiece in the later cycle must be processed after the previous cycle has been processed. j,r C represents the start time of the r-th cycle of workpiece j. j,r The end time of the r-th cycle of workpiece j represents the processing time; Equation (9) indicates that batch processing is performed before processing to ensure that each job is assigned to only one batch during batch processing, where b represents the batch index, B s,k The set of batches processed on machine k in stage s represents the batch set. This indicates whether workpiece j is assigned to batch b. If so, ,otherwise Equation (10) indicates that a batch can only be processed on one batch processing machine, and BS represents the set of batch processing stages. This indicates whether batch b is processed on machine k in stage s. If yes... ,otherwise, Equation (11) indicates that only workpieces j belonging to the same process type can form a batch b. This indicates whether workpiece j and workpiece j' belong to the same process type. If they do... ,otherwise Equation (12) represents the capacity limit of the batch processing machine, ensuring that the total number of jobs in the batch processed by the machine does not exceed its maximum capacity, V. s,k S represents the processing capacity of machine k in stage s; Equation (13-14) represents the sequence priority constraint, ensuring that different batches processed by the same batch processing machine follow the order of processing. b,k C represents the start time of batch b on machine k. b,k This indicates the end time of batch b on machine k. This indicates whether batch b is processed before batch b'. If so, ;otherwise ; C. A hierarchical dynamic scheduling method based on strategy optimization DDQN is used to solve the dynamic scheduling problem of reentrant hybrid flow shop; First, a hierarchical structure based on the policy optimization DDQN algorithm is proposed, and a batching agent BA and a scheduling agent SA are constructed to solve the workpiece batching and scheduling subproblems in the dynamic scheduling problem of reentrant hybrid flow shop with batch processing machine, respectively. The batching agent BA decides whether to execute the work batch, while the scheduling agent SA determines the priority order of the work or batch and the machine selection. Secondly, based on the two intelligent agents mentioned above, a Markov decision process model suitable for the dynamic scheduling problem of reentrant hybrid flow shop with batch processing machine is designed. It combines multi-stage batch processing and reentrancy characteristics, defines state, reward and action, and combines the heuristic variable threshold batch strategy VTBP with dynamic workpiece batching. Finally, to further enhance the effectiveness and generalization ability of the hierarchical dynamic scheduling method, two improvement strategies are proposed: a) Combining a mask-based action selection strategy with an ε-greedy strategy to limit the action space of each agent at different decision points and improve the efficiency of collaborative decision-making; b) A soft-start network update strategy is used for offline training to accelerate learning and convergence.
2. The hierarchical dynamic scheduling method based on the policy optimization DDQN algorithm according to claim 1, characterized in that, The Markov decision process model is specifically as follows: (1) Status: Considering the stage-based processing characteristics of a hybrid assembly line workshop, the features of each agent are designed based on the processing stage at the decision-making time point. Six state features are extracted as inputs for the Business Intelligence (BA), specifically: ； Twelve state features are extracted as input to the SA, specifically: ； (2) Action: A heuristic variable threshold batching strategy (VTBP) is used to construct the action space of the batching agent. The VTBP strategy is as follows: First, the process type of the job is determined; then, the batch capacity threshold is determined; next, the jobs are sorted according to certain rules; finally, a batch is formed by selecting a threshold number of jobs. During scheduling, all jobs in the processing stage buffer are sorted in ascending order according to their arrival time at that stage and classified by job type. Then, the average waiting time for each type of job is calculated; the type q with the longest average waiting time is selected for batch processing. Next, the threshold V of the current batch needs to be determined. b Let O s,q,t This represents the set of job operations of type q that need to be batch processed in stage s but have not yet reached the current time t, in BN. s,q Indicates the type and number of jobs in the current stage s; if O s,q,t If BN ≠ 0, it means that there are still batch processing operations for jobs that have not been completed in the current stage; in this case, if BN s,q >|O s,q,t |, then V b =min(BN s,q V s,k If BN s,q ≤|O s,q,t |, then V b =0; otherwise, if O s,q,t =0, then V b =min(BN s,q V s,k ); The batching scheme constructed based on different batching rules directly affects the optimization results. When batch processing jobs within the buffer, the consistency of processing time is given priority, that is, jobs with similar processing times are grouped into the same batch, thereby reducing wasted processing time. In addition, in order to minimize total latency, completion time and reduce the waiting time of jobs within the buffer, three batching rules are proposed to balance processing time consistency with other time-based performance metrics. A weighted batch quality index is also introduced. , , The index takes into account the consistency of delivery time and processing time of operations within the buffer zone, as shown in Equation (15): ； In the formula, buffer s q represents the set of jobs in stage s buffer during the batch scheduling decision process. j = q j' This indicates that only jobs belonging to the same job family can be grouped for batch processing. θ is the weighting coefficient, and P... j,o,s,k D represents the processing time of operation o for workpiece j on machine k in stage s. j c represents the due date of assignment j. j A represents the remaining shortest processing time for task j. s,j Indicates the arrival time of task j in stage s; batch quality index , , The smaller the value, the higher the priority for batch processing of the job; jobs are processed according to... , , Sort in ascending order to determine batch sequence; The action space of BA is as follows: ； When V b When =0, BA selects the "None" action, and the job will continue to wait; The scheduling point is set to machine idle time; the SA's action space includes job selection and machine selection; in the batch processing stage, SA schedules batches already formed by BA; in the individual processing stage, the job selection rules and machine selection rules are as follows: ； (3) Rewards: The total weighted delay of the workpiece is broken down into each scheduling decision of the BA (Balanced Decision Model). After each step is executed by the BA, an immediate reward is generated to reflect the impact of the action. The instantaneous reward at each moment is shown in Equation (16); to ensure that the two objectives of minimizing the maximum completion time and maximizing machine utilization are reflected in the actions of the same decision model, the difference between the difference in average machine utilization and the difference in completion time between each scheduling step is taken as the instantaneous reward of the decision model SA. The instantaneous reward at each moment is shown in Equation (17). To ensure that the average machine utilization and the time difference between each scheduling step have the same influence on the decision model's behavior, an additional reward is introduced. and ; ； ； in, This represents the number of jobs in the job pool at time t in stage p. This indicates the number of workpieces in batch b. Z represents the next decision time. If the workpiece's due date is less than the next decision time, Z is 1; otherwise, Z is 0. If the workpiece completes its current process at the next decision time, Z represents the next decision time. It equals the completion time of that process for that workpiece; otherwise... Equal to the next decision time .