A large-scale flexible job shop scheduling method based on DDQN algorithm

Abstract

This invention discloses a large-scale flexible job shop scheduling method based on the DDQN algorithm. Specifically, the large-scale scheduling problem is formalized, including defining state features, action sets, and reward functions. To ensure effective model training within the framework, the problem is decomposed, and a method for feeding back features from the action set and reward function is proposed to design state features. A composite scheduling rule with high sensitivity to minimizing completion time is selected as the action set, and a heuristic reward method is designed to guide algorithm convergence. Finally, the DDQN algorithm is used to solve the large-scale flexible job shop scheduling problem. This invention can efficiently and effectively solve large-scale flexible job shop scheduling problems.

Description

Technical Field

[0001] The present invention belongs to the technical field of flexible job shop scheduling, and in particular relates to a large-scale flexible job shop scheduling method based on a DDQN algorithm. Background Art

[0002] The flexible job shop scheduling problem is an extension of the traditional job shop scheduling problem and a prototype of a large number of production lines. Due to its flexible scheduling, multiple processing machines can be selected for each workpiece processing step, better suiting the actual production conditions of the workshop. Flexible job shop scheduling problems where the number of workpieces × the number of machines exceeds 1000 are called large-scale flexible job shop scheduling problems. In fact, the production of complex products often results in large-scale flexible job shop scheduling problems due to factors such as the large number of machines and complex processes. This type of problem has a large solution space and is difficult to optimize, so studying it in light of actual production practices is of great significance.

[0003] Numerous approaches exist for solving flexible job shop scheduling problems, but heuristic scheduling rules and meta-heuristic intelligent optimization algorithms are commonly used. Priority-based heuristic algorithms are simple to implement and can quickly and stably generate feasible solutions. However, heuristic scheduling rules themselves lack optimization capabilities, resulting in low-quality solutions. While meta-heuristic intelligent optimization algorithms offer high accuracy, they require time-consuming iterative searches when solving large-scale scheduling problems. Even for static scheduling problems, solutions can be found in advance, but they involve numerous resources. Premature scheduling fails to account for subsequent resource changes, resulting in poor execution of the resulting solution. Summary of the Invention

[0004] In response to the above problems, the present invention provides a large-scale flexible job shop scheduling method based on the DDQN algorithm.

[0005] A large-scale flexible job shop scheduling method based on the DDQN algorithm of the present invention comprises the following steps:

[0006] Step A: Identify the large-scale flexible job shop scheduling problem.

[0007] Suppose there are n workpieces {J1, J2, ..., J n}On m machines {M1,M2,…,M m} processing, each workpiece has n i Process {O i,1 ,O i,2 ,…,O i,ni}, O i,j represents the jth process of workpiece i, and each process O i,j There are multiple machines available for processing; i,j,k Indicates process O i,j Processing time on the kth device.

[0008] When the following three situations occur, it is called a large-scale flexible job shop scheduling problem:

[0009] a) When workpiece n>50 and machine m>20;

[0010] b) When workpiece n≤50, machine m>20, n×m>1000;

[0011] c) When workpiece n>50, machine m≤20, n×m>1000.

[0012] Step B: Establish a mathematical model for optimizing scheduling objectives in large-scale flexible job shops.

[0013] The objective function is to minimize the completion time, as follows:

[0014]

[0015] Constraints:

[0016]

[0017]

[0018]

[0019]

[0020] n·m>1000 (6)

[0021]

[0022] Among them, n represents the total number of workpieces to be processed, m represents the number of machine tools, and J i represents the i-th workpiece, M k represents the kth machine, M i,j Indicates process O i,j The set of available machines, n i represents the number of processes for workpiece i, C i,j Indicates process O i,j Completion time, C i represents the completion time of job i.

[0023] Among them, formula (1) represents minimizing the maximum completion time, formula (2) represents that the completion time of each process is non-negative, formula (3) represents that each process can only be processed on one machine, formula (4) represents that each machine can only process one process at a time, formula (5) represents that the next process can only be started after the previous process is completed, formula (6) represents the large-scale constraint condition, and formula (7) represents the decision variable.

[0024] Step C: Transformation of the scheduling problem.

[0025] (a) State feature design

[0026] The flexible job shop scheduling problem is decomposed into two sub-goals: job scheduling completion and minimization of completion time. For the scheduling completion sub-goal, the design state feature f1 represents the job completion rate, and f2 represents the standard deviation of the job completion rate. For the minimization of completion time, the design state feature f3 represents the average machine utilization rate, and f4 represents the standard deviation of the average machine utilization rate.

[0027] In order to enable the scheduling agent to obtain action-related information, the action set state feature f5 is designed to represent the normalized processing of the maximum remaining working time, f6 represents the normalized processing of the maximum number of remaining processes, f7 represents the normalized processing of the minimum machine tool load, f8 represents the normalized processing of the minimum processing time, and f9 represents the normalized processing of the minimum completion time of the available machine tools.

[0028] (b) Action set design

[0029] Two workpiece selection rules and three machine selection rules that perform well in optimizing completion time are selected, and composite scheduling rules are obtained through permutations and combinations.

[0030] (c) Reward function design

[0031] A heuristic reward method is designed to guide the agent to obtain a larger reward value in a shorter time, and a set of scheduling rules is obtained to minimize the maximum completion time. The reward value setting takes into account the round reward and the immediate reward. At the end of the round, the reward value shown in formula (8) is designed. Since the higher the machine utilization rate, the shorter the completion time, in order to avoid sparse rewards, the immediate reward shown in formula (9) is adopted.

[0032]

[0033] Where makepan(t) represents the completion time of the current round.

[0034]

[0035] Where Uave(t) represents the average machine utilization at time t.

[0036] Step D: DDQN algorithm is used to optimize and solve the large-scale flexible job shop scheduling problem.

[0037] The scheduling agent selects a composite scheduling rule based on the state of the workshop. After sorting the workpieces and assigning them to machine tools, the workshop environment changes, and the scheduling agent is rewarded accordingly. If the reward value is high, the scheduling agent will choose this composite scheduling rule. Through continuous trial and error learning and continuous interaction with the environment, it ultimately ensures that the optimal solution is obtained when the cumulative reward value is maximized. The DDQN algorithm solves the large-scale flexible job shop scheduling problem as follows:

[0038] Step 1: Initialize the memory pool capacity to D, the maximum number of iterations to Max_episode, the initial learning rate to α, the discount rate to γ, and initialize the parameters of the target network and the estimated network.

[0039] Step 2: At the beginning of each round, the workshop environment is reset to the initial state S0.

[0040] Step 3: Input the state into the network, the network outputs the Q value of the action, and uses the greedy strategy to select the action, that is, select the composite scheduling rule to schedule the workpiece and select the machine to process the workpiece, and execute the action a t .

[0041] Step 4: After executing this action, the workshop environment enters the next state S t+1 , calculate the state feature value, and the intelligent experience will get the reward value r after the environment changes t , then S t+1 Assign S t .

[0042] Step 5: Data (S t ,a t ,r t ,S t+1 ) is stored in the memory pool D, and then batchsize samples are randomly sampled from the memory pool to learn. During the learning process, the weight parameters in the network are updated by gradient descent.

[0043] Step 6: Assign the parameters of the estimated network to the target network every C steps.

[0044] Step 7: Determine whether all workpiece processes of the case have been scheduled. If so, proceed to the next round. If not, continue to step 3.

[0045] Step 8: Determine whether the round is over. If so, output a better scheduling model. If not, continue to step 2.

[0046] The beneficial technical effects of the present invention are:

[0047] (1) This method designs a DDQN algorithm framework to train a model to solve the problem. In the solution example, the trained model can obtain a better solution than the scheduling rules and the optimization time is shorter than the metaheuristic algorithm, indicating that the algorithm framework can solve the large-scale flexible job shop scheduling problem efficiently and with high quality.

[0048] (2) This method designs a method to decompose the scheduling problem and feed back the designed state characteristics based on the designed action set and reward function to solve the problem of difficult state feature extraction; for the flexible job shop scheduling problem with two sub-actions, machine selection and workpiece sorting, the design uses a composite scheduling rule as the action set to improve the accuracy of the solution; at the same time, a heuristic reward function is designed to accelerate the convergence of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the framework of the DDQN of the present invention for solving the large-scale flexible job shop scheduling problem.

[0050] Figure 2 This is a graph that records rewards and punishments during the learning process of the scheduling agent.

[0051] Figure 3 Iteration diagram of the completion time of MK01 in the scheduling agent learning process.

[0052] Figure 4 The optimal scheduling Gantt chart for solving the MK01 example using the DDQN algorithm of the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] This paper proposes using the DDQN algorithm framework to train a model to solve the problem. First, a large-scale scheduling problem is formalized, including the definition of state features, action sets, and reward functions. Secondly, to ensure effective model training within the framework, the problem is decomposed, and a method is proposed to design state features by feeding back features from the action set and reward function. A composite scheduling rule with a high sensitivity to minimizing completion time is selected as the action set, and a heuristic reward method is designed to guide algorithm convergence.

[0055] A large-scale flexible job shop scheduling method based on the DDQN algorithm of the present invention comprises the following steps:

[0056] Step A: Identify the large-scale flexible job shop scheduling problem.

[0057] Suppose there are n workpieces {J1, J2, ..., J n}On m machines {M1,M2,…,M m} processing, each workpiece has n i Process {O i,1,O i,2 ,…,O i,ni}, O i,j represents the jth process of workpiece i, and each process O i,j There are multiple machines available for processing; i,j,k Indicates process O i,j Processing time on the kth device.

[0058] When the following three situations occur, it is called a large-scale flexible job shop scheduling problem:

[0059] a) When workpiece n>50 and machine m>20;

[0060] b) When workpiece n≤50, machine m>20, n×m>1000;

[0061] c) When workpiece n>50, machine m≤20, n×m>1000.

[0062] Therefore, the large-scale flexible job shop scheduling problem needs to meet the following constraints: (1) Each machine tool can only process one process at a certain time, and each process cannot be interrupted once it starts; (2) The next process of a workpiece can only be started after the previous process is completed; (3) At the same time, each process of the workpiece is only processed on one machine; (4) The machine tool can start working at time zero, and the workpiece can be processed at time zero; (5) The product of the number of workpieces and the number of machines cannot be less than one thousand.

[0063] Step B: Establish a mathematical model for optimizing scheduling objectives in large-scale flexible job shops.

[0064] The objective function is to minimize the completion time, as follows:

[0065]

[0066] Constraints:

[0067]

[0068]

[0069]

[0070]

[0071] n·m>1000 (6)

[0072]

[0073] Among them, n represents the total number of workpieces to be processed, m represents the number of machine tools, and J i represents the i-th workpiece, Mk represents the kth machine, M i,j Indicates process O i,j The set of available machines, n i represents the number of processes for workpiece i, C i,j Indicates process O i,j completion time.

[0074] Among them, formula (1) represents minimizing the maximum completion time, formula (2) represents that the completion time of each process is non-negative, formula (3) represents that each process can only be processed on one machine, formula (4) represents that each machine can only process one process at a time, formula (5) represents that the next process can only be started after the previous process is completed, formula (6) represents the large-scale constraint condition, and formula (7) represents the decision variable.

[0075] Step C: Transformation of the scheduling problem.

[0076] (a) State feature design

[0077] To address the difficulty of extracting state features, and the fact that too many or too few can reduce algorithm efficiency, this paper conducts in-depth analysis of the flexible job shop scheduling problem and optimization objectives. Based on the designed reward function and action set that feed back state features, the paper extracts the following nine state features as network inputs. The flexible job shop scheduling problem is first decomposed into two sub-goals: complete workpiece scheduling and minimized completion time. For the completion sub-goal, state feature f1 is designed to represent the workpiece completion rate, and f2 is the standard deviation of the workpiece completion rate. For the minimum completion time sub-goal, state features f3 are designed to represent the average machine utilization rate, and f4 is the standard deviation of the average machine utilization rate.

[0078] In order to allow the scheduling agent to obtain action-related information, the state feature f5 is designed based on the workpiece and machine-related information such as working hours, number of processes, machine tool load, etc., combined with the action set to represent the normalized processing of the maximum remaining working hours, f6 represents the normalized processing of the maximum number of remaining processes, f7 represents the normalized processing of the minimum machine tool load, f8 represents the normalized processing of the minimum processing hours, and f9 represents the normalized processing of the minimum completion time of the available machine tools. Taking normalized features is conducive to network learning and enhancing the generalization of the model, and the solution effect is better, so the state features of the present invention adopt normalized processing. The detailed description of each state feature is shown in Table 1.

[0079] Table 1 Detailed description of status characteristics

[0080]

[0081] (b) Action set design

[0082] At present, in the workshop scheduling problem, two workpiece selection rules and three machine selection rules that perform well when the completion time is optimal have been screened out. The composite scheduling rules shown in Table 2 are obtained by permutation and combination.

[0083] Table 2 Action set table

[0084]

[0085] (c) Reward function design

[0086] Designing a reward function is key to determining whether deep reinforcement learning can learn a strategy and directly affects the algorithm's final performance. To accelerate algorithm convergence, a heuristic reward method is designed to guide the agent to obtain a larger reward value in a shorter period of time, resulting in a set of scheduling rules that minimize the maximum completion time. The reward value setting takes into account both round rewards and immediate rewards. At the end of the round, the reward value shown in Equation (8) is designed. Since higher machine utilization results in shorter completion times, the immediate reward shown in Equation (9) is used to avoid sparse rewards.

[0087]

[0088] Where makepan(t) represents the completion time of the current round.

[0089]

[0090] Where Uave(t) represents the average machine utilization at time t.

[0091] Step D: DDQN algorithm is used to optimize and solve the large-scale flexible job shop scheduling problem.

[0092] When the workshop is in a certain state, the scheduling agent selects a composite scheduling rule based on the workshop state. After the workpieces are sorted and the machine tools are assigned, the workshop environment changes, and the scheduling agent is given a reward value accordingly. If the reward value is high, the scheduling agent will choose this composite scheduling rule. Through continuous trial and error learning and continuous interaction with the environment, it is ultimately guaranteed to obtain a better solution when the cumulative reward value is maximized.

[0093] DDQN algorithm solves large-scale flexible job shop scheduling problems such as Figure 1 As shown, the details are as follows:

[0094] Step 1: Initialize the memory pool capacity to D, the maximum number of iterations to Max_episode, the initial learning rate to α, the discount rate to γ, and initialize the parameters of the target network and the estimated network.

[0095] Step 2: At the beginning of each round, the workshop environment is reset to the initial state S0.

[0096] Step 3: Input the state into the network, the network outputs the Q value of the action, and uses the greedy strategy to select the action, that is, select the composite scheduling rule to schedule the workpiece and select the machine to process the workpiece, and execute the action a t .

[0097] Step 4: After executing this action, the workshop environment enters the next state S t+1 , calculate the state feature value, and the intelligent experience will get the reward value r after the environment changes t , then S t+1 Assign S t .

[0098] Step 5: Data (S t ,a t ,r t ,S t+1 ) is stored in the memory pool D, and then batchsize samples are randomly sampled from the memory pool to learn. During the learning process, the weight parameters in the network are updated by gradient descent.

[0099] Step 6: Assign the parameters of the estimated network to the target network every C steps.

[0100] Step 7: Determine whether all workpiece processes of the case have been scheduled. If so, proceed to the next round. If not, continue to step 3.

[0101] Step 8: Determine whether the round is over. If so, output a better scheduling model. If not, continue to step 2.

[0102] This paper verifies the effectiveness of the proposed algorithm framework and the generalization of the model on the standard Brandimarte example of the flexible job shop scheduling problem and randomly generated large-scale examples.

[0103] (1) Experimental design

[0104] The number of layers and hidden layer nodes of the neural network will greatly affect the performance of the algorithm. Since the flexible job shop scheduling problem is relatively complex, in the experiment, the present invention constructed a neural network with a single input layer, four hidden layers and a single output layer, wherein the number of input layer nodes is the number of designed state features, the number of hidden layer nodes is 30, and the number of output layer nodes is the number of action sets designed above, wherein each layer activation function uses the ReLU activation function. At the same time, the correct selection of hyperparameters has a great impact on the learning ability of the intelligent agent and the performance of the algorithm. However, the range of hyperparameters is very wide, and it is difficult to select suitable parameters. The present invention sets the relevant parameters according to general principles, see Table 3. The experiment uses Python language, Tensorflow1.4 version, and the equipment configuration is Intel (R) Core (TM) i5-9400 CPU @ 2.90GHz 2.90GHz, with 8GB of machine RAM.

[0105] Table 3 Offline training algorithm parameters

[0106]

[0107] (2) Analysis of experimental results

[0108] Figure 2 In order to schedule the reward and punishment record chart during the intelligent agent's learning process, the intelligent agent continues to learn within 1,000 generations and is in the trial-and-error stage. After that, it can choose the action that maximizes the reward with a high probability, and the entire learning process tends to be stable. Figure 3 This is the iterative curve of the completion time of the MK01 example during the learning process. It can be seen from the figure that as the reward value increases, the completion time continues to decrease. When the reward value obtained by the agent tends to be stable, the completion time can converge to a better value, thereby verifying the effectiveness of the designed state characteristics and reward values. Figure 4 This is the Gantt chart for the scheduling with the minimum completion time in the MK01 example.

[0109] Based on the set parameters, the algorithm was tested on a classic example of the FJSP problem and compared with existing end-to-end deep reinforcement learning algorithms and the composite scheduling rule designed for action concentration. The results are shown in Table 4. The GAP in Table 4 represents the percentage difference between the solutions obtained by various algorithms and the optimal solution of the existing MK standard example.

[0110] Table 4 Comparison of experimental results

[0111]

[0112]

[0113] As shown in Table 4, when the algorithm proposed in this invention solves the MK01-MK10 cases, the average solution is 17.4 time units higher than the end-to-end deep reinforcement learning algorithm. Among the results obtained, 90% are better than the end-to-end deep reinforcement algorithm solution, and the solution results of all cases are better than the composite scheduling rule, verifying that the algorithm proposed in this invention can obtain better solutions compared with the composite scheduling rule and the existing end-to-end deep reinforcement algorithm.

[0114] Deep neural networks require a certain amount of time to train, but offline trained models can be applied to examples to obtain a feasible solution stably and quickly, thus avoiding the time-consuming optimization problem of intelligent algorithms solving large-scale problems.

[0115] Table 5 shows the test results of the model obtained by training with 50×20 scale examples on 70×30 scale examples and 100×50 scale examples. It is compared with the improved genetic algorithm (IGA) and the improved weed algorithm (IIWO) for scheduling rules designed with action sets. In Table 5, GAP represents the percentage difference between the solutions obtained by the composite scheduling rule and metaheuristic algorithm and the solution obtained by the algorithm framework proposed in this paper.

[0116] Table 5 Large-scale test generalization results

[0117]

[0118] As can be seen, the offline training model can quickly obtain a feasible solution that outperforms the composite scheduling rule, thus verifying the model's generalization. Although the solution accuracy is slightly lower than that of the metaheuristic algorithm, when solving a large-scale 100×50 problem, the solution time is only 1 / 58 of that of the metaheuristic intelligent algorithm, obtaining a feasible solution in just 22.33 seconds. This further demonstrates that the proposed algorithm framework can effectively solve the time-consuming problem of solving large-scale problems.

Claims

1. A large-scale flexible job shop scheduling method based on DDQN algorithm, characterized in that: The following steps are involved: Step A: Identify the large-scale flexible job shop scheduling problem; Suppose there are n workpieces {J1, J2, ..., J n }On m machines {M1,M2,…,M m } processing, each workpiece has n i Process {O i,1 ,O i,2 ,…,O i,ni }, O i,j represents the jth process of workpiece i, and each process O i,j There are multiple machines available for processing; i,j,k Indicates process O i,j Processing time on the kth device; When the following three situations occur, it is called a large-scale flexible job shop scheduling problem: a) When workpiece n>50 and machine m>20; b) When workpiece n≤50, machine m>20, n×m>1000; c) When workpiece n>50, machine m≤20, n×m>1000; Step B: Establish a mathematical model for optimizing the scheduling objectives of a large-scale flexible job shop; The objective function is to minimize the completion time, as follows: Constraints: n·m>1000 (6) Among them, n represents the total number of workpieces to be processed, m represents the number of machine tools, and J i represents the i-th workpiece, M k represents the kth machine, M i,j Indicates process O i,j The set of available machines, n i represents the number of processes for workpiece i, C i,j Indicates process O i,j Completion time, C i represents the completion time of workpiece i; Among them, formula (1) represents minimizing the maximum completion time, formula (2) represents that the completion time of each process is non-negative, formula (3) represents that each process can only be processed on one machine, formula (4) represents that each machine can only process one process at a time, formula (5) represents that the next process can only be started after the previous process is completed, formula (6) represents the large-scale constraint condition, and formula (7) represents the decision variable; Step C: Transformation of scheduling problem; (a) State feature design The flexible job shop scheduling problem is decomposed into two sub-goals: job scheduling completion and minimum completion time. For the scheduling completion sub-goal, the design state feature f1 represents the job completion rate, and f2 represents the standard deviation of the job completion rate. For the minimum completion time sub-goal, the design state feature f3 represents the average machine utilization rate, and f4 represents the standard deviation of the average machine utilization rate. In order to allow the scheduling agent to obtain action-related information, the action set state features f5 are designed to represent the normalization of the maximum remaining working time, f6 represents the normalization of the maximum number of remaining processes, f7 represents the normalization of the minimum machine tool load, f8 represents the normalization of the minimum processing time, and f9 represents the normalization of the minimum completion time of the available machine tools. (b) Action set design Two workpiece selection rules and three machine selection rules that perform well in optimizing completion time were selected, and then the composite scheduling rules were obtained through permutations and combinations. (c) Reward function design A heuristic reward method is designed to guide the agent to obtain a larger reward value in a shorter time, and a set of scheduling rules is obtained to minimize the maximum completion time. The reward value setting takes into account the round reward and the immediate reward. At the end of the round, the reward value shown in formula (8) is designed. Since the higher the machine utilization rate, the shorter the completion time, in order to avoid sparse rewards, the immediate reward shown in formula (9) is adopted. Where makepan(t) represents the completion time of the current round; Where Uave(t) represents the average machine utilization at time t; Step D: DDQN algorithm is used to optimize and solve the large-scale flexible job shop scheduling problem; When the workshop is in a certain state, the scheduling agent selects a composite scheduling rule based on the workshop state. After the workpieces are sorted and the machine tools are assigned, the workshop environment changes, and the scheduling agent is given a reward value accordingly. If the reward value is high, the scheduling agent will choose this composite scheduling rule. Through continuous trial and error learning and continuous interaction with the environment, it is ultimately guaranteed to obtain a better solution when the cumulative reward value is maximized.

2. A large-scale flexible job shop scheduling method based on DDQN algorithm according to claim 1, characterized in that: The DDQN algorithm solves the large-scale flexible job shop scheduling problem as follows: Step 1: Initialize the memory pool capacity to D, the maximum number of iterations to Max_episode, the initial learning rate to α, the discount rate to γ, and initialize the parameters of the target network and the estimated network; Step 2: At the beginning of each round, the workshop environment is reset to the initial state S0; Step 3: Input the state into the network, the network outputs the Q value of the action, and uses the greedy strategy to select the action, that is, select the composite scheduling rule to schedule the workpiece and select the machine to process the workpiece, and execute the action a t ; Step 4: After executing this action, the workshop environment enters the next state S t+1 , calculate the state feature value, and the intelligent experience will get the reward value r after the environment changes t , then S t+1 Assign S t ; Step 5: Data (S t ,a t ,r t ,S t+1 ) is stored in the memory pool D, and then batchsize samples are randomly sampled from the memory pool to learn. During the learning process, the weight parameters in the network are updated by gradient descent; Step 6: Assign the parameters of the estimated network to the target network every C steps; Step 7: Determine whether all workpiece processes of the case have been scheduled. If yes, proceed to the next round. If not, continue to step 3. Step 8: Determine whether the round is over. If so, output a better scheduling model. If not, continue to step 2.

Images